developing extensions for hue
TRANSCRIPT
![Page 1: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/1.jpg)
Developing eXtensions for HUE
Maksym DoroshenkoLviv Hadoop User Group 19 February 2015
![Page 2: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/2.jpg)
Agenda
● What is HUE?
● Why use it?
● HUE architecture
● HUE features
● Add extensions to HUE
![Page 3: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/3.jpg)
● Aggregates the most common
Hadoop components into single
interface
● Simplify and integrate
● Free and open source
Goal of HUE
![Page 4: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/4.jpg)
Apps
![Page 5: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/5.jpg)
Hive & Pig
![Page 6: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/6.jpg)
Metastore & HDFS
![Page 7: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/7.jpg)
Additional applications
![Page 8: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/8.jpg)
Backend
Python + Django
(2.6+ / 1.4.5)
Frontend
jQuery
Bootstrap
Knockout.js
Stack
![Page 9: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/9.jpg)
View from 30K feet
You
and your
colleagues
Hue, as a "container" web application, sits in between your
Hadoop installation and the browser. It hosts all the Hue Apps,
including the built-in ones, and ones that you may write
yourself.
![Page 10: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/10.jpg)
The HUE Server
Django running on WSGI
container/web server:
● manages the url dispatch
● executes app logic code
● puts together the views
from their templates
● use DB to manage
session data and Hue
apps can use it as well
for their “models”
Some Hue applications run
daemon processes "on the
side". For example, Beeswax
runs a daemon
("beeswax_server") that keeps
track of query states.
![Page 11: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/11.jpg)
Install SDK app to HUE
For example we use CDH 5.3.
1. Go to HUE directory – cd /usr/lib/hue
2. Run "create_desktop_app" to Set up a New Source Tree –
“./build/env/bin/hue create_desktop_app calculator”
3. Install SDK application – “./build/env/bin/python
tools/app_reg/app_reg.py --install calculator --relative-
paths”
The "app_reg.py" tool manages the applications that are installed.
![Page 12: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/12.jpg)
Result of installation
After installing we restart HUE – “sudo service hue restart”
And then visit http://127.0.0.1:8888/calculator/ you should see:
![Page 13: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/13.jpg)
Issue to solve
Suppose we need track history of workflow
and for that we need some dashboard with
specific logic which we have not in HUE.
![Page 14: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/14.jpg)
Create Django application
setup(
name="dashboard",
version="0.1",
description='Dashboard',
author='Maksym Doroshenko',
packages=find_packages('src'),
package_dir={'': 'src'},
install_requires=['setuptools', 'desktop'],
entry_points={'desktop.sdk.application': 'dashboard=dashboard'},
package_data={
'ctc_sla_dashboard': expand_package_data(
["src/dashboard/templates", "src/dashboard/static"],
"src/dashboard/")
}
)
![Page 15: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/15.jpg)
Walk-through of a Django view
![Page 16: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/16.jpg)
How add your application to HUE ?
As the previous one :)
1. Copy your application to HUE directory
(or use path to application in next step)
2. Install application –
“./build/env/bin/python
tools/app_reg/app_reg.py --
install dashboard --relative-
paths”
![Page 17: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/17.jpg)
![Page 18: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/18.jpg)
Our results
![Page 19: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/19.jpg)
Dashboard example
![Page 20: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/20.jpg)
Accessing HDFS
The “webhdfs.py” module give the file system object that exposes
operations that manipulate HDFS.
It is pre-configured to access HDFS as the user that's currently logged in.
Operations available on FS object are similar to the file operations typically
available in python.
The list of functions available is as follows:
● chmod, chown
● exists
● isdir, isfile
● listdir (and listdir_stats)
● mkdir
● open (which exposes a file-like object with read(), write(), seek() methods)
● remove, rmdir, rmtree
![Page 21: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/21.jpg)
Web apps in any language
To create a new app:
● sudo build/env/bin/hue create_proxy_app Tableau
https://online.tableausoftware.com/
● sudo tools/app_reg/app_reg.py --install Tableau --relative-paths
If you want to update the url later, change it in the ini:
[Tableau]
url=http://new_url
![Page 22: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/22.jpg)
Show time
![Page 23: Developing eXtensions for HUE](https://reader031.vdocuments.us/reader031/viewer/2022020218/55ab0e6d1a28ab880c8b48f3/html5/thumbnails/23.jpg)
Q&A