spark summit europe: building a rest job server for interactive spark as a service
TRANSCRIPT
![Page 1: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/1.jpg)
BUILDING A REST JOB SERVER FOR INTERACTIVE SPARK AS A SERVICERomain Rigaux - Cloudera Erick Tryzelaar - Cloudera
![Page 2: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/2.jpg)
WHY?
![Page 3: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/3.jpg)
![Page 4: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/4.jpg)
![Page 5: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/5.jpg)
NOTEBOOKS
EASYACCESSFROMANYWHERE
SHARESPARKCONTEXTSANDRDDs
BUILDAPPS
SPARKMAGIC
…
WHY SPARKAS A SERVICE?
![Page 6: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/6.jpg)
MARRIEDWITHFULLHADOOPECOSYSTEM
WHY SPARKIN HUE?
![Page 7: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/7.jpg)
HISTORYV1: OOZIE
• Itworks
• Codesnippet
THE GOOD
• SubmitthroughOozie
• Shellac:on
• VerySlow
• Batch
THE BAD
workflow.xmlsnippet.py
stdout
![Page 8: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/8.jpg)
HISTORYV2: SPARK IGNITER
• ItworksbeAer
THE GOOD
• CompilerJar
• Batchonly,noshell
• NoPython,R
• Security
• Singlepointoffailure
THE BAD Compile
Implement
Upload
jsonoutput
Batch
Scala
jar
Ooyala
![Page 9: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/9.jpg)
HISTORYV3: NOTEBOOK
• Likespark-submit/sparkshells
• Scala/Python/Rshells
• Jar/PythonbatchJobs
• NotebookUI
• YARN
THE GOOD
• Beta?
THE BAD
Livy
codesnippet batch
![Page 10: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/10.jpg)
GENERAL ARCHITECTURE
Spark
Spark
Spark
Livy YARN
!"
# $
![Page 11: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/11.jpg)
Livy
Spark
Spark
Spark
YARN
API
!"
# $
GENERAL ARCHITECTURE
![Page 12: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/12.jpg)
LIVY SPARK SERVER
![Page 13: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/13.jpg)
LIVYSPARK SERVER
•RESTWebserverinScalaforSparksubmissions
• Interac:veShellSessionsorBatchJobs
•Backends:Scala,Java,Python,R
•NodependencyonHue
•OpenSource:hAps://github.com/cloudera/
hue/tree/master/apps/spark/java
•Readaboutit:hAp://gethue.com/spark/
![Page 14: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/14.jpg)
ARCHITECTURE
• Standardwebservice:wrapperaroundspark-submit/Sparkshells• YARNmode,Sparkdriversruninsidethecluster(supportscrashes)• Noneedtoinheritanyinterfaceorcompilecode• Extendedtoworkwithadditionalbackends
![Page 15: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/15.jpg)
LIVY WEB SERVERARCHITECTURE
LOCAL“DEV”MODE YARNMODE
![Page 16: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/16.jpg)
LOCAL MODE
LivyServer
Scalatra
SessionManager
Session
SparkContextSpark
Client
SparkClient
SparkInterpreter
![Page 17: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/17.jpg)
LOCAL MODE
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkClient
SparkContext
SparkInterpreter
![Page 18: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/18.jpg)
LOCAL MODE
SparkClient
1
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkContext
SparkInterpreter
![Page 19: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/19.jpg)
LOCAL MODE
SparkClient
1
2
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkContext
SparkInterpreter
![Page 20: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/20.jpg)
LOCAL MODE
SparkClient
SparkInterpreter
1
2
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkContext
3
![Page 21: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/21.jpg)
LOCAL MODE
SparkClient
1
2
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkContext
3
4 SparkInterpreter
![Page 22: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/22.jpg)
LOCAL MODE
SparkClient
1
2
LivyServer
Scalatra
SessionManager
Session
SparkClient
SparkContext
3
4
5
SparkInterpreter
![Page 23: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/23.jpg)
YARN-CLUSTERMODE
PRODUCTION SCALABLE
![Page 24: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/24.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 25: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/25.jpg)
LivyServer
YARNMaster
Scalatra
SparkClient
SessionManager
Session
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
YARN-CLUSTERMODE
SparkInterpreter
![Page 26: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/26.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 27: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/27.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 28: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/28.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 29: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/29.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4
5
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 30: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/30.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1
2
3
4
5
6
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 31: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/31.jpg)
YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
1 7
2
3
4
5
6
LivyServer
Scalatra
SessionManager
Session
YARN-CLUSTERMODE
SparkInterpreter
![Page 32: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/32.jpg)
SESSION CREATION AND EXECUTION%curl-XPOSTlocalhost:8998/sessions\-d'{"kind":"spark"}'{"id":0,"kind":"spark","log":[...],"state":"idle"}
%curl-XPOSTlocalhost:8998/sessions/0/statements-d'{"code":"1+1"}'{"id":0,"output":{"data":{"text/plain":"res0:Int=2"},"execution_count":0,"status":"ok"},"state":"available"}
![Page 33: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/33.jpg)
Jar
Py
Scala
Python
R
Livy
Spark
Spark
Spark
YARN
/batches
/sessions
BATCH OR INTERACTIVE
![Page 34: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/34.jpg)
SHELL OR BATCH?YARNMaster
SparkClient
YARNNode
SparkInterpreter
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
![Page 35: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/35.jpg)
SHELLYARNMaster
SparkClient
YARNNode
pyspark
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
![Page 36: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/36.jpg)
BATCHYARNMaster
SparkClient
YARNNode
spark-submit
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
![Page 37: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/37.jpg)
LIVY INTERPRETERSScala,Python,R…
![Page 38: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/38.jpg)
REMEMBER?YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
![Page 39: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/39.jpg)
INTERPRETERS
• Pipestdin/stdouttoarunningshell
• Executethecode/sendtoSparkworkers
• Performmagicopera:ons
• Oneinterpreterperlanguage• “Swappable”withotherkernels(python,spark..)
Interpreter
>println(1+1)2
println(1+1)
2
![Page 40: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/40.jpg)
LivyServer
INTERPRETER FLOW
Interpreter
![Page 41: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/41.jpg)
LivyServer
>1+1
Interpreter
INTERPRETER FLOW
![Page 42: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/42.jpg)
LivyServer
{“code”:“1+1”}
>1+1
Interpreter
INTERPRETER FLOW
![Page 43: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/43.jpg)
LivyServer Interpreter
1+1{“code”:“1+1”}
>1+1
INTERPRETER FLOW
![Page 44: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/44.jpg)
LivyServer Interpreter
1+1{“code”:“1+1”}
>1+1
Magic
INTERPRETER FLOW
![Page 45: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/45.jpg)
LivyServer
2
Interpreter
1+1{“code”:“1+1”}
>1+1
Magic
INTERPRETER FLOW
![Page 46: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/46.jpg)
{“data”:{“application/json”:“2”}}
LivyServer
2
Interpreter
1+1{“code”:“1+1”}
>1+1
Magic
INTERPRETER FLOW
![Page 47: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/47.jpg)
{“data”:{“application/json”:“2”}}
LivyServer
2
Interpreter
1+1{“code”:“1+1”}
>1+1
2 Magic
INTERPRETER FLOW
![Page 48: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/48.jpg)
INTERPRETER FLOW CHART
ReceivelinesSplitintoChunks
Sendoutputtoserver
Senderrortoserver
Success
ExecuteChunkMagic!
Chunksle[?
Magicchunk?
No
Yes
NoYes
Exampleofparsing
![Page 49: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/49.jpg)
INTERPRETER MAGIC
• table• json• plotting• ...
![Page 50: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/50.jpg)
NO MAGIC
>1+1
Interpreter
1+1
sparkIMain.interpret(“1+1”)
{"id":0,"output":{"application/json":2}}
![Page 51: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/51.jpg)
[('',506610),('the',23407),('I',19540)...]
JSON MAGIC
>countssparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
vallines=sc.textFile("shakespeare.txt");valcounts=lines.flatMap(line=>line.split("")).map(word=>(word,1)).reduceByKey(_+_).sortBy(-_._2).map{case(w,c)=>Map("word"->w,"count"->c)}
%jsoncounts
![Page 52: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/52.jpg)
JSON MAGIC
>countssparkIMain.valueOfTerm(“counts”)
.toJson()
Interpreter
{"id":0,"output":{"application/json":[{"count":506610,"word":""},{"count":23407,"word":"the"},{"count":19540,"word":"I"},...]...}
vallines=sc.textFile("shakespeare.txt");valcounts=lines.flatMap(line=>line.split("")).map(word=>(word,1)).reduceByKey(_+_).sortBy(-_._2).map{case(w,c)=>Map("word"->w,"count"->c)}
%jsoncounts
![Page 53: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/53.jpg)
[('',506610),('the',23407),('I',19540)...]
TABLE MAGIC
>counts
Interpreter
vallines=sc.textFile("shakespeare.txt");valcounts=lines.flatMap(line=>line.split("")).map(word=>(word,1)).reduceByKey(_+_).sortBy(-_._2).map{case(w,c)=>Map("word"->w,"count"->c)}
%tablecounts
sparkIMain.valueOfTerm(“counts”).guessHeaders().toList()
![Page 54: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/54.jpg)
TABLE MAGIC
>countssparkIMain.valueOfTerm(“counts”)
.guessHeaders().toList()
Interpreter
vallines=sc.textFile("shakespeare.txt");valcounts=lines.flatMap(line=>line.split("")).map(word=>(word,1)).reduceByKey(_+_).sortBy(-_._2).map{case(w,c)=>Map("word"->w,"count"->c)}
%tablecounts"application/vnd.livy.table.v1+json":{"headers":[{"name":"count","type":"BIGINT_TYPE"},{"name":"name","type":"STRING_TYPE"}],"data":[[23407,"the"],[19540,"I"],[18358,"and"],...]}
![Page 55: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/55.jpg)
PLOT MAGIC
>
sparkIMain.interpret(“png(‘/tmp/plot.png’)barplotdev.off()”)
Interpreter
...barplot(sorted_data$count,names.arg=sorted_data$value,main="Resourcehits",las=2,col=colfunc(nrow(sorted_data)),ylim=c(0,300))
![Page 56: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/56.jpg)
PLOT MAGIC
>
sparkIMain.interpret(“png(‘/tmp/plot.png’)barplotdev.off()”)
Interpreter
...barplot(sorted_data$count,names.arg=sorted_data$value,main="Resourcehits",las=2,col=colfunc(nrow(sorted_data)),ylim=c(0,300))
![Page 57: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/57.jpg)
PLOT MAGIC
>png(‘/tmp/..’)>barplot>dev.off()
sparkIMain.interpret(“png(‘/tmp/plot.png’)barplotdev.off()”)
Interpreter
...barplot(sorted_data$count,names.arg=sorted_data$value,main="Resourcehits",las=2,col=colfunc(nrow(sorted_data)),ylim=c(0,300))
![Page 58: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/58.jpg)
PLOT MAGIC
>png(‘/tmp/..’)>barplot>dev.off()
sparkIMain.interpret(“png(‘/tmp/plot.png’)barplotdev.off()”)
File(’/tmp/plot.png’).read().toBase64()
Interpreter
...barplot(sorted_data$count,names.arg=sorted_data$value,main="Resourcehits",las=2,col=colfunc(nrow(sorted_data)),ylim=c(0,300))
![Page 59: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/59.jpg)
PLOT MAGIC
>png(‘/tmp/..’)>barplot>dev.off()
sparkIMain.interpret(“png(‘/tmp/plot.png’)barplotdev.off()”)
File(’/tmp/plot.png’).read().toBase64()
Interpreter
...barplot(sorted_data$count,names.arg=sorted_data$value,main="Resourcehits",las=2,col=colfunc(nrow(sorted_data)),ylim=c(0,300))
{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAe…"...}...}
![Page 60: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/60.jpg)
• PluggableBackends• Livy'sSparkBackends– Scala– pyspark– R
• IPython/Jupytersupportcomingsoon
PLUGGABLE INTERPRETERS
![Page 61: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/61.jpg)
• Re-usingit• GenericFrameworkforInterpreters
• 51Kernels
JUPYTER BACKEND
![Page 62: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/62.jpg)
SPARK AS A SERVICE
![Page 63: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/63.jpg)
REMEMBER AGAIN?YARNMaster
SparkClient
YARNNode
SparkContext
YARNNode
SparkWorker
YARNNode
SparkWorker
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
![Page 64: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/64.jpg)
MULTI USERS
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter YARN
Node
SparkContext
SparkInterpreter
YARNNode
SparkContext
SparkInterpreter
SparkClient
SparkClient
SparkClient
![Page 65: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/65.jpg)
SHARED CONTEXTS?
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
SparkClient
SparkClient
SparkClient
![Page 66: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/66.jpg)
SHARED RDD?
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
SparkClient
SparkClient
SparkClient
RDD
![Page 67: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/67.jpg)
SHARED RDDS?
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
SparkClient
SparkClient
SparkClient
RDD
RDD
RDD
![Page 68: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/68.jpg)
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
SparkClient
SparkClient
SparkClient
RDD
RDD
RDD
SECURE IT?
![Page 69: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/69.jpg)
YARNNode
SparkContext
LivyServer
Scalatra
SessionManager
Session
SparkInterpreter
SparkClient
SparkClient
SparkClient
RDD
RDD
RDD
SECURE IT?
![Page 70: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/70.jpg)
LivyServer
Spark
SparkClient
SparkClient
SparkClient
SPARK AS SERVICE
Spark
![Page 71: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/71.jpg)
SHARING RDDS
![Page 72: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/72.jpg)
PySparkshell
RDD
ShellPythonShell
![Page 73: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/73.jpg)
PySparkshell
RDD
ShellPythonShell
![Page 74: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/74.jpg)
PySparkshell
RDD
ShellPythonShell
r=sc.parallelize([])srdd=ShareableRdd(r)
![Page 75: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/75.jpg)
PySparkshell
RDD{'ak':'Alaska'}
{'ca':'California'}
ShellPythonShell
r=sc.parallelize([])srdd=ShareableRdd(r)
![Page 76: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/76.jpg)
PySparkshell
RDD{'ak':'Alaska'}
{'ca':'California'}
ShellPythonShell
curl-XPOST/sessions/0/statement{'code':srdd.get('ak')}
r=sc.parallelize([])srdd=ShareableRdd(r)
![Page 77: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/77.jpg)
PySparkshell
RDD{'ak':'Alaska'}
{'ca':'California'}
ShellPythonShell
states=SharedRdd('host/sessions/0','srdd')states.get('ak')
r=sc.parallelize([])srdd=ShareableRdd(r)
curl-XPOST/sessions/0/statement{'code':srdd.get('ak')}
![Page 78: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/78.jpg)
DEMO TIME
https://github.com/romainr/hadoop-tutorials-examples/tree/master/notebook/shared_rdd
![Page 79: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/79.jpg)
• SSLSupport• PersistentSessions• Kerberos
SECURITY
![Page 80: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/80.jpg)
SPARK MAGIC
• FromMicrosop
•PythonmagicsforworkingwithremoteSpark
clusters
•OpenSource:hAps://github.com/jupyter-
incubator/sparkmagic
![Page 81: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/81.jpg)
FUTURE
•Movetoextrepo?
• Security• iPython/Jupyterbackendsandfileformat
• SharednamedRDD/contexts?
• Sharedata• Sparkspecific,languagegeneric,both?• LeverageHue4
https://issues.cloudera.org/browse/HUE-2990
![Page 82: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/82.jpg)
• OpenSource:hAps://github.com/cloudera/
hue/tree/master/apps/spark/java
• Readaboutit:hAp://gethue.com/spark/
•Scala,Java,Python,R
•TypeIntrospec:onforVisualiza:on
•YARN-clusterorlocalmodes
•Codesnippets/compiled
•RESTAPI
•Pluggablebackends
•Magickeywords
•Failureresilient
•Security
LIVY’SCHEAT SHEET
![Page 83: Spark Summit Europe: Building a REST Job Server for interactive Spark as a service](https://reader030.vdocuments.us/reader030/viewer/2022020410/587332341a28ab596c8b6d3d/html5/thumbnails/83.jpg)
BEDANKT!
@gethue
USER GROUP
hue-user@
WEBSITE
hAp://gethue.com
LEARN
hAp://learn.gethue.com