apache nifi meetup - princeton nj 2016
TRANSCRIPT
![Page 1: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/1.jpg)
1 ©HortonworksInc.2011–2016.AllRightsReserved
ApacheNiFiOverviewTimothySpannSolutionsEngineerJuly14,20166:00PMFutureofData– PrincetonMeetup
![Page 2: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/2.jpg)
2 ©HortonworksInc.2011–2016.AllRightsReserved
Agenda
• 6:00pm – 6:45pm Registration and Food• 6:45pm – 7:00pm Introduction and Welcome• 7:00pm – 8:00pm Introduction to Apache NiFi by Tim Spann
• WhatisNiFi?• Demo• Hands-On
• 8:00pm – 8:45pm Creating a Custom Apache NiFi Processor by Milind Pandit
• 8:45pm – 9:00pm Wrap-up, Upcoming Kafka Talk
![Page 3: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/3.jpg)
3 ©HortonworksInc.2011–2016.AllRightsReserved
NIFITimothy Spann
![Page 4: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/4.jpg)
4 ©HortonworksInc.2011–2016.AllRightsReserved
Terminology
FlowFile• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)Processor• Performsthework,canaccessFlowFilesConnection• Linksbetweenprocessors• QueuesthatcanbedynamicallyprioritizedProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports
![Page 5: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/5.jpg)
5 ©HortonworksInc.2011–2016.AllRightsReserved
Installation
Download the binary from here: http://hortonworks.com/downloads/#dataflowOr here:https://nifi.apache.org/download.htmlOr on Mac:brew install nifi
https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#starting-nifi
bin/nifi.sh start
![Page 6: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/6.jpg)
6 ©HortonworksInc.2011–2016.AllRightsReserved
TheFutureofData:ActionableIntelligence
D A T A I N M O T I O N
ST
OR
AG
EST
OR
AG
E
GROUP 2GROUP 1
GROUP 4GROUP 3
D A T A A T R E S T
INTERNETOF
ANYTHING
Hortonworks’uniqueapproachtodata-in-motionanddata-at-restpowersActionableIntelligence
![Page 7: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/7.jpg)
7 ©HortonworksInc.2011–2016.AllRightsReserved
DATAATREST
DATAINMOTION
ACTIONABLEINTELLIGENCE
MODERNDATAAPPLICATIONS
ActionableIntelligencefromConnectedDataPlatforms
Capturingperishableinsightsfromdatainmotion
Ensuringrich,historicalinsightsondataatrest
Necessary formoderndataapplications
HortonworksDataFlow
HortonworksDataPlatform
![Page 8: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/8.jpg)
8 ©HortonworksInc.2011–2016.AllRightsReserved
Data-in-motion:HortonworksDataFlowPoweredbyApacheNiFi
Collect,conductandcuratereal-timedata
End-to-endsecuritywithencryptionandrules
Traceabilityandreal-timeprovenance
DeliversInstant,PerishableInsights
HortonworksDataFlow
Data-in-motion
![Page 9: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/9.jpg)
9 ©HortonworksInc.2011–2016.AllRightsReserved
Accumulate,Analyze,ActonAllData
CentralizedArchitectureforMulti-tenancy
EnterpriseOperations,GovernanceandSecurity
DeliversRichHistoricalInsights
Data-at-rest:HortonworksDataPlatformPoweredbyApacheHadoop
HortonworksDataPlatform
Data-at-rest
![Page 10: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/10.jpg)
10 ©HortonworksInc.2011–2016.AllRightsReserved
ApacheNiFi• Powerfulandreliablesystemtoprocessanddistributedata
• Directedgraphsofdataroutingandtransformation
• Web-basedUserInterfaceforcreating,monitoring,&controllingdataflows
• Highlyconfigurable- modifydataflowatruntime,dynamicallyprioritizedata
• DataProvenancetracksdatathroughentiresystem
• Easilyextensiblethroughdevelopmentofcustomcomponents
[1]https://nifi.apache.org/
![Page 11: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/11.jpg)
11 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - Terminology
à FlowFile• Unitofdatamovingthroughthesystem• Content+Attributes(key/valuepairs)
à Processor• Performsthework,canaccessFlowFiles
à Connection• Linksbetweenprocessors• Queuesthatcanbedynamicallyprioritized
à ProcessGroup• Setofprocessorsandtheirconnections• Receivedataviainputports,senddataviaoutputports
![Page 12: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/12.jpg)
12 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - UserInterface
• Draganddropprocessorstobuildaflow• Start,stop,andconfigurecomponentsinrealtime• Viewerrorsandcorrespondingerrormessages• Viewstatisticsandhealthofdataflow• Createtemplatesofcommonprocessor&connections
![Page 13: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/13.jpg)
13 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - Provenance
• Tracksdataateachpointasitflowsthroughthesystem
• Records,indexes,andmakeseventsavailablefordisplay
• Handlesfan-in/fan-out,i.e.mergingandsplittingdata
• Viewattributesandcontentatgivenpointsintime
![Page 14: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/14.jpg)
14 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - QueuePrioritization
• Configureaprioritizer perconnection
• Determinewhatisimportantforyourdata– timebased,arrivalorder,importanceofadataset
• Funnelmanyconnectionsdowntoasingleconnectiontoprioritizeacrossdatasets
• Developyourownprioritizer ifneeded
![Page 15: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/15.jpg)
15 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - Extensibility
à Builtfromthegroundupwithextensionsinmind
à Service-loaderpatternfor…• Processors• ControllerServices• ReportingTasks• Prioritizers
à ExtensionspackagedasNiFi Archives(NARs)• DeployNiFi libdirectory andrestart• ProvidesClassLoader isolation• Samemodelasstandardcomponents
![Page 16: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/16.jpg)
16 ©HortonworksInc.2011–2016.AllRightsReserved
NiFi - Architecture
OS/Host
JVM
FlowController
WebServer
Processor1 ExtensionN
FlowFileRepository
ContentRepository
ProvenanceRepository
LocalStorage
OS/Host
JVM
FlowController
WebServer
Processor1 ExtensionN
FlowFileRepository
ContentRepository
ProvenanceRepository
LocalStorage
OS/Host
JVM
NiFiClusterManager– RequestReplicator
WebServer
MasterNiFiClusterManager(NCM)
OS/Host
JVM
FlowController
WebServer
Processor1 ExtensionN
FlowFileRepository
ContentRepository
ProvenanceRepository
LocalStorage
SlavesNiFiNodes
![Page 17: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/17.jpg)
17 ©HortonworksInc.2011–2016.AllRightsReserved
RapidEcosystemAdoption:130+Processors
HTTP
Syslog
HTML
Image
Hash Encrypt
Extract
TailMerge
Evaluate
Duplicate Execute
Scan
GeoEnrich
Replace
ConvertSplit
Translate
HL7
FTP
UDP
XML
SFTP
RouteContent
RouteContext
RouteText
ControlRate
DistributeLoadAMQP
![Page 18: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/18.jpg)
18 ©HortonworksInc.2011–2016.AllRightsReserved
Addprocessorfordataintake.Time:1minute1 Draganddropprocessoriconfromthetopmenu
![Page 19: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/19.jpg)
19 ©HortonworksInc.2011–2016.AllRightsReserved
![Page 20: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/20.jpg)
20 ©HortonworksInc.2011–2016.AllRightsReserved
à https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/
à https://github.com/jfrazee/awesome-nifi
à https://dzone.com/articles/getting-started-with-apache-nifi-and-hdf
à https://nifi.apache.org/docs.html
à https://community.hortonworks.com/articles/4356/getting-started-with-nifi-expression-language-and.html
LearningMore
![Page 21: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/21.jpg)
21 ©HortonworksInc.2011–2016.AllRightsReserved
Contact:
[email protected]/futureofdata-princeton
community.hortonworks.com/users/9304/tspann.html
![Page 22: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/22.jpg)
22 ©HortonworksInc.2011–2016.AllRightsReserved
HortonworksCommunityConnection
Read access for everyone, join to participate and be recognized
• FullQ&APlatform(likeStackOverflow)
• KnowledgeBaseArticles
• CodeSamplesandRepositories
![Page 23: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/23.jpg)
23 ©HortonworksInc.2011–2016.AllRightsReserved
CommunityEngagement
Participate now at: community.hortonworks.com©HortonworksInc.2011–2015.AllRightsReserved
4,000+RegisteredUsers
10,000+Answers
15,000+TechnicalAssets
One Website!
![Page 24: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/24.jpg)
24 ©HortonworksInc.2011–2016.AllRightsReserved
WePowertheEcosystem
1600+ Partners
3000+members
11,000+Weekly visitors
![Page 25: Apache NiFi Meetup - Princeton NJ 2016](https://reader031.vdocuments.us/reader031/viewer/2022030316/587a344a1a28abdb1c8b549b/html5/thumbnails/25.jpg)
25 ©HortonworksInc.2011–2016.AllRightsReserved