reference architecture subgroup nist big data public working group reference architecture subgroup...
TRANSCRIPT
Reference Architecture Subgroup
NIST Big Data Public Working Group
Reference Architecture Subgroup
September 30, 2013
Co-chairs:Orit Levin MicrosoftJames Ketner AT&TDon Krapohl Augmented Intel
Reference Architecture Subgroup 2
Agenda
• Deliverable #1: White Paper: Survey of Existing Big Data RAs
• Deliverable #2: NIST Big Data Reference Architecture• Next Steps
Reference Architecture Subgroup
NIST SurveyBig Data Architecture Models
Input Document M0151
Reference Architecture Subgroup 4
List Of Surveyed Architectures
• Vendor-neutral and technology-agnostic proposals– Bob Marcus ET-Strategies– Orit Levin Microsoft– Gary Mazzaferro AlloyCloud– Yuri Demchenko University of Amsterdam
• Vendors’ Architectures– IBM– Oracle– Booz Allen Hamilton– EMC– SAP– 9sight– LexusNexis
Reference Architecture Subgroup 5
Vendor-neutral and Technology-agnostic Proposals
Data Processing Flow
M0039
Data Transformation Flow
M0017
IT StackM0047
Reference Architecture Subgroup 6
Vendor-neutral and Technology-agnostic Proposals
Data Processing Flow
M0039
Data Transformation Flow
M0017
IT StackM0047
Reference Architecture Subgroup 7
Vendor-neutral and Technology-agnostic Proposals
Data Processing Flow
M0039
IT StackM0047
Data Transformation Flow
M0017
Reference Architecture Subgroup 8
Vendor-neutral and Technology-agnostic Proposals
Data Transformation Flow
M0017
IT StackM0047
Data Processing Flow
M0039
Reference Architecture Subgroup 9
Draft Agreement / Rough Consensus
• Transformation includes– Processing functions– Analytic functions– Visualization functions
• Data Infrastructure includes– Data stores– In-memory DBs– Analytic DBs
Sources
Transformation
Usage Data
In
frast
ruct
ure
Secu
rity
Man
ag
em
en
t
Clo
ud
Com
pu
tin
g
Netw
ork
Reference Architecture Subgroup
NIST BIG DATAReference Architecture
Input Document M0226
Reference Architecture Subgroup 11
• A superset of a “traditional data” system
• A representation of a vendor-neutral and technology-agnostic system
• A functional architecture comprised of logical roles
• Applicable to a variety of business models– Tightly-integrated enterprise
systems– Loosely-coupled vertical
industries
• A business architecture representing internal vs. external functional boundaries
• A deployment architecture
• A detailed IT RA of a specific system implementation
All of the above will be developed in the next stage in the context of specific use cases.
What the Baseline Big Data RAIs Is Not
Reference Architecture Subgroup
Main Functional Blocks
12
Big Data Application Provider
System Orchestrator
Data
C
on
su
mer
Data
P
rovid
er
Big Data Framework Provider
• Application Specific• Identity Management &
Authorization• Etc.
• Discovery of data• Description of data• Access to data• Code execution on data• Etc.
• Discovery of services• Description of data• Visualization of data• Rendering of data• Reporting of data• Code execution on data• Etc.
• Analytic processing of data• Machine learning• Code execution on data et situ• Storage, retrieval, search, etc.
of data• Providing computing
infrastructure• Providing networking
infrastructure• Etc.
Reference Architecture Subgroup
Big Data Lifecycle
13
Big Data Application Provider
System Orchestrator
DATA
DATA
Visualization Access
AnalyticsCuration Collection D
ata
C
on
su
mer
Data
P
rovid
er
Big Data Framework Provider
DA
TA
Reference Architecture Subgroup
Big Data Frameworks
14
Big Data Application Provider
Visualization Access
AnalyticsCuration Collection
System Orchestrator
DATA
DATA
Data
C
on
su
mer
Data
P
rovid
er
Horizontally Scalable (VM clusters)
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Processing Frameworks (analytic tools, etc.)
Platforms (databases, etc.)
Infrastructures
Physical and Virtual Resources (networking, computing, etc.)
DA
TA
Big Data Framework Provider
Reference Architecture Subgroup
Bringing Tools to the Data
15
Big Data Application Provider
Visualization Access
AnalyticsCuration Collection
System Orchestrator
DATASW
DATASW
Data
C
on
su
mer
Data
P
rovid
er
Horizontally Scalable (VM clusters)
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Big Data Framework ProviderProcessing Frameworks (analytic tools, etc.)
Platforms (databases, etc.)
Infrastructures
Physical and Virtual Resources (networking, computing, etc.)
DA
TA S W
Reference Architecture Subgroup
Ma
na
ge
me
nt
Se
cu
rit
y &
P
riv
ac
y
16
Big Data Application Provider
Visualization Access
AnalyticsCuration Collection
System Orchestrator
DATASW
DATASW
INFORMATION VALUE CHAIN
IT V
AL
UE
C
HA
IN
Data
C
on
su
mer
Data
P
rovid
er
Horizontally Scalable (VM clusters)
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Horizontally Scalable
Vertically Scalable
Big Data Framework ProviderProcessing Frameworks (analytic tools, etc.)
Platforms (databases, etc.)
Infrastructures
Physical and Virtual Resources (networking, computing, etc.)
DA
TA S W
Reference Architecture Subgroup 17
Outline
Executive Summary1 Introduction2 Big Data System Requirements3 Conceptual Model4 Main Components
4.1 Data Provider4.2 Big Data Application Provider4.3 Big Data Framework Provider4.4 Data Consumer4.5 System Orchestrator
5 Management5.1 System Management5.2 Lifecycle Management
6 Security and Privacy7 Big Data TaxonomyAppendix A: Terms and DefinitionsAppendix B: AcronymsAppendix C: ReferencesAppendix D: Deployment Considerations1 Big Data Framework Provider1.1 Traditional On-Premise Frameworks1.2 Cloud Service Providers
Reference Architecture Subgroup 18
Summary
• Summary– The NIST Big Data functional reference architecture (RA
v.1.0) is available for review as input document M0226.
• Next Steps– Continue the editorial and alignment effort– Map generic Big Data use cases to RA– Map specific collected Big Data cases to RA
Let’s exchange additional ideas this afternoon at the breakout session!