sharing data, hiding complexity · 2012. 12. 6. · james henderson techmesh 2012. sharing data,...
TRANSCRIPT
![Page 1: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/1.jpg)
Sharing Data, Hiding Complexitywith RDF and Clojure
Malcolm SparksJames Henderson
TechMesh 2012
![Page 2: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/2.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Data: Setting the scene
Financial, regulatory pressures in banking are presenting ITwith new challenges.
How can we process data :-
I Faster (as always)
I At greater volumes (risk sensitivities - HistSim)
I More precisely (finer granualarity - Dodd Frank initialmargin)
![Page 3: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/3.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Data: Setting the scene
Financial, regulatory pressures in banking are presenting ITwith new challenges.How can we process data :-
I Faster (as always)
I At greater volumes (risk sensitivities - HistSim)
I More precisely (finer granualarity - Dodd Frank initialmargin)
![Page 4: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/4.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Data: Setting the scene
Financial, regulatory pressures in banking are presenting ITwith new challenges.How can we process data :-
I Faster (as always)
I At greater volumes (risk sensitivities - HistSim)
I More precisely (finer granualarity - Dodd Frank initialmargin)
![Page 5: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/5.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Data: Setting the scene
Financial, regulatory pressures in banking are presenting ITwith new challenges.How can we process data :-
I Faster (as always)
I At greater volumes (risk sensitivities - HistSim)
I More precisely (finer granualarity - Dodd Frank initialmargin)
![Page 6: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/6.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Data: Setting the scene
Financial, regulatory pressures in banking are presenting ITwith new challenges.How can we process data :-
I Faster (as always)
I At greater volumes (risk sensitivities - HistSim)
I More precisely (finer granualarity - Dodd Frank initialmargin)
![Page 7: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/7.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Conway’s Law
“Organizations which design systems ... areconstrained to produce designs which are copies ofthe communication structures of theseorganizations”
Mel Conway (1968)
![Page 8: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/8.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Conway in action
Processes
StorageBob
Trade Database
Custom
Alice
Extraction
NFS
CORBA/SQL
Pricing
Risk Database
RMI
Publication
XML
Loading
Oracle CoherenceJava
SFTP
Figure : An example communication structure
Q: Do Bob and Alice get the same view of the data?
![Page 9: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/9.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
What does it all mean?
CommonReference Narrative NumericId LegalEntityManagementArea GLRegion LocalBankingTradingAggregationLevel SettlementEngineSource ISIN CUSIPPrivatePlacement OtherSecurityId FinancialProductTypeReportingCurve InternalExternalFlag BuySell Fas133FlagSpvId MtnId LinkageReference LinkageReference2 NodeRefReasonCode YNFlag
Observation: Data doesn’t travel well
![Page 10: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/10.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
What went wrong?
Processes
StorageBob
Trade Database
Custom
Alice
Extraction
NFS
CORBA/SQL
Pricing
Risk Database
RMI
Publication
XML
Loading
Oracle CoherenceJava
SFTP
Conclusion: We need our data models to span across oursystems.
![Page 11: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/11.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 12: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/12.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 13: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/13.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 14: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/14.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 15: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/15.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 16: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/16.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 17: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/17.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 18: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/18.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Does modelling scale?
As we scale up :-
I Complexity increases
I Deviations increase
I Concepts become more abstract
I Understanding decreases
As data escapes its domain model :-
I It loses assumptions (meaning)
I Local identifiers fail to become global ones.
![Page 19: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/19.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Implications
I Our primary tool for flexible data processing (datamodelling) does not scale!
I We are fire-fighting this problem with a code hosepipe!
![Page 20: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/20.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Implications
I Our primary tool for flexible data processing (datamodelling) does not scale!
I We are fire-fighting this problem with a code hosepipe!
![Page 21: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/21.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Introduction to RDF
The cat the matsat on
http://animals.com/ginger
http://actions.com/satOn http://tech-mesh-hotel.co.uk/mat ;
rdf:type http://animals.com/Cat ;
.
http://actions.com/satOn
rdfs:label "sat on" ;
rdfs:comment "The action of sitting on a given object." ;
.
![Page 22: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/22.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
RDF features
I URIs - globally unique identification and transparency
I Namespaces (for federated governance)
I Storage and transfer neutrality
I Metadata
![Page 23: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/23.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
RDF features
I URIs - globally unique identification and transparency
I Namespaces (for federated governance)
I Storage and transfer neutrality
I Metadata
![Page 24: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/24.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
RDF features
I URIs - globally unique identification and transparency
I Namespaces (for federated governance)
I Storage and transfer neutrality
I Metadata
![Page 25: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/25.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
RDF features
I URIs - globally unique identification and transparency
I Namespaces (for federated governance)
I Storage and transfer neutrality
I Metadata
![Page 26: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/26.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
RDF features
I URIs - globally unique identification and transparency
I Namespaces (for federated governance)
I Storage and transfer neutrality
I Metadata
![Page 27: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/27.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Observations
RDF has the power to break silos
RDF is an non-disruptive way of applying federated datagovernance
![Page 28: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/28.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
The RAP
I Exposing the data in the ‘ODS’
I The tables in the ODS are built from a meta-model
I We get the meta-model in XML format
![Page 29: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/29.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
The RAP
I Exposing the data in the ‘ODS’
I The tables in the ODS are built from a meta-model
I We get the meta-model in XML format
![Page 30: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/30.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
The RAP
I Exposing the data in the ‘ODS’
I The tables in the ODS are built from a meta-model
I We get the meta-model in XML format
![Page 31: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/31.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
The RAP
I Exposing the data in the ‘ODS’
I The tables in the ODS are built from a meta-model
I We get the meta-model in XML format
![Page 32: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/32.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
So why do we transform it to RDF?
I Disambiguation of terms
I Merging different data sources
![Page 33: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/33.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
So why do we transform it to RDF?
I Disambiguation of terms
I Merging different data sources
![Page 34: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/34.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
So why do we transform it to RDF?
I Disambiguation of terms
I Merging different data sources
![Page 35: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/35.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Exposing a data dictionary
![Page 36: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/36.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Forming a View
![Page 37: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/37.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Forming a View
A View in graph form:
Author
Surname
attribute
Book
join - Book-Author
Title
attribute
![Page 38: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/38.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Forming a View
RDF storage of a view:
views:demo-view
<select> [ <node> rap:Author ;
<selectedAttribute> [ <attribute> rap:Author-Name ; ] ;
<childNode> [ <node> rap:Book ;
<relationship> palace:Book-Author ;
<selectedAttribute>
[ <attribute> rap:Book-Title ; ] ; ] ; ]
.
![Page 39: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/39.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Delivering the View
Generate the SQL
Receive the SQL results
Convert to intermediate data structure
JSON XML RDF CSV
![Page 40: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/40.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
The Power of Graphs
![Page 41: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/41.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Storing config in RDF
<hosts/p21> <instance> <instances/PROD1> .
<hosts/u11> <instance> <instances/UAT2> .
<hosts/d30> <instance> <instances/SIT1> .
![Page 42: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/42.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Storing config in RDF
<hosts/p21> cmdb:hostName "p21.db.com" .
<instances/PROD1> <service> <services/PROD1/http> .
<services/PROD1/http> cmdb:port 8080 .
![Page 43: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/43.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Why RDF for config?
I Disambiguation
I Meta-data
I Querying the config
I Multiple config sources
![Page 44: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/44.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Why RDF for config?
I Disambiguation
I Meta-data
I Querying the config
I Multiple config sources
![Page 45: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/45.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Why RDF for config?
I Disambiguation
I Meta-data
I Querying the config
I Multiple config sources
![Page 46: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/46.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Why RDF for config?
I Disambiguation
I Meta-data
I Querying the config
I Multiple config sources
![Page 47: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/47.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
Graph-Zip:
https://github.com/james-henderson/graph-zip
![Page 48: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/48.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
An example config graph:<hosts/p21>
<instances/PROD1>
:instance
’p21.db.com’
:cmdb/hostname
<services/PROD1/http>
:service
<services/PROD1/swank>
:service
<jvms/PROD1>
:jvm
8080
:cmdb/port
<HttpService>
:rdf/type
4005
:cmdb/port
127.0.0.1
:cmdb/bindTo
<SwankService>
:rdf/type
2048m
:jvmMaxHeap
![Page 49: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/49.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
Importing the graph:
(def config-map
(build-in-memory-graph
[{:subject "<hosts/p21>" :property :hostname :object "p21.db.com"}
{:subject "<hosts/p21>" :property :instance :object "<instances/PROD1>"}
{:subject "<instances/PROD1>" :property :service :object "<services/PROD1/swank>"}
{:subject "<services/PROD1/swank>" :property :port :object "4005"}
{:subject "<services/PROD1/swank>" :property :rdf/type :object "<SwankService>"}
{:subject "<instances/PROD1>" :property :service :object "<services/PROD1/http>"}
{:subject "<services/PROD1/http>" :property :port :object "8080"}
{:subject "<services/PROD1/http>" :property :binds-to :object "127.0.0.1"}
{:subject "<services/PROD1/http>" :property :rdf/type :object "<HttpService>"}
{:subject "<instances/PROD1>" :property :jvm :object "<jvms/PROD1>"}
{:subject "<jvms/PROD1>" :property :jvmMaxHeap :object "2048m"}]))
![Page 50: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/50.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
An example config graph:<hosts/p21>
<instances/PROD1>
:instance
’p21.db.com’
:cmdb/hostname
<services/PROD1/http>
:service
<services/PROD1/swank>
:service
<jvms/PROD1>
:jvm
8080
:cmdb/port
<HttpService>
:rdf/type
4005
:cmdb/port
127.0.0.1
:cmdb/bindTo
<SwankService>
:rdf/type
2048m
:jvmMaxHeap
Building the zipper:
(def prod-host-zipper (graph-zip config-map "<hosts/p21>"))
![Page 51: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/51.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
An example config graph:<hosts/p21>
<instances/PROD1>
:instance
’p21.db.com’
:cmdb/hostname
<services/PROD1/http>
:service
<services/PROD1/swank>
:service
<jvms/PROD1>
:jvm
8080
:cmdb/port
<HttpService>
:rdf/type
4005
:cmdb/port
127.0.0.1
:cmdb/bindTo
<SwankService>
:rdf/type
2048m
:jvmMaxHeap
Navigating the graph:
(zip-> prod-host-zipper
:instance
:service
node)
;; -> ("<services/PROD1/swank>" "<services/PROD1/http>")
![Page 52: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/52.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
An example config graph:<hosts/p21>
<instances/PROD1>
:instance
’p21.db.com’
:cmdb/hostname
<services/PROD1/http>
:service
<services/PROD1/swank>
:service
<jvms/PROD1>
:jvm
8080
:cmdb/port
<HttpService>
:rdf/type
4005
:cmdb/port
127.0.0.1
:cmdb/bindTo
<SwankService>
:rdf/type
2048m
:jvmMaxHeap
Finding out the Swank port:
(zip1-> prod-host-zipper
:instance
:service
[(prop= :rdf/type "<SwankService>")]
:cmdb/port
node)
;; -> "4005"
![Page 53: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/53.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Querying the config
Graph-Zip:
https://github.com/james-henderson/graph-zip
Neo-Zip:
https://github.com/james-henderson/neo-zip
![Page 54: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/54.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
![Page 55: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/55.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
In XML:
<hosts>
<host>
<hostname>p21.db.com</hostname>
<instances>
<instance>
<label>PROD1</label>
<services>
<service>
<type>swank</type>
<port>4005</port>
<bindsTo>127.0.0.1</bindsTo>
</service>
<service>
<type>http</type>
<port>8080</port>
</service>
</services>
</instance>
</instances>
</host>
...
</hosts>
![Page 56: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/56.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
Using traditional zip syntax:
(for [host (xml-> doc :host)]
{:hostname (xml1-> host :hostname text)
:instances (for [instance (xml-> host :instances :instance)]
{:label (xml1-> instance :label text)
:services (for [service (xml-> instance :services :service)]
{:type (xml1-> service :type text)
:port (xml1-> service :port text)
:bindsTo (xml1-> service :bindsTo text)})})})
![Page 57: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/57.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
Using a datalog syntax:
(let [db (d/db conn)]
(for [[host hostname] (d/q ’{:find [?h ?name]
:where [[?h :type :host]
[?h :hostname ?name]]}
db)]
{:hostname hostname
:instances (for [[instance label]
(d/q ’{:find [?inst ?label]
:in [$ ?host]
:where [[?host :instance ?inst]
[?inst :label ?label]]}
db host)]
{:label label
:services (for [[service type port binds-to]
(d/q ’{:find [?service ?type ?port ?binds-to]
:in [$ ?instance]
:where [[?instance :service ?service]
[?service :port ?port]
[?service :type ?type]
[?service :binds-to ?binds-to]]}
db instance)]
{:type type
:port port
:binds-to binds-to})})}))
![Page 58: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/58.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
An enhanced zipper query:
(xml-> doc
:host
{:hostname ^:1 [:hostname text]
:instances [:instances
:instance
{:instance-name ^:1 [:label text]
:services [:services
:service
{:type ^:1 [:type text]
:port ^:1 [:port text]
:interface ^:1 [:binds-to text]}]}]})
![Page 59: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/59.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
Combining Zippers with RAP Views
An enhanced zipper query:
(xml-> doc
:host
{:hostname ^:1 [:hostname text]
:instances [:instances
:instance
{:instance-name ^:1 [:label text]
:services [:services
:service
{:type ^:1 [:type text]
:port ^:1 [:port text]
:interface ^:1 [:binds-to text]}]}]})
Resulting data structure:
[{:hostname "p21.db.com"
:instances [{:instance-name "PROD1"
:services [{:type "swank"
:port "4005"
:interface "127.0.0.1"}
{:type "http"
:port "8080"}]}]}
{:hostname "u11.db.com"
:instances [{:instance-name "UAT1" :services ...}
{:instance-name "UAT2" :services ...}]}
{:hostname "d30.db.com"
:instances [{:instance-name "SIT1" :services ...}]}]
![Page 60: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/60.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
![Page 61: Sharing Data, Hiding Complexity · 2012. 12. 6. · James Henderson TechMesh 2012. Sharing Data, Hiding Complexity Malcolm Sparks James Henderson Data Loses Fidelity As You Move It](https://reader033.vdocuments.us/reader033/viewer/2022052105/60401ea39c27e702c91fad13/html5/thumbnails/61.jpg)
Sharing Data,Hiding Complexity
Malcolm SparksJames Henderson
Data Loses FidelityAs You Move It
RDF and the RAP
The Power ofGraphs
James Henderson@jhenderson 89https://github.com/james-henderson
Malcolm Sparks
@malcolmsparkshttps://github.com/malcolmsparks