the big metadata
TRANSCRIPT
![Page 1: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/1.jpg)
The Big Metadata
![Page 2: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/2.jpg)
Stories from the dark underbelly of data
operations
![Page 3: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/3.jpg)
By Daniela Tomova
![Page 4: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/4.jpg)
Origin Story
![Page 5: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/5.jpg)
![Page 6: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/6.jpg)
ID=112056 Name=Julia’s file
![Page 7: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/7.jpg)
What was Julia’s file?
![Page 8: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/8.jpg)
Who was Julia?
![Page 9: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/9.jpg)
What is metadata?
![Page 10: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/10.jpg)
Data: qualifies or quantifies a concept or a real-world occurence, often in the form of a variable across time. Used to measure and understand.
Metadata: classifies and describes data. Used to understand, structure, track and manipulate data.
What is metadata?
![Page 11: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/11.jpg)
Metadata and ”dumb” data
This commentator is basically your average data.
![Page 12: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/12.jpg)
What is metadata?
ID Time Dimension 1 Time Dimension 2 Value
112056 27-11-2006 23:00:00 28-11-2006 01:00:00 830
112056 27-11-2006 23:00:00 28-11-2006 02:00:00 12.7
Descriptive or Semantic Metadata:
CommodityVariableContract typeFacility typeTechnologyGeographySectorEtc.
Structural or Technical Metadata:
Creation dateOrigin systemSet IDPublication freq.Value freq.Variable typeChange dateSourceSource fileEtc.
![Page 13: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/13.jpg)
Precisely!
We cannot afford not to use metadata:
- Structure, traceability and common standards save time and resources. The more data – the greater the savings.
- Matadata removes the human bottleneck. Enables data usage and reusage by both people and processes.
But that’s even more data! Don’t we have enough/too much already?
![Page 14: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/14.jpg)
No.
-Aggregation. Easier to process than the underlying data even across sets and dimensions.
- Abstraction. Easier for people with different levels of experience to understand.
- Tool. It has a bi-directional relationship with its subject and can be used to manipulate it.
Just data about data?
![Page 15: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/15.jpg)
- Julia’s File or WeaCity.ECeENS_Europe.Precip;;WeaCity;PC;EC.Ens;F;H.12;UTC;SVK.SK01.BRATIS;Wea.Precip;mm;H.6;;03;
How do we use it?
![Page 16: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/16.jpg)
Common standard
![Page 17: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/17.jpg)
Result
Multiple tuples linked to a curve ID:
![Page 18: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/18.jpg)
Application dictionaryEasy, powerful, and robust Matlab quieries.
Easy groupings of data in containers: charts, files, tables.
Reusable and pivotable code.
Efficient manipulation of groups of curves.
Powerful and scalable monitoring and debugging of large amounts of heterogenous data.
![Page 19: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/19.jpg)
Human dictionary
A store of analyst knowledge about the data in a common vocabulary.
Searchable
![Page 20: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/20.jpg)
Some cool stuff which would be impossible without meta
Smart homes and IoTMachine learningNatural language processingBitcoin operations and new uses for the blockchain meta Tergeted online contentSmart gridsBig data analysisModern video and audio librariesiTunes
![Page 21: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/21.jpg)
Future usesEmergent algorithms – like those underpinning swarm intelligence behavior and artificial neural networks
Emergent technology – technology the effects of which are greated than its building blocks
Singularity?
![Page 22: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/22.jpg)
SummaryHumans are not optimised for raw data processing.
We think in abstractions, relationships and tool manipulation. If we want to keep up with data, we need to shape it to the way our brains work.
That’s what metadata does.
![Page 23: The Big Metadata](https://reader036.vdocuments.us/reader036/viewer/2022062412/58d1ece51a28ab51448b5a8d/html5/thumbnails/23.jpg)
“I've seen things you people wouldn't believe…” – Roy in Blade Runner
Questions?