![Page 2: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/2.jpg)
Outline
1. What is energy disaggregation?
2. My dataset: UKDALE
3. Opensource Python tool: NILMTK
4. Metadata
![Page 3: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/3.jpg)
What is energy disaggregation?
![Page 4: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/4.jpg)
![Page 5: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/5.jpg)
![Page 6: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/6.jpg)
![Page 7: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/7.jpg)
UKDALE: UK Disaggregated ApplianceLevel Energy data
● 5 homes● Duration: 39 days to 3.5 years of data per house● Once every 6 seconds records wholehouse
apparent power and appliance power demand● CSV with YAML metadata● HDF5 version● Records wholehouse voltage and current at 16kHz
for 3 homes (stored as hourly FLAC files)● 6 terabytes● Recording hardware: offtheshelf and DIY● Available on UKERC EDC (with DOIs) and FTP● Paper in NPG's Scientific Data journal
![Page 8: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/8.jpg)
![Page 9: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/9.jpg)
![Page 10: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/10.jpg)
![Page 11: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/11.jpg)
![Page 12: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/12.jpg)
![Page 13: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/13.jpg)
![Page 14: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/14.jpg)
![Page 15: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/15.jpg)
![Page 16: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/16.jpg)
![Page 17: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/17.jpg)
Opensource Python disaggregation toolkit: NILMTK
● nilmtk.github.io● (NILM = NonIntrusive Load Monitoring)● Imports 11 datasets to standard format using
HDF5 and detailed metadata● Data cleaning, summary statistics, plotting etc.● 6 NILM algorithms (1 hosted by algo author)● NILM metrics● Can process more data than can fit into RAM● Documentation and code examples● Issue queue (501); mailing list; twitter; wiki
![Page 18: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/18.jpg)
![Page 19: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/19.jpg)
![Page 20: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/20.jpg)
NILMTK: What worked well?
● 11 dataset importers: lots of contributed code● Standalone data conversion scripts are simple to write● People definitely tinker with NILMTK and cite it● Not clear how many people use it for productive work!● 6 NILM algorithms (although this required a lot of effort)● Lots of discussions on the issue queue● Lazy loading & outofcore is powerful (but complex)● Built on existing packages (Pandas etc.)● TESTS! ● Continuous integration (TravisCI)
![Page 21: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/21.jpg)
NILMTK: What didn't work well?
● Outofcore is perhaps not worth the effort: makes the code much more complex. Instead use Blaze?
● Too “monolithic”. Should interact with existing tools more naturally. Perhaps separate NILMTK into separate tools?
● Documentation needs to be extensive but uptodate. This is hard work!
● Issue queue is hard work. Lots of “I don't know how to use Python; please hold my hand”.
● “Need to move to stand still”: keeping up to date with new versions of dependencies (Pandas etc.)
![Page 22: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/22.jpg)
NILMTK: Conclusions
● KISS (especially if you want 3rd party contributions)● Focus on highquality documentation for users: will save
you time in the longrun● Make full use of automatic package managers so users can
install your code & dependencies easily
![Page 23: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/23.jpg)
Datasets are complex
● Many meters● Each connected to some number of appliances● Arbitrary wiring between meters● Different meter models● Specify details of appliances● Preprocessing applied to data● etc.
![Page 24: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/24.jpg)
NILM Metadata schema
● YAML (YAML Ain't Markup Language)● Controlled vocab, object inheritance● Schema has 2 components:
Schema for describing individual datasetsDataset, buildings, meters, appliances, measurements
Common metadata about appliance typesCategories, priors, models of appliances, countries
![Page 25: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/25.jpg)
![Page 26: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/26.jpg)
![Page 27: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/27.jpg)
![Page 28: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/28.jpg)
dataset.yaml
![Page 29: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/29.jpg)
![Page 30: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/30.jpg)
![Page 31: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/31.jpg)
![Page 32: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/32.jpg)
![Page 33: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/33.jpg)
![Page 34: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/34.jpg)
meter_devices.yaml
![Page 35: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/35.jpg)
![Page 36: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/36.jpg)
![Page 37: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/37.jpg)
![Page 38: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/38.jpg)
# building1.yaml
appliances: type: washer dryer instance: 1 meters: [10, 20] components: type: motor meters: [10] type: electric heating element meters: [20]
Example of an appliance containing other 'appliances'
![Page 39: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/39.jpg)
![Page 40: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/40.jpg)
![Page 41: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/41.jpg)
![Page 42: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/42.jpg)
![Page 43: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/43.jpg)
![Page 44: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/44.jpg)
![Page 45: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/45.jpg)
![Page 46: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/46.jpg)
Example of dataset metadata
![Page 47: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/47.jpg)
Example of dataset metadata
![Page 48: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/48.jpg)
Example of dataset metadata
![Page 49: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/49.jpg)
Example of dataset metadata
![Page 50: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/50.jpg)
Example of dataset metadata
![Page 51: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/51.jpg)
Example of dataset metadata
![Page 52: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/52.jpg)
Example of dataset metadata
![Page 53: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/53.jpg)
Example of dataset metadata
![Page 54: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/54.jpg)
Example of dataset metadata
![Page 55: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/55.jpg)
Example of dataset metadata
![Page 56: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/56.jpg)
Example of dataset metadata
![Page 57: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/57.jpg)
Example of dataset metadata
![Page 58: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/58.jpg)
Example of central metadata
![Page 59: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/59.jpg)
NILM Metadata schema in use
● Used by NILMTK● At least 11 datasets use NILM Metadata● github.com/NILMTK/NILM_Metadata● Lots of contributions to controlled vocabulary
![Page 60: Metadata for Energy and working with Python - Bath · Metadata for Energy and working with Python Jack Kelly Imperial College London jack.kelly@imperial.ac.uk. Outline 1. ... CSV](https://reader030.vdocuments.us/reader030/viewer/2022021802/5b6a26f77f8b9a8d058bab30/html5/thumbnails/60.jpg)
NILM Metadata: future work?
● Make it as easy as possible to use: simple profile?● e.g. detailed appliance metadata not very useful?● Reskin as RDF and OWL?!● Defaults● Wizard● Validator● Micro generation