metadata for material science - at the lightsource bessy ii...bessy ii experiments - general...
TRANSCRIPT
![Page 1: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/1.jpg)
Metadata for material scienceat the Lightsource BESSY II
T. Birke1,H. Gorzig1, V. Laux1, T. Mertens1,R. Muller1, M. Ries1, A. Schaelicke1, P.Schnizer1, T. Unold1, L. Vera Ramirez1, J. Viefhaus1
1Helmholtz-Zentrum Berlin (HZB), Germany
11/07/2019
![Page 2: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/2.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 2
Table of Contents
![Page 3: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/3.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 3
Table of Contents
![Page 4: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/4.jpg)
courtesy of M. Ries
Many 6= measurement deviceswith own (meta)data + beamlines
(settings!) to get the light fromthe machine to the experiment.
![Page 5: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/5.jpg)
courtesy of M. Ries
Many 6= measurement deviceswith own (meta)data + beamlines
(settings!) to get the light fromthe machine to the experiment.
![Page 6: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/6.jpg)
courtesy of M. Ries
![Page 7: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/7.jpg)
courtesy of M. Ries
![Page 8: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/8.jpg)
courtesy of M. Ries
![Page 9: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/9.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 8
Table of Contents
![Page 10: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/10.jpg)
Rawdata
Level 3
Meta dataMetadatastore
Process Process Procdata
RawPre-
process Process Pre-procdata
Exp Level 2
Meta dataMetadatastore
Level 3
Meta data
Level 1
Meta dataMetadatastore
no
yes
no
yes
yes
no
Device and settings, notes ...Data Reduction/CleaningBandwidth!!!
Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!
Metadata at BESSY II, HZB, 11/06/2019 9
Data flow schematic
![Page 11: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/11.jpg)
Rawdata
Level 3
Meta dataMetadatastore
Process Process Procdata
RawPre-
process Process Pre-procdata
Exp Level 2
Meta dataMetadatastore
Level 3
Meta data
Level 1
Meta dataMetadatastore
no
yes
no
yes
yes
no
Device and settings, notes ...
Data Reduction/CleaningBandwidth!!!
Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!
Metadata at BESSY II, HZB, 11/06/2019 9
Data flow schematic
![Page 12: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/12.jpg)
Rawdata
Level 3
Meta dataMetadatastore
Process Process Procdata
RawPre-
process Process Pre-procdata
Exp Level 2
Meta dataMetadatastore
Level 3
Meta data
Level 1
Meta dataMetadatastore
no
yes
no
yes
yes
no
Device and settings, notes ...
Data Reduction/CleaningBandwidth!!!
Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!
Metadata at BESSY II, HZB, 11/06/2019 9
Data flow schematic
![Page 13: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/13.jpg)
Rawdata
Level 3
Meta dataMetadatastore
Process Process Procdata
RawPre-
process Process Pre-procdata
Exp Level 2
Meta dataMetadatastore
Level 3
Meta data
Level 1
Meta dataMetadatastore
no
yes
no
yes
yes
no
Device and settings, notes ...Data Reduction/CleaningBandwidth!!!
Further Data Reduction/CleaningAnalysis / Machine LearningBandwidth AND Storage!!!
Metadata at BESSY II, HZB, 11/06/2019 9
Data flow schematic
![Page 14: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/14.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 10
Table of Contents
![Page 15: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/15.jpg)
I What do we consider as metadata?I Auto-generated? → MostI Manual? → Some
I Digital (elog entries)I Analog (handwritten)
I Data lives! → Persistent IDentification ofdata (unique, versioned,...)
I Logbook interface → SometimesI Storage: SQL/noSQL, file formats →
NEXUS + mongodb + ElasticSearchI Scalability: can we handle the future? →
Elastic Stack + NEXUS?I Data Model?I Findability vs amount of metadata
Some side notes (issues acc phys):I Accelerator Physics is
experience based (an Art)(https://www.youtube.com/watch?v=A1L2xODZSI4)
I Trains, cell phones, cars, ...I Running software on different
systemsI Version controlI File formatsI Maintainability of packages/tools
(cookiecutter, code certificates, )I Containerization : Singularity
with a SCIentific File System(SCIF)
Metadata at BESSY II, HZB, 11/06/2019 11
Some (meta)data considerations
![Page 16: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/16.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 12
Table of Contents
![Page 17: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/17.jpg)
I Control System BESSY: Epics 3.14(3.15 under way)
I Epics variables with unique namingconvention (location, device type,etc...) metadata
I all channels logged in archiverI accelerator metadata = dataI Question: which data is metadata
for some expI Meaning meta(data) depends on
settings (changed manually and canNOT be inferred from data)
I We want: Near Real TimeSimulation / Analysis
I Why? Performance optimization,fast recovery, machine protectionand maintenance
I Experiments: beam commissioning
Metadata at BESSY II, HZB, 11/06/2019 13
BESSY II - the machine
![Page 18: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/18.jpg)
I PythonI Ophyd for device abstraction (epics, labview, but
also extendable)I Bluesky for experiment control and planningI Nice data model (see right)I Databroker available (base: sqlite and MongoDb)I Suitcase for elasticsearch developed in house
(ohters can be easily produced)I possibility to store data in external files but keep
links to data in database (adaptors can be writtenin straightforward way to load/save the data)
I Generates unique ID for each experimentI Can talk to OlogI METADATA and data hintsI live plotting and fitting
Metadata at BESSY II, HZB, 11/06/2019 14
Example: Bluesky and Ophyd
![Page 19: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/19.jpg)
Metadata at BESSY II, HZB, 11/06/2019 15
Example bluesky metadata
![Page 20: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/20.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 16
Table of Contents
![Page 21: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/21.jpg)
I Different Experiment Control Softwares with different ways to control SampleEnvironment
I Complex software interface protocolsI Time consuming integration of new equipment with little mobility.I No metadata standards for Sample Environments (SECoP with NEXUS?)I Physicists programming drivers.I Human Readable variable names?I No generic work-/dataflow - user/beamline specific, what about
authentication?I New ”workflows” (related to (meta)data) should not change users habits or
create extra overheadI Metadata database for static metadata (see further)I Experiment controls: EPICS, Tango, Sardana → Python?I Common Data Model Architecture vs NoMaD → Data Model?
Metadata at BESSY II, HZB, 11/06/2019 17
Beamlines - Let’s complain a bit.
![Page 22: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/22.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 18
Table of Contents
![Page 23: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/23.jpg)
courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 19
![Page 24: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/24.jpg)
courtesy of H. Gorzig
ICAT : metadata catalogue
Metadata at BESSY II, HZB, 11/06/2019 20
![Page 25: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/25.jpg)
courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 21
![Page 26: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/26.jpg)
courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 22
![Page 27: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/27.jpg)
courtesy of H. GorzigMetadata at BESSY II, HZB, 11/06/2019 23
![Page 28: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/28.jpg)
BESSY II
Experiments - general schematic
Some considerations
BESSY II - Accelerator Experiments
BESSY II - beamlines issues
BESSY II - beamlines ongoing work
Summary
Metadata at BESSY II, HZB, 11/06/2019 24
Table of Contents
![Page 29: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/29.jpg)
I Many open questions.I Initial work started both from machine side as from experiments side.I EPICS with Bluesky (Python) + Archiver for the machine.I ICAT database for static metadata under development.I NEXUS interfaces for data/metadata under development.I Some limitations can not be solved easily (bandwidth, standardization,
storage, access rights,...)I Working towards FAIR data.
Metadata at BESSY II, HZB, 11/06/2019 25
Summary
![Page 30: Metadata for material science - at the Lightsource BESSY II...BESSY II Experiments - general schematic Some considerations BESSY II - Accelerator Experiments BESSY II - beamlines issues](https://reader035.vdocuments.us/reader035/viewer/2022081620/610951e8b7dbf03fd85d943d/html5/thumbnails/30.jpg)
I Singularity :https://sylabs.io/docs/I Bluesky, Ophyd :https://github.com/blueskyI Scientific Filesytem :https://sci-f.github.io/
Metadata at BESSY II, HZB, 11/06/2019 26
Links