usability issues facing 21st century data archives joey mukherjee and david winningham [email protected]
TRANSCRIPT
![Page 2: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/2.jpg)
Current Archiving Goal
Mission TeamRawData Processed
Data
Write Papers
DataIteration
QualityData
ArchiveFuture Scientists
QualityData
![Page 3: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/3.jpg)
Current Archiving Reality
Mission TeamRawData Processed
Data
Write Papers
DataIteration
DataSubsets
Permanent Archive
Future Scientists
UncheckedData
Home Institution
Archive
PublicData
![Page 4: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/4.jpg)
New Goal
Mission TeamRawData Processed
Data
Write Papers
DataIteration
ProcessedData
ArchiveFuture Scientists
ProcessedData
![Page 5: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/5.jpg)
Standardizing HOWTO
Make it easyMake it usefulMake it extensible
![Page 6: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/6.jpg)
Make it Easy
Reading / writing files must be super easy (i.e. cheap!)
– Either with tools or libraries
Tools can be command line or GUI
![Page 7: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/7.jpg)
Make it Useful
How do I look at it?– Plots/Analysis
What else can I do with it?– Read into IDL, Matlab, Excel, etc.
Must have immediate benefits
![Page 8: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/8.jpg)
Make it Extensible
Must be possible for others to add value added servicesMust be able to hold varieties of dataMust agree to give up control on content
![Page 9: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/9.jpg)
Case Studies: HTML
Easy to create!Once done, look at in browserEmbrace / Extend
![Page 10: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/10.jpg)
Case Studies: SPASE
Creation is slow and difficultOnce created, no real benefits yetVxOs have embraced, no one extended yet
![Page 11: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/11.jpg)
Case Studies: IDFS
Until recently, difficult to create, complexOnce in, easy to look at, use, archive, etc.Somewhat extensible
![Page 12: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/12.jpg)
Things right with IDFS
EfficientSelf documentingCalibrations stored in text file Science units derived instead of storedLittle to no reprocessing ever needed
![Page 13: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/13.jpg)
Other IDFS Benefits
Can store most types of space physics data from raw telemetry to highly processed science unitsReversible from science units to raw telemetryUsable by data processor, scientist, and data archiver
![Page 14: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/14.jpg)
Things wrong with IDFS
Overly complex format and APINot enough support in other tools - poor buy-inAnalysis routines merged with the file format - tried to do too much!
![Page 15: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/15.jpg)
Implementation Plan
Develop a simple file format that can contain any and all types of time series space physics dataDevelop tools that allow someone to create and inspect files in this format Merge in the best parts of IDFS, CDF, netCDF, HDF, FITS, etc... without breaking paradigm of simplicity
![Page 16: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/16.jpg)
Simple File Format
Format might already exist:– HDF5– XML– JSON– Other data models?
![Page 17: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/17.jpg)
Making it useful
Get buy-in from visualization tools (SDDAS, DataShop, VisBard, IDL DLM, etc.)Get buy-in from archives sites (PDS, PSA, NSSDC, etc.)Seed money is essential
![Page 18: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/18.jpg)
Advantages
ProvidersUsersManagement
![Page 19: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/19.jpg)
Advantages: Providers
Instrument teams now have something to work towardCan develop expertise
![Page 20: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/20.jpg)
Advantages: Users
Quick ways to create plots or access dataExpertise again!
![Page 21: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/21.jpg)
Advantages: Management
Homogenous archives are infinitely easier to manage and maintainValue added services are a natural extension of quality archives
![Page 22: Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org](https://reader030.vdocuments.us/reader030/viewer/2022032805/56649ef15503460f94c01ce9/html5/thumbnails/22.jpg)
Conclusion
Why now? Because SPASE is gaining traction, this is the next logical step.This will save money for everyone in the long run.Everyone benefits with value added services.