lockss software re-architecturestorageconference.us/2018/presentations/lockss-tutorial-4.pdf ·...
TRANSCRIPT
![Page 1: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/1.jpg)
LOCKSS Software Re-Architecture
34th International Conference on Massive Storage Systems and TechnologyMay 14, 2018
Santa Clara, California
Thib Guicherd-Callin <[email protected]>Technical Manager, LOCKSS Program
Digital Library Systems and Services, Stanford University Libraries
"Lego Building Blocks" by Traumsicherheit (CC0 1.0 https://pixabay.com/en/lego-building-blocks-colorful-2285065/)
![Page 2: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/2.jpg)
Overview1. Shifting Sands2. Legacy3. Changing LOCKSS4. Code and APIs
![Page 3: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/3.jpg)
Shifting Sands
"Shifting, Howling Sand" by Bill Morrow (CC BY 2.0 https://www.flickr.com/photos/billmorrow/37520367760)
![Page 4: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/4.jpg)
On the Web● Dynamic Web● Web replay
![Page 5: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/5.jpg)
In IT● Appliance model
○ Storage○ Computation
● Management○ Machine environment○ Virtualization○ Outsourcing
![Page 6: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/6.jpg)
Legacy
"Love Padlocks on the Corktown Footbridge in Ottawa, August 14th, 2017" by William Chen (CC BY-SA 4.0 https://commons.wikimedia.org/wiki/File:Love_Padlocks_on_the_Corktown_Footbridge_in_Ottawa,_August_14th,_2017.jpg)
![Page 7: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/7.jpg)
What's In the (LOCKSS) Box?● 1,100+ Java classes● 750+ test classes● Single codebase● Single stack
![Page 8: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/8.jpg)
Custom Implementations● LOCKSS Web Crawler● Web replay● Repository (storage)
![Page 9: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/9.jpg)
Things Change● Project hosting● Source control management● Build system● Unit testing● Continuous integration● Binary distribution● Development environment
![Page 10: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/10.jpg)
Changing LOCKSS
"Photo 638725" by Pxhere (CC0 1.0 https://pxhere.com/en/photo/638725)
![Page 11: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/11.jpg)
LAAWS Initiative● "LOCKSS Architected As Web Services"● Mellon Foundation grant● Modernization effort
![Page 12: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/12.jpg)
Goals● Re-architect LOCKSS daemon as collection of Web Services
○ WARC storage○ Memento access
● Use off-the-shelf open source components● Minimize LOCKSS-only components
![Page 13: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/13.jpg)
Desiderata● LOCKSS protocol compatibility● LOCKSS plugin compatibility
![Page 14: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/14.jpg)
Splitting Up Functionality● Components -> services● REST API● Considerable refactoring
![Page 15: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/15.jpg)
TechnologiesIn progress
● Maven, Maven Central● JUnit 4, JUnit 5, Mockito● Spring Framework 4, Spring Boot 1
○ Spring Framework 5, Spring Boot 2● Swagger 2
○ OpenAPI 3● OpenWayback, Pywb● Solr● Docker, Docker Hub
Under consideration
● Orchestration
![Page 16: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/16.jpg)
Repository Architecture
![Page 17: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/17.jpg)
Disaggregated Architecture
![Page 18: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/18.jpg)
Code and APIs
"Photo 1172040" by Pxhere (CC0 1.0 https://pxhere.com/en/photo/1172040)
![Page 19: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/19.jpg)
LOCKSS Developer Pages● https://lockss.github.io/● https://lockss.github.io/laaws/code-artifacts/
![Page 20: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/20.jpg)
Core● LOCKSS Core (org.lockss.laaws:lockss-core)
○ Stable version: 1.1.1○ Development version: 1.2.0-SNAPSHOT○ GitHub: https://github.com/lockss/lockss-core
![Page 21: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/21.jpg)
Repository● LOCKSS Repository Core (org.lockss.laaws:laaws-repository-core)
○ Stable version: 1.11.4○ Development version: 1.12.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-repository-core
● LOCKSS Repository Service (org.lockss.laaws:laaws-repository-service)○ Stable version: 1.7.0○ Development version: 1.8.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-repository-service
● LOCKSS WARCImport Client (org.lockss.laaws:laaws-repository-client)○ Stable version: 1.5.0○ Development version: 1.6.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-repository-service
![Page 22: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/22.jpg)
Metadata● LOCKSS Metadata Extraction Service
(org.lockss.laaws:laaws-metadata-extraction-service)○ Stable version: 1.0.0○ Development version: 1.1.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-metadataextractor
● LOCKSS Metadata Service (org.lockss.laaws:laaws-metadata-service)○ Development version: 1.0.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-metadataservice
● LOCKSS Metadata Extractor Common Code (org.lockss.laaws:laaws-metadataextractor-common)
○ Stable version: 1.2.0○ Development version: 1.3.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-metadataextractor-common
![Page 23: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/23.jpg)
Polling and Repair● LOCKSS Poller Service (org.lockss.laaws:laaws-poller)
○ Development version: 1.0.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-poller
![Page 24: LOCKSS Software Re-Architecturestorageconference.us/2018/Presentations/LOCKSS-tutorial-4.pdf · LOCKSS Software Re-Architecture 34th International Conference on Massive Storage Systems](https://reader030.vdocuments.us/reader030/viewer/2022041116/5f28b014a7d8db25c80e2f78/html5/thumbnails/24.jpg)
Infrastructural● LOCKSS Configuration Service
(org.lockss.laaws:laaws-configuration-service)○ Stable version: 1.0.0○ Development version: 1.1.0-SNAPSHOT○ GitHub: https://github.com/lockss/laaws-configservice