Download - Disposable Environments at Scale
![Page 1: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/1.jpg)
/
Disposable Environments at Scale
or: How I Learned to Stop Worrying and Love ZFS
![Page 2: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/2.jpg)
Eric Sproul
Build EngineerSysadmin
Consultant
Twitter:@eirescot
![Page 3: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/3.jpg)
Recipe for Success
enablement (n): the act of providing (someone)with adequate power, means, opportunity or authority(to do something)
This is a story of how ZFS enabled business success
![Page 4: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/4.jpg)
Background
Etsy is the world's handmade marketplace
Experiencing rapid growth
Every department needed to understandhow the business was evolving
Asked OmniTI for help
![Page 5: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/5.jpg)
The Situation
No data warehouse
Analytical queries againstPostgreSQL OLTP
Large initial size (~250 GB)
Forecast to reach 1 TB+within a year
![Page 6: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/6.jpg)
Problems
Long-running queries to OLTPdestroy web performance
Re-running reportsproduces different results
Inflexible reporting interface
![Page 7: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/7.jpg)
Requirements
Relieve pressure fromOLTP database
Make production datacontinuously available
Enable correlationof other sources
Flexible reporting UI
![Page 8: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/8.jpg)
Solution
Create separate BI analytics DB
Build it on ZFS
![Page 9: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/9.jpg)
Initial Capabilities
ETL to collate table-level datafrom multiple databases
Run deep analytic querieswithout impacting website
New web UI enables ad-hoc reporting
![Page 10: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/10.jpg)
Reaping the Benefits of ZFS
Snapshots
Faster backups
Simple replica creation
![Page 11: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/11.jpg)
Reaping the Benefits of ZFS
CompressionExtend usable life of storage:
PgSQL logical: 1.3TOn-disk: 653G (2.0x)
Intelligent resilverShorter rebuilds ==
Reduced risk of data loss
![Page 12: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/12.jpg)
We Want More!
Monthly reports now on BI system
Occasional problems requirere-running reports
Still get different results,same as before
![Page 13: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/13.jpg)
We Want More!
Need to test report changes
Fine to dev with small mock-up
Staging requires somethingthat looks like production
![Page 14: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/14.jpg)
The Next Level
Disposable environments
Run on slave replica
R/W copy of BI data
Discard when finished
![Page 15: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/15.jpg)
Disposable Environment
Use a non-global zone
set zonepath=/zones/bistageset autoboot=trueset limitpriv=default,dtrace_proc,dtrace_userset ip-type=sharedadd netset address=10.1.2.3set physical=bnx0endadd datasetset name=bi01tank/stageend
![Page 16: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/16.jpg)
Disposable Environment
Starting state: ZFS datasets
bi01tank/pgsql/databi01tank/pgsql/data/91bi01tank/pgsql/wal_archivebi01tank/pgsql/wal_archive/91
![Page 17: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/17.jpg)
Disposable Environment
Take snapshots
bi01tank/pgsql/data@stagebi01tank/pgsql/data/91@stagebi01tank/pgsql/wal_archive@stagebi01tank/pgsql/wal_archive/91@stage
zfs snapshot -r bi01tank/pgsql/data@stage
![Page 18: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/18.jpg)
Disposable Environment
Create clones
bi01tank/pgsql/data@stagebi01tank/pgsql/data/91@stagebi01tank/pgsql/wal_archive@stagebi01tank/pgsql/wal_archive/91@stage
bi01tank/stage/databi01tank/stage/data/91bi01tank/stage/wal_archivebi01tank/stage/wal_archive/91
zfs clone <src_dataset>@stage <dst_dataset>
![Page 19: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/19.jpg)
Disposable Environment
Zone now sees a full, writable copy of data
Unchanged data is referenced to origin
Changes accumulate to clone
![Page 20: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/20.jpg)
Disposable Environment
pgsql/data/91 pgsql/data/91@stage stage/data/91
Change accountedto clone
Unchanged datareferenced fromsnapshot
Live FS Snapshot Clone
![Page 21: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/21.jpg)
Next-Level Results
Any report can be re-runon the same data
Massage existing data orbring more in for ad-hoc report
Test changes to reports and web UI
![Page 22: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/22.jpg)
Next-Level Results
When finished with the environment:
Shut down zone
Delete clone & origin snap
![Page 23: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/23.jpg)
Return on Investment
BI database runs on 2 machines
OLTP database lifetime extendedtwo years past expectation
Faster, more granular, and ad-hoc reportingenables better decisions by management
![Page 24: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/24.jpg)
Bonus!
With the same technique,we can safely test:
PostgreSQL upgrades
Schema changes
![Page 25: Disposable Environments at Scale](https://reader031.vdocuments.us/reader031/viewer/2022020207/55981d221a28ab36048b47fb/html5/thumbnails/25.jpg)
Thank You
ZFS, Zones and much moreare available to the community via
illumos and its distributions
Go forth and enable your business!