all about storeconfigs
DESCRIPTION
Storeconfigs is not a popular feature among Puppet admins, because most don’t know how to use it or fear performance issues. Attend this talk to know how to enhance your Puppet deployments with easy cross-nodes interactions and collaborations, while conserving system efficiency.TRANSCRIPT
![Page 1: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/1.jpg)
StoreconfigsAll About
Brice Figureaumasterzen (#puppet)
![Page 2: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/2.jpg)
Who am I?
![Page 3: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/3.jpg)
![Page 4: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/4.jpg)
Puppet Community Developerplanetpuppet.org maintainer
![Page 5: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/5.jpg)
![Page 6: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/6.jpg)
Whatʼs Storeconfigs?
![Page 7: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/7.jpg)
Storeconfigs stores to an RDBMSuses Railsʼ Active Record
![Page 8: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/8.jpg)
2 Possible Usages
Inventory Export/Collect
![Page 9: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/9.jpg)
Export/CollectNode collaborations
![Page 10: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/10.jpg)
Syntax: export
@@resource { ...}
![Page 11: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/11.jpg)
Syntax: collect
Resource <<| query |>>
![Page 12: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/12.jpg)
Collect: Query•Resource Parameters•Tags•and, or
![Page 13: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/13.jpg)
Tip
Always tag exported resources
![Page 14: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/14.jpg)
The BIG Picture
masterDB
collector 1
Exporter
collector 2
![Page 15: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/15.jpg)
Simple exampleWeb server replication, deployment server
![Page 16: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/16.jpg)
Simple exampleWeb server replication, web server part
![Page 17: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/17.jpg)
Intermediate exampleBuilding /etc/resolv.conf from exported snippets
![Page 18: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/18.jpg)
Intermediate exampleBuilding /etc/resolv.conf, aggregating snippets
![Page 19: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/19.jpg)
Advanced ExampleTinydns master
![Page 20: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/20.jpg)
Advanced ExampleTinydns master
![Page 21: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/21.jpg)
Advanced ExampleTinydns slave
![Page 22: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/22.jpg)
Limitations
![Page 23: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/23.jpg)
Inner Workings
![Page 24: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/24.jpg)
Inner Workings1. Read all resources, tags for a node (slow)
![Page 25: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/25.jpg)
Inner Workings2. Compare with in-memory catalog (fast)
![Page 26: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/26.jpg)
Inner Workings3. Store/remove missing/extraneous resources/tags (slow)
![Page 27: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/27.jpg)
Inner WorkingsDB schema
![Page 28: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/28.jpg)
Performance Impact
![Page 29: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/29.jpg)
Performance ImpactCollecting is fast
![Page 30: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/30.jpg)
Performance ImpactStoreconfigs write can be slow
![Page 31: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/31.jpg)
Performance Impact
0
6
13
19
25
Time (seconds)
Off On1st run (1950 resources)
![Page 32: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/32.jpg)
Performance Impact
0
2
4
6
8
Time (seconds)
Off OnSubsequent runs (1950 resources)
![Page 33: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/33.jpg)
First Run Impact
![Page 34: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/34.jpg)
First Run ImpactFailure
err: Could not retrieve catalog: execution expired
![Page 35: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/35.jpg)
Scaling: some rough math1 HDD = 250 IOP/s
![Page 36: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/36.jpg)
Scaling: some rough mathAll nodes have 500 resources4 params/ resources4 tags/resources=> 4500 writes=> 1000 IOP
![Page 37: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/37.jpg)
Scaling: some rough mathMaster concurrency: 5
![Page 38: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/38.jpg)
Scaling: some rough mathIOP/s left per node:
250/5 = 50 IOP/s
![Page 39: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/39.jpg)
Scaling: some rough mathTime per node
1000 IOP = 20s
![Page 40: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/40.jpg)
Scaling: some rough mathAnd for a 500 nodes setup...
500/5 = 100 “runs”
![Page 41: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/41.jpg)
Scaling: some rough mathWith a sleep time of 30min
30 min = 1800s1800s/20s = 90
90 nodes per 30 min
![Page 42: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/42.jpg)
Scaling: some rough mathHouston we have a problem
90 nodes when we needed 100
![Page 43: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/43.jpg)
Scaling the back-endDedicated RDBMS server(s)
![Page 44: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/44.jpg)
Scaling the back-endPowerful I/O subsystem (RAID,BBU...)
![Page 45: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/45.jpg)
Scaling the back-endDB setup tuning
![Page 46: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/46.jpg)
Scaling the back-endMySQL, scaling for insertsinnodb_buffer_pool_size = 70% of physical RAM
![Page 47: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/47.jpg)
Scaling the back-endMySQL, scaling for insertsinnodb_log_file_size = up to 5% of physical RAM
![Page 48: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/48.jpg)
Scaling the back-endMySQL, scaling for insertsinnodb_flush_method = O_DIRECT
![Page 49: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/49.jpg)
Scaling the back-endMySQL, scaling for insertsinnodb_flush_log_at_trx_commit = 2
![Page 50: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/50.jpg)
0.25 to the help!
![Page 51: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/51.jpg)
0.25 to the help!Async Storeconfigs - no more thundering herd
![Page 52: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/52.jpg)
0.25: Async Storeconfigs
masterDB
node
puppetqdMessage Queue
![Page 53: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/53.jpg)
0.25: Thin StoreconfigsOnly exported resources, tags and facts
![Page 54: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/54.jpg)
0.25: Thin Storeconfigs
0
1
3
4
6
Time (seconds)
Off On1st run (1950 resources)
![Page 55: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/55.jpg)
Storeconfigs future?
![Page 56: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/56.jpg)
More AR adapters
![Page 57: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/57.jpg)
More AR tuning (pooling)
![Page 58: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/58.jpg)
Non RDBMS back-ends
![Page 59: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/59.jpg)
Non RDBMS back-ends•Tokyo Cabinet/Tyrant
•Mongo DB
![Page 60: All About Storeconfigs](https://reader033.vdocuments.us/reader033/viewer/2022060111/55666f94d8b42a3d3f8b522b/html5/thumbnails/60.jpg)
Questions ?Please speak loud and slowly and excuse me
by advance if I donʼt understand a word of your question.
Being French has some advantages (like health care access or a cute if not stupid accent)it also has drawbacks, especially our educational system for
learning foreign languages simply sucks :-D