webinar: dyn + datastax - helping companies deliver exceptional end-user experience
TRANSCRIPT
![Page 1: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/1.jpg)
Dyn + DataStax: Helping Companies Deliver Exceptional End-User ExperienceMay 17, 2016Tim Chadwick, Principal Engineer, Infrastructure, DynRick Bross, Principal Engineer, Scalability, Dyn
![Page 2: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/2.jpg)
The Story at Dyn
The Road to Production
Lessons and Direction
Journey to DataStax Enterprise
![Page 3: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/3.jpg)
The Story at DynDyn is a cloud-based
Internet Performance Management (IPM) company that provides unrivaled visibility and
control into cloud and public Internet resources.
Dyn’s platform monitors, controls and optimizes applications and infrastructure through Data,
Analytics, and Traffic Steering, ensuring traffic gets delivered faster, safer, and more reliably
than ever.
http://techcrunch.com/2016/05/10/dyn-series-b/
![Page 4: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/4.jpg)
DNS Overview
![Page 5: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/5.jpg)
Dyn Global: 20+ Data Centers
![Page 6: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/6.jpg)
tchadwick@piedmont:~$ dig SOA ifc.com | grep -A 1 "ANSWER SECT";; ANSWER SECTION:ifc.com. 7175 IN
SOA ns1.p28.dynect.net. postmaster.ifc.com. 2016042900 3600 600 604800 1800Build a sustainable system that
can track usage by customer and zone (domain).
The consumers are our customers, our billing department, and Chris Baker.
Who Needs These Data?
![Page 7: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/7.jpg)
For each five minute interval of an invoice period, determine the Queries per Second (QPS) and sort in descending order.
Discard the top 5%, and it is the maximum value remaining which is a customer’s 95th Percentile, or monthly bill rate.
http://dyn.com/blog/the-95th-percentile-burstable-billing-model-managed-dns/https://en.wikipedia.org/wiki/Burstable_billing#95th_percentile
Traffic Telemetry
![Page 8: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/8.jpg)
1. Operations-Flexible Topology-Resilient Clusters-Visibility and Administration
2. Data Model-Idempotent Writes-Low Concurrency-Application Redundancy
Oh, and it must perform well.
Benchmarking Cassandra Scalability on AWS
Over a million writes per second
Priorities that Led to DataStax Enterprise
![Page 9: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/9.jpg)
Consult the Experts
![Page 10: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/10.jpg)
Oh Baby!
![Page 11: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/11.jpg)
One sec, new priority...
![Page 12: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/12.jpg)
FidelityCustomer
Enterprise Requirement Ahead!
![Page 13: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/13.jpg)
Sunnyvale (USSNN1)
North Bergen (USNBN1)
CREATE KEYSPACE qld WITH replication = { 'class': 'NetworkTopologyStrategy', ...};
USE qld;
CREATE TABLE qld_logs ( key text, row_seq bigint, logline text, PRIMARY KEY ((key), row_seq)) WITH COMPACT STORAGE AND ... compaction={'class': 'SizeTieredCompactionStrategy'} AND ...
Detailed DNS Query Log - DSE Cluster
![Page 14: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/14.jpg)
Success!
![Page 15: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/15.jpg)
Back to our original goal....
![Page 16: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/16.jpg)
• Customers • Zones (Domains)• Zone Record Types• Fully Qualified Domain Names
(qnames)• Regions (ANYCAST)• Data Centers• Nameservers• “Top 10s”
Many, many more customers.Many, many more dimensions.
I Want More From You....
![Page 17: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/17.jpg)
Datastax Enterprise Provided the Tools
![Page 18: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/18.jpg)
North BergenSunnyvale
CREATE TABLE "QueryCountSummaryCF”CREATE TABLE "QueryZoneCountCF"CREATE TABLE "QueryHostCountCF"CREATE TABLE “QueryCountSummaryRollupsCF"CREATE TABLE "QueryZoneCountRollupsCF"CREATE TABLE "QueryHostCountRollupsCF"CREATE TABLE "QueryPlatformCountCF"
WITH DEFAULT_TIME_TO_LIVE = 31536000;WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication' : 2 };
The Working Solution
![Page 19: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/19.jpg)
Factoids
Throughput:○ > 12k w / s○ 99th percentile < 5ms○ Avg read latency < 10ms
Size:○ 200GB -> 1.2TB, steady○ ~ 12B data points
![Page 20: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/20.jpg)
How DataStax Enterprise Provided Value● Support in Every Phase
○ Proof of Concept○ Design○ Operations○ Optimization
● Integrated Toolkit○ OpsCenter○ SPARK
We get the value of many, many people at the cost of about 1/2 FTE.
![Page 21: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/21.jpg)
Lessons Learned
![Page 22: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/22.jpg)
Top Lessons Learned
1. Include all teams in planning, deployment and implementation.2. Consult knowledgeable people before making decisions and “optimizations”.3. Understand compaction strategies to immediately eliminate those that are not a fit.4. Ensure that client load balancing policies and consistency levels match DC
topology and schema replication factors.5. Model and understand all failure scenarios.6. Use Spark to aggregate data in order to save storage and improve performance.
![Page 23: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/23.jpg)
#1: Include all teams
● Product management● Application engineering● DBAs● Operations ● Network engineering● System engineering● Finance and Management
![Page 24: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/24.jpg)
#2: Consult knowledgeable people . . .
● Schema● Cluster topology and tuning● Tuning● Compaction algorithms● Client interaction
Talk to Datastax! They’ve probably seen it before!
![Page 25: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/25.jpg)
#3: Understand Compaction Strategies!
DTCS was our first choice. It didn’t work . . . .
Tim Goodaire September 02, 2015 17:10
We have changed the compaction strategy,
concurrent_compactors, compaction_throughput, and
heap size. It took a while for the cluster to complete
the compactions, but it's done now. The cluster is up
and appears to be healthy.
Today, we've been adding a few more nodes and
resetting the heap size back to 8 GB.
![Page 26: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/26.jpg)
#4: Ensure client and cluster settings match
Load balancing policies, read and write consistency, schema replication factor, cluster topology . . .
![Page 27: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/27.jpg)
#5: Model Failure ScenariosWhat happens when a node fails? Two? The DC?
Will the client fail? How will queries be satisfied?
![Page 28: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/28.jpg)
700 rows for a single 5 minute interval
Daily billing went from 14 hours, to 2 hours on DSE/C*, and 12 minutes with DSE/SPARK
#6: Use DSE Spark to aggregate20 rows for an hour interval
![Page 29: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/29.jpg)
What’s Next?
© DataStax, All Rights Reserved. 29
● Rely on best practices to support more analytical use cases across products.
● Complete development of generic C* solution, for quicker time to market, and greater scale in our hybrid cloud.
● Consider new opportunities for relying on DSE for products delivering services.
![Page 30: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/30.jpg)
Contacts and Thanks!Tim ChadwickPrincipal Engineer, Infrastructurehttps://www.linkedin.com/in/[email protected]@DynData
Rick BrossPrincipal Engineer, Data Analyticshttps://www.linkedin.com/in/[email protected]
Dyn, Inc.150 Dow St – Tower TwoManchester, NH 03101603-668-4998
© DataStax, All Rights Reserved. 30
![Page 31: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/31.jpg)
Coming Soon!
● June 8: How to Half Hour - Building Data Pipelines with SMACK: Storage Strategy using Cassandra and DSE
● July 6: How to Half Hour - Building Data Pipelines with SMACK: Analyzing Data with Spark
● For the latest schedule of webinars, check out our Webinars page: http://www.datastax.com/resources/webinars.
![Page 32: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/32.jpg)
Appendix
![Page 33: Webinar: Dyn + DataStax - helping companies deliver exceptional end-user experience](https://reader035.vdocuments.us/reader035/viewer/2022062522/586f83af1a28ab54768b4a77/html5/thumbnails/33.jpg)
Client● Client cluster and session object configuration
○ Cluster seeds (DCAwareRoundRobinPolicy implications)
○ Other load balancing policies to wrap○ Read and write consistency setting○ # connections per host ○ # requests per connection○ Pool timeout
● Client query settings○ Read and write consistency (may override default
for specific query)○ Batches (rarely if ever should be used)○ Stored procedures (usually best practice for groups
of queries - ex. we use for high velocity inserts)○ Sync or Async? Depends on the specific query, but
usually best practice with stored procedures.○ Write with a consistent TTL per table.○ How many threads should share the client session
object? We’ve found that balancing the DC capabilities, client latency, and a (native) thread pool turbocharges inserts.
Cassandra Cluster● Network topology
○ Colocated latency? Inter DC latency?○ Replication factor per DC per schema
● Schema○ Don’t mix schemas with different use cases!○ Dyn’s usage pattern
■ Optimize INSERTs.■ Ensure READs succeed.■ Avoid UPDATEs (“out of order” TTLs)■ Ban DELETEs (turn off the repair service)
○ Attempt to have all (voluminous) tables use the same compaction strategy.
○ Use consistent TTLs for writes. If you override the default, always override with the same value.
● Compaction algorithms○ With time series data, no deletes, no ✓ ✓ ✓
updates and consistent TTLs: you can use ✓DTCS, which will simply drop old sstables.
Client/Cluster Settings - Must Work Together!