skb kontur: digging cassandra cluster
TRANSCRIPT
![Page 1: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/1.jpg)
DIGGING CASSANDRA CLUSTER
Ivan Burmistrov
![Page 2: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/2.jpg)
Ivan BurmistrovTech Lead at SKB Kontur
5+ years Cassandra experience (from Cassandra 0.7)
WHO AM I?
@isburmistrov
https://www.linkedin.com/in/isburmistrov/en
![Page 3: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/3.jpg)
• Services for businesses
• B2B: e-Invoicing
• B2G: e-reporting of tax returns to government
SKB KONTUR
![Page 4: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/4.jpg)
RETAIL
![Page 5: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/5.jpg)
• 24 x 7 x 365
• Guarantee of delivering
REQUIREMENTS
![Page 6: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/6.jpg)
• 24 x 7 x 365
• Guarantee of delivering
• Delivery time <= 1 minute
REQUIREMENTS
![Page 7: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/7.jpg)
When Cassandra just works
![Page 8: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/8.jpg)
When Cassandra just works
![Page 9: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/9.jpg)
When Cassandra just works
![Page 10: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/10.jpg)
SMART GUY
![Page 11: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/11.jpg)
• 150+ different tables in cluster (Cassandra 1.2)
• Client read latency (99th percentile): 100ms – 2.0s
• Affected almost all tables
• CPU: 40% – 80%
• Disk: not a problem
THE PROBLEM
2 sec.
![Page 12: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/12.jpg)
• ReadLatency.99thPercentile
node’s latency of processing read request
• ReadLatency.OneMinuteRate
node’s read requests per second
• SSTablesPerReadHistogram
how many SSTables node reads per read request
HYPOTHESIS 1: ANOMALIES IN METRICS
![Page 13: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/13.jpg)
• ReadLatency.99thPercentile
node’s latency of processing read request
• ReadLatency.OneMinuteRate
node’s read requests per second
• SSTablesPerReadHistogram
how many SSTables node reads per read request
• Tables were pretty similar in these metrics
• What values are good, which are bad?
HYPOTHESIS 1: ANOMALIES IN METRICS
![Page 14: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/14.jpg)
• Decrease/increase compaction throughput
• Change compaction strategy
HYPOTHESIS 2: COMPACTION
![Page 15: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/15.jpg)
• Decrease/increase compaction throughput
• Change compaction strategy
• Nothing changed
HYPOTHESIS 2: COMPACTION
![Page 16: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/16.jpg)
• ParNew GC – 6 seconds per minute (10%!)
• Read good articles about Cassandra and GC• http://tech.shift.com/post/74311817513/cassandra-tuning-
the-jvm-for-read-heavy-workloads
• http://aryanet.com/blog/cassandra-garbage-collector-tuning
• Tried to tune
HYPOTHESIS 3: GC
![Page 17: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/17.jpg)
• ParNew GC – 6 seconds per minute (10%!)
• Read good articles about Cassandra and GC• http://tech.shift.com/post/74311817513/cassandra-tuning-
the-jvm-for-read-heavy-workloads
• http://aryanet.com/blog/cassandra-garbage-collector-tuning
• Tried to tune
• Nothing changed
HYPOTHESIS 3: GC
![Page 18: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/18.jpg)
• Built-in profiling tool from Oracle JDK 7 Update 40
• Low performance overhead: 1-2%
• Useful for CPU profiling: hot threads, hot methods,
call stacks,…
• Profiling results: 70% of time – SSTablesReader
Java Mission Control and Java Flight Recorder
![Page 19: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/19.jpg)
• SSTablesPerReadHistogram did not help
• We needed another metric
• SSTablesPerSecond
how many SSTables each table read per second
SSTablesPerSecond = SSTablesPerReadHistogram.Mean *
ReadLatency.OneMinuteRate
What tables cause most reads of SSTables?
![Page 20: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/20.jpg)
SSTablesPerSecond
![Page 21: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/21.jpg)
• 7 leading tables = only 7 candidates for deep investigation
• Large difference between leaders and others
• Almost all leaders were surprises
• 3 types of problems
SSTablesPerSecond: results
![Page 22: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/22.jpg)
Problem 1: Invalid timestamp usage
CREATE TABLE users_lastaction (
user_id uuid,
subsystem text,
last_action_time timestamp,
PRIMARY KEY (user_id)
);
subsystem: ‘API‘,‘WebApplication‘,…
![Page 23: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/23.jpg)
Problem 1: Invalid timestamp usage
First subsystem:
INSERT INTO users_lastaction
(user_id, subsystem, last_action_time)
VALUES (62c36092-82a1-3a00-93d1-46196ee77204,‘API',‘2011-02-03T04:05:00');
Second subsystem:
INSERT INTO users_lastaction
(user_id, subsystem, last_action_time)
VALUES (62c36092-82a1-3a00-93d1-46196ee77204,‘WebApp',‘2011-02-08T07:05:00')
USING TIMESTAMP 635774040762020710;
Time in ticks,
10000 ticks = 1 millisecond
![Page 24: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/24.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
SSTables
Memtable
![Page 25: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/25.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
1. Looks at Memtable
SSTables
Memtable
![Page 26: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/26.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
1. Looks at Memtable
2. Filters SSTables using bloom filter
SSTables
Memtable
![Page 27: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/27.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Filters SSTables by timestamp
(CASSANDRA-2498)
SSTables
Memtable
![Page 28: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/28.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Filters SSTables by timestamp
(CASSANDRA-2498)
4. Reads remaining SSTables
SSTables
Memtable
![Page 29: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/29.jpg)
Problem 1: Invalid timestamp usage
SELECT last_action_time FROM users_lastaction
WHERE user_id = 62c36092-82a1-3a00-93d1-46196ee77204
AND subsystem = ‘API'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Filters SSTables by timestamp
(CASSANDRA-2498)
4. Reads remaining SSTables
5. Merges resultSSTables
Memtable
![Page 30: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/30.jpg)
Problem 1: Invalid timestamp usage
First subsystem:
INSERT INTO users_lastaction
(user_id, subsystem, last_action_time)
VALUES (62c36092-82a1-3a00-93d1-46196ee77204,‘API',‘2011-02-03T04:05:00');
Second subsystem:
INSERT INTO users_lastaction
(user_id, subsystem, last_action_time)
VALUES (62c36092-82a1-3a00-93d1-46196ee77204,‘WebApp',‘2011-02-08T07:05:00')
USING TIMESTAMP 635774040762020710;
Time in ticks,
10000 ticks = 1 millisecond
![Page 31: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/31.jpg)
Problem 1: Invalid timestamp usage
Fix:
started to use equal timestamp sources for one
table
![Page 32: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/32.jpg)
Problem 2: Few writes, many reads
• Reads dominates over writes (example – user accounts)
• Each read – from SSTable (Memtable already flushed)
![Page 33: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/33.jpg)
Problem 2: Few writes, many reads
• Reads dominates over writes (example – user accounts)
• Each read – from SSTable (Memtable already flushed)
• Fix: just enabled row cache
![Page 34: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/34.jpg)
Problem 3: Aggressive time series
CREATE TABLE activity_records(
time_bucket text,
record_time timestamp,
record_content text,
PRIMARY KEY (time_bucket, record_time)
);
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
![Page 35: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/35.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
SSTables
Memtable
![Page 36: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/36.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
SSTables
Memtable
![Page 37: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/37.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
2. Filters SSTables using bloom filter
SSTables
Memtable
![Page 38: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/38.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Can’t use CASSANDRA-2498
SSTables
Memtable
![Page 39: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/39.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Can’t use CASSANDRA-2498
4. CASSANDRA-5514!
SSTables
Memtable
![Page 40: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/40.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Can’t use CASSANDRA-2498
4. CASSANDRA-5514!
5. Reads remaining SSTables
SSTables
Memtable
![Page 41: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/41.jpg)
Problem 3: Aggressive time series
SELECT record_content FROM activity_records
WHERE time_bucket = ‘2015-05-10 12:00:00'
AND record_time > ‘2015-05-10 12:30:10'
1. Looks at Memtable
2. Filters SSTables using bloom filter
3. Can’t use CASSANDRA-2498
4. CASSANDRA-5514!
5. Reads remaining SSTables
6. Merges result SSTables
Memtable
![Page 42: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/42.jpg)
Problem 3: Aggressive time series
Fix: just upgraded to Cassandra 2.0+
![Page 43: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/43.jpg)
SSTablesPerSecond: before
![Page 44: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/44.jpg)
SSTablesPerSecond: after
![Page 45: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/45.jpg)
Before:• Client read latency (99th percentile): 100ms – 2s
• CPU: 40% – 80%
After:• Client read latency (99th percentile): 50ms – 200ms
• CPU: 20% – 50%
WHAT ABOUT OUR GOAL?
![Page 46: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/46.jpg)
• Reading SSTables vs reading Memtable – 50/50
• SliceQuery – 70%
PROFILE AGAIN
![Page 47: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/47.jpg)
• LiveScannedHistogram
how many live columns node scans per slice query
• TombstonesScannedHistogram
how many tombstones node scans per slice query
LOOK AT METRICS AGAIN
![Page 48: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/48.jpg)
• LiveScannedHistogram
how many live columns node scans per slice query
• TombstonesScannedHistogram
how many tombstones node scans per slice query
• Not found any anomalies
LOOK AT METRICS AGAIN
![Page 49: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/49.jpg)
• LiveScannedHistogram
how many live columns node scans per slice query
• TombstonesScannedHistogram
how many tombstones node scans per slice query
• Not found any anomalies
• Why not use the successful trick?
LOOK AT METRICS AGAIN
![Page 50: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/50.jpg)
LiveScannedPerSecond
how many live columns Cassandra scans per second for each table
LiveScannedHistogram.Mean * ReadLatency.OneMinuteRate
![Page 51: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/51.jpg)
• 1 obvious leader
• Large difference between leader and others
• Leader – big surprise
LiveScannedPerSecond: results
![Page 52: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/52.jpg)
• 1 obvious leader
• Large difference between leader and others
• Leader – big surprise
• Fix: fixed the bug
LiveScannedPerSecond: results
![Page 53: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/53.jpg)
Initial:• Client read latency (99th percentile): 100ms – 2.0s
• CPU: 40% – 80%
After SSTablesPerSecond fixes:• Client read latency (99th percentile): 50ms – 200ms
• CPU: 20% – 50%
After LiveScannedPerSecond fixes:• Client read latency (99th percentile): 30ms – 100ms
• CPU: 10% – 30%
WHAT ABOUT OUR GOAL?
![Page 54: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/54.jpg)
Compaction – 30%
PROFILE AGAIN
![Page 55: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/55.jpg)
Compaction – 30%
Fix:
throttled down compactions during high load period,
throttled up during low load period
PROFILE AGAIN
![Page 56: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/56.jpg)
WHAT ABOUT OUR GOAL?
![Page 57: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/57.jpg)
Initial:• Client read latency (99th percentile): 100ms – 2.0s
• CPU: 40% – 80%
After LiveSkannedPerSecond fixes:• Client read latency (99th percentile): 30ms – 100ms
• CPU: 10% – 30%
After Compaction fixes:• Client read latency (99th percentile): 10ms – 50ms
• CPU: 5% – 25%
WHAT ABOUT OUR GOAL?
![Page 58: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/58.jpg)
• TombstonesScannedPerSecond
• KeyCacheMissesPerSecond
• …
MORE METRICS!
![Page 59: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/59.jpg)
• TombstonesScannedPerSecond
• KeyCacheMissesPerSecond
• …
MORE METRICS!
Initial:• Client read latency (99th percentile): 100ms – 2.0s
• CPU: 40% – 80%
After all fixes:• Client read latency (99th percentile): 5ms – 25ms 50 times less at average!
• CPU: 5% – 15% 7 times less at average
![Page 60: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/60.jpg)
THANK YOU
![Page 61: SKB Kontur: Digging Cassandra cluster](https://reader031.vdocuments.us/reader031/viewer/2022022200/58a9d5ed1a28aba05b8b4b0b/html5/thumbnails/61.jpg)
Extra: The effect of the slow queries
pending tasks concurrent_reads