cassandra day sv 2014: basic operations with apache cassandra
DESCRIPTION
Matt Stump, Solutions Architect at DataStax talks basic operations with Apache Cassandra.TRANSCRIPT
Operations & Tuning
Data Model First
Operations Can’t Fix a Bad Data Model
Sizing for Latency
CPU
Memory
Disk
Sizing for Latency
CPU
Buffer Cache
Disk
JVM
Sizing for Operations
(C * Nodes * Cores) / Replication Factor = Ops/Second
C is a hardware constant!• 3000 AWS Non-SSD!• 4000 Commodity SAS Hardware!• 12000 Commodity SSD Hardware
Sizing for Latency
•30-130 microseconds in memory!•100 microseconds to 12 milliseconds for SSD!•9 milliseconds to 60 milliseconds for mechanical disks
Shared Storage
DO NOT USE SHARED STORAGE!!!!!
Disable Access Time
/dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2
Warm The Buffer Cache
➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null \;
Disable NUMA Zone Reclaim
➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode
Use SSD
25% more expensive!10x Faster
Tuning SSD: Read Ahead
➜ ~ sudo blockdev --report RO RA SSZ BSZ StartSec Size Devicerw 256 512 4096 0 6442450944 /dev/xvdarw 256 512 4096 1920 6144000000 /dev/xvda1rw 8 512 4096 0 343568023552 /dev/xvdbrw 256 512 4096 0 343568023552 /dev/xvdc
➜ ~ sudo blockdev --setra 8 /dev/xvdb
Tuning SSD: Scheduler
➜ ~ echo 0 > /sys/block/xvdb/queue/rotational➜ ~ echo noop > /sys/block/xvdb/queue/scheduler
Compaction
Mechanical Disk = Size Tiered!SSD = Leveled Compaction
TTL
TTL Compaction!https://issues.apache.org/jira/browse/CASSANDRA-5228!!
Default TTL!https://issues.apache.org/jira/browse/CASSANDRA-3974
Cassandra Heap Settings
8G MAX_HEAP !2G NEW_HEAP
Solr/Cassandra Heap Settings
14G MAX_HEAP !2G NEW_HEAP
Hadoop/Cassandra Heap Settings
10G MAX_HEAP !2G NEW_HEAP
Kernel
Modern Kernel = 30% performance boost
Monitoring
nodetool cfhistograms
Offset SSTables Write Latency Read Latency Row Size Column Count 1 3579 0 0 0 0 2 0 0 0 0 0 . . . 35 0 0 0 0 0 42 0 0 27 0 0 50 0 0 187 0 0 60 0 10 460 0 0 72 0 200 689 0 0 86 0 663 552 0 0 103 0 796 367 0 0 124 0 297 736 0 0 149 0 265 243 0 0 179 0 460 263 0 0 . . . 25109160 0 0 0 0 0
nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 15 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 0 0 3674 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemoryMeter 0 0 1 0 0 MemtablePostFlusher 0 0 267 0 0 FlushWriter 0 0 9 0 1 MiscStage 0 0 0 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 !Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0
Warning Signs• Pending compactions > 15!• Parnew collections > 1 second!• CMS collections > 5 minutes!• Disk utilization above 40% or 70%!• CFHistograms write throughput > 150 μs!• CFHistograms read throughput > 50 ms!• Really large rows in CFHistograms!• Anything blocked in TPStats
Tools