strangeloop 2012 apache cassandra anti patterns
DESCRIPTION
random list of Apache Cassndra Anti Patterns. There is a lot of info on what to use Cassandra for and how, but not a lot of information on what not to do. This presentation works towards filling that gap.TRANSCRIPT
![Page 1: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/1.jpg)
Apache CassandraAnti-Patterns
Matthew F. Dennis // @mdennis
![Page 2: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/2.jpg)
C* on a SAN
● fact: C* was designed, from the start, for commodity hardware
● more than just not requiring a SAN, C* actually performs better without one
● SPOF● unnecessary (large) cost● “(un)coordinated” IO from nodes ● SANs were designed to solve problems C*
doesn’t have
![Page 3: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/3.jpg)
Commit Log + Data Directory(on the same volume)
● conflicting IO patterns● commit log is 100% sequential append only● data directory is (usually) random on reads
● commit log is essentially serialized● massive difference in write
throughput under load● NB: does not apply to SSDs or EC2
![Page 4: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/4.jpg)
Oversize JVM Heaps
● 4 – 8 GB is good(assuming sufficient ram on your boxen)
● 10 – 12 GB is not bad(and often “correct”)
● 16GB == max● > 16GB => badness● heap >= boxen RAM => badness
![Page 5: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/5.jpg)
oversized JVM heaps(for the visually oriented)
![Page 6: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/6.jpg)
not using -pr on scheduled repairs
● -pr is kind of new● only applies to scheduled repairs● reduces work to 1/RF (e.g. 1/3)
![Page 7: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/7.jpg)
low file handle limit● C* requires lots of file handles
(sorry, deal with it)● Sockets and SSTables mostly● 1024 (common default) is not sufficient● fails in horrible miserably unpredictable ways
(though clear from the logs after the fact)● 32K - 128K is common● unlimited is also common, but personally I
prefer some sort of limit ...
![Page 8: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/8.jpg)
Load Balacners(in front of C*)
● clients will load balance(C* has no master so this can work reliably)
● SPOF● performance bottle neck● unneeded complexity● unneeded cost
![Page 9: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/9.jpg)
restricting clients to a single node
● why? ● no really, I don’t understand how
this was thought to be a good idea● thedailywtf.com territory
![Page 10: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/10.jpg)
Unbalanced Ring● used to be the number one
problem encountered● OPSC automates the resolution of
this to two clicks (do it + confirm) even across multiple data centers
● related: don’t let C* auto pick your tokens, always specify initial_token
![Page 11: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/11.jpg)
Row Cache + Slice Queries
● the row cache is a row cache, not a query cache or slice cache or magic cache or WTF-ever-you-thought-it-was cache
● for the obvious impaired: that’s why we called it a row cache – because it caches rows
● laughable performance difference in some extreme cases (e.g. 100X increase in throughput, 10X drop in latency, maxed cpu to under 10% average)
![Page 12: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/12.jpg)
Row Cache + Large Rows
● 2GB row? yeah, lets cache that !!!
● related: wtf are you doing trying to read a 2GB row all at once anyway?
![Page 13: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/13.jpg)
OPP/BOP● if you think you need BOP, check
again● no seriously, you’re doing it wrong● if you use BOP anyway:
● IRC will mock you● your OPS team will plan your disappearance● I will setup a auto reply for your entire domain
that responds solely with “stop using BOP”
![Page 14: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/14.jpg)
Unbounded Batches● batches are sent as a single message● they must fit entirely in memory
(both server side and client side)● best size is very much an empirical
exercise depending on your HW, load, data model, moon phase, etc(start with 10 – 100 and tune)
● NB: streaming transport will address this in future releases
![Page 15: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/15.jpg)
Bad Rotational Math
● rotational disks require seek time● 5ms is a fast seek time for a rotational disk● you cannot get thousands of random seeks
per second from rotational disks● caches/memory alleviate this, SSDs solve it● maths are teh hard? buy SSDs● everything fits in memory? I don’t care what
disks you buy
![Page 16: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/16.jpg)
32 Bit JVMs
● C* deals (usually) with BigData● 32 bits cannot address BigData● mmap, file offsets, heaps, caches● always wrong? no, I guess not ...
![Page 17: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/17.jpg)
EBS volumes● nice in theory, but ...● not predictable● freezes common● outages common● stripe ephemeral drives instead● provisioned IOPS EBS?
future hazy, ask again later
![Page 18: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/18.jpg)
Non-Sun (err, Oracle) JVM
● at least u22, but in general the latest release(unless you have specific reasons otherwise)
● this is changing● some people (successfully) use
OpenJDK anyway
![Page 19: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/19.jpg)
Super Columns● 10-15 percent overhead on reads and writes● entire super column is always held in memory
at all stages● most C* devs hate working on them● C* and DataStax is committed to maintaining
the API going forward, but they should be avoided for new projects
● composite columns are an alternative
![Page 20: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/20.jpg)
Not Running OPSC
● extremely useful postmortem● trivial (usually) to setup● DataStax offers a free version
(you have no excuse now)
![Page 21: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/21.jpg)
Q?Matthew F. Dennis // @mdennis
![Page 22: strangeloop 2012 apache cassandra anti patterns](https://reader033.vdocuments.us/reader033/viewer/2022060115/557a35c3d8b42a32248b48fc/html5/thumbnails/22.jpg)
Thank You!Matthew F. Dennis // @mdennishttp://slideshare.net/mattdennis