caching - percona · database & application harmony • a well designed database schema may not...
Post on 19-Jul-2018
223 Views
Preview:
TRANSCRIPT
Caching
Memcached & Redis
MMMMMM Databases
• We love them, otherwise we would not be here
Sometimes though, they suck
Need it (-5)
Need it (-4)
Need it (-3)
Need it (-2)
Need it (-1)
Need it N
ow!
Still need it
0
1000
2000
3000
4000
5000
6000
Database Performance
• Sometimes our databases under perform when we need them most!
• The mix of a heavy workload & lots of users can make performance unpredictable
Database & Application Harmony
• A well designed database schema may not store data in a way that is optimal for the application to consume it
• A well designed application does not always use objects that lend itself to optimal schema design• i.e. think ORM's like Hibernate or Active
Record
Example
• An Employee “Object” could be comprised of• Employee table• Department table• Position table • Office table• Office location table• Cost Center table• etc
Object Centric Schema
• Building a more object like schema can often lead to problems:• Bigger database sizes – i.e. table bloat,
duplicated data• Problems accessing data outside of
applications initial design • i.e. complex queries, weird access patterns
These can lead to performance issues!
Database Centric Schema
• Building a classic relational schema can also lead to issues:• Problems building and accessing the core
application object• i.e. having to access lots of tables, pulling back
data you may not need, etc
These can lead to performance issues too!
Enter External Caching
• Can speed up access substantially• Mask complex schema's
• Reduce workload on database servers
•
Why can external caching be faster?
It's really an unfair race
Reducing work
• Less Joins• Less Tables to access
• PK like access on “Objects”
Memory is faster then disk
VS
Component Access Times
• L1 cache reference
• Branch mispredict
• L2 cache reference
• Mutex lock/unlock
• Main memory reference
• Compress 1K bytes with Zippy
• Send 2K bytes over 1 Gbps network
• Read 1 MB sequentially from memory
• Round trip within same datacenter
• Disk seek
• Read 1 MB sequentially from disk
• Send packet CA->Netherlands->CA
• 0.5 ns
• 5 ns
• 7 ns
• 25 ns
• 100 ns
• 3,000 ns
• 20,000 ns
• 250,000 ns
• 500,000 ns
• 10,000,000 ns
• 20,000,000 ns
• 150,000,000 ns
See: http://www.linux-mag.com/cache/7589/1.html and Google http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
Network + Memory faster
+ >
External Caching Architecture
Web/App Servers
Cache Servers Database Server(s)
External Caching Architecture
DB
CacheIn
Cache?
No
Yes
ReturnData
Check Cache
Get Data ●Is the Data in Cache?
●If Yes Return it.●If no hit the database
● Store in cache● Return data
Note a Cache miss requires 3 ops instead of 1
Issues with external caching
• Thundering herd/stampede• Cache Miss is 3 hops
• Access Stale Data• Many layers here, consistency between db
and cache is only 1
• Cache warm-up• Over Cache
• Too many cache hits on a single page
Cache Stampede
• The cache key called CARS_SOLD_COUNT is set to expire at 9:07am.
• At 9:07am, the first apache request to discover no cache starts regenerating the cache key.
• This key is used on the homepage - which serves 60 apache requests/second.
• .. it takes 3 seconds to do this.• .. what happens to all other requests in this
time?
Cache Stampede (cont.)
• 60 requests / second * 3 seconds = 180.• In most naive caching implementations, this
is how many threads will be regenerating the cache key simultaneously!
• This problem is real, and can often be seen as ‘blips’ in cacti graphs.
Memcached
• The well established - battle tested cache solution• Solid• Well Known• Huge deploy base• Dead simple• Fast
How it works
• In simple terms its key-value (Value can be serialized object, text, etc)
• get(key) returns value• set(key,value)
• In Memory, no Persistence
• Supports distributed servers
• Consistent Hashing• No authentication
• By default objects < 1MB in size
• Objects can be pushed from cache or expire
Commands
• Limited commands help keep things simple
• Supported commands:• set,add,replace,append,prepend,cas• get,gets• delete• incr/decr• stat
Example
• Employee Object• $employee_id = 1234• $object ={name => 'Matt Yonkovit', Title =>
'Director of Consulting', Location => 'St Johns, MI USA', Age => 35, Department=> Consulting, Americas, Reports to=>'Peter Zaitsev', Reports to title=>'CEO' }
• set($employee_id,$object)• get($employee_id)
• Returns $object
More details
• Getting my employee object is a minimum of 194 bytes.
• Even if I need only my name, the full 194 bytes is returned.
• I then must parse this in code before I move on
Features
• Non-blocking sets• Multi-threaded
• Multi-operations
• Consistent hashing
Add Ons/Commercial
• There are add-ons for memcached that help do things like• Add replication• Add tags to your obects• Etc.
Where Memcached works
• When you have no control over the number of objects or growth
• When you want to keep things simple• Established applications with hooks
already built in• Consistent reliable speed
Redis
• Feature packed cache solution, designed for more then just caching• Advanced feature set
• Replication• Multiple data types• Persistence
• Via Journal or snapshot
• Gaining popularity•
Redis Datatypes
• Strings• Similar to memcached access• Support for
• Incr/decr• Get bit• Get/Set Range (i.e. substr)• Expiration time• Multi get/set
Redis Datatypes
• Lists • Stored in insert order• Can push at the end or beginning• Fast head/tail access• Access by index• Complex built in functions , i.e. move from 1
list to another
List Example
LPUSH employee_list "Matt"
:1
LPUSH employee_list "Peter"
:2
LRANGE employee_list 0 1
*2
$7
"Peter"
$6
"Matt"
RPUSH employee_list "Baron":3LRANGE employee_list 0 1*3$7"Peter"$6"Matt"$7"Baron"
Redis Datatypes
• Sets• Unordered collections of strings• There are no repeated members of a set• Support for
• diffing sets• Intersecting/merging• Moving form 1 set to another
Set Example
sadd matts.friends yves peter
baron kenny
:4
smembers matts.friends
*4
$4
yves
$5
peter
$5
kenny
$5
baron
sadd kenny.friends yves baron matt:3sdiff matts.friends kenny.friends
*2$5peter$5kenny
Sorted Sets
• Sorted sets• Non-repeated• Ordered by a score• Supports
• most set functions• Ranges of scores•
Redis Datatypes
• Hashes • Object like access
• HGET <id> <key> returns Value• HGETALL <id> returns a list of key->value pairs
Revisit our Example
• Employee Object• $employee_id = 1234• $object ={name => 'Matt Yonkovit', Title =>
'Director of Consulting', Location => 'St Johns, MI USA', Age => 35, Department=> Consulting, Americas, Reports to=>'Peter Zaitsev', Reports to title=>'CEO' }
Using a hash
• HMSET 1234 name “Matt Yonkovit” Title “Director of Consulting” Location “St Johns, MI USA” Age “35” Department “Consulting, Americas” Reports_to “Peter Zaitsev” Reports_to_title “CEO”
• HGET 1234 name • returns “Matt Yonkovit”
More details
• In Memcached • Getting my employee object is a minimum of
194 bytes. • Even if I need only my name, the full 194
bytes is returned. I then must parse this in code before I move on
• Using a Redis Hash• I can return ~13 Bytes containing my name
Shortcomings
• Redis is single threaded
• Can work around it, like we used to with memcached starting multiple daemons
• Going to be fixed in upcoming Redis cluster release
• Not every client has consistent hashing
• Some operations require extra memory, i.e. snapshots
• Bound to memory on server
TBD: Where Redis works
• When storing rankings• sorted sets are great
• When you only need small pieces of an object• i.e. hash
• When you need ranges, lists, or queue• Need persistence
• Replication is helpful
Benchmarks
• Single node benchmarks are easy, but the because the server is single threaded the benchmark tools are not setup to benchmark multiple redis servers.
• Used redis-benchmark and mc-benchmark• Tests were run in a loop• Datasets where small• 2K objects, 500K requests/distribution• These will randomly return set/gets
Single Threaded Benchmarks
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 430
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
# of Writes per second
Single Threaded Server
Redis
Memcached
Single Threaded Benchmarks
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 430
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
# of Writes per second
Single Threaded Server
Redis
Memcached
Whats Up with This?
Benchmarks over time
2 22 42 62 82 102
122
142
162
182
202
222
242
262
282
302
322
342
362
382
402
422
442
462
482
502
522
542
562
0
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
Long Run Benchmarks
redis -vs- memc
redis set
redis get
memc set
memc get
Change Versions
2 42 82 12212
2232 52
6272 92
102112 132
142152
162172
182192
202212
222232
242252
262272
282292
302312
322332
342352
362372
382392
402412
422432
442452
462472
482492
502512
522532
542552
562572
582592
602612
622632
642652
662672
682
80000
100000
120000
140000
160000
180000
200000
Long Running New Version
Redis 2.9.7 – Set Redis 2.4 – Set Memc – Set
Different Hardware – Much Slower CPU, 3.2 Kernel
1 21 41 61 81 1016
1116 26
3136 46
5156 66
7176 86
9196 106
111116
121126
131136
141146
151156
161166
171176
181186
191196
201206
211216
221226
231236
241246
251256
261266
271276
281286
291296
301306
311316
321326
331336
341346
351356
361366
371376
381
0
20000
40000
60000
80000
100000
120000
140000
Redis V Memcached
Ubuntu 12.04
Redis GET Redis SET Memc Get Memc Set
Ec2 Test
13
57
911
1315
1719
2123
2527
2931
3335
3739
4143
4547
4951
5355
5759
6163
6567
6971
7375
7779
8183
8587
8991
9395
9799
101103
105107
109111
113115
117119
121123
125127
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
ec2 tests Against Redis 2.4, Ubuntu 11.10
Get Set
There is more here that needs to be looked into
Persistence
• Redis config Options• Snapshots:
• save <seconds> <changes>
• Append Only (Journal)• appendonly <no/yes>• appendfsync <always/everysec/no>• no-appendfsync-on-rewrite <no/yes>•
•
Cost of Redis Persistance
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 430
20000
40000
60000
80000
100000
120000
140000
160000
180000
200000
Redis Benchmarks with Persistance
No Persistence
Snapshots
Append only
Need more Benchmarking
• Datatype specific benchmarks
• So I will need to build a test using the ruby client libraries that do support multiple servers via consistent hashing
• Need big dataset tests! Need to test with 64GB or 128GB of memory
• Impact of replication
Redis Issues
• Had several crashes under heavy load • Looked to be related to memory usage with
the append file
• Weird Performance Fluctuations repeated on multiple servers
• Missing consistent hashing in some clients• Rewrite underway to support cluster
top related