tuning solr for logs
DESCRIPTION
Tuning Solr for indexing and searching logs.TRANSCRIPT
Tuning Solr for Logs
Radu Gheorghe
@radu0gheorghe @sematext
/me does...
Logsenesearch consulting + = logging consulting
.com/logsene
Tuning. Is it worth it?
baseline last run
# of logs 10M 310M
EC2 bill/month 700 450
What to optimize for?
http://www.seasonslogs.co.uk/images/products/SL_001.pnghttps://openclipart.org/image/300px/svg_to_png/169833/Server_1U.png
capacity: how many logs
the same hardware can keep
while still providing decent
performance
What's decent performance? “It depends”
Assumptions
indexing: enough to keep up with generated logs*
search concurrency
search latency: 2s for debug queries, 5s for charts
*account for spikes!
Enough theory, let's start testing!
Solr instance
m3.2xlarge (8CPU, 30GB RAM, 2x80GB SSD)
Solr 4.10.1
Feeder instance
c3.2xlarge (8CPU, 15GB RAM, 2x80GB SSD)
apache access logs
python script to parse and feed them
Baseline test
15GB heap
debug query
status:404 in the last hour
charts query
all time status counters
all time top IPs
user agent word cloud
http://blog.sematext.com/2013/12/19/getting-started-with-logstash/
Baseline result
100K 2.5M 4M 6M 9M 10M0
2000
4000
6000
8000
10000
12000
debugchartsEPS
100K 2.5M 4M 6M 9M 10M0
2000
4000
6000
8000
10000
12000
debugchartsEPS
Baseline result
capacity
100K 2.5M 4M 6M 9M 10M0
2000
4000
6000
8000
10000
12000
debugchartsEPS
Baseline result
capacitybottleneck: facets eat CPU
100K 2.5M 4M 6M 9M 10M0
2000
4000
6000
8000
10000
12000
debugchartsEPS
Baseline result
capacitybottleneck: facets eat CPU
on average,CPU is OK
100K 2.5M 4M 6M 9M 10M0
2000
4000
6000
8000
10000
12000
debugchartsEPS
Baseline result
capacitybottleneck: facets eat CPU
indexing limitedbecause pythonscripts eatsfeeder CPU
on average,CPU is OK
Indexing throughput: is it enough?
“it depends”
how long do you keep your logs?
1M logs/day * 10 days <> 0.3M logs/day * 30 days. Both need 10M capacity
1M logs/day * 30 days? Needs 3 servers, each getting 0.3M logs/day
Baseline run: 10M index fills up in <1/2h at 7K EPS
Indexing throughput: is it enough?
“it depends”
how long do you keep your logs?
1M logs/day * 10 days <> 0.3M logs/day * 30 days. Both need 10M capacity
1M logs/day * 30 days? Needs 3 servers, each getting 0.3M logs/day
how big are your spikes? (assumption: 10x regular load)
7K EPS is enough for 10M capacity if you keep logs >5h
1.5M 3M 5M 8M 11M0
1000
2000
3000
4000
5000
6000
7000
8000
chartsEPSdebug
Rare commits
10% above baseline
auto soft commits every 5 seconds
auto hard commits every 30 minutes
RAMBufferSize=200MB; maxBufferedDocs=10M
Same results with
even rarer commits (auto-soft every 30s, 500MB buffer)
omitNorms + omitTermFreqAndPositions
larger caches
cache autowarming
THP disabled
mergeFactor 5
mergeFactor 20
but indexingwas cheaper
manually ranqueries, too
1.5M 3M 5M 8M 10M 12M0
1000
2000
3000
4000
5000
6000
7000
8000
chartsEPSdebug
DocValues on IP and status code
20% above baseline
3M 10M 18M 24M 31M 36M0
10002000300040005000600070008000
chartsEPSdebug
Detour: what if user agent was string?
3.6x baseline
8M 16M 24M 32M 40M 48M 56M 64M 67M 69M 70M 70.5M0
1000
2000
3000
4000
5000
6000
7000
8000
chartsEPSdebug
… and if user agent used DocValues?
6.7x baseline
reducing indexingadds 5% capacity
3M 7M 11M 15M 19M 23M 27M 28M0
5000
10000
15000
20000
25000
30000
35000
chartsEPSdebug
Time based collections (1 minute)
2.7x baseline
OOM (150 collections)
10M 40M 70M 100M 130M 160M 190M 213M0
1000
2000
3000
4000
5000
6000
7000
8000
chartsEPSdebug
Time based collections (10 minutes)
21x baseline
still OOM(~100 collections)
50M 100M 150M 200M 250M 300M 310M 330M 340M0
1000
2000
3000
4000
5000
6000
7000
8000
chartsEPSdebug
10min collections: 20GB heap; optimize old
31x baseline,5 days projected retentionwith 10x spikes
no more OOM,just slower queries
34x baseline,10 days projectedretention (10x)
Software optimizations recap
Definitely worth it Nice to have I wouldn't bother
time-based collections
noop I/O scheduler merge policy tuning
DocValues omit norms, term frequencies and positions
autowarm
rare soft commits optimize “old” collections
super-rare soft commitsdisable THP
20M 70M 120M 170M 220M 270M 320M 372M0
1000
2000
3000
4000
5000
6000
7000
chartsEPSdebug
r3.2xlarge: +30GB RAM, +$0.14/h, 1x160GB SSD
37x baseline,9 days projected retentionwith 10x spikes
less indexing throughputthan m3.2xlarge
20M 50M 80M 110M 140M 170M 177M0
100020003000400050006000700080009000
chartsEPSdebug
c3.2xlarge: -15GB RAM, -$0.14/h
17x baseline,5 days projected retentionwith 10x spikes
Monthly EC2 cost per 1M logs*
m3.2xlarge: $1.3r3.2xlarge: $1.33c3.2xlarge: $1.78
TODO (a.k.a. truth always messes with simplicity): more/expensive facets => more CPU => c3 looks better less/cheap facets => not enough instance storage => EBS (magnetic/SSD/provisioned IOPS)? => storage-optimized i2? => old-gen instances with magnetic instance storage? use different instance types for “hot” and “cold” collections?
*on-demand pricing at 2014-11-07
How NOT to build an indexing pipeline
custom script: reads apache logs from files
parses them using regex
takes 100% CPU and 100% RAM from a c3.2xlarge instance
maxes out at 7K EPS
Enter Apache Flume*
*Or Logstash. Or rsyslog. Or syslog-ng. Or any other specialized event processing tool
agent.sources = spoolSrcagent.sources.spoolSrc.type = spooldiragent.sources.spoolSrc.spoolDir = /var/log
agent.sources.spoolSrc.channels = solrChannelagent.channels = solrChannelagent.channels.solrChannel.type = fileagent.sinks.solrSink.channel = solrChannel
agent.sinks = solrSinkagent.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSinkagent.sinks.solrSink.morphlineFile = conf/morphline.confagent.sinks.solrSink.morphlineId = 1
put Solr and Morphlinejars in lib/
channel
source
sink
morphline.conf (think Unix pipes)morphlines : [ { id : 1 commands : [ { readLine { charset : UTF-8 } } { grok { dictionaryFiles : [conf/grok-patterns] expressions : { message : """%{COMBINEDAPACHELOG}""" } } } { generateUUID { field : id } } { loadSolr { solrLocator : { collection : collection1 solrUrl : "http://10.233.54.118:8983/solr/" } } } ] }]
same ID as in the flume.confsink definition
process one line at a time(there's also readMultiLine)
https://github.com/cloudera/search/blob/master/samples/solr-nrt/grok-dictionaries/grok-patterns
parses each property(eg: IP, status code)in its own field
Solr cando it, too*
use zkHostfor SolrCloud
*http://solr.pl/en/2013/07/08/automatically-generate-document-identifiers-solr-4-x/
Result: 2.4K EPS, feeder machine almost idle
2.4K EPS is typically enough for this
application server+ Flume agent
application server+ Flume agent
application server+ Flume agent
scales nicely with # of serversbut all buffering and processingis done here
but not for this
application server+ Flume agent
application server+ Flume agent
application server+ Flume agent
centralized bufferingand processing
Flume agent
Flume agent
or this
application server+ Flume agent
application server+ Flume agent
application server+ Flume agent
buffer, then process (separately)
Flume agent
Flume agent
Flume agent
Increase throughput: batch sizes; memory channel
agent.sources = spoolSrcagent.sources.spoolSrc.type = spooldiragent.sources.spoolSrc.spoolDir = /var/logagent.sources.spoolSrc.batchSize = 5000
agent.sources.spoolSrc.channels = solrChannelagent.channels = solrChannelagent.channels.solrChannel.type = file memoryagent.channels.solrChannel.capacity = 1000000agent.channels.solrChannel.transactionCapacity = 5000agent.sinks.solrSink.channel = solrChannel
agent.sinks = solrSinkagent.sinks.solrSink.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSinkagent.sinks.solrSink.morphlineFile = conf/morphline.confagent.sinks.solrSink.morphlineId = 1agent.sinks.solrSink.batchSize = 5000
solrLocator : { collection : collection1 solrUrl : "http://10.233.54.118:8983/solr/" batchSize : 5000}
make sure you have enough heap
Result: 10K EPS, 6%CPU usage (2x baseline)
More throughput? Parallelize
Depends* on the bottleneck
*last time I use this word, I promise
source channel sink
more threads (if applicable)
more sources
multiplexingchannel selector
load balancingsink processor
more threads (if applicable)
Source1 C1
Source1
C1
Source2
Source1
C1
C2
C1 Sink1
C1
Sink1
Sink2
Result: default Solr install maxed out at 24K EPS
TODO: log in JSON where you can
Then, in morphline.conf, replace the grok command with the much ligher: readJson {}
Easy with apache logs, maybe not for other apps:
LogFormat "{ \ \"@timestamp\": \"%{%Y-%m-%dT%H:%M:%S%z}t\", \ \"message\": \"%h %l %u %t \\\"%r\\\" %>s %b\", \... \"method\": \"%m\", \ \"referer\": \"%{Referer}i\", \ \"useragent\": \"%{User-agent}i\" \ }" ls_apache_jsonCustomLog /var/log/apache2/logstash_test.ls_json ls_apache_json
More details at:http://untergeek.com/2013/09/11/getting-apache-to-output-json-for-logstash-1-2-x/
Conclusions
Use time-based collections and DocValues
Rare soft&hard commits are good Pushing them too far is probably not worth it
Hardware: test and see what works for you A balanced, SSD-backed machine (like m3) is a good start
Use specialized event processing tools Apache Flume is a fine example
Processing and buffering on the application server side scales betterBuffer before [heavy] processingMind your batch sizes, buffer types and parallelizationLog in JSON where you can
Thank you!
Feel free to poke me @radu0gheorghe
Check us out at the booth, sematext.com and @sematext
We're hiring, too!