![Page 1: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/1.jpg)
© 2015 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. 15-2752.
Dan Aiello Principal Cyber Security Engineer, MITRE
Dan Aiello, Principal Cyber Security Engineer
Splunk Search Pro Tips
Splunk .conf2015
![Page 2: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/2.jpg)
| 2 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Agenda
§ My background
§ Comments § Search by index § In the year 2000 § Red Card § Watch Lists § Search Job Inspector § More fun with subsearches § Carrier signal § Imaginary events
§ Summary
2
![Page 3: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/3.jpg)
| 3 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
My Splunk background
§ 4 years Splunk experience § SOC is primary user base
§ 6 indexers § 350 GB data/day § 90 indexes § 170 sourcetypes
![Page 4: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/4.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 4 |
/ / N o c o m m e n t
comments
![Page 5: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/5.jpg)
| 5 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Comment your Splunk search
sourcetype=access_combined_wcookie | eval COMMENT="This is my comment" or sourcetype=access_combined_wcookie | rename COMMENT -‐> "This is my comment" * There’s nothing special about the word “COMMENT”, use whatever you like
![Page 6: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/6.jpg)
| 6 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
rename vs. eval for comments?
§ In practice, it does not seem to matter § In a few odd circumstances, I have seen rename be faster than eval
![Page 7: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/7.jpg)
| 7 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Why would you need comments?
![Page 8: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/8.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 8 |
Search by index
![Page 9: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/9.jpg)
| 9 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Searching by index and sourcetype
Specifying an index in your search speeds it up
This difference is less pronounced in Fast Mode
![Page 10: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/10.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 10 |
I n t h e y e a r 3 0 0 0
In the year 2000
![Page 11: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/11.jpg)
| 11 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Get with the times
§ Timestamps are extremely important for Splunk data § Detected at index time, set forever § Cannot be fixed if they’re wrong
§ Common errors: – Incorrect time zone interpretation – Host clock incorrect
![Page 12: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/12.jpg)
| 12 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Past, present, future
If this search ever returns events, you have timestamp problems1: index=* earliest=+30m latest=+9y This requires some tweaking, depending on your expected delay: index=* | eval delta=_indextime-‐_time | where delta>300 1 Or a flux capacitor2
2 Or a TARDIS
![Page 13: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/13.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 13 |
y o u r a p p r o x i m a t e w a i t t i m e i s…
red card
![Page 14: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/14.jpg)
| 14 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Calculate average delay proxy logs
index=main | eval delta = _indextime -‐ _time | timechart span=1h avg(delta) Problem: that’s a lot of events
![Page 15: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/15.jpg)
| 15 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
![Page 16: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/16.jpg)
| 16 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Calculate average delay proxy logs
Solution: */5 * * * * wget http://testdomain.zzz index=main testdomain.zzz | eval delta = _indextime -‐ _time | timechart span=1h avg(delta) Search terms Duration
index=main 453 s index=main testdomain.zzz 6 s
![Page 17: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/17.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 17 |
b e t t e r t h a n g r e p - f
watchlists
![Page 18: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/18.jpg)
| 18 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Watchlist examples
§ Known “evil” IP addresses § Known “evil” domain names § List of your DMZ web servers § Known allowed IP/port combinations in your DMZ
![Page 19: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/19.jpg)
| 19 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Example IP watchlist
![Page 20: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/20.jpg)
| 20 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Conventional way to use watchlists
This is essentially grep -F
![Page 21: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/21.jpg)
| 21 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Let’s try a subsearch
![Page 22: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/22.jpg)
| 22 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
What’s the difference? Both return the same events
![Page 23: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/23.jpg)
| 23 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
What’s the difference? 71% less time
![Page 24: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/24.jpg)
| 24 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Saving time on a search can be important for large
or frequent searches
Small watchlists and large datasets make
this difference greater
![Page 25: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/25.jpg)
| 25 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Just the subsearch
How does it work?
![Page 26: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/26.jpg)
| 26 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Subsearches implicitly end with
| format Add it explicitly to
see what’s happening
![Page 27: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/27.jpg)
| 27 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
sourcetype=access_combined_wcookie [ | inputlookup ip_watchlist.csv | search type=malicious | fields clientip ] …after the subsearch is evaluated becomes this: sourcetype=access_combined_wcookie ( ( clientip="131.178.233.243" ) OR ( clientip="212.58.253.71" ) OR … ) i.e., the results of the subsearch are appended
![Page 28: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/28.jpg)
| 28 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
The Search Job Inspector shows us this.
![Page 29: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/29.jpg)
| 29 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Why is sourcetype=access_combined_wcookie ( ( clientip="131.178.233.243" ) OR ( clientip="212.58.253.71" ) OR … ) Better than sourcetype=access_combined_wcookie | lookup ip_watchlist.csv clientip | search type=malicious
![Page 30: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/30.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 30 |
…e x p l a i n s i t a l l
Search Job Inspector
![Page 31: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/31.jpg)
| 31 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
This icon means there’s some debugging message
you should examine
Inspect Job is always here
![Page 32: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/32.jpg)
| 32 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
![Page 33: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/33.jpg)
| 33 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Debugging message
Profiling information
![Page 34: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/34.jpg)
| 34 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
The slowest parts of a Splunk search are usually field extraction
and reading events from disk.
![Page 35: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/35.jpg)
| 35 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Approximate order of operations for searches
1. Search index for keywords 2. Read matching events from disk 3. Extract fields (as necessary) 4. Match keywords to fields (as necessary) 5. Filter (e.g. additional “where” or “search” pipes) 6. Send data to search head
![Page 36: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/36.jpg)
| 36 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
What are keywords?
![Page 37: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/37.jpg)
| 37 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
A stitch in time saves nine
lookup subsearch 2 21 Check index for keywords
39,000 2,700 Read matching events from disk 39,000 2,700 Extract fields (i.e. regex) 39,000 2,700 Match keywords to fields 39,000 2,700 Filter
* This is illustrative and approximate, not precise
Pare your data early to save time late
![Page 38: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/38.jpg)
| 38 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
sourcetype=access_ combined_wcookie
( ( clientip="131.178.233.243" ) OR ( clientip="212.58.253.71" ) OR … )
lookup method reads and regexes all this data
subsearch method reads only this data
![Page 39: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/39.jpg)
| 39 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Compare “lookup” and “subsearch” methods
![Page 40: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/40.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 40 |
More fun with subsearches
![Page 41: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/41.jpg)
| 41 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Field name mismatch with subsearch
For lookup and subsearch, sometimes fields need to be renamed
lookup method | lookup watchlist.csv foo AS bar
subsearch method
[ | inputlookup watchlist.csv | rename foo AS bar ]
![Page 42: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/42.jpg)
| 42 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
If you rename a field to “query”, you can search
anywhere in the event rather than a single field
![Page 43: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/43.jpg)
| 43 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Large subsearches
43
If your watchlist is >10000 lines, the subsearch method
chokes
![Page 44: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/44.jpg)
| 44 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Large subsearches
44
Add “|format” explicitly to fix it! Warning is
gone
We have events
![Page 45: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/45.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 45 |
carrier signal
![Page 46: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/46.jpg)
| 46 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
What’s the problem with watchlists?
When they don’t alert, is it because: § the watchlist is broken? § there’s nothing to alert on?
![Page 47: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/47.jpg)
| 47 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Label your test domain as type=test in the watchlist
And adjust your subsearch accordingly
Add test cases to your watchlist
![Page 48: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/48.jpg)
| 48 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Label your test domain as type=test in the watchlist
And adjust your subsearch accordingly
Add test cases to your watchlist
What’s so great about that? We could have just used google.com for that test
![Page 49: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/49.jpg)
| 49 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
With your wget, you know precisely how many to
expect and you can alert only when it’s erroneous
![Page 50: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/50.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 50 |
w e l a n d e d o n t h e m o o n
imaginary events
![Page 51: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/51.jpg)
| 51 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Creating data on the fly
![Page 52: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/52.jpg)
| 52 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
How is this helpful?
You can add to a watchlist inline
![Page 53: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/53.jpg)
| 53 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Creating data on the fly with timestamps
53
This will also provide a current
timestamp
![Page 54: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/54.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 54 |
Are we there yet?
![Page 55: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/55.jpg)
| 55 |
© 2015 The MITRE Corporation. All rights reserved. © 2015 The MITRE Corporation. All rights reserved.
Overall lessons
§ Read Splunk Search Manual § Try multiple methods § Use the Job Inspector § Understand what Splunk is doing “under the hood”
![Page 56: Splunk Search Pro Tips...Approved for Public Release; Distribution Unlimited. 15-2752. © 2015 The MITRE Corporation. All rights reserved. Dan Aiello Principal Cyber Security](https://reader034.vdocuments.us/reader034/viewer/2022052015/602cb0eed0230b31f94ae66a/html5/thumbnails/56.jpg)
© 2015 The MITRE Corporation. All rights reserved.
| 56 |
Thank you!