managing*search*heads* using*rest* - splunkconfto*working*with*splunk*idid*webmethods*&*java...
TRANSCRIPT
Copyright © 2013 Splunk Inc.
Jason Hupka Technical Architect, Staples
Managing Search Heads Using REST
#splunkconf
About Me ! Worked for Staples for 12 years
! Working with Splunk for 2 years
! Prior to working with Splunk I did webMethods & Java development
! I can add nearly any food to a quesadilla and convince my children to eat it
2
! World’s largest office supply company
! 2nd largest on-‐line retailer
! Over 1500 retail stores in North America
About .
3
What We’re Trying to Solve…
! We don’t like manually configuring anything in Splunk
! Majority of configuraUon can be controlled by .conf files, but a few have caveats
! Deployment server can’t deploy everything we want to manage remotely
5
High-‐Level SoluUon…
! Use REST endpoints to perform core configuraUon instead of manually via GUI
! Script the REST configuraUons so they are: – Automatable – Repeatable – Self-‐Healing – EASY!
6
Splunk Environment at Staples ! We have mulUple, idenUcal search heads ! We do not use search head pooling due to datacenter constraints
! We could have lots of people with their hands in the pot
7
Splunk Instances at Staples
x 8 search heads
x 8 indexers
4000+ forwarders
x 5 search heads
x 3 indexers
~ 150 forwarders
Non-‐produc2on Produc2on
8
How We Deploy Splunk ArUfacts ! We create two tar files per environment each build:
– [env]_search_head.tar – [env]_deployment_server.tar
– prod_search_head.tar prod_deployment_server.tar – devtest_search_head.tar devtest_deployment_server.tar
! So this REST configuraUon soluUon is a bit self-‐imposed, but we sUll have the issue of certain sejngs in .conf files that must be configured via a Splunk interface
9
Why Not Just Use .conf Files? ! Not everything can be configured correctly via .conf:
– Search peers – LDAP configuraUon – Users/passwords
! Some global context sejngs we want to fully control and not let anyone change via .conf overlays or manually changing in the GUI
– Sejngs in $SPLUNK_HOME/etc/system/local can be difficult to set per-‐environment
– Self-‐healing
! We want to avoid a proliferaUon of environment-‐specific apps or tar-‐files
10
REST for Search Head Management ! Remember, Splunk maps the HTTP methods for their endpoints like the following:
– GET Read value – POST Update (create) value – DELETE Delete value
! We decided to use REST endpoint that configure many of the global context configuraUons
– Stuff in $SPLUNK_HOME/etc/system/local – a.k.a stuff on the right-‐hand side of Splunk manager
11
Exploring REST ! Not everything is listed in REST API documentaUon:
– htp://docs.splunk.com/DocumentaUon/Splunk/latest/RESTAPI/RESTlist
! This is due to deeper endpoints tend to be specific to your search head's current configuraUon
! You will need to explore the REST interface with a browser to see what is there and what can be modified
13
Exploring REST Example – LDAP (1)
! htps://localhost:8089/services/authenUcaUon
What’s in providers?
14
Exploring REST Example – LDAP (2)
! htps://localhost:8089/services/authenUcaUon/providers
Ooh -‐ LDAP
15
Exploring REST Example – LDAP (3)
! htps://localhost:8089/services/authenUcaUon/providers/LDAP
Not everyone may have the LDAP endpoint if the Search Head isn’t configured for LDAP
16
Exploring REST Example – LDAP (4) ! htps://localhost:8089/services/authenUcaUon/providers/LDAP/ped ! This ped endpoint is specific to our environment:
17
A Few Endpoints We Use ! Configure LDAP providers(s)
– /services/authen.ca.on/providers/LDAP
! Manage LDAP group-‐role mappings – /services/admin/LDAP-‐groups
! Manage Splunk roles – /services/authoriza2on/roles
! Manage Splunk users (update/delete local users) – /services/authen.ca.on/users
18
A Few Endpoints We Use (cont)
19
! Configure various global context sejngs – /services/proper2es/
! Setup search peers (e.g. index servers) – /services/search/distributed/peers
Now Let’s Automate What We Know ! We have a list of servers we need to manage
– At Staples, all of our search heads
! We have a list of endpoints we want to use for configuraUon – We just went through the example of LDAP and listed others we
use at Staples
! We should have an idea of what we want for our configuraUon – It helps to manually setup a search head and explore the resultant
configuraUon sejngs via REST
20
High-‐Level Pseudocode ! For each search head
– For each REST endpoint ê Apply configuraUon items
21
Sample Code ! Code can be found on GitHub:
– h@ps://github.com/jhupka/splunk_conf2013
! manageSearchHeads.py is a simple python script that implements the pseudocode and is driven by python config files
22
More InformaUon ! Script should log before view and what it is doing
– Splunk this! ! Include script as part of our conUnuous integraUon builds
– ! Script could be app-‐iUzed
– App could live on deployment server – Script is scheduled to run periodically to self-‐heal
Splunk environment
26
More InformaUon (cont) ! Extend this to control not just search heads:
– REST configuraUon of index servers – REST configuraUon of forwarders
27
Key Take Away Managing Splunk Search Heads Using REST
SoluUon: • UUlize REST interface to remotely
configure those pesky sejngs
• Centralize and automate these REST calls to ensure a consistent environment and remove manual steps and human-‐error
• Put problem foods into quesadillas to trick them into eaUng them
Problem: • Not everything is easily configured
via .conf files
• Deployment server works at the app level, but some sejngs we want to live in $SPLUNK_HOME/etc/system/local
• SomeUmes children won’t eat certain foods
28
Next Steps
Download the .conf2013 Mobile App If not iPhone, iPad or Android, use the Web App
Take the survey & WIN A PASS FOR .CONF2014… Or one of these bags! Go to “Shu@l – Get More Value from Your Splunk Data” Room: Brera 2&3, Level 3 Today, 1:45-‐2:45pm
1
2
3
29