supporang*and* upgrading*large* deployments* · pdf fileexpedia’s*story*and*nextsteps*...
TRANSCRIPT
Copyright © 2014 Splunk Inc.
Erik Andresen System Engineer II Expedia, Inc.
SupporAng and Upgrading Large Deployments
Disclaimer
2
During the course of this presentaAon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauAon you that such statements reflect our current expectaAons and
esAmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,
please review our filings with the SEC. The forward-‐looking statements made in the this presentaAon are being made as of the Ame and date of its live presentaAon. If reviewed aPer its live presentaAon, this presentaAon may not contain current or accurate informaAon. We do not assume any obligaAon to update any forward-‐looking statements we may make. In addiAon, any informaAon about our roadmap outlines our general product direcAon and is subject to change at any Ame without noAce. It is for informaAonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaAon either to develop the features or funcAonality described or to
include any such feature or funcAonality in a future release.
PresentaAon Agenda
! About the Speaker ! Expedia, Inc.: A Global Enterprise ! Components and Symbols
! StarAng Out Small – Single Server ! SeparaAng FuncAons – MulAple Tiers
! Expanding Horizontal – MulAple Servers ! Expanding Datacenters – MulA-‐Datacenter Instances ! Headed to the Cloud – AWS Instances
! SeparaAng Teams – Pod Instances ! Up Next for Expedia’s Splunk Environment 3
About the Speaker
5
! System Engineer II, Monitoring Tools Team, Expedia, Inc. – As Splunk Administrator, helps to manage the enAre Splunk Environment – The team provides monitoring, analyAcs, and automaAon soluAons for
Expedia – Has worked for Expedia since September 2012
! Previously contracted at MicrosoP – OperaAons Engineer with the Xbox OperaAons Center (XOC) – Worked for four years in various roles from Tier I to Tier 2.5 – SME
! Graduated from and worked at the University of Washington – Business AdministraAon Degree – Management InformaAon Systems, 2008 – Windows System Administrator, Electrical Engineering Department
Expedia, Inc.: A Global Enterprise
8 1Includes eLong bookable properties
2comScore Worldwide Average Monthly UV data, TTM 2Q14
Components and Symbols
11
Icon Descrip,on
Splunk Server – A server that acts as both a search head and an indexer that allows users to search, aggregate, and manipulate machine data that is locally stored.
Search Head – A dedicated Splunk instance uAlized by users to search, aggregate, and manipulate machine data that is stored on Splunk indexer(s).
Indexer – A dedicated Splunk instances that holds machine data in a proprietary map-‐reduce repository and enables it to be searched/returned to the search head.
Forwarder – A package installed onto a client server that sends data into Splunk in a format that allows it to be indexed and subsequently searched by a user.
Splunk Components
Components and Symbols
12
Icon Descrip,on
Deployment Server – A dedicated Splunk instance uAlized by the operaAon team to deploy new and update exisAng Splunk based apps.
Licensing Server – A dedicated or combined Splunk instance that hosts the licensing file(s) that search heads and Indexers connect to report their usage.
Master Cluster Node – A dedicated Splunk instance that manages a cluster, handles were searches are directed and how data is indexed onto the Indexers.
Job Server – A dedicated Splunk instance that runs scheduled searches only and outputs the results to summary indexes, alerts, and reports.
Splunk Components
Components and Symbols
13
Splunk FuncAonaliAes
Icon Descrip,on
Search – An ad-‐hoc search, usually started by a user through the Splunk UI or API, that is started on a search head and runs against the indexer(s).
Saved Search – A search, either scheduled or not scheduled, that is saved on a search head and runs against the indexer(s).
Alert – A saved and scheduled search that is on a search head and runs against the Indexer(s), which is setup to alert, via email, scripts, etc. when a criteria is met.
Dashboard – A visual display that shows the results from an inline or saved search through bar charts, line charts, radial gauges, single values, etc.
App and Add-‐On – A package that is installed onto a search head or indexer that provides addiAonal funcAonaliAes and features on top of the base install.
Components and Symbols
14
Splunk FuncAonaliAes
Icon Descrip,on
Indexes – A locaAon that stores the events sent from the forwarders to the indexers that are normally stored for short Ameframes.
Summary Indexes – A locaAon that stores summary events that are retrieved from the indexers that are normally stored for longer Ameframes.
Accelerated Reports – A data backed summary created off of an accelerated search that automaAcally creates a data stored alongside the buckets in the original index.
Components and Symbols
15
Splunk FuncAonaliAes
Icon Descrip,on
Log File Input – An input from a server, normal in the form of a log or text file, that indexes events into a Splunk instance
Network Input – An input from networking devices, normally through Syslog streaming, that indexes events into a Splunk instance
Scripted Input – An input that pulls informaAon from a source, including log files, databases, etc., that indexes events into a Splunk instance
Components and Symbols
16
User Symbols
Icon Descrip,on
Read-‐Only Users – A Splunk role that grants minimal access to run UI-‐based searches, save private searches, and create private dashboards
Power Users – A Splunk role that adds to the permissions of the read-‐only users role to include more concurrent searches and longer search Ameframes
Power Plus Users – A Splunk role that enables addiAonal like real-‐Ame and accelerated searches, public dashboards, etc. features to the other roles
Splunk Admin – A Splunk role that has full admin rights and capabiliAes within the Splunk instances
Splunk Support – Technically not a Splunk role, instead a group of people that work for Splunk and provide product support
Components and Symbols
17
LocaAons Symbols
Icon Descrip,on
Datacenter – A Splunk environment that is locally hosted within a company owned or leased locaAon
Cloud – A Splunk environment that is remotely hosted within a cloud-‐based locaAon like Amazon Web Services (AWS)
Office Building – A locaAon were employees work and interact with and support a Splunk environment
Network – ConnecAons between Splunk servers and forwarders, datacenters, the cloud, and users
StarAng Out Small – Single Server SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
StarAng Out Small – Single Server
19
! A single installaAon of Splunk, running either on a physical server or virtual machine ! Normally no customizaAon of user access roles, parsing sehngs, or other
configuraAons
! Due to simple need, normally running a default installaAon is all that is needed
SecAon Overview
StarAng Out Small – Single Server
20
Data Tier
Search and Indexing Tier
Environment Diagrams Splunk FuncAonality
StarAng Out Small – Single Server
21
Data Tier
Search and Indexing Tier
Environment Diagrams Splunk Roles
StarAng Out Small – Single Server
22
Drawbacks • Limited by the resources (cpu,
memory, storage, etc.) of a single physical server or virtual machine
• Cannot grow horizontal or verAcally without standing up a separate instance
• Everybody’s search and indexing load is on the same server
• No way to take parAal outages for maintenances or refreshes
Benefits • Simple to setup and manage
configuraAons • Ideal for proof of concepts and
demos of features and funcAonaliAes
• Ideal for local installs (laptop and desktop) to complete tesAng and troubleshooAng
• InstallaAon can be done on a physical server or virtual machine
StarAng Out Small – Single Server
23
Expedia’s Story: ! Single server instances were primarily used as a proof of concept in the early days
of using Splunk
! There are a couple single server instances of Splunk running within AWS for producAon and lab uses
! Single server instances are also used on our laptops to test configuraAons and troubleshooAng issues with logs formahng
Up Next: ! The first “ProducAon” instance of Splunk was setup as mulAple Aer environment.
This was due to the planning esAmates to index two to three terabytes a day
Expedia’s Story and Next Steps
SeparaAng FuncAons – MulAple Tiers SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
SeparaAng FuncAons – MulAple Tiers
25
! A Splunk installaAon is broken up into two parts, searching and indexing, that are broken up verAcally into two Aers
! Normally referred to has a three Aer environment, which is made up of the search and indexing Aers, with the third Aer being the data or forwarder Aer.
! The search Aer consists of Splunk search heads and the indexing Aer consists of Splunk indexers
! Each Aer consists of a single server with similar hardware specificaAons, except for more storage space in the server acAng as an Indexer
SecAon Overview
SeparaAng FuncAons – MulAple Tiers
26
Environment Diagrams
Data Tier
Indexing Tier
Search Tier
Splunk FuncAonality
SeparaAng FuncAons – MulAple Tiers
27
Environment Diagrams
Data Tier
Indexing Tier
Search Tier
Splunk Roles
SeparaAng FuncAons – MulAple Tiers
28
Drawbacks • SAll limited by the resources (cpu,
memory, storage, etc.) of a single physical server or virtual machine for each Aer
• Cannot grow horizontal without standing up a separate instance
• Everybody’s search and indexing load is on the same set of servers
• No way to take parAal outages for maintenances or refreshes
Benefits • SAll prely simple to setup and
manage configuraAons • Break up the funcAonaliAes that
could impact each other • Provides addiAonal security as
access to the indexer can be locked down to admins only
• InstallaAon can be done on a physical server or virtual machine
SeparaAng FuncAons – MulAple Tiers
29
Expedia’s Story: ! Several mulAple Aer instances, with single servers in each Aer, for the first year or
so of using Splunk ! There are a couple mulAple Aer instances, with single servers in each Aer, of Splunk
running within AWS for producAon and lab uses
Up Next: ! APer a handful of teams had successfully stood up several mulAple Aer instances,
they were combined together into a single “ProducAon” instance. ! Due to the combined search and index load, we also started expanding both Aers
horizontally
Expedia’s Story and Next Steps
Expanding Horizontal – MulAple Servers
SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
Expanding Horizontal – MulAple Servers
31
! The two Aers, search and indexing Aers, are now expanded horizontally with addiAonal installaAons of Splunk
! Each Aer can consists of the same or different number of servers depending on where the demand is coming
! If search load is heavier then index load, add addiAonal search heads into the search Aer
! If index load is heavier then search load, add addiAonal indexers into the indexing Aer
! Normally a good Ame to start customizing user access roles, parsing sehngs, or other configuraAons
! Normally a good Ame to start looking at deploying a separate license and deployment server(s) to handle the addiAonal Splunk servers and forwarders
SecAon Overview
Expanding Horizontal – MulAple Servers
32
Environment Diagrams
Data Tier
Indexing Tier
Search Tier
Splunk FuncAonality
Expanding Horizontal – MulAple Servers
33
Environment Diagrams Splunk FuncAonality
Data Tier
Indexing Tier
Search Tier Splunk Infrastructure Tier
Expanding Horizontal – MulAple Servers
34
Environment Diagrams
Data Tier
Indexing Tier
Search Tier
Splunk Roles
Expanding Horizontal – MulAple Servers
35
Drawbacks • InstallaAon recommended to be
done only on a physical servers • Setup and management becomes
more complex with mulAple servers in each Aer
• Everybody’s search and indexing load is sAll on the same set of servers
Benefits • SAll a prely simple to setup and
manage configuraAons • Expansion of one or both Aers to
beler support load • Considerably less limited by the
resources (cpu, memory, storage, etc.) as each Aer can be expanded as needed
• ParAal outage can be done for maintenances or refreshes
• Start taking advantage of Splunk’s other funcAonaliAes (deployment, license, heavy forwarder, etc.)
Expanding Horizontal – MulAple Servers
36
Expedia’s Story: ! Three major instances that are both mulAple Aers and servers including our
producAon, PPE/DR, and lab environments ! In addiAon to our main instances, we have three other small mulAple Aers and
servers instances hosted within our datacenters ! There is also a Splunk instances within AWS that is running with mulAple indexers
that are separate from a single search head
Up Next: ! With several team moving towards a mulA-‐datacenter setup for their services,
worked started to update our environment to allow cross-‐site availability and searching
Expedia’s Story and Next Steps
Expanding Datacenters – MulA-‐Datacenter Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
Expanding Datacenters – MulA-‐Datacenter Instances
38
! A mulA-‐Aer and server instance is setup within two or more separate datacenters that have customer facing service running in them
! It is recommended that the forwarders only send data to the indexing Aer within their datacenter
! Once the data is indexed, you can either use index and forward or the new mulAsite clustering feature to send across mulAple sites
! Setup the search heads within each datacenter to have both sets of indexers as search peers to allow mulAsite searching
! Remember to look at high availability of the license and deployment servers to handle the Splunk servers and forwarders within each environment
SecAon Overview
Expanding Datacenters – MulA-‐Datacenter Instances
39
Environment Diagrams Splunk FuncAonality
Data Tier
Indexing Tier
Search Tier
Expanding Datacenters – MulA-‐Datacenter Instances
40
Environment Diagrams Splunk Roles
Data Tier
Indexing Tier
Search Tier
Expanding Datacenters – MulA-‐Datacenter Instances
41
Drawbacks • Setup and management becomes
more complex with mulAple servers in each Aer and mulAple instances that are interconnected
• InstallaAon recommended to be done only on a physical servers
• Depending on setup, the WAN link between datacenters could become overloaded
• Everybody’s search and indexing load is on the same set of servers
Benefits • Adds the ability to expand one or
both Aers to beler support and index load
• Expansion can also be done within each datacenter as search and index load increases
• Minimal, if any, outages comes from maintenances or refreshes as the clustering ensures the data is always available
• High availability can be setup for all funcAons of Splunk
Expanding Datacenters – MulA-‐Datacenter Instances
42
Expedia’s Story: ! There are only a limited number of search heads that can search across our two
datacenters
! The primary purpose for mulA-‐datacenter searching is for operaAonally and security purposes and it is available to a select few users
! There is no cross-‐site indexing of data, instead all data from servers within each datacenter is stored on indexers in the same datacenter
! There are no mulAsite clustering or high availability setup within our environment for any piece of the Splunk environment
Expedia’s Story and Next Steps
Expanding Datacenters – MulA-‐Datacenter Instances
43
Up Next: ! ConAnued research of the opAons for allowing addiAonal users to use cross-‐site
searching and possibly opening it up to every user
! The new mulAsite clustering feature is being looked at as part of our post-‐upgrade project to Splunk 6.1
! With team’s moving into the cloud, instances are being created within each AWS VPC were we have producAon and lab servers
! Due to rapid growth of the environment, impact due to team’s search and index load is being seen, so a re-‐architecAng of the producAon instances is under way to setup pod-‐based instances
Expedia’s Story and Next Steps
Headed to the Cloud – AWS Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
Headed to the Cloud – AWS Instances
45
! A single or mulA-‐Aer instance that is setup with the cloud, i.e. AWS, that indexes data from forwarders installed on the other virtual servers
! It is recommended that the forwarders only send data into instance hosted within the same VPC to reduce networking cost between VPCs or datacenters
! Remember to setup Splunk license and deployment server(s), can be on a search head, single instance, or standalone, within each VPC unless you are able to open up access to your datacenters
! Remember to setup local Splunk accounts, for each role, within each VPC unless you are able to open up corporate authenAcaAon
SecAon Overview
Headed to the Cloud – AWS Instances
46
Environment Diagrams Splunk FuncAonality
Data Tier
Indexing Tier
Search Tier
Headed to the Cloud – AWS Instances
47
Environment Diagrams Splunk FuncAonality
Data Tier
Indexing Tier
Search Tier
Headed to the Cloud – AWS Instances
48
Drawbacks • If you follow the recommendaAon
to have instances within each VPC, you will have many environments that people have to search against
• Need separate license and deployment servers in each VPC, unless you open up connects to the your datacenter
Benefits • Index AWS logs without paying for
networking costs to get it back to your datacenter
• Single Aer and mulA-‐Aer with single or many servers can be created depending on search and index load
• Expansion can be done on each or both Aers within each VPC as needed
Headed to the Cloud – AWS Instances
49
Expedia’s Story: ! There are eight separate instances of Splunk running within several AWS VPCs
throughout the world ! The Splunk instances support both producAon and lab environments running within
the AWS VPCs
Up Next: ! Current plans include upgrading the Splunk instances to Splunk 6.1 and expanding
either or both Aers as needed ! Ongoing invesAgaAon into open up the centralized license and deployment servers
to communicate with the various AWS VPCs
Expedia’s Story and Next Steps
SeparaAng Teams – Pod Instances SecAon Overview Environment Diagrams Benefits and Drawbacks Expedia’s Story and Next Steps
SeparaAng Teams – Pod Instances
51
! Several mulA-‐Aer and server instances are setup within the same datacenter that allows team specific data to be indexed into separate pods
! It is recommended that the forwarders only send data to the indexing Aer for that team’s pod to reduce duplicate licensing usage
! Remember to look at high availability of the license and deployment servers to handle the Splunk servers and forwarders within each environment and pod
SecAon Overview
SeparaAng Teams – Pod Instances
52
Environment Diagrams Splunk FuncAonality
Data Tier
Indexing Tier
Search Tier
SeparaAng Teams – Pod Instances
53
Environment Diagrams Splunk Roles
Data Tier
Indexing Tier
Search Tier
SeparaAng Teams – Pod Instances
54
Drawbacks • Will most likely involve addiAonal
servers to support the Splunk infrastructure
• Setup and management becomes even more complex with pods in each datacenter and instances that are interconnected
• Complexity also comes if other teams need to search against a pod that is not for their team
Benefits • Expansion of one or both Aers to
beler support search or index load for each team
• Adds the ability to separate the possible impact of expensive search load or large indexing volumes
• Separate maintenances can be completed on each pod when it is best for the team
• Some pods can be setup with clustering or new mulAsite clustering feature
SeparaAng Teams – Pod Instances
55
Expedia’s Story: ! Working on building out five pod environments within our producAon datacenter ! The new environment will be made up of four team specific pods and an “other”
pod that will contain the rest of the teams ! All pod environments will be fully searchable for our Enterprise InformaAon
Security Team and Global OperaAons Center to monitor for security and service issues
! All users within Expedia will have limited searchability to each pod instances to allow upstream or downstream invesAgaAons
Expedia’s Story
SeparaAng Teams – Pod Instances
56
Up Next: ! The process being completed in our producAon datacenter will be repeat in our
PPE/DR datacenter next year so that the two datacenters mirror each other
! The “other” pod will be monitored for search and index capacity issues that would warrant creaAng another team specific pod
! Looking at ways to expand our usage of Splunk to provide addiAonal service offerings and possibly decommission old applicaAons and tools
! Working on conAnual improvements to the Splunk environment, funcAonaliAes, features, and users
Expedia’s Story
Up Next for Expedia’s Splunk Environment
58
! SecAon Overview ! Usage of Splunk and Community Apps ! MulAsite FuncAonaliAes Between Instances
! Improved Access Control ! AutomaAon of App Deployments and ConfiguraAons
! AutomaAon of Forwarder Deployments ! Centralized License, Deployment, and AuthenAcaAon Support for AWS
SecAon Topics
Up Next for Expedia’s Splunk Environment
59
! Splunk’s usage within Expedia was and sAll is to provide a centralized locaAon for log aggregaAon. It also allows users without access to the producAon servers to look at their service’s logs
! Its organic growth has caused the environment to grow from 3TB when it was first stood up four years ago to 4TB in September 2012 to 8TB in September 2013 to 16TB in September 2014
! It has recently become a stateless monitoring planorm of the logs indexed into the environment through dashboards, alerts, and reports
! With our recent and ongoing upgrades to the current version and improvements to the architecture, the following slides explain what we are currently doing or would like to do next
SecAon Overview
Up Next for Expedia’s Splunk Environment
60
Background: ! Usage of the Splunk and community created apps is limited to a couple that provide
simple funcAonality for both the users and admins ! Around 15 addiAonal applicaAons and tools are supported by our team and several
of them could be replaced with Splunk apps
Requirement: ! Provide Splunk apps that offer monitoring of our infrastructure, capacity
management for Windows, Linux, and VMware servers, and PCI and security compliance
! Apps need to offer direct replacements of the features currently being used or extend the funcAonality past what the current applicaAons and tools can provide
Usage of Splunk and Community Apps
Up Next for Expedia’s Splunk Environment
61
Solu,ons: ! Monitoring of Infrastructure
! Capacity Management
Usage of Splunk and Community Apps
AppDynamics Cisco Security Suite Extra Hop Monitoring of JVM Solr Monitoring Splunk App for MicrosoP Exchange
Splunk App for Puppet Splunk Deployment Monitor Splunk for Keynote Splunk for Nagios Splunk Support for AcAve Directory
Splunk App for Windows Infra. Splunk Add-‐on for MicrosoP Windows Splunk app for Unix and Linux
Splunk Add-‐on for Unix and Linux Splunk app for VMware
Up Next for Expedia’s Splunk Environment
62
Solu,ons: ! PCI and Security Compliance
! Extend FuncAonality
Usage of Splunk and Community Apps
Splunk App for AWS Splunk App for Enterprise Security Splunk App for PCI Compliance Splunk for Blue Coat ProxySG
Splunk for Juniper Firewalls Splunk for Juniper SA Splunk for Sourcefire Splunk for Squid
AutomaAon TesAng -‐ SyntheAc TransacAons System Center OperaAons Manager IntegraAon Sideview UAls Splunk App for HadoopOps Splunk DB Connect
Splunk for Asset Discovery Splunk for ServiceNow Splunk Hadoop Connect Splunk ODBC
Up Next for Expedia’s Splunk Environment
63
Background: ! With limited excepAon, there are no cross-‐site searching setup that allows a user to
search for their logs in each datacenter from a single locaAon ! There is no high availability or mulA-‐site indexing setup that crosses between our
three datacenters
Requirement: ! Provide the ability to search across each datacenter without impacAng the
e-‐commerce traffic that is on the same WAN link ! Provide mulA-‐site indexing that allows seamless failover between instances and
minimal impact during maintenances
MulAsite FuncAonaliAes Between Instances
Up Next for Expedia’s Splunk Environment
64
Solu,ons: ! InvesAgate into taking a slice of the WAN link between our three datacenters for cross-‐
site search and index traffic ! The slice would have a throlling limit so that e-‐commerce traffic would not be impact by
Splunk running wild, similar our Big Data setup ! Even with the throlling limit, Aght user permissions will be upheld to ensure concurrent
search limits and search result size do not cause other users impact
! With the pod setup, high availability and mulA-‐site clustering would be deployed on a team-‐by-‐team basis
! Setup automated failover and maintenance processes to enable minimal impact to the users
MulAsite FuncAonaliAes Between Instances
Up Next for Expedia’s Splunk Environment
65
Background: ! Only the default access roles (users, power, and admin) that are built into Splunk,
along with their default sehngs, are used in our environment ! The sehngs for searchable indexes by default is enabled for every index, which
causes issues with large search results if an index value is not included
Requirement: ! Ensure that all users have the features and funcAonaliAes they need to do their jobs
while safeguarding the Splunk environment ! Create custom access roles to allow the separaAon of Splunk’s features
and funcAonaliAes
Improved Access Control
Up Next for Expedia’s Splunk Environment
66
Solu,ons: ! Research into the need for addiAonal, non-‐default, access roles that limit or add
to the exisAng default access roles ! UpdaAng access roles that, for example, would remove real-‐Ame searching
from the power role and creaAng a new role that grants real-‐Ame searching to select users
! CreaAng new access roles that, for example, would increasing concurrent search limits, adding access to security specific indexes and apps, etc
! New and updated access roles would be specific to individual instances, instead of for the enAre environment
! Regular audits of the memberships of all access roles would be completed to ensure accurate access to the environment
Improved Access Control
Up Next for Expedia’s Splunk Environment
67
Background: ! Only a limited number of search head and indexer apps are checked into Perforce
and none of them use the deployment server ! The majority of the saved searches, dashboards, alerAng, etc. are setup on the
search heads without peer review or code checks before they are deployed
Requirement: ! Ensure that all search head and indexer apps are checked into Perforce to allow
proper version control and back-‐ups ! Provide fast and seamless deployments of search head and Indexer apps while
sAll safeguarding the Splunk environment with automated peer reviews and code checks
AutomaAon of App Deployments and ConfiguraAons
Up Next for Expedia’s Splunk Environment
68
Solu,ons: ! Re-‐enable the built-‐in deployment funcAonality on the search heads and indexers
which is currently disabled
! Setup a secAon in our Perforce depot to handle search head and indexer apps and configuraAons in a centralized locaAon and enable back-‐ups
! InvesAgate into an automated process to sync from Perforce to the deployment servers once an app is checked in
! InvesAgate into an automated peer review and code check process that ensure no common issues (i.e. inline searches in dashboards, no index/sourcetype/host values in searches, etc.) are deployed out
AutomaAon of App Deployments and ConfiguraAons
Up Next for Expedia’s Splunk Environment
69
Background: ! A semi-‐automated process to deploy monitoring apps is setup that pulls checked in
apps from Perforce and syncs them to the deployment servers ! The creaAon of all monitoring apps goes through a request process that is handled
and completed by my team
Requirement: ! Ensure that all monitoring apps are checked into Perforce to allow proper version
control and back-‐ups ! Provide fast and seamless deployments of monitoring apps while sAll safeguarding
the Splunk environment
AutomaAon of Forwarder Deployments
Up Next for Expedia’s Splunk Environment
70
Solu,ons: ! Setup a new secAon in our Perforce depot to handle monitoring apps and
configuraAons in a centralized locaAon and enable back-‐ups
! All exisAng monitoring apps are fully audited for proper setup and usefulness and the serverclass.conf file is rebuilt from scratch
! InvesAgate into an automated process to sync from Perforce to the deployment servers once an app is checked in
! InvesAgate into an automated peer review and code check process that ensure no common issues (i.e. backup files indexed, parsing issues, etc.) are deployed out
AutomaAon of Forwarder Deployments
Up Next for Expedia’s Splunk Environment
71
Background: ! The various AWS instances run their own local licenses and deployment servers and
do not Ae into our central license and deployment servers ! AuthenAcaAon for the producAon instances is local and does not Ae into our
Corporate AD/LDAP systems
Requirement: ! Securely allow connecAons from the various AWS instances to our central license
and deployment servers hosted in our datacenters ! Remove the need for local accounts and allow authenAcaAon to our corporate AD/
LDAP systems like our main instances
Centralized License, Deployment, and AuthenAcaAon Support for AWS
Up Next for Expedia’s Splunk Environment
72
Solu,ons: ! InvesAgate into the possibiliAes to open up the external firewalls to allow inbound
connecAons only from our AWS instances of Splunk
! Create external CNAMEs for each funcAonality so that all of the cloud-‐based Splunk instances can point to them and not need to be updated for changes
! Setup a secAon in our Perforce depot to handle search head, indexer, and forwarder apps for all of the AWS instances
! Delete all local accounts and non-‐standard access roles, then add in the normal security group to role mappings like our main instances
Centralized License, Deployment, and AuthenAcaAon Support for AWS