Business  Analy,cs  Paradigm  Change  

 

Dmitry  Anoshin  

Target  Market  Trends  

•  “Feeding  transac,onal  data  into  a  tradi,onal  data  warehouse  no  longer  represents  the  extent  of  capabili,es  necessary  for  BI.”  

 •  “The  simple  idea  of  building  a  tradi,onal  data  warehouse  to  

support  a  BI  plaEorm  is  no  longer  sufficient.”    •  “….require  new  informa,on  management  capabili,es  to  integrate  

informa,on  from  disparate,  external  and  unstructured  informa,on  sources.”  

Tradi,onal  Analy,cs  

Types:  •  Business  Intelligence  •  Data  mining  

•  OLAP  •  Plain  Analy,cs  

Uses:  •  Get  beNer  sense  of  their  opera,ons  •  Cut  costs  •  Improve  decision  making  

•  Iden,fy  inefficient  processes,  which  can  lead  to  iden,fy  new  business  opportuni,es  and  reengineering  their  processes  

Challenges:      •  Raw  informa,on  lives  are  usually  

decoupled  or  spread  across    distributed  systems  

•  Difficult  to  consolidate  •  Involves  an  effort  going  through  the  

typical  SDLC,  which  takes  lots  of  ,me  

Typical  Process  for  Structured  Data  

Applica,on  

Applica,on  

Applica,on   Connector  

Data  base  

ETL   Data  Warehouse  

Analy,cs  Tool  

Direct  Insert  

Early  Structure  Binding  •  Decide  what  ques,ons  to  ask  •  Design  the  data  schema  •  Normalize  the  data  

•  Write  database  inser,on  code  •  Create  the  queries  •  Feed  the  results  into  an  analy,cs  tool  

Business  Analy,cs  –Before  Splunk  IT/Business  Challenges  •  Most  organiza,ons  only  rely  on  structured  data  

for  business  analy,cs  –  not  sufficient  today!    •  New  data  sources  such  as  machine  increasingly  

cri,cal  sources  of  insight  –  not  leveraged  by  organiza,ons  

 •  Inability  to  scale  /  handle  data  volume  of  new  

sources  as  data  con,nues  to  grow  Inability  to  deliver  real-­‐,me  insights  to  the  business.    

 •  Most  today  rely  on  ETL  causing  latency  in  

analy,cs  Exis,ng  solu,ons  unable  to  do  data  mash-­‐up  across  structured  and  machine  data  

Business  Consequence  •  Inability  to  gain  real-­‐,me  business  insights  

from  new  data  sources    •  Business  users  across  func,ons  (sales  ops,  

product  managers,  marke,ng,  and  customer  support  users  cannot  leverage  new  data  sources  for  analy,cs  

 •  Compe,,ve  disadvantage  as  other  

companies  increasingly  leverage  machine  data  for  business  insights  

 •  Unable  to  get  insights  from  new  data  

sources  with  their  tradi,onal  structured  analy,cs  tools  

Business  Analy,cs  –  A]er  Splunk    IT/Business  Vision  

•  Deliver  real-­‐,me  business  insight  from  machine  data  

•  Enrich  machine  data  with  structured  data  to  provide  business  context  

•  Complement  exis,ng  BI  technologies  for  insight  into  a  new  class  of  data  

•  Leverage  search,  interac,ve  dashboards  in  Splunk  or  other  3rd  party  visualiza,on  tools  

•  Rapid  ,me  to  value  in  gaining  business  insights  from  machine  data  

Business  Benefits  •  Applica,on  Analy,cs  –  to  understand  how  customers  

are  interac,ng  with  various  online  applica,ons.  •  Content  &  Search  Analy,cs  –  to  understand  how  

customers  are  accessing  and  searching  for  content  served  up  over  CDNs    

•  Real-­‐,me  Sales  Analy,cs  –  to  gain  real-­‐,me  visibility  into  products  and  services  that  customers  are  purchasing.    

•  Service  Cost  Analy,cs  –  to  gain  insight  (for  example)  into  call  detail  records  and  cost  associated  with  comple,ng  each  call.    

•  Online  Mone,za,on  Analy,cs  –  an  example  of  this  is  online  gaming  companies  where  they  are  introducing  virtual  goods  and  charging  for  them.    

•  Marke,ng  Analy,cs  –  understanding  customer  click-­‐through  for  ads  helps  improve  placement,  pricing  and  click  through  rates.  

Splunk  Delivers  Value  Across  IT    and  the  Business  

Business  Analy,cs  

Digital  Intelligence  

Security  and  

Compliance  

IT  Opera,ons  

App  Manageme

nt  

Industrial  Data  

Developer  PlaEorm  (REST  API,  SDKs)  

>SPLUNK  Small  Data.  Big  Data.  Huge  Data.  

Splunk  Turns  Machine  Data  into    Opera,onal  Intelligence  

Customer    Facing  Data  

Outside  the  Datacenter  

ApplicaDons  "  Web  logs  "  Log4J,  JMS,  JMX  "  .NET  events  "  Code  and  scripts  

Networking  "  Configura,ons  "  syslog  "  SNMP  "  neElow  

Databases  "  Configura,ons  "  Audit/query  logs  

"  Tables  "  Schemas  

VirtualizaDon    &  Cloud  "  Hypervisor  "  Guest  OS,  Apps  "  Cloud  

Linux/Unix  "  Configura,ons  "  syslog  "  File  system  "  ps,  iostat,  top  

Windows  "  Registry  "  Event  logs  "  File  system  "  sysinternals  

Logfiles   Configs   Messages   Traps      Alerts  

Metrics   Scripts   Tickets  Changes  

"  Click-­‐stream  data  "  Shopping  cart  data  "  Online  transac,on  data  

"  Manufacturing,  logis,cs…  

"  CDRs  &  IPDRs  "  Power  consump,on  "  RFID  data  "  GPS  data  

Early  vs.  Late  Binding  Schema  Early  Structure  Binding  -­‐  Tradi,onal    

SELECT  customers.*  FROM  customers  WHERE    customers.customer_id  NOT  IN(SELECT  customer_id  FROM  Orders  WHERE  year(orders.order_date)  =  2004)  

Structure   Data  

•  Schema  –  created  at  design  ,me

   

•  Homogeneous–  must  fit  into  tables  or  be  converted  to  fit  into  tables  

•  Queries  –  understood  at  design  ,me  for  maximum  performance  

•  Must  exactly  match  constraints  

Early  vs.  Late  Binding  Schema  Late  Structure  Binding    -­‐  Splunk    

Structure   Data  

•  Schema-­‐less    

•  Heterogeneous–  can  come  from  any  textual  source

   

•  Created  at  search  ,me  

•  Constantly  changing  

•  Queries/searches  can  be  ad-­‐hoc  

•  No  conversion  required,  no  constraints  

Analy,cs  Early  Structure  Binding   Late  Binding  Schema  

Decide  the  ques,on(s)  you  want  to  ask  

Design  the  Schema  

Normalize  the  data  and  write  DB  inser,on  code  

Create  SQL  &  Feed  into  Analy,cs  Tool  

Write  data  (or  events)  to  log  files    

Collect  the  log  files    

Create  searches,  graphs,  and  reports  using  Splunk    

(Days,  Weeks  or  Months  &  Destruc,ve)  

(Minutes  &  Non-­‐Destruc,ve)  

Example:  Business  Visibility  From  Machine  Data  

Machine  Data  (from  customer  interacDon)   Product  InformaDon   Geo  locaDon  Data  

Customer  interacts  with  service  online  or  from  any  device  

Real-­‐Time  Business  Insights  from  Machine  Data  

66.57.19.112 ..[05/Dec/2011 07:05:22:152]”GET /card.do?

action=addtocart&itemid=EST-17& product_id=K9-

BD-01&JSESSIONID.SD7SLSFF8ADFF8HTTP 1.1” 200 3923

AppleWebKit/535.2 (KHTML.like Gecko) Chrome/15.0.874.121

Safari535.2

Product  Ac,on  User  

session  User  browser  informa,on  

Product_id=K9-BD-01 Product Name=2 TB Portable Drive

Manufacturer=iomega Geo location data

Correlated  with  product  informa,on  from  database  

Loca,on  data  based  on  where  the  customer  purchased  /  interacted  with  service  

–  What  products  are  popular  in  what  region?  –  Which  product  are  customers  leaving  in  cart?  

–  What  are  interac,on  paths  by  devices?  –  How  can  we  improve  customer  experience?  

Gepng  Structured  Data  In  Splunk  

CSV  lookup  

Splunk  Connector  

•  Access  data  at  scale  •  In  real-­‐,me  •  Easy  set-­‐up  &  maintenance  

Log  files  

Structured  databases  

Applica,ons  

Web  Servers  

Other  systems  

DB  Connect:  Business  Context  to  Machine  Data  

Rate  plans,  customer  profile,  geo  loca,on  

Customer  profile,  Service  subscrip,on  

Product  descrip,ons,  Customer  profile  

Device  ac,va,on,    Radius,  applica,on  logs  

Applica,on,  server  and  network  logs  

Applica,on  logs,    authen,ca,on  logs  

Structured  Data   >Machine  Data   >Business  AnalyDcs  

Sales  Analy,cs  

Customer  Analy,cs  

Product  Analy,cs  

Gepng  Business  Insights  from    Splunk  

User  Interface:  Splunk  

User  Interface:  Third  Party  

Dashboards   Searches   Pivot  

Schedule   SDK/APIs   ODBC  

Posi,oning  Splunk  for    Business  Analy,cs  

>New  class  of  data  for  business  analy,cs  

>Enrich  machine  data  with  structured  data  

>Real-­‐,me  business  insights  

>Complement  tradi,onal  BI  Tools  

Features   Splunk   Leading  BI  Tools  

Focus   PlaEorm  for  real-­‐,me  opera,onal  intelligence    

Data  visualiza,on  and  business  intelligence  so]ware    

Value   Collect,  index,  search,  monitor,  report  on,  analyze  massive  streams  of  machine  data  

Analyze,  visualize  and  share  structured  data    

Users   IT,  Opera,ons,  Security,  Developers,  Analysts,  Business  Users  (as  consumers)    

Business  Users  and  Analysts  (already  using  data  discovery  tool)    

Use  Cases   IT  Ops,  App  Management,  Security,  Digital  Intelligence,  Business  Analy,cs  from  machine  data,  Internet  of  Things    

Marke,ng,  HR,  Sales  Repor,ng,  Supply  Chain  Analysis  

Splunk  Complements  Exis,ng    BI  Tools  

Scales  to  TBs/day  and  Thousands  of    Users  

18  

"   Automa,c  load  balancing  linearly  scales  indexing  

"   Distributed  search  and  MapReduce  linearly  scales  search  and  repor,ng  

> Real  Time  Architecture  > Universal  Machine  Data  PlaWorm  > Schema  on  the  Fly  > Agile  ReporDng  and  AnalyDcs  > Scales  from  Desktop  to  Enterprise  > Fast  Time  to  Value  > Passionate  and  Vibrant  Community  

Summary