[d-1] spunk · 2019-12-21 · © 2019 splunk inc. [d-1] spunk , staff sales engineer, splunk...

[D-1] ��Spunk��

��, Staff Sales Engineer, Splunk Services Japan

2019/9/6

During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.

Forward-Looking Statements

THIS SLIDE IS REQUIRED FOR ALL 3 PARTY PRESENTATIONS.

Schema on The Fly

"��#� ��

!�� or ��

1. Splunk�$)#%

2. Splunk��'"

3. � ��+� (�� vs.�� )

4. �(!��

5. �*�� Splunk&��

��

► AWS► Splunk Version 7.3.1► Search Head x 1 �

• ��: C5.4xlarge (16core,32GB)• ��: 80GiB 400IOPS

► Indexer x 6�• ��: C5.4xlarge (16core,32GB)• ��: 200GiB 800IOPS

►��: auto (750MB)► limits.conf

• ��

��

►��• ��: 315,508,701�• ��: 2019/7� 1��• ��

• ��• mac_id:�� 443,774 (0.0002%)

• station_id:�� 203,824 (0.0005%)

• area_id:�� 220 (0.45%)

��

2019/07/31 23:00:00 mac_id=000:001:053:43:33:104 station_id = K102116 area_id=104 ,2.5ghz,nil,D,534333,,10.304,19,1562.632,1057.999,504.272,67.906,46.392,21.236,25.851,30.09,5.724,7373.742,285.24,7136.532,237.177,8.221,1.986,11.03,7.076,1.413,10.693,0.859,0.665

• ��: 32

��index=bigdata sourcetype=bigdata_traffic

[bigdata_traffic]BREAK_ONLY_BEFORE_DATE =DATETIME_CONFIG =KV_MODE = autoLINE_BREAKER = ([\r\n]+)NO_BINARY_CHECK = trueSHOULD_LINEMERGE = falseTIME_PREFIX = ^TIME_FORMAT = %Y/%m/%d %H:%M:%STZ = Asia/Tokyocategory = Customdescription = Search time field extractions for bigdatadisabled = falsepulldown_type = trueREPORT-00 = bigdata_traffic_csv_fields

��

��index=bigdata_idx sourcetype=bigdata_traffic_idx

[bigdata_traffic_idx]BREAK_ONLY_BEFORE_DATE =DATETIME_CONFIG =KV_MODE = noneLINE_BREAKER = ([\r\n]+)NO_BINARY_CHECK = trueSHOULD_LINEMERGE = falseTIME_PREFIX = ^TIME_FORMAT = %Y/%m/%d %H:%M:%STZ = Asia/Tokyocategory = Customdescription = Index time field extractions for bigdatadisabled = falsepulldown_type = trueTRANSFORMS-00 = bigdata_traffic_mac_idTRANSFORMS-01 = bigdata_traffic_station_idTRANSFORMS-02 = bigdata_traffic_area_idTRANSFORMS-03 = bigdata_traffic_others ��

Splunk�� Splunk��

�� $(�&'

► ��• %�� (Dense)

► �� • !.�� • )#�� • ��

► ��• "*��• /,��

► ��-+!.► ��

► �� =�! ��#��

► �� =��

��1 Indexer "��

��

index=bigdata

��

index=bigdata| streamstats count

��*

index=bigdata| stats count

��

��1 Indexer��

��

�� 305K��(172.4�)

24K��(2191.0�)

828K��(63.5�)

�� 20K��(2629.1�)

3K(16745.9�)

20K��(2620.5�)

�� >�� >��

q K��%�*�� CPUIP��]09('�X\�V�� 1 IndexerG�� 50K� 200K 1:,/Z�'!4:�<_@

q %�*��Ua �/��8-J�QD �$2:-) 0��IP�;^�

q 1',06"+�' 1• %�*�Ua �/��8-J�THFCY�`��• R�SE',7�3:#$2:-��Wf�/��8-��de�;�

• SE',7�3:#$2:-��%�*�?O� 1:,NJ�b�L��B=

• Splunk .�&5:�A��>�[�

SplunkMcIP��

Splunk� ��Splunk� ��

► ��$��"&� ��#�

► ��!�� %�

��

$ cd var/lib/splunk/$ ls -l

auditauthDbbigdatabigdata_idxconf2019conf2018defaultdbhistorydb

► db��"�� %*

► ($��!��colddb�)&

�S2�#��&'�+��

��!��

$ cd bigdata/$ ls -l

colddbdatamodel_summarydbthaweddb

► ��(��"��"��*��

► �"��('��#$�- ��&,%��"�

�� =��"��+)�� !��

$ cd db/$ ls –l

.bucketManifestCreationTimedb_1485388800_1483228800_0db_1498867200_1501545599_0hot_v1_0GlobalMetaData

Splunk ��

��"��!

��$: likes (vodka OR cognac)

(vodka OR cognac) = (4) or (2) = (2,4)

likes = (0,1,2,3,4,5,6)

(2,4) and (0,1,2,3,4,5,6) = (2,4)

2% � 4% ��

��&#��(��)

��)/��.%�#��+$

hash(1, ”Armit") == 0hash(2, ”Roger") == 7

▶ Lexicon Term��(-▶ ��*��&"▶ 0! Term�� ,'�� 1�� 0

1 1 0 0 1 0 1 0 0

mac_id� ��*

index=bigdatamac_id=000:001:004:30:17:001

station_id� ��*

index=bigdatastation_id=X209019

area_id� ��*

index=bigdataarea_id=88

��

��"��-!

mac_id station_id area_id

�� 12.2( 2.2( 36.2(

�� 1,464& 52,895& 134,029&

�� 418,835& 52,895& 9,789,654&

�� 1,464&�)�� 52,895 &)��,��%'&$�� &$�#+*��

q 1&-0@��/�+�Z��B��Xc(+�6)�� Z!�#5;0�HR � �_N/�+ (:$)';)��

q (�,=��:$)';/�+ Ua��^�J�#5;0 S?

q (�,�DM�KPLF��*$7;LF�Y] E��

q `T�+�6 O��:$)';�39�62"9+��b�J��Q[�CM��<\

q 5*048%."* 2• #;/-%*I�G?• =>�G?• #5;0 V?��+�6 O�

Splunk �/�+WA��

�� vs. ��

� ��

► 1,464 ��419,801��

►��

��index=bigdata mac_id=000:001:004:30:17:001

�� "'�2#

��mac_id=000:001:000:17:30:001mac_id=000:001:001:17:30:004mac_id=000:001:004:30:17:001mac_id=000:001:004:30:17:004

Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3

��):mac_id=000:001:004:30:17:001

Posting:1,2,3

�� 1,2,3 �� 0�$�KV_MODE=auto�� mac_id�1!%(�� 1 3 ��*& .��"��,/⇒+-��

� ��$�-�4��"�'+

Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3mac_id::000:001:000:17:30:001 0mac_id::000:001:001:17:30:004 1mac_id::000:001:004:30:17:001 2mac_id::000:001:004:30:17:004 3

��0:mac_id::000:001:004:30:17:001

6!��%� 2�� %��5 . ��$�8&37��#%�2/�

��%��(��$��mac_id=000:001:004:30:17:001��fileds.conf�,)�1*

mac_id��*

index=bigdata_idxmac_id::000:001:004:30:17:001

station_id��

index=bigdata_idxstation_id::X209019

area_id��

index=bigdata_idxarea_id::88

��

��)$#�!-&��(%9* ( �"-&��(%9*)

�� 2.22 (12.22) 2.22 (2.22) 8.22 (36.22)

�� 1,4640 52,8950 134,0290

station_id� �"-&��(%9*��17�!�')��4�*��17�!�')�+��)$#�!-&��(%�,�5��

�� 3 ��Splunk� �"�� 86��

( ).� �"-&��(%9*�5/

��

�� 2 ��

��

MAJOR = [ ] < > ( ) { } | ! ; , ' " * \n \r \s \t &

? + %21 %26 %2526 %3B %7C %20 %2B %3D --%2520 %5D %5B

%3A %0A %2C %28 %29

MINOR = / : = @ . -$ # % \\ _

��+�)�/-".�;5�$�,�=��

Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3mac_id=000:001:000:17:30:001 0mac_id=000:001:001:17:30:004 1mac_id=000:001:004:30:17:001 2mac_id=000:001:004:30:17:004 3

�0'& #6*��/(E1��A��-".�$�,�=��

-".�$�,�24�*��/(=?��<�� Splunk�B��:>C��8@��!�%��mac_id=000:001:004:30:17:001��9��+�)�$�,7D�3��

mac_id�� *

index=bigdataTERM(mac_id=000:001:004:30:17:001)

station_id��

index=bigdataTERM(station_id=X209019)

area_id��

index=bigdataTERM(area_id=88)

��TERM ��

��!-%��)$:+�'�(� �&��19

�� 2.27 (12.27) - (2.27) 6.17 (36.27)

�� 1,4643 03 134,0293

�� 1,4643 03 134,0293�� 2.27 2.27 8.27

TERM�5� mac_id��*#"��-%��)$:+�,�04�station_id��26 0�

mac_id�� *#"��-%��)$:+��8��

( ).� TERM �5��8/

station_id�� 0��

2019/07/29 23:00:00 mac_id=000:001:035:90:82:211 station_id = X209019 area_id=1 ,2.1ghz,15,R,359082,211,0.066,1,0.136,0.083,0,0.162,0.098,0,14.938,14.588,,72.938,5,72.938,5,0.067,0,0.5,,,13.591,,

station_id = X209019station_idX209019

��

area_id��

| dbinspect index=bigdata*| stats count as num_buckets sum(sizeOnDiskMB) as sizeOnDiskMB by index| eval sizeOnDiskGB = round(sizeOnDiskMB/1024, 2)| fields index num_buckets sizeOnDiskGB

��

bigdata 146 78.53GB

bigdata_idx 333 232.02GB

��

�� index=bigdata

�� index=bigdata_idx

��

�� index=bigdata| streamstats count

�� index=bigdata_idx| streamstats count

��

�� index=bigdata| stats count

�� index=bigdata_idx| stats count

�� vs. ��

��!1 Indexer��#��

� ��

��"� 305K�� 24K�� 828K��

�� "� 279K�� 24K�� 661K��

"�� $��!

��

�� index=bigdata| eval dl_gbytes = (dl_mbyte/1024)

�� index=bigdata_idx| eval dl_gbytes = (dl_mbyte/1024)

��

�� index=bigdata| streamstatsavg(dl_mbyte)

�� index=bigdata_idx| streamstatsavg(dl_mbyte)

��

�� index=bigdata| stats avg(dl_mbyte)

�� index=bigdata_idx| stats avg(dl_mbyte)

�� vs. ��

�� 1 Indexer��"� ��

��

��!� 134K�� 17K�� 378K��

��!� 153K�� 17K�� 560K��

��!�� #��

q 0%+/2")�% 3• .�4&��F�.��4,

• IM�%!15��5*("%=.��4,Q7�H@��

• IM�%!15 �:9�� TERM�>O��5*("%=.��4,Q7�H@��

• IM�%!15�D;� TERM�>O��:9��5*("%=.��4,Q7�GK

• JC�F�.��4,• �*�-3)��<�JC�F�.��4,��5*("%=.��4,Q7��$�'L?�EN��

• �5*("%=.��4,Q7�8��5*("%.��4�6��GK�#�%BA��$�'=.��4,Q7��P��3%"��

.��4,Q7$�'= vs. �5*("%=

��

��%�!�$�"

��%�

��%�+

�#� ��

��%�!�$�"

��%��%�

+�#� ��

��

) 2 1p c I s c I c xpo l D I x po l c c

l l s x W ) - lhk n Ip ) - Ha ( - x po l

n S n p c n c x po l x

po l p W x p nc x po l u i r i r c I p

x po l W i r i ru x Sa F s c D

Sd n x po llhk n I Ha x po l I p

Ii r 422 i rc erto x c

��

r u e aI r u e a

u e a e H S h 5

uacuac x I e

d ua nu uh I I uh a r nuuh S

d d l SH S I d e

Il u r H ku S Ia d H

c H SdI pu

��

r eTX pr s x 6 DS d a h u d a

s asd a ce

eps h ls I 6d a d e seTX pr s Dd a d h u

se H h

d Xt d a Dks

��x

t m tkr C )2 Msx t e S m t

s S 3v e

nr 1 I S e Co e l Ce S

1 d C S Me S7 4 d C S Me S

m tIcS em t CLm u

loadjob

1a H 1

b h a de211 2 2jo S b lr l 2

��

ar ID e f mI n Ho

n m m 1 1 1 /3 1d_ i bI n SH x

t 1 1 P 1 /3 1 6 / I . /3 1 /3 6 1 1s

o ma 1 1 I S cU lU aC ID) C hI S pID

d_ i bI xID

redistribute

dilk ) . R maK 74 3 x.1 ) . Rnu ) . K p Hma

x m mdilk ) . R R I

s u 7 Rh 4 2 71 .1 Rdilk ) . t 2 2 maRx

y m _Sb S.1 e kc R

r ma r

��

index=bigdata| stats avg(dl_mbyte) by area_id

� ��

index=bigdata_idx| stats avg(dl_mbyte) by area_id

| tstatsavg(bigdata_traffic.dl_mbyte) from datamodel=DM_bigdata_traffic_dl_mbyte by bigdata_traffic.area_id

�� vs. � �� vs. DMA

��)�� vs. �� vs. DMA

�� DMA

�� 159.4& 112.8& 64.7&

��%!��#"�� (��$'

q '�"&-� �� 4• �1!��6%��/#O2�� 6%��/#O2�5M9�&�,1�@�3��FB8CK��• 67�G<��43�5M9

− �).��1!��− 0(�"��-0��,1

• 67G<��JAI:�5M9− !��+!/��-0��,1

• .�/��*>�H=��5M9− loadjob

• .��ELD;��• redistribute• !��$�&-�1N?9

5M9�&�,1��

��Splunk��

statschart

xyseriesuntable

transponse

makemvmvcombinemvexpand

eval MV ��

rename A_* as B_*foreach

rexstreamstatseventstats

�� !��& %��

)$�#$� �"' �� (

��

(index=A A.val=*)| join � ��

[| search (index=B B.val=*) ]| stats avg(A.val) as avg_A_val,avg(B.val) as avg_B_valby � ��

��

(index=A A.val=*) OR(index=B B.val=*)| stats avg(A.val) as avg_A_val,avg(B.val) as avg_B_valby � ��

| search avg_A.val=* avg_B.val=*

�� vs. ��

��

�� 11.2� 5.4�

�� 49,172� 120,045�

Splunk ��

Treemap

Sankey Diagram

Punchcard Calendar Heat Map

Parallel Coordinates

Bullet GraphLocation Tracker

Horseshoe Meter

Machine Learning Charts

Timeline

Horizon Chart

Multiple use cases across IT, security, IoT, and business analytics

Box Plot

3D scatter plot

��

Wordcloud

Donut Chart

Heat Map

��

Custom Cluster Map Missile Map

q Splunk�,&�6%;*39#�bXZ0�-!L�O�?`��]W�)�.!=�_N��S\

q C��0�-�� -#7,-;4�i<AG�E��,&�6%;*39#��"�"L�O�

q �"��!��B��f��0�-49/13$�7�Splunk�

q M�U��c��"�49/15�7��]W�dR�0�-!>g ��

q Splunk�0�-YJ�_N�!JQ��@^I%4+8;�)�.(6;2!HaD��eP ��hF!120%VK

q SplunkT[4:'97https://www.splunk.com/ja_jp/training.html

,&�6%;*39#��

4 Days of Innovation 350 Education Sessions 20 Hours of Networking

“Hands down the most beneficial and attendee focused conference I have attended!”

– Michael Mills, Senior Consultant, Booz Allen Hamilton

�� conf.splunk.com

.conf19October 21-24, 2019

Splunk UniversityOctober19-21, 2019

Las Vegas, NVThe Venetian Sands Expo

October 21-24

Thank You.

Schema on The Fly is Always The Best Friend for Your Machine Data� �� J

[d-1] spunk · 2019-12-21 · © 2019 splunk inc. [d-1] spunk , staff sales engineer, splunk...

Documents

splunk for it operational intelligence · • splunk apps...

splunk conf2014 - splunk monitoring - new native tools for...

timeacertime*–** comparingtimerangesinsplunk* ·...

splunk developer & admin certification training...this...

splunkingyour mobile apps - .conf19 | splunk · introducing...

tenable and splunk integration · splunk...

monitoring docker containers with splunk - splunk .conf ·...

splunk .conf2011: splunk for fraud and forensics at intuit

© 2019 splunk inc. and elk let's chat about splunk

[poster] menswear collection: spunk at the 2014 edinburgh...

splunk for it operational...

islandsofsplunk* - .conf2017 | the 8th annual splunk...

agency chargeback models to enable enterprise splunk...

splunk in rakuten: splunk as a service for all

reversinglabs explainable threat intelligence enriches ......

rivium splunk windows · o splunk enterprise security * o...

referent / redner benjamin tiggemann · 2015-07-08 ·...

splunk conf2010: corporate express presents splunk with sap

spunk: a machine-readable transcription

fireeye splunk interfireeye + splunk: intermediate...