benchmarking runs on cdh5.4 - detailed report

8/18/2019 Benchmarking Runs on CDH5.4 - Detailed Report

1/9

PerfKitBenchmarker Installation and Run (Got Errors highlighted below)

[ec2-user@CDH ~]$ cd Perfit!e"ch#r%er-&''&[ec2-user@CDH Perfit!e"ch#r%er-&''&]$ lltotl &2*-rw-rw-r-- & ec2-user ec2-user &+* Dec & &*,2* ./H01-rw-rw-r-- & ec2-user ec2-user 2332 Dec & &*,2* CH4GE'#d-rw-rw-r-- & ec2-user ec2-user 5677 Dec & &*,2* C04/18!./84G'#ddrw9rw9r-9 5 ec2-user ec2-user 7:+ Dec & &*,2* hoo%s-rw-rw-r-- & ec2-user ec2-user &&536 Dec & &*,2* ;8CE4E-rw-rw-r-- & ec2-user ec2-user &37 Dec & &*,2* ?-rw-rw-r-- & ec2-user ec2-user 5357 Dec & &*,2* 1ED'cfg-rw-rw-r-- & ec2-user ec2-user :6* Dec & &*,2* setu>'>?-rw-rw-r-- & ec2-user ec2-user +63 Dec & &*,2* test-reuire#e"ts't9tdrw9rw9r-9 * ec2-user ec2-user 7:+ Dec & &*,2* testsdrw9rw9r-9 7 ec2-user ec2-user 7:+ Dec & &*,2* tools-rw-rw-r-- & ec2-user ec2-user +3* Dec & &*,2* to9'i"i[ec2-user@CDH Perfit!e"ch#r%er-&''&]$ sudo >i> i"stll -r reuire#e"ts't9tDow"lodi"gu">c%i"g >?tho"-gflgsAA2' (fro# -r reuire#e"ts't9t (li"e &7)) Dow"lodi"g >?tho"-gflgs-2''tr'gB (+3%!), +3%! dow"loded 1u""i"g setu>'>? eggi"fo for >c%ge >?tho"-gflgsDow"lodi"gu">c%i"g i"2A2'6 (fro# -r reuire#e"ts't9t (li"e &3)) Dow"lodi"g Fi"2-2'*'tr'gB (536%!), 536%! dow"loded 1u""i"g setu>'>? eggi"fo for >c%ge i"2

wr"i"g, "o files fou"d #tchi"g ru"-tests'>? wr"i"g, "o files fou"d #tchi"g u"der director? custo#fi9ers wr"i"g, "o files fou"d #tchi"g u"der director? i"2testsuiteres wr"i"g, "o >reIiousl?-i"cluded files #tchi"g fou"d u"der director? docsbuild wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?c fou"d u"der director? i"2 wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?c fou"d u"der director? docs wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?o fou"d u"der director? i"2


2/9

wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?o fou"d u"der director? docs1euire#e"t lred? stisfied (use --u>grde to u>grde), distribute i" usrlib>?tho"2'+site->c%ges (fro# -rreuire#e"ts't9t (li"e &+))Dow"lodi"gu">c%i"g colorlog[wi"dows]AA2'+' (fro# -r reuire#e"ts't9t (li"e &6)) Dow"lodi"g colorlog-2'+''tr'gB 1u""i"g setu>'>? eggi"fo for >c%ge colorlog

8"stlli"g e9tr reuire#e"ts, wi"dowsDow"lodi"gu">c%i"g bli"%erA&'5 (fro# -r reuire#e"ts't9t (li"e &*)) Dow"lodi"g bli"%er-&'7'tr'gB (&&&%!), &&&%! dow"loded 1u""i"g setu>'>? eggi"fo for >c%ge bli"%erDow"lodi"gu">c%i"g futuresA5''5 (fro# -r reuire#e"ts't9t (li"e &:)) Dow"lodi"g futures-5''5'tr'gB 1u""i"g setu>'>? eggi"fo for >c%ge futuresDow"lodi"gu">c%i"g P?Jc%ge P?J>i"g e9t?#l'c C?tho" e9te"sio" (u>-to-dte)Dow"lodi"gu">c%i"g cs>iA''6 (fro# -r reuire#e"ts't9t (li"e 2&)) Dow"lodi"g cs>i-''6'tr'gB

1u""i"g setu>'>? eggi"fo for >c%ge cs>i1euire#e"t lred? stisfied (use --u>grde to u>grde), >r#i%o i" usrlib>?tho"2'+site->c%ges (fro# -rreuire#e"ts't9t (li"e 22))Dow"lodi"gu">c%i"g fe (fro# i"2A2'6--r reuire#e"ts't9t (li"e &3)) Dow"lodi"g fe-'25'tr'gB 1u""i"g setu>'>? eggi"fo for >c%ge fe1euire#e"t lred? stisfied (use --u>grde to u>grde), color# i" usrlib>?tho"2'+site->c%ges (fro#colorlog[wi"dows]AA2'+'--r reuire#e"ts't9t (li"e &6))1euire#e"t lred? stisfied (use --u>grde to u>grde), reuests i" usrlib>?tho"2'+site->c%gesreuests-&'2'5->?2'+'egg (fro# cs>iA''6--r reuire#e"ts't9t (li"e 2&))Dow"lodi"gu">c%i"g doco>t (fro# cs>iA''6--r reuire#e"ts't9t (li"e 2&)) Dow"lodi"g doco>t-'+'2'tr'gB 1u""i"g setu>'>? eggi"fo for >c%ge doco>t

1euire#e"t lred? stisfied (use --u>grde to u>grde), >?cr?>toA&': i" usrlib>?tho"2'+site->c%ges>?cr?>to-2'+'&->?2'+-li"u9-9*++7'egg (fro# >r#i%o--r reuire#e"ts't9t (li"e 22))8"stlli"g collected >c%ges, >?tho"-gflgsK i"2K colorlogK bli"%erK futuresK P?JiK feK doco>t 1u""i"g setu>'>? i"stll for >?tho"-gflgs 1u""i"g setu>'>? i"stll for i"2 wr"i"g, "o files fou"d #tchi"g ru"-tests'>? wr"i"g, "o files fou"d #tchi"g u"der director? custo#fi9ers


3/9

wr"i"g, "o files fou"d #tchi"g u"der director? i"2testsuiteres wr"i"g, "o >reIiousl?-i"cluded files #tchi"g fou"d u"der director? docsbuild wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?c fou"d u"der director? i"2 wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?c fou"d u"der director? docs wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?o fou"d u"der director? i"2 wr"i"g, "o >reIiousl?-i"cluded files #tchi"g '>?o fou"d u"der director? docs

1u""i"g setu>'>? i"stll for colorlog 1u""i"g setu>'>? i"stll for bli"%er 1u""i"g setu>'>? i"stll for futures =ou"d e9isti"g i"stlltio", P?J? i"stll for P?Jrotector-->r#Ass>-buffer-siBeA7 -#+7 -#tu"eAge"eric -DG4.0.1CE -fP8C -fwr>I -D4DE!.G -02 -g ->i>e -Lll -L>K-D=01/8=J0.1CEA2 -fe9ce>tio"s -fstc%->rotector -->r#Ass>-buffer-siBeA7 -#+7 -#tu"eAge"eric -DG4.0.1CE -fP8C-fwr>I -fP8C -8usri"clude>?tho"2'+ -c buildte#>'li"u9-9*++7-2'+chec%lib?#l'c -o buildte#>'li"u9-9*++7-2'+chec%lib?#l'o

buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,2,&*, error, ?#l'h, 4o such file or director? buildte#>'li"u9-9*++7-2'+chec%lib?#l'c, 8" fu"ctio" M#i"M, buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,3, error, M?#l>rsertM u"declred (first use i" this fu"ctio") buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,3, error, (Ech u"declred ide"tifier is re>orted o"l? o"ce buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,3, error, for ech fu"ctio" it >>ers i"') buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,3, error, e9>ected MNM before M>rserM buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,+, error, M?#le#ittertM u"declred (first use i" this fu"ctio") buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,+, error, e9>ected MNM before Me#itterM buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,*, wr"i"g, i#>licit declrtio" of fu"ctio"M?#l>rseri"itiliBeM buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,*, error, M>rserM u"declred (first use i" this fu"ctio") buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,:, wr"i"g, i#>licit declrtio" of fu"ctio" M?#l>rserdeleteM buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,&&, wr"i"g, i#>licit declrtio" of fu"ctio"

M?#le#itteri"itiliBeM buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,&&, error, Me#itterM u"declred (first use i" this fu"ctio") buildte#>'li"u9-9*++7-2'+chec%lib?#l'c,&2, wr"i"g, i#>licit declrtio" of fu"ctio" M?#le#itterdeleteM

lib?#l is "ot fou"d or co#>iler error, forci"g --without-lib?#l (if lib?#l is i"stlled correctl?K ?ou #? "eed to s>ecif? the o>tio" --i"clude-dirs or u"co##e"t "d


4/9

#odif? the >r#eter i"cludedirs i" setu>'cfg) s%i>>i"g e9t?#l'c C?tho" e9te"sio" (u>-to-dte) 1u""i"g setu>'>? i"stll for cs>i 1u""i"g setu>'>? i"stll for fe buildi"g #r%u>sfe's>eedu>s e9te"sio" gcc ->thred -f"o-strict-lisi"g -02 -g ->i>e -Lll -L>K-D=01/8=J0.1CEA2 -fe9ce>tio"s -fstc%->rotector

-->r#Ass>-buffer-siBeA7 -#+7 -#tu"eAge"eric -DG4.0.1CE -fP8C -fwr>I -D4DE!.G -02 -g ->i>e -Lll -L>K-D=01/8=J0.1CEA2 -fe9ce>tio"s -fstc%->rotector -->r#Ass>-buffer-siBeA7 -#+7 -#tu"eAge"eric -DG4.0.1CE -fP8C-fwr>I -fP8C -8usri"clude>?tho"2'+ -c #r%u>sfes>eedu>s'c -o buildte#>'li"u9-9*++7-2'+#r%u>sfes>eedu>s'o gcc ->thred -shred buildte#>'li"u9-9*++7-2'+#r%u>sfes>eedu>s'o -;usrlib+7 -l>?tho"2'+ -obuildlib'li"u9-9*++7-2'+#r%u>sfes>eedu>s'so 1u""i"g setu>'>? i"stll for doco>tuccessfull? i"stlled >?tho"-gflgs i"2 colorlog bli"%er futures P?Ji fe doco>tCle"i"g u>'''

[ec2-user@CDH Perfit!e"ch#r%er-&''&]$ '>%b'>? --cloudAL --be"ch#r%sAi>erf -#chi"et?>eAt2'#ediu#/rcebc% (#ost rece"t cll lst), =ile O'>%b'>?OK li"e &:K i" #odule

fro# >erf%itbe"ch#r%er'>%b i#>ort erf%itbe"ch#r%er>%b'>?OK li"e +K i" #odule fro# >erf%itbe"ch#r%er i#>ort rchiIe =ile Oho#eec2-userPerfit!e"ch#r%er-&''&>erf%itbe"ch#r%errchiIe'>?OK li"e 22K i" #odule fro# >erf%itbe"ch#r%er'>roIiders'ws'util i#>ort LP/H =ile Oho#eec2-userPerfit!e"ch#r%er-&''&>erf%itbe"ch#r%er>roIidersi"it'>?OK li"e &*K i" #odule fro# >erf%itbe"ch#r%er i#>ort i#>ortutil =ile Oho#eec2-userPerfit!e"ch#r%er-&''&>erf%itbe"ch#r%eri#>ortutil'>?OK li"e &6K i" #odule i#>ort i#>ortlib8#>ortError, 4o #odule "#ed i#>ortlib[ec2-user@CDH Perfit!e"ch#r%er-&''&]$ '>%b'>? --cloudAL --be"ch#r%sAi>erf -#chi"et?>eAt2'#ediu#/rcebc% (#ost rece"t cll lst), =ile O'>%b'>?OK li"e &:K i" #odule

fro# >erf%itbe"ch#r%er'>%b i#>ort erf%itbe"ch#r%er>%b'>?OK li"e +&K i" #odule fro# >erf%itbe"ch#r%er i#>ort li"u9be"ch#r%s =ile Oho#eec2-userPerfit!e"ch#r%er-&''&>erf%itbe"ch#r%erli"u9be"ch#r%si"it'>?OK li"e 2: for #odule i" !E4CH


5/9

[ec2-user@CDH Perfit!e"ch#r%er-&''&]


6/9

Hi-Bench Benchmarking Runs on Hadoop [ec2-user@CDH ~]$ cd hibe"chHi!e"chsrc[ec2-user@CDH src]$ >wdho#eec2-userhibe"chHi!e"chsrc

[ec2-user@CDH src]$ #I" cle" >c%ge -D s>r%&'3 -D che#Ie"#Ie"-error-dig"ostics2'#Ie"-error-dig"ostics-2''>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"-error-dig"ostics2'#Ie"-error-dig"ostics-2''>o# (*&2 ! t 26'5 !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorg>che#Ie"#Ie"->lugi"-registr?2'#Ie"->lugi"-registr?-2''>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"->lugi"-registr?2'#Ie"->lugi"-registr?-2''>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"->lugi"-registr?2'#Ie"->lugi"-registr?-

2''>o# (2 ! t 7&'* !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorg>che#Ie"#Ie"->lugi"-descri>tor2'#Ie"->lugi"-descri>tor-2''>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"->lugi"-descri>tor2'#Ie"->lugi"-descri>tor-2''>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"->lugi"-descri>tor2'#Ie"->lugi"-descri>tor-2''>o# (2 ! t 76'+ !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorg>che#Ie"#Ie"-#o"itor2'#Ie"-#o"itor-2''>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"-#o"itor2'#Ie"-#o"itor-2''>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"#Ie"-#o"itor2'#Ie"-#o"itor-2''>o# (7 ! t&5'3 !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorgcodehus>le9us>le9us-utils&'7'2>le9us-

utils-&'7'2'>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-utils&'7'2>le9us-utils-&'7'2'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-utils&'7'2>le9us-utils-&'7'2'>o# (2 !t +5'* !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorgcodehus>le9us>le9us-rchiIer&'-l>h-6>le9us-rchiIer-&'-l>h-6'>o#


7/9

Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-rchiIer&'-l>h-6>le9us-rchiIer-&'-l>h-6'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-rchiIer&'-l>h-6>le9us-rchiIer-&'-l>h-6'>o# (2 ! t 57' !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorgcodehus>le9us>le9us-co#>o"e"ts&'&'+>le9us-co#>o"e"ts-&'&'+'>o#

Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-co#>o"e"ts&'&'+>le9us-co#>o"e"ts-&'&'+'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us-co#>o"e"ts&'&'+>le9us-co#>o"e"ts-&'&'+'>o# (2 ! t +&'3 !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorgcodehus>le9us>le9us&''*>le9us-&''*'>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us&''*>le9us-&''*'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2orgcodehus>le9us>le9us&''*>le9us-&''*'>o# (* ! t 253'7!sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorg>che#Ie"re>orti"g#Ie"-re>orti"g-i#>l2''7#Ie"-re>orti"g-i#>l-2''7'>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"re>orti"g#Ie"-re>orti"g-i#>l2''7#Ie"-re>orti"g-i#>l-2''7'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"re>orti"g#Ie"-re>orti"g-i#>l2''7#Ie"-

re>orti"g-i#>l-2''7'>o# (2 ! t 3'& !sec)Dow"lodi"g, htt>s,oss'so"t?>e'orgco"te"tgrou>sscl-toolsorg>che#Ie"re>orti"g#Ie"-re>orti"g2''7#Ie"-re>orti"g-2''7'>o#Dow"lodi"g, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"re>orti"g#Ie"-re>orti"g2''7#Ie"-re>orti"g-2''7'>o#Dow"loded, htt>,re>o'#Ie"'>che'org#Ie"2org>che#Ie"re>orti"g#Ie"-re>orti"g2''7#Ie"-re>orti"g-2''7'>o# (6 ! t 25'+ !sec)'''

[84=0] ------------------------------------------------------------------------[84=0] 1ector u##r?,[84=0][84=0] hibe"ch ''''''''''''''''''''''''''''''''''''''''''' .CCE ['2*6s]

[84=0] "utchi"de9i"g ''''''''''''''''''''''''''''''''''''' .CCE [&,+':&6s][84=0] #hout '''''''''''''''''''''''''''''''''''''''''''' .CCE [5:'63*s][84=0] hiIebe"ch ''''''''''''''''''''''''''''''''''''''''' .CCE [&*'+62s][84=0] >r%be"ch '''''''''''''''''''''''''''''''''''''''' .CCE [2,'5*5s][84=0] PEG., Pet-cle Gr>h


8/9

[84=0] tre#i"gbe"ch dt ge"ertio" tools '''''''''''''' .CCE [+'37s][84=0] tre#i"g !e"ch#r% Soo%ee>er Hel>er .tils '''''''' .CCE [2'::+s][84=0] tre#i"g !e"ch#r% >r% ''''''''''''''''''''''''' .CCE [&,23'&:*s][84=0] tre#i"g !e"ch#r% tor# ''''''''''''''''''''''''' =8;.1E [73'&s][84=0] tre#i"g !e"ch#r% #B ''''''''''''''''''''''''' 8PPED[84=0] ------------------------------------------------------------------------

[84=0] !.8;D =8;.1E[84=0] ------------------------------------------------------------------------[84=0] /otl ti#e, +,35'*6&s[84=0] =i"ished t, =ri Dec && &,5*,26 ./C 2&3[84=0] =i"l ut for #ore i"for#tio"' - [Hel> &][E1101][E1101] /o see the full stc% trce of the errorsK re-ru" che'orgco"flue"cedis>l?roble#sK ?ou c" resu#e the build with the co##"d[E1101] #I" gols -rf ,stre#i"g-be"ch-stor#

Since only Storm Build failed, proceeding withwordcount

[ec2-user@CDH >re>re]$ >wdho#eec2-userhibe"chHi!e"chwor%lodswordcou"t>re>re

[ec2-user@CDH >re>re]$ '>re>re'sh>tchi"g rgsAPrsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f-defult->ro>erties'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f&-defult-stre#i"gbe"ch'co"f


9/9

Prsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f&-dt-scle->rofile'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f2-s#B-co##o"'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f5-s#B-wor%lods'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chco"f::-userdefi"ed>ro>erties'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chwor%lodswordcou"tco"f-wordcou"t-defult'co"fPrsi"g co"f, ho#eec2-userhibe"chHi!e"chwor%lodswordcou"tco"f&-wordcou"t-userdefi"e'co"f

/rcebc% (#ost rece"t cll lst), =ile Oho#eec2-userhibe"chHi!e"chbi"fu"ctio"slod-co"fig'>?OK li"e 33+K i" #odule lodco"fig(co"frootK wor%lodrootK wor%lodfolderK >tchi"gco"fig) =ile Oho#eec2-userhibe"chHi!e"chbi"fu"ctio"slod-co"fig'>?OK li"e &+&K i" lodco"fig ge"erteo>tio"lIlue() =ile Oho#eec2-userhibe"chHi!e"chbi"fu"ctio"slod-co"fig'>?OK li"e 2:6K i" ge"erteo>tio"lIlue ssert hdoo>Iersio"K OE1101K e9ecute Vs with "o retur"K >lese co"fir# hdoo> e"Iiro"#e"t is co"figured>ro>erl?'O V c#dssertio"Error, E1101K e9ecute usrbi"hdoo>bi"hdoo> Iersio" W hed -& W cut -d X -f 2 with "o retur"K>lese co"fir# hdoo> e"Iiro"#e"t is co"figured >ro>erl?'ho#eec2-userhibe"chHi!e"chbi"fu"ctio"swor%lod-fu"ctio"s'sh, li"e 5:, ', file"#e rgu#e"t reuired', usge, ' file"#e [rgu#e"ts]strt Hdoo>Pre>reLordcou"t be"ch

'>re>re'sh, li"e 23, 84P./HD=, u"bou"d Irible

benchmarking runs on cdh5.4 - detailed report

Documents