upgrading from-hdp-21-to-hdp-24
TRANSCRIPT
![Page 1: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/1.jpg)
UpgradingfromHDP2.1toHDP2.4
2016/03/24@wyukawa
HadoopSCR#hadoopreading
![Page 2: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/2.jpg)
About1yearago
hFp://www.slideshare.net/tagomoris/upgrading-from-hdp-21-to-hdp-22
![Page 3: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/3.jpg)
2014/12-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Inputdatafluentd-plugin-webhdfs
shib
Executequeriesoverhiveserver2onMR
hivebatchonMR
![Page 4: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/4.jpg)
HDP2.1.5.0
• InstalloverAnsible,withoutAmbari• Hadoop2.4.0• YARNResourceManagerHA• NameNodeHA• Hive0.13
![Page 5: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/5.jpg)
HDP2.3.4
• InstalloverAmbaributuseAnsible• Hadoop2.7.1– YARNResourceManagerRESTAPI(includekill)– shibuseyarncommandtokillonHDP2.1.5.0
• Hive1.2.1– backportHive-12084HivequerieswithORDERBYandlargeLIMITfailswithOutOfMemoryErrorJavaheapspace.
• Tez0.7.0
![Page 6: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/6.jpg)
changeHadoop/Hivesebngbyambari
• dfs.datanode.failed.volumes.tolerated=1• hive.server2.authendcadon=NOSASL• hive.server2.enable.doAs=false• hive.auto.convert.join=false• changememorysizebecauseofprestoandfluentd– increasehadoop/datanode/resourcemanager/hiveserver2heap
– decreaseyarn.nodemanager.resource.memory-mbandyarn.scheduler.maximum-allocadon-mb
![Page 7: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/7.jpg)
bluegreendeployment
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Inputdatafluentd-plugin-webhdfs
shib
Executequeriesoverhiveserver2onMR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.3.4)
hivebatchonTezhivebatchonMR
![Page 8: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/8.jpg)
OOMEoccuredatHiveonTez
• reportasHIVE-13238– increasehive.tez.container.size
• reportasHIVE-13239
SwitchtohivebatchonMR!IhavethereasontouseMRbecauseofshibstatusfeature...
![Page 9: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/9.jpg)
WhydoI useshibonMR?
• canexecuteHiveonTez– sethive.execudon.engine=tez
• can’tmonitorquerystatus• can’tkillquery• shibusemapreduce.job.nametoidendfysubmiFedquery
![Page 10: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/10.jpg)
OMMEoccuredatHiveonMR
• reportasHIVE-13273– hiveserver2memoryleak
![Page 11: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/11.jpg)
needtoupgradeHDP2.4
hFps://community.hortonworks.com/quesdons/20116/logfdscacheflushdmer-thread-increase.html
![Page 12: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/12.jpg)
HDP2.4.0
• almostsameasHDP2.3.4exceptforKala,Spark
• backportHIVE-12766!!!!!!!!!!!!!!!!!!!!!!!!!• AMBARI-14389
![Page 13: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/13.jpg)
upgradingfromHDP2.3.4toHDP2.4.0
• reportasAMBARI-15418
Iwanthortonworkstofixthisissue.But,anyway,switchtohivebatchonTez!
![Page 14: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/14.jpg)
hiveserver2isstable!!!
![Page 15: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/15.jpg)
2016/3-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Inputdatafluentd-plugin-webhdfs
shib
Executequeriesoverhiveserver2onMR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.4.0)
hivebatchonTezhivebatchonMR
![Page 16: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/16.jpg)
somedmescorrupt/missingblock...
monitoringwithgrafana,prometheus,hadoop_exporter
![Page 17: Upgrading from-hdp-21-to-hdp-24](https://reader031.vdocuments.us/reader031/viewer/2022020410/58ed3b731a28ab201c8b4593/html5/thumbnails/17.jpg)
Conclusion
• Don’tuseHDP2.3.4ifyouusehiveserver2• useHDP2.4.0• seeApachepatchinformadon• I wanttoseeHDPsourcerepository• somedmesneedtochangequery(HIVE-9390)• IwantHivecommiFer