WebHDFS at King - May 2014 Hadoop MeetUp

Download WebHDFS at King - May 2014 Hadoop MeetUp

Post on 25-May-2015

450 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

The latest developments at King on their work with WebHDFS .

TRANSCRIPT

  • 1. 2 How to turbo charge your data transfers with WebHDFS Andy Done, Data Platform Lead andy.done@king.com

2. Last time 3. Since then 4. 100 40 Hadoop 5. 1 0.5 Storage 6. 15 10 Events 7. 10 4 ExaSol 8. 2.5 6 Load times 9. Problem WebHDFS 12 10. Old way WebHDFS 11. Old way hadoop fs cat /some/path/* | bulk_load my_table WebHDFS 12. WebHDFS way WebHDFS 13. WebHDFS way IMPORT INTO TABLE my_table FROM FILE http://namenode/webhdfs/v1/some/path/file_1 FILE http://namenode/webhdfs/v1/some/path/file_2 FILE http://namenode/webhdfs/v1/some/path/file_n WebHDFS 14. WebHDFS benefits Simple Efficient Ubiquitous Parallelisable Bidirectional Fast WebHDFS 15. 18 Conclusion WebHDFS 16. Thank you 19 17. We're hiring! 20