thoughtworks technology radar roadshow - melbourne
TRANSCRIPT
![Page 1: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/1.jpg)
TECHNOLOGYRADARMay 2015 — Our thoughts on the technology and trends that are shaping the future
1
![Page 2: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/2.jpg)
2
![Page 3: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/3.jpg)
3
TECHNOLOGY ADVISORY BOARD
![Page 4: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/4.jpg)
4
![Page 5: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/5.jpg)
5
![Page 6: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/6.jpg)
6
![Page 7: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/7.jpg)
THEMES FOR THIS ISSUE
7
![Page 8: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/8.jpg)
TECHNIQUES8
![Page 9: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/9.jpg)
TECHNIQUES8
![Page 10: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/10.jpg)
9
ADOPT 1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging
TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW
ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW
HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team
TECHNIQUES
![Page 11: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/11.jpg)
10
TECHNIQUESArchitectures for the digital world (theme: innovations in architecture)
![Page 12: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/12.jpg)
10
TECHNIQUES18
15
14
12
REACTIVE ARCHITECTURES
FLUX
ENTERPRISE DATA LAKE
APPEND-ONLY DATA STORE
Architectures for the digital world (theme: innovations in architecture)
![Page 13: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/13.jpg)
Attrition
Acquisition
Retention
Activation
Referral
Ads
ARCHITECTURES FOR THE DIGITAL JOURNEY
11
Emails
![Page 14: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/14.jpg)
Legacy Systems
RIGHT DATA, RIGHT PLACE, RIGHT TIME
12
Legacy SystemsLegacy Systems Web Analytics Operational Metrics Insights
![Page 15: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/15.jpg)
13
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events Event Queue
(Time Series Database, Apache Kafka, AWS Kinesis,
Eventstore, …)
![Page 16: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/16.jpg)
13
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
![Page 17: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/17.jpg)
14
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
![Page 18: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/18.jpg)
14
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
![Page 19: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/19.jpg)
15
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
![Page 20: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/20.jpg)
15
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
File Store (S3)Microservices
![Page 21: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/21.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
Microservices
Append-only
Database
Flux-based Web Application
User Actions
View Rendering
(react.js)
Transactions, Web Analytics,
Operational Logs
Subscribed Events
Business Logic
(functions)
Legacy Systems
Archivers
![Page 22: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/22.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
![Page 23: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/23.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
![Page 24: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/24.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
![Page 25: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/25.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
![Page 26: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/26.jpg)
16
Insights Analytics
Reports, Model
Parameters
Spark, Hadoop
File Store (S3)
Event Queue (Time Series Database,
Apache Kafka, AWS Kinesis,
Eventstore, …)PUSHES RESPONSIBILITY FOR DATA QUALITY BACK ON THE SOURCE SYSTEMS
ALL DATA IN MOTION IS IMMUTABLE
FIT-FOR-PURPOSE “STATE” IS COMPUTED BY THE CONSUMER
MANAGING AND PUBLISHING EVENTS BRINGS COMPLEXITY
![Page 27: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/27.jpg)
17
ADOPT 1. Consumer-driven contract testing NEW
2. Focus on mean time to recovery 3. Generated infrastructure diagrams NEW 4. Structured logging
TRIAL 5. Canary builds 6. Datensparsamkeit 7. Local storage sync 8. NoPSD 9. Offline-first web applications NEW 10. Products over projects NEW 11. Threat Modelling NEW
ASSESS 12. Append-only data store 13. Blockchain beyond Bitcoin 14. Enterprise Data Lake 15. Flux NEW 16. “git-based CMS” NEW 17. Phoenix environments NEW 18. Reactive architectures NEW
HOLD 19. Long lived branches with Gitflow 20. Microservice envy 21. Programming in your CI/CD tool 22. SAFe™ 23. Security sandwich 24. Separate DevOps team
TECHNIQUES
![Page 28: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/28.jpg)
TOOLS18
![Page 29: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/29.jpg)
TOOLS18
![Page 30: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/30.jpg)
19
TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman
TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul
55. Cursive 56. Gitlab 57. Hamms NEW
58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW
ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW
HOLD 76. Citrix for development
![Page 31: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/31.jpg)
20
TOOLS
![Page 32: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/32.jpg)
20
TOOLS
57 HAMMS
![Page 33: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/33.jpg)
WE BUILD DISTRIBUTED SYSTEMS BY DEFAULT
21
![Page 34: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/34.jpg)
22
![Page 35: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/35.jpg)
22
FAILURE IS THE NORM…but testing it is tough
![Page 36: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/36.jpg)
HAMMSSimulating badly behaved servers
https://github.com/kevinburke/hamms
![Page 37: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/37.jpg)
STARTING
24
![Page 38: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/38.jpg)
STARTING
24
![Page 39: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/39.jpg)
IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6
![Page 40: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/40.jpg)
IF AT FIRST YOU DON’T SUCCEED…
25
http://127.0.0.1:5512/?key=demo&tries=6
![Page 41: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/41.jpg)
NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25
![Page 42: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/42.jpg)
NOBODY IS 100% PERFECT
26
http://127.0.0.1:5513/?failrate=0.25
![Page 43: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/43.jpg)
OTHER FAILURE MODES
27
Invalid content lengths and types Non-http responses
Slow responses Partial responses
![Page 44: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/44.jpg)
WHEN WOULD YOU USE HAMMS?
28
Automated testing Usability testing
![Page 45: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/45.jpg)
BENEFIT OF COMMON PROTOCOLS
29
HTTP, TCP/IP
Well-known failure cases
Build on other people’s hard work!
![Page 46: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/46.jpg)
GOING BEYOND UNIT TESTING…
30
ZED ATTACK PROXY
CONSUMER-DRIVEN CONTRACT TESTING
REST-ASSURED
GORHAMMSTEST.GENERATIVE
QUICKCHECK
SIMULANT
![Page 48: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/48.jpg)
32
TOOLSADOPT 48. Composer 49. Go CD 50. Mountebank 51. Postman
TRIAL 52. Boot2docker 53. Brighter NEW 54. Consul
55. Cursive 56. Gitlab 57. HAMMS NEW
58. IndexedDB 59. POLLY NEW 60. Rest-assured NEW 61. Swagger 62. Xamarin 63. ZAP NEW
ASSESS 64. Apache Kafka NEW 65. Blackbox 66. Bokeh/Vega NEW 67. Gor NEW 68. NaCL NEW 69. Origami NEW 70. Packet beat 71. pdfmake NEW 72. PlantUML NEW 73. Prometheus NEW 74. Quick NEW 75. Security Monkey NEW
HOLD 76. Citrix for development
![Page 49: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/49.jpg)
LANGUAGES & FRAMEWORKS
33
![Page 50: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/50.jpg)
LANGUAGES & FRAMEWORKS
33
![Page 51: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/51.jpg)
34
LANGUAGES & FRAMEWORKS
ADOPT 77. Nancy
TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot
ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift
HOLD 92. JSF
![Page 52: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/52.jpg)
35
LANGUAGES & FRAMEWORKS
![Page 53: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/53.jpg)
35
LANGUAGES & FRAMEWORKS
85 SPRING BOOT
NANCY77
![Page 54: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/54.jpg)
A TALE OF TWO WEB FRAMEWORKS
36
Java/Spring C#/.NET
Lightweight ✓ ✓
Low-ceremony ✓ ✓
Self-hosted ✓ ✓
Opinionated ✓ ✓
boot
![Page 55: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/55.jpg)
ON THE SURFACE, VERY SIMILAR
37
![Page 56: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/56.jpg)
ON THE SURFACE, VERY SIMILAR
37
![Page 57: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/57.jpg)
BUT WHAT’S UNDER THE COVERS?
38
Spring Boot’s pom.xml
1847 lines in total!
![Page 58: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/58.jpg)
BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page
![Page 59: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/59.jpg)
BUT WHAT’S UNDER THE COVERS?
38
Nancy’s Nuget page
![Page 60: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/60.jpg)
FRAMEWORKS VS. COMPOSITION
39
Spring Framework
Your Spring Boot App
JettyYour App
Code
Owin
Nancy.Owin
Nancy
Composes
Calls higher-order functions
![Page 61: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/61.jpg)
40
LANGUAGES & FRAMEWORKS
ADOPT 77. Nancy
TRIAL 78. Dashing 79. Django Rest 80. Ionic Framework 81. Nashorn 82. Om 83. React.js 84. Retrofit 85. Spring Boot
ASSESS 86. Ember.js NEW 87. Flight.js 88. Haskell Hadoop library 89. Lotus 90. Reagent 91. Swift
HOLD 92. JSF
![Page 62: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/62.jpg)
PLATFORMS41
![Page 63: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/63.jpg)
PLATFORMS41
![Page 64: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/64.jpg)
42
PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication
HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW
ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
![Page 65: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/65.jpg)
43
PLATFORMS
![Page 66: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/66.jpg)
43
PLATFORMS26CLOUDERA IMPALA
![Page 67: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/67.jpg)
44
IMPALA Elegant, nimble querying of
big, lumbering data
![Page 68: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/68.jpg)
45
HADOOP Fast processing
Scalable Commodity hardware
Elegant failure handling HDFS
![Page 69: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/69.jpg)
MAPREDUCE
46
• Powerful … but complicated • Not always the right model
Input
Splitting Mapping
SortingReducing
![Page 70: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/70.jpg)
SQL
47
SELECT …
FROM …
WHERE …
GROUP BY …
Declarative
Simple
Well known
![Page 71: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/71.jpg)
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
![Page 72: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/72.jpg)
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
• Data needs to be optimised for consumption
ETL
?
?
![Page 73: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/73.jpg)
RELATIONAL DATABASES
48
Not always good choice for adhoc analysis:
• Data needs to be optimised for consumption
• Scaling is tough
?
?
??
ETL
?
?
![Page 74: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/74.jpg)
IMPALA
49
An SQL engine that runs on Hadoop
![Page 75: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/75.jpg)
IMPALA IN ACTION
50
External tables
create EXTERNAL TABLE books( id BIGINT, isbn STRING, category STRING, publish_date TIMESTAMP, publisher STRING, price FLOAT )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LOCATION '/data/books/';
![Page 76: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/76.jpg)
IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
Real-time querying of large datasets
![Page 77: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/77.jpg)
IMPALA IN ACTION
51
SELECT category,
count(*)
FROM books
GROUP BY category
+---------------------------+----------+| category | count(*) |+---------------------------+----------+| COMPUTERS | 312604 || SPORTS-RECREATION | 312766 || PETS | 313039 || CRAFTS-HOBBIES | 312438 || FAMILY-RELATIONSHIPS | 312435 || HOUSE-HOME | 313365 || EDUCATION | 313014 |
Runtime : 5 seconds(~15 million rows)
Real-time querying of large datasets
![Page 78: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/78.jpg)
IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity) as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON books.id = transactions.book_id
GROUP BY category;
![Page 79: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/79.jpg)
IMPALA IN ACTION
52
Arbitrary joins
SELECT category, sum(quantity) as total_quantity
FROM books
JOIN [SHUFFLE] transactions ON books.id = transactions.book_id
GROUP BY category;
+---------------------------+----------------+| category | total_quantity |+---------------------------+----------------+| STUDY-AIDS | 10171749 || SELF-HELP | 10238182 || TECHNOLOGY-ENGINEERING | 10207003 || SOCIAL-SCIENCE | 10177075 || HISTORY | 10238243 || LAW | 10229825 || PERFORMING-ARTS | 10231744 || POLITICAL-SCIENCE | 10193267 |
Runtime : 19 seconds
![Page 80: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/80.jpg)
WHEN WOULD YOU USE IMPALA?
53
Adhoc/exploratory data analysis
Existing Business Intelligence tools via
ODBC/JDBC
![Page 81: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/81.jpg)
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
![Page 82: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/82.jpg)
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI Tools
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
![Page 83: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/83.jpg)
DIVERSIFY CONSUMERS OF YOUR DATA
54
?
BI Tools
Data Scientists
Graph Fans
Map Reducers
?
?
?
?
Our Data
Curious Person
Financial Analyst
Customer Experience
Guru
Dashboard Whizz
![Page 84: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/84.jpg)
MADE POSSIBLE BY V2.0 DESIGN CHOICES
55
Diagram taken from: http://hortonworks.com/blog/apache-hadoop-2-is-ga/
![Page 85: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/85.jpg)
FURTHER RESOURCES
56
http://impala.io/
Tutorials from Cloudera: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/impala_tutorial.html
Impala and Amazon EMR: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html
Hadoop: http://hadoop.apache.org/
Yarn + hadoop v2: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
![Page 86: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/86.jpg)
57
PLATFORMSADOPT TRIAL 25. Apache Spark NEW 26. Cloudera Impala NEW 27. DigitalOcean 28. TOTP Two-Factor Authentication
HOLD 45. Application Servers NEW 46. OSGi 47. SPDY NEW
ASSESS 29. Apache Kylin NEW 30. Apache Mesos 31. CoreCLR and CoreFX NEW 32. CoreOS 33. Deis NEW 34. H2O NEW 35. Jackrabbit Oak 36. Linux security modules 37. MariaDB 38. Netflix OSS Full stack 39. OpenAM 40. SDN 41. Spark.io 42. Text it as a service / Rapidpro.io 43. Time-series Databases NEW 44. U2F
![Page 87: ThoughtWorks Technology Radar Roadshow - Melbourne](https://reader035.vdocuments.us/reader035/viewer/2022062304/55b39bacbb61eb863d8b4737/html5/thumbnails/87.jpg)
58
Scott Shaw
@scottwshaw
Jen Smith
@JenniferSmithCo
thoughtworks.com/radar