grid computing and trade analytics with elastic - jay chin's presentation at elasticontour 2015
TRANSCRIPT
![Page 1: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/1.jpg)
Jay Chin – [email protected]
Principal Consultant, Excelian
3 November 2015
0
Grid Computing and Trade Analytics
with Elastic
![Page 2: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/2.jpg)
Excelian Technical Consulting – Who we are
1
Financial Services specialists
Distributed computing specialists since 2006
Experts in niche and emerging technologies
![Page 3: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/3.jpg)
Financial Services – Insatiable appetite for Compute
• Algorithms (Computers) that actually do the trading - Remember the Flash Crash of May 6 2010? This is a result of HFT stopping and not trading causing the Market to drop 6% in mere minutes.
• Financial modelling - Use complex mathematical models to deal with asset prices, market movement, portfolio returns, etc.
• Huge amounts of data to process - e.g. connecting to one of the exchanges FIX, there will be up to 100,000 messages to process per second.
2
Source: Information Week, Wall Street & Technology Source: The Telegraph
![Page 4: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/4.jpg)
What do compute grids look like ?
3
Typical Numbers For A Standard Grid
- 40k cores/engines
- 30m tasks
- 120 GB of Log metrics
- 60 – 80% Average Utilisation
- Data retention up to 6 Months
https://flic.kr/p/ydnEvw
![Page 5: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/5.jpg)
Grid Maturity in Financial Services
4
HPC Maturity
Benchmark
2014
Tier I = Tier I banks
Tier II = Tier II banks
Point = point solutions used only for a specific use case (e.g. behind a software package, only for one business line…)
Mat
uri
ty L
evel
It is fairly common for bank to have grids. Larger banks tend to have at least 30,000 cores.
![Page 6: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/6.jpg)
Case Study: ELK for Enterprise Grid Reporting Framework
Requirements:
• Enterprise Grid with 40,000 Cores across 4 Data centers in 2 Countries
• Reporting Dashboard for Grid Metrics
• Scalable up to 100,000 cores and 200 million Grid tasks per day
5
Goal: Architect an Enterprise Grid and design a Grid metrics reporting framework
for a top-tier investment bank.
![Page 7: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/7.jpg)
The Case for ELK
6
Features ElasticSearchIntuitive Interface
Ease of Use
Security Integration
Scalability
Support
Pricing
Features
Integration with Grid Middleware
Elasticsearch met all the requirements except for the last one,which required some work on our part.
![Page 8: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/8.jpg)
Initial Architecture: Single cluster across 2 regions
7
curl -XPUT localhost:9200/GridA_metrics/_settings -d '{"index.routing.allocation.include.tag" : “region_A" }'
![Page 9: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/9.jpg)
Architecture After Consultation with Elastic Platinum Support
8
![Page 10: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/10.jpg)
Challenges
9
• Bespoke deployment due to security restrictions in Bank’s Datacentre
• https://github.com/Excelian/ansible_fs_elkstack
• Development of custom ETL to query Grid Metrics database and load them into ElasticSearch
https://flic.kr/p/eqJHbr
![Page 11: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/11.jpg)
More ELK Goodness
• Bank was very impressed with the reporting capabilities
• Support team at Elastic was also superior compared to some of the big vendors we were dealing with
10
AS A RESULT
1. We were tasked to do log centralization using Logstash
2. Explore Watcher for monitoring Grid and applications
![Page 12: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/12.jpg)
Feedback from Investment Bank
• For the first time ever, developers were able to view Grid metrics and correlate them with logging events from a single interface
• Application teams are experimenting with Elastic
• Developers rethinking logging
11
![Page 13: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/13.jpg)
Key Takeaways
• Lots of opportunities and interest in ElasticSearch in Financial services
• Single tool to do log analytics, alerting, events, searching, and metrics
• Elastic ticks all the right boxes for financial services: Security, scalability, support SLAs, etc.
• Elastic Platinum support has been fantastic
• Advanced Use Cases : Fraud Detection, Trade surveillance, Market Sentiment Analysis
12
![Page 14: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/14.jpg)
13
Thank you!
If you have any feedback, please get in touch:
If you would like to join our community of technologists at Excelian please have a look at our careers page for the latest vacancies:
www.excelian.com/careers/
@Excelian
@Excelian
@ExcelianLTD
![Page 15: Grid computing and trade analytics with elastic - Jay Chin's presentation at ElasticOnTour 2015](https://reader031.vdocuments.us/reader031/viewer/2022030315/587c8b511a28ab93568b730f/html5/thumbnails/15.jpg)
www.elastic.co
14