matlab in business critical applications...2 outline problem statement the big picture / financial...
TRANSCRIPT
![Page 1: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/1.jpg)
1© 2014 The MathWorks, Inc.
MATLAB in Business Critical Applications
Arvind HosagraharaPrincipal Technical [email protected]
![Page 2: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/2.jpg)
2
Outline
Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations
– TIBCO® Spotfire®
– IBM® PureData™ for Analytics (powered by Netezza)– Microsoft® SQL Server®
– Handling Big Data and out-of-memory problems (Hadoop, NoSQL databases)
MATLAB in Business Critical Applications Conclusions
![Page 3: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/3.jpg)
3
Code
Build / Test
Release
Deploy
Operate/ Monitor
Plan / Design
Problem Statement
Agility: Is it possible to accelerate the development of data analytics?
Democratization: Is it possible to make the results and insights from these analytics available to all stakeholders in an organization?
Production: Is it possible to build data analytics algorithms in a flexible, scalable manner that is suitable for production usage and rigor?
![Page 4: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/4.jpg)
4
Taking MATLAB into production
Performance Scalability Reliability Governance Auditability Maintenance Recoverability Support and
Services
Agility Lightweight
processes Visualization Access to data Expressive high-
level language Integration with
best-in-class tooling
Prototyping Test Production
Verification and validation– Correctness– Performance
Automation of testing
Test-driven development
![Page 5: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/5.jpg)
5
The Big Picture
Reference architectures offer solutions to finance industry applications with an emphasis on selection of the right tool for the task leading to efficient workflows.
Cloud Solutions
(eg: Amazon, Azure, PaaS,
IaaS)Analytics Platforms (eg: TIBCO Spotfire®)
Enterprise Messaging
Queues(eg: AMQP)
Visualization(eg:
HighChartsJS)
In-house Enterprise Software
Big Data Frameworks(eg: Hadoop, Databases)
![Page 6: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/6.jpg)
6
Finance Industry Applications
Energy Trading – Load Forecasting– Model and Storage
Assets– Price Energy Options
Risk Modeling
Portfolio Optimization
…
“The tools we developed with MATLAB are more reliable, scalable, and maintainablethan our spreadsheet-based approach. We know the tools will work, we can add new capabilities, and we can update the production system without getting IT involved.” - Manuel Arancibia, Horizon Wind Energy
“MathWorks Consultants were well-qualified, professional, and fast. They understood not only the technical issues but also the business goals, which is essential when working on a core business system. We got more than we expected from MathWorks Consulting.” - Dr. Norbert Tönder, RWE
![Page 7: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/7.jpg)
7
Reference Architecture
MATLAB Analytics with TIBCO Spotfire
![Page 8: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/8.jpg)
8
MATLAB Analytics with TIBCO® Spotfire™
![Page 9: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/9.jpg)
9
DevelopDevelop
TestTestRefineRefine
MATLAB
Rapidly create analytics for use in desktop, web, and mobile visualizations and dashboards– Run MATLAB scripts or programs within MATLAB sessions (small scale)
– Access MATLAB analytics on MATLAB Production Server (large scale)
TIBCOSpotfire
MATLAB Production Server
MATLABAnalyticsMATLABAnalytics
DevelopDevelop
TestTestRefineRefine
MATLAB
MATLAB Analytics with TIBCO Spotfire
![Page 10: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/10.jpg)
10
MathWorks. MPSExtension
MathWorks. MPSExtension
Reference Diagram
Mobile Web Desktop
TIBCOSpotfire
Web Player
TIBCOSpotfireServer
MATLAB Production Server
MATLABAnalyticsMATLABAnalytics
HTTP(s)
![Page 11: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/11.jpg)
11
MATLAB Analytics Library
MATLAB analytics available for use in Spotfire clients
LargeTableOutSmallTableOutOptionsPriceOptionsPriceShort
![Page 12: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/12.jpg)
12
TIBCOSpotfireServer
MATLAB Production ServerTIBCOSpotfire
Web Player
TIBCOSpotfire
Web Player
MathWorks. MPSExtension
Increasing Capacity and Redundancy
Mobile Web Desktop
MathWorks. MPSExtension
TIBCOSpotfire
Web Player
TIBCOSpotfireServer
MATLAB Production Server
MATLABAnalyticsMATLABAnalytics
Load BalancerLoad B
alancer
![Page 13: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/13.jpg)
13
Benefits
Develop
Develop
TestTestRefine
Refine
MATLABMATLAB Production Server
MATLABAnalyticsMATLABAnalytics
• Rapid development and deployment of MATLAB analytics
MATLAB
• Central management of locked down analytics• Scalable and reliable architecture
MATLAB Compiler SDKMATLAB Production Server
• MATLAB Analytics available to all business users TIBCO Spotfire Server
![Page 14: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/14.jpg)
14
Reference Architecture
MATLAB Analytics with IBM® PureData™ / Netezza
![Page 15: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/15.jpg)
15
Off-Database
MATLAB Analytics with IBM PureData
IBM PureData(Netezza)
ODBC JDBC
Database Toolbox
On-Database
Low Performance “Processes” (UDAP or AE)
Python Java MATLAB……
Pure Data ODBC/JDBC Driver High-Performance “Function”(UDF / UDTF / UDA)
MDCS MPS MCR
MATLAB Compiler
Compute Cluster
C++ Lua
MATLAB Coder
Wor
kflo
w
IBM PureData(Netezza)
![Page 16: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/16.jpg)
16
MATLAB Analytics with IBM PureData
![Page 17: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/17.jpg)
17
Building and Running the MATLAB Analytics
Generate Code and move to the IBM PureData Appliance
Compile the generated code into a shared object for the host and the SPU Register the shared object as a library Use the MATLAB analytic in the User-Defined Function / Table Function /
Aggregate [ SELECT Histogram(*) ]
1
2
3
4
![Page 18: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/18.jpg)
18
Benefits
• Rapid development and deployment of MATLAB analytics
MATLAB
• High Performance Analytics written in MATLAB and generated as C++
MATLAB Coder and/or Simulink Coder with Embedded Coder
• MATLAB Analytics available to downstream users as a Stored Procedure/Function
IBM PureData for Analytics powered by NetezzaorMicrosoft Netezza
![Page 19: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/19.jpg)
19
Reference Architecture
MATLAB Analytics with Microsoft SQL Server
![Page 20: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/20.jpg)
20
Off-Database
MATLAB Analytics with Microsoft SQL Server
Microsoft SQL Server
ODBC JDBC
Database Toolbox
On-Database
“Processes”
C# Common Language Runtime(CLR)
ODBC/JDBC Driver
.NET Assembly MPS Client Library
MATLAB Compiler SDK
“Transactions”
MATLAB Component Runtime
Develop
Develop
TestTestRefine
Refine
MATLABMATLAB Production Server
MATLABAnalyticsMATLABAnalytics
![Page 21: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/21.jpg)
21
MATLAB Analytics with Microsoft SQL Server
Develop
Develop
TestTestRefine
Refine
MATLABMATLAB Production Server
MATLABAnalyticsMATLABAnalytics
1
2
3
![Page 22: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/22.jpg)
22
Benefits
• Rapid development and deployment of MATLAB analytics
MATLAB
• Analytics expressed in MATLAB with nearly all available toolbox functionality
MATLAB Compiler SDKMATLAB Production Server
• MATLAB Analytics available to downstream users as a Stored Procedure/Function
Microsoft SQL Server
Develop
Develop
TestTestRefine
Refine
MATLABMATLAB Production Server
MATLABAnalyticsMATLABAnalytics
![Page 23: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/23.jpg)
23
Big Data with MATLAB
![Page 24: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/24.jpg)
24
Techniques for Big Data in MATLAB
Complexity
EmbarrassinglyParallel
Non-Partitionable
MapReduce
Distributed MemorySPMD and distributed arrays
Load,Analyze,Discard
parfor, datastore,
out-of-memory
in-memory
![Page 25: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/25.jpg)
25
Big Data Capabilities in MATLABMemory and Data Access 64-bit processors Memory Mapped Variables Disk Variables Databases Datastores
Platforms Desktop (Multicore, GPU) Clusters Cloud Computing (MDCS on EC2) Hadoop
Programming Constructs Streaming Block Processing Parallel-for loops GPU Arrays SPMD and Distributed Arrays MapReduce
![Page 26: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/26.jpg)
26
MATLAB in Business Critical Applications
![Page 27: MATLAB in Business Critical Applications...2 Outline Problem Statement The Big Picture / Financial Applications Reference Architectures and integrations – TIBCO® Spotfire® –](https://reader030.vdocuments.us/reader030/viewer/2022040907/5e7e437527ae7161bc3fe5f9/html5/thumbnails/27.jpg)
27
Conclusions: MATLAB Analytics in ProductionAgility: Yes. Democratization: Yes. Production Quality: Yes.
Access and explore data from within MATLAB during prototype development
Develop and integrate analytics with enterprise systems– Compact, precise language for expression of complex analytics– High-quality libraries and support
Open and extensible platform for analytics– Data Visualization: TIBCO Spotfire,
– Hadoop: Apache
– Data warehouses: IBM PureData, Microsoft SQL Server
MATLAB products and Services provide a single-stack solution when used with supporting technologies to address production data analytics demands.