activeeon’s workflows: from hpc to data analytics to...
TRANSCRIPT
![Page 1: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/1.jpg)
Paris, Sophia Antipolis, London, San Jose USA Paris, Sophia Antipolis, London, San Jose USA, Montreal CA
ActiveEon’s Workflows:from HPC to Data Analytics to Machine Learning
Denis Caromel
![Page 2: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/2.jpg)
AE Vision
Workflows are Now Strategic
Convergence of Usage in Academia & Industry
Convergence in usage:HPC to Data Analytics to Machine Learning
For Scientist, Application People, Data Scientists, ML + AI Experts
![Page 3: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/3.jpg)
AE Architectural Vision
Studio: Easy to Use Workflow Designer,with Palettes and Presets + APIs
PROACTIVE STUDIO
PROACTIVERESOURCES MANAGER
PROACTIVE SCHEDULER
PROACTIVE CLOUD AUTOMATION
Automation Dashboard: PaaS, Job Planner, Cloud Watch, Analytics
Scheduling & Meta-Scheduling: on all Infra for Legacy, Cloud & Hybrid
Resource Manage: “Infrastructure as Code”, Cost + Resource saving
Studio:
Automation Dashboard:
Scheduling & Meta-Scheduling:
Resource Manager:
![Page 4: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/4.jpg)
Scalable & Elastic resources
Incremental resource deployment
100% resources usage, no waste
Smart scale down
Provides cloud computing power according to your needs.Minimize costs by deploying VMs only when needed (configurableload factor). Never exceed your budget (min/max VMs threshold).
Smart and fully configurable elastic policy. Shutdown unusedVMs whenever it's possible. Prevent time-consuming re-deployments by adjusting idle nodes’ release delay (avoid scaleup/down cycles).
![Page 5: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/5.jpg)
Hybrid Cloud Customer Cases
![Page 6: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/6.jpg)
SituationMake Sentinel data available to the greatest number and encourage the development of applications using them (agriculture, maritime field...)
SolutionProactive Solution provided by ActiveEon to execute on Azure in hybrid modeallows enhancing PEPS data and making them available to API providers :• Multi-Cloud Ecosystem Platform• Remove complexity for Data Scientists• Provide Cloud performance
Benefits• Faster execution, Optimisation of On-Prem ressources & Clouds, • Easier to use by end-users
Space & Image Processing
![Page 7: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/7.jpg)
PEPS: Sentinel Satellite Image Analysis
ProActive task
ProActive nodes
Key points to choose AE:OnPrem & Clouds with a unique Task definitionSingle Interface for Job ManagementOut-of-the-box effective error managementProduct integration with DockerBusiness tasks can trigger VM provisioning on Azure Cloud
![Page 8: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/8.jpg)
Genomic Platformby
![Page 9: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/9.jpg)
Hybrid Cloud Architecture OverviewWeb Portal and Integration with Scientific tools
Total DNA QC/Library preparation
Proton/Illumina Sequencing
1.3PBDataBase
1TB / Sequence Analysis
WindowsCluster 1192 cores
Linux Cluster 2366 cores
Pre, Post Processing of Data AnalysisFlexibility, Speed of AnalysisGranular executionDistribution for fast executionSecure data transfer
Quantitative Metagenomics Platformfor gene profiling and statistical analysis
On-Prem Azure Cloud
CPU cores on demandCloud Bursting
REST / HTTPS
Secure data transfer
![Page 10: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/10.jpg)
Non Destructive TestingSimulation and Analysisby
![Page 11: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/11.jpg)
CEA List Architecture
On-Prem Azure Cloud
CPU cores on demandCloud Bursting
REST / HTTPS
Secure data transfer
Dev. Cluster72 cores
Partner Cluster144 cores
Prod. Cluster192 cores
Result visualization
Job monitoringCIVA softwareto launch NDT
Multiple NDTTechniques
RT : weld inspection
CT : tomographic reconstruction
of complex parts
ET : 2D map of a complex defect
![Page 12: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/12.jpg)
Finance (Solvency Risk) on Microsoft Azure
Time (hours)105 160
1024
Wor
kers
Profile Results
1024
Wor
kers
Profile Results
I/O intensive tasksAggregation &Reporting
Low PriorityCPU-Intensive tasksRisk Watch Simulation
Medium PriorityCPU-Intensive tasksRisk Watch Simulation
High PriorityCPU-Intensive tasksRisk Watch Simulation
“More Value, Faster”
Full computation without intermediate resultHigh Priority Results Full Report
2H 5H
Batch optimization with ProActive:• Enforce strong priorities• Optimal compact execution• Start tasks as early as possible• Pipeline and co-allocate• CPU-intensive with I/O intensive
tasks
18H
Achievements & Benefits:High Priority Results in 2 H instead of 18 H !Compacting Executions for savings on the Cloud
What Legal & General said about Batch optimization with ProActive:“It enforces strong priorities, optimally compacts execution, starts tasks as early as
possible, pipelines and co-allocates CPU-intensive with I/O intensive tasks.”
32 V
Ms
or M
achi
nes
32 V
Ms
or M
achi
nes
Profile Results without ProActive Profile Results with ProActive
![Page 13: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/13.jpg)
AI – Machine Learning Customer Cases
![Page 14: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/14.jpg)
Resource Management
Scientists• Provide new statistical models on demand• Develop ML models and algorithms at scale• Manage distributed ML model training• Deploy solution everywhere
Clinicians• Request additional diagnostic on demand• Identify new patterns for prediction and
advices
Hospital - Human Brain Project
Clinicians
on-prem
Scientists
![Page 15: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/15.jpg)
Machine Learning Analytics for Aerospace
![Page 16: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/16.jpg)
Each campaign generates large amount of data on more than 2000 sensors
Data generationScheduler
Resource Manager
Fault ToleranceCloud burstingResource agnosticMicro-serviceEtc.
Local Cloud
Data
exp
ositi
on
ProActive allows to parallelize data
analytics and scale the infrastructure to
optimize compute times.
Tests
2 ML Models: a Streaming Version during the Test and a Post-Mortem for fine grain detections It also allows to Industrialize, Manage and continuously improve the Models.
Machine Learning Analytics for aeronautics
![Page 17: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/17.jpg)
FEATURES
● Data analysis: data cleaning and merging● Signal processing● Big data workflows for automation of test scenarios● Automatic detection of the different categories of
anomalies (out of order, noisy, axis inversion)● Data visualization in-browser
BENEFITS
● Data fetching from many sources● Extraction of relevant features● Faster results with parallel parsing and processing● Automatic anomaly detection with machine learning● Extensible & Evolving solution using Workflow tasks
ProActive workflow for Anomaly Detection for Satellite SignalsData visualization
No Anomaly
Noise
Out of order
Axis inversion
phase amplitude
Anomaly Detection on Satellite Sensors
![Page 18: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/18.jpg)
FEATURES
● Introduce Artificial Intelligence technologies in the aerial/satellite image chain
● Give them autonomy, opening up many opportunities for remote sensing with a capacity for reaction
● Creation of large database of labeled aerial images dedicated to learning
● Evaluate performances on concrete settings (use cases, data, hardware)
BENEFITS
● A collaborative platform to design, evaluate and share ML algorithms
● Direct access to diverse hardware (different GPU, FPGA, Clouds, On-premise)
● On-demand scalability for diverse experiments
ML for Image Analysis Onboard Satellites
![Page 20: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/20.jpg)
TechnologyProcessing & Automation Workflows
Scheduling & Orchestration+ Meta-Scheduling
ResourceManagement& Monitoring
WorkflowStudio
Job Console
Resource Manager
![Page 21: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/21.jpg)
Automation Dashboard - Catalog
Workflows stored in buckets in the CatalogRBAC support for each bucket / Users can share workflows and templates
Keep track of the revisions with a versioning feature integrated
![Page 22: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/22.jpg)
Cloud Automation: On-demand PaaS
On-Demand PaaS Services with full Life-Cycle Management
![Page 23: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/23.jpg)
Machine Learning Open Studio
https://www.youtube.com/watch?v=mbrQxCf4lqM
![Page 24: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/24.jpg)
Job Planner – Interactive GANTT Monitoring
![Page 25: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/25.jpg)
UK Ministry of Interior is using ActiveEon for 2 critical applications:• Visa Delivery Process, and • Big Data & Analytics platform for Crime Reduction (HODAC).
Situation25 different sources of Data.uild a consolidated Data Lake and analytics platform to be used for many Home Land security applications.
SolutionActiveEon used as the central Orchestrator to Schedule and Meta-Schedule all the Big Data, ETL, Analytics, Machine Learnigs software appliance of the platform (Hadoop, SAS, TIBCO Spotfire, Python, Anaconda, GreenPlum, ElasticSearch, …).
Gov.: UK Ministry of Interior
![Page 26: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/26.jpg)
Meta-Scheduler, Big Data ETL + new ELT
Execution prod
Analytical prod
Staging Dev
Virtualized Infrastructure using Docker4 Thousands of physical cores
Main BenefitsCentral Orchestration Tool Workflow Expressiveness: universal & comprehensiveManagement of Security forhighly sensitive environmentsManagement of Resources for all appliances (SAS, GREENPLUM, TIBCO, …)
« The only solution capable to Schedule any Big Data Analytics, mono-threaded, multi-threaded, multi-core, parallel and distributed »Cap Gemini Lead Engineer for Home Office
EC2, RDS DB instance
![Page 27: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/27.jpg)
Komatsu is a Japanese multinational corporation. It manufactures construction, mining, industrial and military equipment.
SituationActiveEon Orchestrates on Cloud execution over hot and cold storage for Streaming and Batch Analytics
1,200 tasks executed per hour
SolutionActiveeon has enabled control over scheduling and execution:• Error Management – Notification, Automated Recovery• Job Planner• Distribution & Parallelization
Benefits• Reliable execution to orchestrate multiple services and resources• Provide consistent results and KPIs to end users and BI Tools
IoT
ActiveEon allowed to migrate from AWS to Azure
![Page 28: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/28.jpg)
Fault-Tolerance & Error Management
![Page 29: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/29.jpg)
Supporting Specialized Processing
CudaOpenCLFPGA (e.g. Altera)ASIC (if needed one day)
Edges for IoT(Raspberry Pi, …)
![Page 30: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/30.jpg)
ConclusionBoth Portals and APIs, CLI
(Java, Python, R, …)
Error Management and Notifications
Machine Learning Open StudioData Connectors Data AnalyticsML + DL
Job Analytics
![Page 31: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/31.jpg)
Paris, Sophia Antipolis, London, San Jose USA Paris, Sophia Antipolis, London, San Jose USA, Montreal CA
![Page 32: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/32.jpg)
![Page 33: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/33.jpg)
Building Blocks in StudioSynchro between Tasks & even Jobs.Tolerance aux Fautes, Versionning Management30 mn:
• Intro: 5 mn• Archi: Automation Dashboard / Studio / Scheduler / RM + API• Features: Node Request, Data Space, Streaming/Interactive• Automation: Call Graphs, • Use Cases:• Demo:• Message: Easy to Use – New Usages in Scientific Computing, Hybrid, Portable, Open
![Page 34: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/34.jpg)
Job Planner – Interactive GANTT Monitoring
![Page 35: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/35.jpg)
UK Ministry of Interior is using ActiveEon for 2 critical applications:• Visa Delivery Process, and • Big Data & Analytics platform for Crime Reduction (HODAC).
Situation25 different sources of Data.uild a consolidated Data Lake and analytics platform to be used for many Home Land security applications.
SolutionActiveEon used as the central Orchestrator to Schedule and Meta-Schedule all the Big Data, ETL, Analytics, Machine Learnigs software appliance of the platform (Hadoop, SAS, TIBCO Spotfire, Python, Anaconda, GreenPlum, ElasticSearch, …).
Gov.: UK Ministry of Interior
![Page 36: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/36.jpg)
Meta-Scheduler, Big Data ETL + new ELT
Execution prod
Analytical prod
Staging Dev
Virtualized Infrastructure using Docker4 Thousands of physical cores
Main BenefitsCentral Orchestration Tool Workflow Expressiveness: universal & comprehensiveManagement of Security forhighly sensitive environmentsManagement of Resources for all appliances (SAS, GREENPLUM, TIBCO, …)
« The only solution capable to Schedule any Big Data Analytics, mono-threaded, multi-threaded, multi-core, parallel and distributed »Cap Gemini Lead Engineer for Home Office
EC2, RDS DB instance
![Page 37: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/37.jpg)
Packaged Solution
Open Source
Activeeon Solution
WorkflowStudio
Intuitive, multi-language, open
Error management, Job Planner, Event Base
Schedule & Control
Resource ManagementHybrid, multi-cloud, auto-scaling
MonitorReal-time job and
resource monitoring
![Page 38: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/38.jpg)
TechnologyProcessing & Automation Workflows
Scheduling & Orchestration+ Meta-Scheduling
ResourceManagement& Monitoring
WorkflowStudio
Job Console
Resource Manager
![Page 39: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/39.jpg)
Workflows: All kinds & All Resources
</>
API
Scheduler & Meta-Scheduler
Resource Manager
Clusters
LSF
ProActive Workflows
Fault ToleranceCloud bursting
Resource agnostic
![Page 40: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/40.jpg)
Scalable & Elastic resources
Incremental resource deployment
100% resources usage, no waste
Smart scale down
Provides cloud computing power according to your needs.Minimize costs by deploying VMs only when needed (configurableload factor). Never exceed your budget (min/max VMs threshold).
Smart and fully configurable elastic policy. Shutdown unusedVMs whenever it's possible. Prevent time-consuming re-deployments by adjusting idle nodes’ release delay (avoid scaleup/down cycles).
![Page 41: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/41.jpg)
ProActive Process Flow
ExecuteDesign
Schedule, Monitor & Connect the resources
IT department, Data scientist, Business Lines,
Activeeon Consulting IT department,Business owner
Operational team
1
2
3
PROACTIVE STUDIO
PROACTIVERESOURCES MANAGER
PROACTIVE SCHEDULER
PROACTIVE CLOUD AUTOMATION
![Page 42: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/42.jpg)
Global Locations
Some Supported Languages and Connectors
AWSAzure GCP Docker OpenShiftLinux Windows Solaris VMware Openstack
Infrastructure
Slurm PBS LSF
LSF
HPC SchedulersGoogle
Cloud Platform
PBS
Works
Cmd Java Scala Javascript Groovy Ruby Jython Python Perle PHP R Cron LDAPPowerShell
Languages and Predefined Tasks
FTPURL SFTP MySQL Oracle
Data Connectors
Linux Bash
MongoDB Cassandra AWS-S3
Clouds
PostgreSQL Greenplum SQL Server Azure StorageAzure
Data Lake
KafkaZookeeper Spark Hadoop
Big Data
AzureDatabricks Hadoop HDFS Twitter
LogstashSAP Elasticsearch
SGE
SGE
CNTK Keras PyTorch YOLO H2OTensoFlow
Machine Learning & Artificial Intelligence
Caffe Spark MLlib Pandas JupyterLab
Visdom
Visualization
KibanaSwarm Storm Clearwater
Cognitive Services
Grafana
Scikit-Learn MXNet
AIX AS/400
C++/C#
Cuda
Specialized
Open/CL FPGA
DLib BigDLDeepLearning
G4J
Kubernetes
![Page 43: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/43.jpg)
Machine Learning Open Studio
https://www.youtube.com/watch?v=mbrQxCf4lqM
![Page 44: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/44.jpg)
Some Major CustomersTelco & IT Bio Tech
& HealthFinanceEngineering Aeronautics Energy
& Space
Some Partners:
MediaDistribution
Government IoTCosmetics
![Page 45: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/45.jpg)
Komatsu is a Japanese multinational corporation. It manufactures construction, mining, industrial and military equipment.
SituationActiveEon Orchestrates on Cloud execution over hot and cold storage for Streaming and Batch Analytics
1,200 tasks executed per hour
SolutionActiveeon has enabled control over scheduling and execution:• Error Management – Notification, Automated Recovery• Job Planner• Distribution & Parallelization
Benefits• Reliable execution to orchestrate multiple services and resources• Provide consistent results and KPIs to end users and BI Tools
IoT
ActiveEon allowed to migrate from AWS to Azure
![Page 46: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/46.jpg)
UK Ministry of Interior is using ActiveEon for 2 critical applications:• Visa Delivery Process, and • Big Data & Analytics platform for Crime Reduction (HODAC).
Situation25 different sources of Data.uild a consolidated Data Lake and analytics platform to be used for many Home Land security applications.
SolutionActiveEon used as the central Orchestrator to Schedule and Meta-Schedule all the Big Data, ETL, Analytics, Machine Learnigs software appliance of the platform (Hadoop, SAS, TIBCO Spotfire, Python, Anaconda, GreenPlum, ElasticSearch, …).
Gov.: UK Ministry of Interior
![Page 47: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/47.jpg)
Meta-Scheduler, Big Data ETL + new ELT
Execution prod
Analytical prod
Staging Dev
Virtualized Infrastructure using Docker4 Thousands of physical cores
Main BenefitsCentral Orchestration Tool Workflow Expressiveness: universal & comprehensiveManagement of Security forhighly sensitive environmentsManagement of Resources for all appliances (SAS, GREENPLUM, TIBCO, …)
« The only solution capable to Schedule any Big Data Analytics, mono-threaded, multi-threaded, multi-core, parallel and distributed »Cap Gemini Lead Engineer for Home Office
EC2, RDS DB instance
![Page 48: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/48.jpg)
Positioning - Differentiation
High-Level Business & Application Owner:
• Important Savings of Cloud Expenses• Time To Value, Speed of Development • Speed of Execution• Muli-System Multi-Data-Source Aggregation• Complex and Large Scale Applications
STRONG TECHNICAL DIFFERENTIATORS
Cloud-Ready: Integrated Resource Manager with one-click elastic Multi-Cloud
Scheduling and Meta-Scheduling of any application
Muli-Language, comprehensive, resilient Workflows at large Scale
![Page 49: ActiveEon’s Workflows: from HPC to Data Analytics to ...orap.irisa.fr/wp-content/uploads/2019/04/F43-Orap-Denis-Caromel.pdf · Signal processing Big data workflows for automation](https://reader034.vdocuments.us/reader034/viewer/2022042219/5ec5627f9c571c0d232230cc/html5/thumbnails/49.jpg)
Paris, Sophia Antipolis, London, San Jose USA @activeeon
+33 988 777 660
Automate Accelerate & Scale10K Nodes, 20K Tasks, 1M Jobs