athmos: a satellite anomaly detection framework microservice … .pdf · 2017-06-23 · athmos: a...

ATHMoS:

A Satellite Anomaly Detection Framework

– Microservice Architecture

Corey O‘MearaGerman Space Operation Center

ESAW 2017

June 21st, 2017

> O'Meara • ATHMoS > 21.06.17www.DLR.de/rb • Slide 1


We want to compare new telemetry data withpast data to see how its behavior has changed

Past Telemetry Todays Telemetry

General Idea of Anomaly Detection


Why Automated Telemetry Checking?


Big Data, Big Computations…

1. Noise Extraction and De-Trending


2. Sub-Cluster Dimension Approximation



3. Anomaly Score Computation

Raw Data

Smoothed

Data

Noise Data


4. Deep Learning NN: Time Series Forecasting

90-min Prediction

45-min Recent Data

ATHMoS: System Architecture Requirements

High Availability

Necessary to ensure 24/7 monitoring in the operational environment

Scalability:

The resulting system should be easily scalable for more

computational power

Performance:

The complete chain from import over processing to storing the data

should complete in a few minutes for a days worth of telemetry data

Modifiability:

The processing chain needs to be easily modifiable to allow for

individual mission-specific adjustments


What Are Microservices? (a.k.a. „fine-grained“ SoA)


TelemetryDB: Apache Cassandra


Used by (among many others):

• Distributed database

• Supports multiple datacenters

• Highly available and scalable

• Developed for Big Data @Facebook

Apache Cassandra

Data Processing: Apache Spark

Apache Spark

• Distributed computational engine

• Highly scalable and asynchronous calculations

• „Next Generation“ Hadoop (~100 times faster)

• Allows real-time processing (data streams)




Language: Scala

Akka (Scala)

• Framework to develop fault tolerant services

• Both OOP and Functional Programming

• Asyncronous communication

• Typesafe better version of Python (imo)



Message Broker: Apache Kafka

Apache Kafka

• Originally developed at LinkedIn

• Open Sourced in 2011

• Distributed messaging (pub/sub) system

• By far, the fastest w.r.t messages/sec



ATHMoS Microservices Overview

Microservice Orchestration: DC/OS


Deployment Infrastructure


x2 x6

x2

Cluster Details


App Scheduler

DC/OS Dashboard


App Launcher and Scheduler


Cluster Details


Allows for a Reactive

System

App Resource Monitoring > O'Meara • ATHMoS > 21.06.17www.DLR.de/rb • Slide 23

Application Instance CPU

Usage

Application Instance RAM

Usage

Application Instance Cached

Memory Usage

Application Instance Network

Usage

Total System Usage Statistics

ATHMoS as a Reactive System


The distributed nature of the system allows us to define limits for CPU,

RAM, network connections/traffic for each component of the software

such that if it is under heavy load it will elastically scale that individual

component


Outlook: Microservice Template

Infrastructure LayerDomain LayerApplication

Layer


Outlook: Continuous Integration and Delivery


In Conclusion…read this!

Thanks for your attention!

@cpomeara


References

1. O’Meara, C., Schlag, L., Faltenbacher, L., Wickler, M., “ATHMoS: Automated Telemetry

Health Monitoring System at GSOC using Outlier Detection and Supervised Machine

Learning,” Proceedings of the AIAA SpaceOps 2016 Conference, May 2016.

2. “Building Microservices”, Sam Newman, O’Reilly Media, Inc., 2015.

3. http://martinfowler.com/articles/microservices.html

4. https://www.linkedin.com/pulse/20141128054428-13516803-monolithic-vs-microservice-

architecture

5. http://www.lab41.org/transformers-rdd-in-disguise/

6. https://www.infoq.com/articles/apache-kafka


http://martinfowler.com/articles/microservices.html

https://www.linkedin.com/pulse/20141128054428-13516803-monolithic-vs-microservice-architecture

http://www.lab41.org/transformers-rdd-in-disguise/

Backup Slides


Frontend Prototype


Example: Battery Voltage Drop


Triggered OOL

Battery Voltage Anomaly first

detected when OOL

triggered during an image

acquisition datatake during

eclipse



ATHMoS correctly identified

not only the anomaly where

the OOL was triggered but

anomalies of the same type

that occurred more than 1

month in advance

OOL

July 2015

Outlie

r S

core

(0-1

00%

)



Outlie

r S

core

(0-1

00%

)

June 2015

ATHMoS correctly identified

not only the anomaly where

the OOL was triggered but

anomalies of the same type

that occurred more than 1

month in advance

athmos: a satellite anomaly detection framework microservice … .pdf · 2017-06-23 · athmos: a...

Documents