wojciech sliwinski [email protected] beams department, controls group cern

THE ROLE OF MESSAGING SOLUTIONS IN COMPLEX SYSTEMS, BASED ON EXPERIENCE IN

THE ACCELERATOR CONTROLS AT CERN

JINR/CERN/MEPHI COMPUTING SCHOOLDUBNA, 26TH OCTOBER 2011

Wojciech [email protected] Department, Controls GroupCERN

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ... 2

Outline

Defining Middleware

Types of Middleware

CERN Acc. Controls Messaging Infrastructure

Middleware Review for CERN Acc. Controls

Conclusions

26th October 2011


Outline

Defining Middleware

Types of Middleware



Conclusions

26th October 2011


Why Middleware?

How do we build/connect applications over a network?

How do we facilitate Distributed Computing (2-tier, …, n-tier)?

How to support the heterogeneous environments?

26th October 2011


What is Middleware? Software, which allows an application to interoperate with other

software, without requiring the user to understand and to code the low-level operations required to achieve interoperability Software layer between OS and the applications Hides complexity & heterogenity of distributed system Handles issues related to OS, Network Protocols & hardware platforms

Application

OperatingSystem1

Communication system

MW

Application

OperatingSystem2

MW

Application

OperatingSystemN

MW

Standard API Standard API

Specific API Specific API

Standard API

Specific API

26th October 2011

Introduction to Middleware

Middleware provides support for:Naming, Location, Service discovery, Replication Protocol handling, Communication faults, QoSSynchronisation, Concurrency, Failover, ScalabilityAccess control, Authentication

Middleware dimensions:Request-Reply vs. Asynchronous MessagingLanguage-specific vs. Language-independentProprietary vs. Standards-basedSmall-scale vs. Large-scaleTightly-coupled vs. Loosely-coupled components


Middleware is everywhere. You might think you can leave it but it just finds you again :-)

26th October 2011


Outline

Defining Middleware

Types of Middleware



Conclusions

26th October 2011


Middleware – large domain … which to choose?

26th October 2011

direct p2p broker

req/rep pub/sub

rpc/orb message

LAN WAN/Internet

Network Embedded


Types of Middleware

Remote Procedure Call (RPC)

Object-Oriented Middleware (OOM)

Message-Oriented Middleware (MOM)

Event-Based Middleware a.k.a Publish-Subscribe

26th October 2011


Remote Procedure Call - RPC Masks remote function calls as being local Client-Server model with Request-Reply paradigm Synchronous communication Marshalling of function parameters and return value Provides Location & Platform transparency Technologies:

DCE-RPC XML-RPC

26th October 2011

Client App.

IDL Stub

Network

Server App.

IDL Skeleton

NetworkReq/Reply


RPC - Disadvantages Synchronous Request-Reply interaction

Tight coupling between client and server Client may block for a long time if server loaded

○ Multi-threaded programming at client Slow/failed clients may delay servers when replying

○ Multi-threaded programming at servers

RPC paradigm is not object-oriented Invoke functions on servers as opposed to methods on objects

26th October 2011


Object-Oriented Middleware - OOM Client-Server model with Request-Reply paradigm Synchronous communication Marshalling of function parameters and return value Provides Location & Platform transparency

Objects can be local or remote Remote objects accesible via proxy objects

Technologies: OMG CORBA MS DCOM Java RMI SOAP

26th October 2011

Client App.

IDL Stub

IIOP

Client ORB

Server App.

IDL Skeleton

IIOP

Server ORB

POA Adapter

Req/Reply


Object-Oriented MW - Disadvantages Synchronous Request-Reply interaction

Tight coupling between client and server Client may block for a long time if server loaded

○ Multi-threaded programming at client Slow/failed clients may delay servers when replying

○ Multi-threaded programming at servers

Distributed garbage collection Releasing memory for unused remote objects Error-prone and difficult to debug

Static mappings and heavy-weight development Difficult to extend an operational system

26th October 2011


Message-Oriented Middleware - MOM Data exchange via Messages & Message Queues Asynchronous communication (Point-to-point One-to-One) Client and Server are loosely coupled Message Servers may do filtering, transforming, logging… Technologies: IBM WebSphereMQ, TIBCO Rendez-vous, JMS, SOAP

26th October 2011

Client App.

local messagequeues

Server App.

local messagequeues

messagequeues

Network Network Network

Message Servers


Message-Oriented MW - Disadvantages Poor programming abstraction (but has evolved)

• Rather low-level• Request-Reply more difficult to achieve, but can be done

Message formats originally unknown to middleware• No type checking (solved in JMS)

Queue abstraction only gives One-to-One communication• Limits scalability (solved in JMS with Publish-Subscribe)

26th October 2011


What we lack, so far

General interaction patterns• We have Request-Reply and One-to-One• Publish-Subscribe? One-to-Many? Many-to-Many?• Dynamic joining and leaving?

Location transparency• Anonymity of communicating entities

26th October 2011


Event-Based Middleware a.k.a Publish-Subscribe Publishers (advertise and) publish Events (Messages) Subscribers express interest in Events with Subscriptions Event Service notifies interested Subscribers of published Events Events can have arbitrary content (typed) Technologies (Brokers): JMS (ActiveMQ, RabbitMQ), AMQP (…)

26th October 2011

Publisher1Event Service

(Broker)

PublisherN

…Publisher2

topics

publish

publish

publish

Subscriber1

SubscriberN

…Subscriber2

subscribe

notify

subscribe

notify

subscribenotify


Properties of Publish-Subscribe

Asynchronous communication• Publishers and Subscribers are loosely coupled

Many-to-Many interaction between Pubs and Subs• Scalable scheme for large-scale systems• Publishers do not need to know Subscribers, and vice-versa• Dynamic join and leave of Pubs & Subs (thanks to Brokers)

Topic based Publish-Subscribe very expressive• Filtered information delivered only to interested parties

26th October 2011


Outline

Defining Middleware

Types of Middleware


CERN Acc. Controls Middleware Review

Conclusions

26th October 2011


CERN Controls Middleware – in all SW tiers…

LHC MACHINE

GENERAL PURPOSE

NETWORK

OPERATORCONSOLES

OPERATORCONSOLES

FIXEDDISPLAYS

CER

N G

IGAB

IT E

THER

NET

TEC

HN

ICAL

NE

TWO

RK

FILE SERVERS APPLICATION SERVERSSCADA SERVERS

RT Lynx/OSVME FRONT ENDS

WORLDFIPFront Ends PLCs

ACTUATORS AND SENSORSCRYOGENICS, VACUUM, ETC…

QUENCH PROTECTION AGENTS,POWER CONVERTERS FUNCTIONSGENERATORS, CRYO TEMPERATURE SENSORS…

BEAM POSITION MONITORS,BEAM LOSS MONITORS,BEAM INTERLOCKS,RF SYSTEMS, ETC…

Wor

ldFI

PSE

GM

EN

T(1

, 2.5

MB

its/s

ec)

PRO

FIB

US

FIP/

IO

TCP/IP communication services

OP

TIC

AL

FIBE

RS



TIMING GENERATION

TTT

T

TT

T T T T

DIR

EC

T I/O

PRESEN

TATIO

N TIER

MID

DLE TIER

RESO

URCE TIER

CMW client (C++/Java)GUIs, LabView, RADE

CMW client (Java servers)Logging, LSA, InCA, SIS

CMW server (C++/Java)Proxy, DIP, AlarmMon, AQ

CMW server (C++)FESA, FGC, GM

CMW server (C++)PVSS (Cryo, Vacuum)

JMS client (Java)GUIs

JMS client (Java)Servers: Logging, InCA, SIS

26th October 2011


JMS for Accelerator Controls

Java Messaging Service Purpose

Reliable transport of data between Java ProcessesControls

○ Alarm System○ Data Rendering Services○ (Control) GUIs○ Logging Services○ Beam Security System(s) No JMS -

No Beam !

26th October 2011


JMS for Accelerator Controls

Devices / Hardware

JMS

C/C++ Process

JAVA Process

JAPCCMWJMS

Front End

GUI Process

JAPCCMWJMS

CMWCMW

~50 Processes ~100 Processes~46K Subscriptions280 Connections

4000 servers @1700 machines

26th October 2011


Examples of JMS ”aware” systems

26th October 2011


History of JMS in Accelerator Controls

Usage of SonicMQ JMS for LHC Alarm System since 2001

ActiveMQ introduced in 2005

Other project started to use JMS service

Today vital part of Beam Operations

26th October 2011


Technical Aspects

Broker ConfigurationNon-persistent messages on topicsNo subscription recovery policyNo durable subscribersNo exclusive consumersNo replyTo mechanismBroker is persistent

ClientsPure JAVA - Wireformat Protocol

26th October 2011


General JMS Deployment Situation

SonicMQ : 12 BrokersTIM, LASER, DIAMONAlways as two node cluster

ActiveMQ : 16 BrokersOne main default Broker ServiceOtherwise project dedicated Brokers

All Brokers on Technical Network

26th October 2011


Overall Broker Service Jan 2011

26th October 2011


JMS Service Data Handling

Central JMS Service (276 days in 2010)58 Million msg / day120 TB IN650 TB OUT

Additional handling of Heavy Connection and

Subscription handlingBursts loads

Connected subscribers grewfrom 2K to 12K in 6 months

=> 440 GB /day=> 2.5 TB /day

YES, ActiveMQ can!

26th October 2011


Data Handling during LHC startup 2010

Total Data Out

Increase in data demand

2.57 TB/day

1TB/day

LHC startup

26th October 2011


JMS Service Availability 2010*

Total hours : 6624

ActiveMQ: 99.99%= 8 minutes downtime (due to kernel update)

SoniqMQ : 99.87 %= 8 hours downtime

* November 201026th October 2011


Monitoring and Troubleshooting Tools

DIAMON & Lemon

JConsole / JVisualVM

Dump of broker information to SQLite DB

Traffic Monitoring Tool

Various Message Listeners

P

R

I

O

R

I

T

Y

26th October 2011


Easy Identification of Bad Producers

Fixed

Problematic Producer

26th October 2011


JMS Service – Lessons Learnt

ActiveMQ was a good choiceRunning reliablyPerforms wellProvides many metrics for diagnostic and monitoringGood integration in BE-CO monitoring tools via JMX

Projects are not always aware of sending so much dataService Level Agreements with projects requiredMore analyzing tools for end users required

26th October 2011



Monitoring is a vital part of operationsMessage reception speed testMemory Percent usageJMX attributes (#topics, #consumers, #subscriptions)

Broker Configuration is not always obvious

26th October 2011



ClusteringGood for failover and load balancingAdditional complexityUpdating clients for new cluster members

○ Automatic rebalancing of clients

Memory is importantTuning of Java GC options

26th October 2011


Outline

Defining Middleware

Types of Middleware


CERN Acc. Controls Middleware Review

Conclusions

26th October 2011


Hardware Devices

GUI operator consoles

Front ends“virtual devices”

CERN Controls Middleware – CMW

26th October 2011

Controls Middleware

Embedded Middleware


CMW Mandate & Scope

Standard set of MW solutions & libraries Centrally managed services Track & optimize runtime parameters

Scope: CERN Accelerator Complex Operational 24*7*365 Must be Reliable & High Quality ~80’000 HW devices, ~4’000 servers ~1700 machines, ~2’000’000 IO points

Huge infrastructure!

GUI Applications

Control Logic

Middleware

Control System

26th October 2011


CMW Data Flow

Devices / Hardware

JAVA Process

JAPCCMW

C/C++ Process

JMS

FEC CMW publisher

CMW

MONITOR

GET / SET

JAVA Process

JAPCCMWJMS

direct

JMS Broker

26th October 2011


Linux/Windows

CERN Controls Middleware

26th October 2011

C++/Java process

Controls Middleware API

TCP/IP

CORBA

Controls Middleware

RT Linux/LynxOS

C++ process

Controls Middleware API

CORBA

Problems:- C++ and Java implementations differ - Heavy in memory usage - Complex error prone API - No direct support for pub/sub - Blocking issues (JacORB) - Shrinking community - Lack of new releases and bug fixes

Long LHC shutdown at the end of 2012 A unique possibility!

Instead of patching let’s provide a new one!


CERN Middleware Requirements

26th October 2011

Desirable

Mandatory

Fundamental

Lightweight

Active community

Friendly API, documentation

Request/reply & pub/sub patterns

Open source, redistributable license

QoS

Stability, Maturity & Longevity

Performance & Scalability

Asynchronous

C++/Java

Linux/Windows

Over TCP/IP LAN


How did we evaluate –> our criteria

26th October 2011

CRITERIA

QoSresources,binary size,

memoryperformance

Communicationspatterns

API, look & feel,documentation

Community,maturity

Appearance

• Creators• specification• documentation

• Users• forums• bug reports

• Internet

Simple usage

• Download• licensing

• Compile• Linux• LynxOS

• Run examples

Testing

• Communication patterns

• Performance• QoS• Exceptional

situations


Surprising how many great new products!

26th October 2011

BSDsockets

RPC

CORBA???

20111969ARPANET

TODAY

1974

TCP

1998

IP v6

1980

UDP

1981

IP v4

1983

1991

2011


Evaluated middleware products

26th October 2011

Ice

Thrift

omniORB

YAMI

OpenSpliceDDS

OpenAMQCoreDXRTI DDS

ZeroMQ

QPid

MQtt RSMBJacORB

Mosquito

All opinions are based only on our knowledge and evaluation. Each of the products, depending on the requirements, may constitute a good solution.

RabbitMQ


CORBA (omniORB, JacORB)

26th October 2011

Object-oriented communication platform standardized by OMG. It is over 20 years old, and the standard is well established but...

Complex, hard to learn and use Differences between implementations Big memory footprint Shrinking community Seldom updates and bug fixes

Complex

Heavy

Shrinking

community

Dying

out

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...

Object-oriented middleware by ZeroC, people who left CORBA to fix its problems. Conceptually similar to CORBA but...

Better design and implementation Modern easier to use API and IDL mapping Concise C++ and Java implementation Support for versioning Active community

Ice

4626th October 2011

Well

designedActiv

e

community


Thrift

26th October 2011

Middleware by Facebook (moved to Apache)

Lightweight, free of unnecessary dependencies Modern, clean API Active community Ongoing development, still incomplete No documentation, empty tutorial

Poor

documentation

Work stil

l

in progress


Message-oriented middleware developed by one of our former colleagues. Used by some CERN Controls services.

Lightweight, free of unnecessary dependencies Well designed, modern API Asynchronous req/rep and pub/sub patterns Dynamic type specification (paid in performance) Small community

YAMI4

26th October 2011

Lightweight

Well

designed

Small

community


Middleware where messages are distributed by a broker. A few independent implementations of the broker and clients.

Big community Designed to support pub/sub Protocol standard issues Broker – single point of failure Broker – additional hop, slower communication Broker – additional, non compliant monitoring tools

AMQP – OpenAMQ, Qpid, RabbitMQ

26th October 2011

Slower for

req/rep Broke

r

issues

Active

community


Data-oriented middleware with p2p communication. Standard defined by OMG, with a few compatible implementations.

Big industrial and military community Possible compatibility with CORBA Designed to support pub/sub Req/rep possible but needs 2 channels Steep learning curve Complex API

DDS – RTI, CoreDX, OpenSplice

26th October 2011

Heavy

Complex

Big industry

presence


Message-oriented middleware by iMatix (they dropped OpenAMQ.) API resembles BSD sockets but it is so much more.

Simple yet powerful API Support for in-proc, inter-proc, TCP, PGM Lightweight, free of unnecessary dependencies

Many communication patterns May be used as a concurrency framework Active community

ZeroMQ

26th October 2011

Lightweight

Well

designed

Active

community


Performance requirements reviewed

26th October 2011

Instead of estimating We asked our users what they will need Gathered statistics from the current system

Most demanding users identified Definition of a few performance tests


C++ server with client in:

Performance tests, reliable Request-Reply

26th October 2011

4000 msg/secPayload = 4B

5 msg/secPayload = 10MB

Many small messages

A few big messages


Performance/Scalability tests, reliable Pub-Sub

26th October 2011

400 msg x 8 B10 clients, <50ms

30 msg x 8 B10 clients, <20ms

Throughput Latency


Products comparison (according to the criteria)

26th October 2011

Sync, async & msg patterns

QoS

Dependencies & memory f-p

Performance

Look & feel, API, docs

Community & maturity

Score

ZeroMQ 6Ice 5

YAMI4 4RTI 3

Qpid 3CORBA 2

Thrift 2

56

Middleware Review Summary Several good middleware solutions available. The choice is dictated by the most critical requirements

for any given application. Not easy QoS policies and performance matter, but also ease of use, community, …

Concerning CERN Controls Middleware… Prototype with the most

promising candidates:

Deploy the new middleware before the long accelerator shutdown at the end of 2012

ZeroMQIce YAMI

Wojciech Sliwinski: The Role of Messaging Solutions in Complex Systems ...26th October 2011


Conclusions

Middleware is an important abstraction for building the distributed systems

Synchronous vs. asynchronous communication

One-to-one vs. many-to-many communication

Better performance with direct communication

Better scalability and less coupling with Brokers

System requirements Messaging patterns to apply26th October 2011

wojciech sliwinski [email protected] beams department, controls group cern

Documents

role of messaging solutions

complex systems

middleware large domain

middleware progress

controls conclusions

accelerator controls

componentswojciech sliwinski

distributed computing