msg status update
DESCRIPTION
MSG Status update. Daniel Rodrigues. Presentation Summary. MSG overview Timeline ActiveMQ Msg -publish-simple Msg-consumer2oracle Current deployments OSG Gridview Next steps. MSG overview. MSG overview. Features of an Messaging System: Flexible architecture: - PowerPoint PPT PresentationTRANSCRIPT
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
MSG Status update
Daniel Rodrigues
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Presentation Summary
• MSG overview• Timeline• ActiveMQ• Msg-publish-simple• Msg-consumer2oracle• Current deployments
– OSG– Gridview
• Next steps
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
MSG overview
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
• Features of an Messaging System:– Flexible architecture:
• Deliver messages, either in point to point (queue)…• … or multicast mode (topics)• Support Synchronous or Asynchronous communication.
– Reliable delivery of messages:• Provide reliability to the senders if required• Configurable persistency / Master-Slave.
– Highly Scalable:• Network of Brokers
• ActiveMQ is an Open Source message broker providing these and many other features.
MSG overview
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
• A real life working example:
MSG overview
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Timeline
• 1st Trimester (Oct/Dez):– Testing of ActiveMQ 4.1, using netcat/bash scripts to test throughput;– Integrated testing with twisted stomp, python framework;
• 2nd Trimester (Jan/Mar):– Simple publishers and python consumers developed by Piotr Nyczyk on place for
OSG;– Msg-publish-simple and msg-consume2oracle (bugfixes + improvements + testing);– Moving to ActiveMQ 5.0, retesting throughput;– Definition of message-classes for gridview-gridftp parsing records; – Testing different broker configurations: Master-Slave, JDBC persistence, file
persistence;– Testing of JMS Java clients (producers and consumers) together with python.
• 3rd Trimester(Apr/):– Network of Brokers testing;– GWT (Google Web Toolkit) Client prototype. – Setting up publishers/consumers for gridview-gridftp
• First 4 machines, now extended for additional 27 machines;
– Definition of message-classes for downtimes with OSG and initial protyping.– Moving to ActiveMQ 5.1
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
ActiveMQ updates
• Move from 4.1 => 5.0– Producer Flow Control!– AMQ Message Store, faster persistence– Command Agent– (Blob Messages)
• Now moving from 5.0 => 5.1– More stable– Default AMQ Message Store + bugfixes
• Issues with recovering from failure: very long time to replay all kept logs!
• No viable Master/Slave with fast persistence otherwise
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
MSG publish simple
• Simple python publisher script, no other dependency besides… python!
• Publishes messages over either stomp or http, depending on network availability.
• Currently version 0.9.3-1:– A few bug fixes since initial version– Priority to stomp over http.– Configurable logging
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
MSG consume2oracle
• Python consumer script– again no other dependency besides… python (and
cx_oracle)!
• How does it work:– Creates a durable subscription to the topics– Receives the messages– Retrieves the records sent along with a message– inserts them into an oracle view
• Currently version 0.9.1-1:– Bug fixes– Configurable logging
• Snapshot: – Implementing Status summaries topic (publishes summary
information back into MSG)
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Current Deployments
• OSG:– Broker at gridmsg001
• activemq 5.0.0• Persistence with journaledJDBC (derby)
– msgConsume2Oracle at gridmsg001• Publishing to SAM metricOutput/metricDescription
– msgPublishSimple• Several publishing from indiana.edu.
• Gridview:– Broker at gridmsg002
• activemq 5.0.0• Persistence with journaledJDBC (derby)
– msgPublishSimple• Called from gridview-publisher• Deployed on 27 gridftp nodes for cms t1transfers.
– msgConsume2Oracle• gridmsg002: publishing to testDatabase• Gridview machine publishing to gridview DB.
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Current Deployments
• Testing :– Brokers:
• lxb6118 / 6117 -> 5.1 Snapshot• Many different configurations along the time
(Master/Slave Network of Brokers, file persistence, etc)
– Publishers:• Java JMS producer (pcitgd24)• lxplus based msg-publish-simple
– Consumers:• msg-consume2oracle (lxb6118/6117) • GWT test (pcitgd24)
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Next Steps
• OSG scheduled downtimes– Message Class defined– Some test messages into a test DB done– Waiting Arvind’s prototype (very soon)
• JobDetails for Dirac3 detailed logging– Interested in getting information such as
cputime, and considering using MSG to do it.– Message class under definition
• MSG System Management:– How to operate a messaging system in
production?– We will need to know what’s going on!
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Thank you for your attention.
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Support Slides
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Some recent test results
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Some recent test results
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Some recent test results
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Some recent test results
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Console, GridMSG001
Broker restarts: 10 Mar.18 Apr, too many open connections -> increase of file descriptors.Since 18 April, 220000 messages.
CERN IT Department
CH-1211 Genève 23
Switzerlandwww.cern.ch/
it
InternetServices
Console, GridMSG002
420087 messages forwarded to each consumer (test + GV).