open source enterprise service management (osesm) system
DESCRIPTION
Open Source Enterprise Service Management (OSESM) System John Emerson, President, Avenue Open, Inc.TRANSCRIPT
Copyright 2010 Avenue Open, Inc.
Open Source Enterprise Service Management (OSESM) System
Open Source Test, Monitoring, and Management System for the Application Tier
Presentation to MIL-OSS 2010
MIL OSS 2010 2
Agenda
• Background – Problem Statement
• Current Research Project
• Full System Vision
• Invitation to Collaborate
MIL OSS 2010 3
Distributed Computing and ESM
• Distributed Computing - Opaque inNature; Requires Application Visibilityo Debug, Test, Monitor, Manage
• Enterprise Service Management (ESM)Systems Emerged in Responseo First Generation – Actional, AmberPoint, others
• Market Consolidation – IBM, Oracle, Progresso Second Generation – Focus on Pinpointing Specific Problems
• Transactional Analysis (BTM)• Infrastructure• Code Analysis
MIL OSS 2010 4
ESM Market Landscape• Ever-Increasing Number of Things to
Monitor End-to-End AND Top-to-Bottomo Services, Distributed Transactions, Application
Components (Java, .NET, others)o Through ESBs, Process Engines, Supporting Infrastructureo Extend to Legacy, Military Tactical, other corner cases
• Significant Need for Cross-Layer Analytics and Root-Cause Analysis
However….
• The Gap between Customer Needsand ESM Vendor Support isWidening, not Decreasing
MIL OSS 2010 5
Imperfect Markets• All Market Options today are COTS, Closed Source
o Limited architectural and functional extensibilityo ESM Products can Only be Extended by the Vendor
• ESM Vendors are the Bottlenecko Proprietary APIs and Instrumentation - Vendor Must Buildo Instruments Hardwired – No Multi-Purpose or Mix & Matcho Limited Support Outside of Proprietary “Vendor Stack”
• Analysis is too Narrow and Requires Duplicate Infrastructureo One Tool for Infrastructure, One for App, One for Codeo Install Different Proprietary Agents for Every System – Even Within
Proprietary Vendor Stacks!
Result – As the Need for Application Monitoring is becoming More Critical to the Success of Distributed Computing, ESM Vendors are Less Willing and Able to Meet Customer Needs
MIL OSS 2010 6
A NEW WAY OF THINKING….
MIL OSS 2010 7
Open Source Enterprise Service Management System (OSESM)
1. Commoditize Proven Capabilities
2. Advance the State-of-the-Art
3. Dramatically Drive Down Cost
MIL OSS 2010 8
What We Are Researching (“R&d”) Today
Initial OSESM Framework1. Mechanisms to Collect Application Tier Data – Various Types
and Forms
2. System to Disseminate Data to the Appropriate Processing points
3. Analysis - Correlation and Root Cause Analysis System
Foundation for Comprehensive Application Runtime Test and Monitoring System
MIL OSS 2010 9
Research Specifics
• New and varied forms of Application Tier Instrumentationo Collectors – Collect Raw Data from Application Infrastructure and
underlying Components – from Application Server, OS, and related− Project to result in the development of Agents or similar− Applicability:
– When the Mission has control of the Technology– When deep, detailed information is required
o Reflectors – Collect Raw Data from Network Devices and related− Non-invasive – Can be used when Mission does not own technology end-
to-end− Higher level data, useful for Visualization, Message Tracking, Inter-System
Analysis
• Data Dissemination Techniqueso From Instruments to appropriate Analytical Engine(s)o Foundation for Federation, needed for “Mission Thread” scale
MIL OSS 2010 10
Advancements in Data Analysis Techniques
• Specifically Targeted to Understand Mission Thread Errors and Breakages
• Researching Sophisticated Correlation Systemo Correlate Interesting and Related Data Across All Tierso What Happened at Each Level at Any Given Point in Time
−Application−Operating System−Runtime Infrastructure−Code
• Root Cause Analysis Applied to Correlated Datao What Broke?o Where, Why, How - Specificallyo When - Proactive Notification, not post-facto Analysis
MIL OSS 2010 11
Project Deliverables• Open, published APIs – Instrumentation and
Analytical Serverso Various forms - SOAP/WSDL/XML, REST, Binary (for Low
Resource, Low Bandwidth situations or Legacy)o Various Levels – Application Layer and Supporting
Resources
• Reference Implementations and Affiliated Documentationo Instruments – Collectors and Reflectorso Intermediary Message Processoro Analytical Servers – Root Cause and Correlation
MIL OSS 2010 12
Benefits• Dramatically Lower Cost - Estimated TCO is 10% of today’s
COTS Solutions• Flat Out - A Better ESM
o Faster Resolution for Problems that fall between OS, Hardware, Network, Application, and Mission Layers
• Extensible - by the Communityo Instrumentation from the Community – Inexpensive, Accessibleo Build New Instruments as Needed – Info Readily Available
• Adaptable – Legacy systems, Packaged Applications, Tactical Military, Hardware Appliances, Cloud Computing, others
• Decouple Instrumentation from Analysiso Mix-and-Match best-of-breed solutions
• New or Expanded Uses – Possibilities includeo Testing & Development - Debugging Distributed Applicationso Application Security Monitoring – New or non-standard Usage Alerting,
Intruder Tracing
MIL OSS 2010 13
Vision and Future Objective• Framework is a Foundation, a Starting Point• Full Vision – Complete System for Testing,
Monitoring, and Managing Applications at Runtime• Additional Requirements for Full System
o Metrics Processingo Discovery Mechanismso Visualization Engineo Logging Facilitieso User Portalo Administrative Console
• Ultimate Objective – Full Open Source System, with a Repository for New Contributionso Open Licensingo Open Repository for New Contributions
MIL OSS 2010 14
Notional End-State Functional Architecture
Service EService D
Service C
Service B
Users
Service A C
C
C
C C
Persistence Store /LogsIntermediary Message Processor
UI / Portal and Reporting Engine
Network Switch R
C
R
- Collector- Reflector
Root Cause
Analysis
Correlation EngineMetrics Visualization
MIL OSS 2010 15
Current Research and Development
Service EService D
Service C
Service B
Users
Service A C
C
C
C C
Persistence Store /LogsIntermediary Message Processor
UI / Portal and Reporting Engine
Network Switch R
C
R
- Collector- Reflector
Root Cause
Analysis
Correlation EngineMetrics Visualization
MIL OSS 2010 16
Invitation to Collaborate
• Net Centric Computing is growing Increasingly more Complex
• OSESM Sorely Needed• We Need:
o Transition Partners – Programs or Organizations Interested in using the Results of the efforts of Current Project
o Collaboration Partners – Willing to Fund and/or Contribute to development of Full Vision System
• Interested? Please Speak to me After the Presentation
MIL OSS 2010 17
Wrap Up
• OSESM will Dramatically Improve the ability to Field and Manage Net Centric Applications and Mission Threadso Gather and Analyze Application Tier Data at Runtime
o Proactive Notification when Mission Thread Breaks
−What Broke, Where, How, Why
−At Every Level – Application, Operating System, Runtime Infrastructure, Code
o At Far Lower Costs – Estimated TCO 10% of COTS
• Initial Framework Under Development Today• Transition and/or Collaboration Partners Strongly
Encouraged!
Copyright 2010 Avenue Open, Inc.
John EmersonPresident, Avenue Open, Inc.
Contact Information