webmd health corp.: agile system automation, integration, and recovery using hp server automation...
DESCRIPTION
Sign up for this session and find out how the WebMD operations team developed and implemented an automatic system for build delivery and deployment, self-healing, and critical content backup with HP Server Automation, HP Operations Orchestration, Serena Teamtrack, EMC Documentum, and integration with middleware framework software to reduce day-to-day maintenance and administration by 40 percent within 6-month development cycle. Presenters will explain how Server Automation and Operations orchestration automate release deployment and critical content backup and make maintenance-free servers and self-serviceable systems possible. And they’ll tell you how they managed a high-load working environment on time with limited resources while ensuring the high availability of a mission critical system. So please attend, and take away tips and best practices to ensure agile and effective implementation of automation systems within tight budgets.TRANSCRIPT
![Page 1: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/1.jpg)
1 ©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
WebMD Health Corp.: agile system automation, integration and recovery using HP Server Automation and HP Operations Orchestration
– Derek Chang
Manager, WebMD
– Roger Hsu
Manager, WebMD
![Page 2: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/2.jpg)
2Introduction & Agenda
Topics
Who are we and where we stand
Infrastructure Layout
Middleware Integration
HP OO preparation
Application Administration Automation
Build Deployment Automation
Unattended WebMD Content Backup
Maintenance Free System
Results from HP SA/OO Implementation
Q/A
![Page 3: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/3.jpg)
3
cmsops
Responsibility– Provide Maintenance and 24x7 support of CMS applications and their
subsystems in production environment– Perform production system patches, bug fixes or software releases and other
build deployments.– Support ongoing releases and developments in non-production environments– Define/document production support requirements, escalation procedures,
issue tracking and guidelines for troubleshooting and build deployments.Resource: 4.5 headcounts*Universe
– 300+ internal users– SDLC environments: dev/devint/qa00/qa01/qa02/perf/production– 130 servers– 4.4 TB of NAS storage for raw contents and site contents– Infrastructure: Zenoss, HPSA, HPOO, Serena teamtrack, MOSS, MSSQL/Oracle
Core technology– EMC Documentum– Proprietary applications
![Page 4: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/4.jpg)
4
Documentum
An enterprise content management platform, now delivered by EMC Corporation, as well as the name of the software company that originally developed the technology.
Flexible, versatile, powerful yet complex platform
Implementation in WebMD
– 2 major portal sites
– 6 Documentum products
– Proprietary content editor for advanced features
– Proprietary page transformer
– Proprietary utilities: 15 applications
![Page 5: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/5.jpg)
5
Challenges
Documentum is a new technology
Documentum is a rare expertise
Complexity of the CMS
Cmsops support users within the company
WebMD is a fast growing company
![Page 6: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/6.jpg)
6
Life in cmsops
Sampling duration:
Oct 11,2007 – Jul 24, 2009
653 days/426 working days
Source: customized teamtrack
reports and emails
Summary
1772 teamtrack requests
479 email requests*
5.3 tickets/working day
![Page 7: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/7.jpg)
7
Our Approach
Develop and utilize process templates
Standardize and adopt the development model
Identify what processes to be automated
– Routine/mundane activities
– Human interactions cause error/failure
– Much longer Lifecycle/Service time than development time
![Page 8: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/8.jpg)
8
Infrastructure Layout
![Page 9: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/9.jpg)
9
Infrastructure Layout
Opsware OO
Central and RAS
SAS OCLI client
Scheduler engine
NRAS
Web interface
Workflow engine
Repository
JRAS
SAS web services client
Build server
RHEL4u4_32BIT VM
Opsware agent
NAS/Build repository
OCLI 1.0
Web interface
Rpm/msi package tools
Opsware SAS
Central and RAS
Twister
OCLI engine
Web services engine
Web interface
Software repository
Server repository
JAVA API
Opsware agent engine
Middleware integration
Jboss 5.0
XML module
Email adapter
Email sender
Web interface
OO client
Teamtrack client
LDAP module
Data modeling
Corporate infrastructure
Active Directory
Teamtrack
Win2K3
Web interface
Business mashup engine
Web services
PAS LAB
RHEL4u6_64BIT VM
Opsware agent
App server(s)
Code base
QA/DEV Clients
Exchange server
![Page 10: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/10.jpg)
10
Middleware Integration
![Page 11: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/11.jpg)
11
Middleware Integration
Java EE web application
– The core of the automation system
– Connections among ticketing, monitoring, and system administration tools within WebMD operations.
– Allows users to use the automation tools via a web browser over network to prevent access to underlying systems/tools such as HP OO directly
– Developed under JBoss Seam framework with technologies such as AJAX, Richfaces, and Hibernate
![Page 12: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/12.jpg)
12
Middleware Integration
Ticketing system integration
– Use web services to connect Serena Business Mashup (TeamTrack)
– Pull information from tickets and pass data to other systems such as HP OO
– Update tickets after automation operation
![Page 13: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/13.jpg)
13
Middleware Integration
System Administration (HP SA/OO) integration
– Java bean uses OO library to trigger OO workflow
– Parse the workflow result (XML format) to get:
• OO flow id and report URL
• Start time and end time
• OO flow response and result
RSFlowInvoke rsf = new RSFlowInvoke();
rsf.setUrl(url+flowName+paraString);
rsf.setUsername(user);
rsf.setPassword(pw);
result = rsf.invoke();
![Page 14: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/14.jpg)
14
Middleware Integration
Security and User Authorization
– Integrates with WebMD LDAP servers that allows users to access the system with their WebMD id/password
– Security Rules engine provides access control based on WebMD LDAP groups of each user
![Page 15: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/15.jpg)
15
HP OO Preparation
![Page 16: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/16.jpg)
16
HP OO Preparation
Identify basic/out of the box OO operations
– SSH
– Windows Remote Command Execution
– Change IIS status
– Change Windows service status
– OCLI to access HP SA
– Iterator, Email CDO, …etc
– Database operations (oracle/mssql)
Modulization and utility workflows
– Use OO operations to build up utility workflows that will be re-used frequently
![Page 17: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/17.jpg)
17
HP OO Preparation
HostsSSH: run Linux commands in a list of hosts
Given a list of hosts to
Iterator (PAS out-of-box
operation)
SSH Command (PAS
out-of-box operation)
Call Error Notice flow
![Page 18: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/18.jpg)
18
HP OO Preparation
IIS Flows:
– HostIISSites: control multiple IIS Sites on single host
– HostsIISSites: control multiple IIS Sites on multiple hosts
Multiple hosts, multiple sites Single host, multiple sites
Given a list of hosts Given a list of sites
![Page 19: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/19.jpg)
19
Application Administration Automation
![Page 20: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/20.jpg)
20
Application Administration Automation
Goal: Develop OO workflows to stop/start WebMD applications and sites
Workflow key features
– Identify target servers
– Windows: stop/start windows svc and IIS sites
– Linux: stop/start applications and run any script if needed
– Send error/success email notices
![Page 21: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/21.jpg)
21
Application Administration Automation
Users pick available host type and environment based on the permission given to their LDAP groups
Login as consumer
QA user
Consumer users are
NOT allowed to pick
professional hosts
QA users controls QA
environments onlyAction buttons
![Page 22: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/22.jpg)
22
Application Administration Automation
Web application then triggers corresponding HP OO workflow
OO workflows connect HP SA with OCLI
HP SA takes actions on target hosts
![Page 23: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/23.jpg)
23
Application Administration Automation
The OO workflows sends the result back to middleware in XML format
Middleware parses the XML and display the result in GUI
dmas qa00 server
![Page 24: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/24.jpg)
24
Application Administration Automation
Users receive email notices
![Page 25: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/25.jpg)
25
Application Administration Automation
Application Administration workflows:
– Documentum Content Servers
– Documentum Application Servers
– ATS: WebMD proprietary content transformer
– PATS: WebMD proprietary content transformer
– Page Builder: WebMD proprietary content editor
![Page 26: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/26.jpg)
26
Application Administration Automation
WebMD Content Servers
Decision: start or
shutdown
Stop when query
servers only
OCLI Query
Servers based on
portal, product,
host type, and
environment
Initiate variables
based on portalStart/stop SCS
(HostsSSH)
Start/stop JMS
(HostsSSH)
Start/stop doc
base (HostsSSH)
Clean up doc
base (HostsSSH)
Send email notice
when finishes
![Page 27: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/27.jpg)
28
Build Deployment Automation
![Page 28: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/28.jpg)
29
Build Deployment Automation
Goal: Develop an OO workflow to build RPM and deploy it to target servers
Workflow key features:
– Identify target servers, software policy and RPM in HP SA
– Build RPM and upload it to HP SA
– Stop/start applications in target servers
– Detach/attach software policies and remediate target servers
– Update RPM in software policies
![Page 29: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/29.jpg)
30
Build Deployment Automation
Workflow inputs:
– Portal
– Product
– Host Type
– Application
– Environment
– Build Version
![Page 30: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/30.jpg)
31
Build Deployment Automation
Identify target servers
– Setup server groups in HP SA: portal groups, product groups, host type groups, and environment groups; then assign servers to appropriate groups
Host type group
Product group
Portal group
Environment group
![Page 31: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/31.jpg)
32
Build Deployment Automation
Identify target servers (Cont.)
– Use OO SSH operation to execute OCLI command to get SAS server list
• OCLI: findServerRefs and getServerVO in server service
• Filter: Use aforementioned server groups as filter
{device_servergroup_name
equal_to "${portal}"} &
{device_servergroup_name
equal_to "${product}"} &
{device_servergroup_name
equal_to "${hostType}"} &
{device_servergroup_name
equal_to "${environment}"}
Filter String
for i in
`/opsw/api/com/opsware/server/S
erverService/method/.findServer
Refs:i
filter='${filterString}'`;
do
/opsw/api/com/opsware/server/Se
rverService/method/getServerVO
self:i="$i";
done
OCLI command
![Page 32: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/32.jpg)
33
Build Deployment Automation
Identify software policy & RPM
– Software Policy naming in HP SA:
{Application} – {Environment}
– Use findSoftwarePolicyRefs OCLI command to identify software policy
– Use findRPMRefs OCLI command to identify RPM
![Page 33: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/33.jpg)
34
Build Deployment Automation
Build RPM and upload it to HP SA
– Required parameters: application and build version
– A Perl application on Apache to build RPM
– Client sends HTTP request with parameters to trigger the Perl application
– Upload the RPM to HP SA with OCLI 1.0
– Get the result back to the client
![Page 34: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/34.jpg)
35
Build Deployment Automation
Stop/start applications in target servers
– Use aforementioned utility workflows to run stop/start command on target hosts
Detach/attach software policies and remediate target servers
– Use OO out-of-box operations
Update RPM in software policies
– Use OCLI update command in software policy service to replace RPM in target software policy
![Page 35: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/35.jpg)
36
Build Deployment Automation
Put it all together!
Build and upload RPM
Identify SP, RPM,
and target servers
Start/stop application
in target servers
Detach/attach SP,
replace RPM in SP,
and Remediate
![Page 36: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/36.jpg)
43
Unattended WebMD Content Backup
![Page 37: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/37.jpg)
44
Unattended WebMD Content Backup
![Page 38: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/38.jpg)
45
Unattended WebMD Content Backup
Goal: Develop two OO workflows:
1. shutdown all components and backup WebMD contents.
2. bring all components up
Workflow key features:
– Identify target servers
– Windows: stop/start windows svc and IIS sites
– Linux: stop/start applications and run any script if needed
– Send error/success email notices
– Utilize OO scheduler to trigger cold backup
– The workflow needs to setup another schedule to trigger another flow to bring up all components
![Page 39: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/39.jpg)
46
Unattended WebMD Content Backup
Workflows OverviewFlow 1:
1. Shut down all components
2. Run file back up
3. Run DB backup
4. Schedule another flow (flow 2) to start all components
Flow 2:
1. Check backup status
2. Start all components
60 min
![Page 40: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/40.jpg)
53
Maintenance Free System
![Page 41: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/41.jpg)
54
Maintenance Free System
Goal: Proactively maintain the health of our applications without shutting them down
Workflow key features:
– Automatically clear cache and stale data without shutting down or restarting applications
– Purge outdated publishing data and logs
– Ensures that the most relevant information is retained.
– Improves both system-level and publishing performance.
– Minimize the need for frivolous restarts.
– Keep our applications online longer
![Page 42: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/42.jpg)
55
Maintenance Free System
Workflow details
– Single SSH Node
– Runs a script to purge data/log files older than 3 days
– Runs on OO scheduler once a day
![Page 43: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/43.jpg)
56
Results from
HP SA/OO Implementation
![Page 44: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/44.jpg)
57
Better Life in cmsops - 1
Sampling duration:
Oct 11,2007 – Jul 24, 2009
653 days/426 working days
Source: customized teamtrack reports
and emails
Summary
1772 teamtrack requests
479 email requests*
5.3 tickets/working day
Sampling duration:
Jul 25,2009 – Dec 10, 2009
135 days/93 working days
Source: customized teamtrack reports
and emails
Summary:
248 teamtrack requests
35 email requests (reduced by 35%)
3.1 tickets/working day
285 cmsai request (self-service)
![Page 45: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/45.jpg)
58
Better Life in cmsops - 2
Non-prod environments are self-serviceable
15% of build deployment is automated
Automatic/Scheduled data/log purging
Scheduled/unattended cold backup*
![Page 46: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/46.jpg)
59
Challenges
Resource constraint
Relatively high initial cost
Maintenance cost of an automation system is often overlooked
![Page 47: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/47.jpg)
60
Q/A
![Page 48: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/48.jpg)
61 ©2010 Hewlett-Packard Development Company, L.P.
To learn more on this topic, and to connect with your peers after
the conference, visit the HP Software Solutions Community:
www.hp.com/go/swcommunity
![Page 49: WebMD Health Corp.: agile system automation, integration, and recovery using HP Server Automation and HP Operations Orchestration](https://reader037.vdocuments.us/reader037/viewer/2022100304/54b409694a7959d12c8b457e/html5/thumbnails/49.jpg)
62