db-14: sometimes a great notion: breaking the logjam glen west software architect, earthlink
TRANSCRIPT
2 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Sometimes A Great Notion
What is EarthLink? One of the world’s largest ISPs 5.4 million subscribers 1.4B annual revenue, 1.4B market cap Received the highest ranking for
customer satisfaction for both high-speed (3rd year) and dial-up Internet services by J.D. Power and Associates in its 2004 Internet Service Provider Residential Customer Satisfaction Study - Sept. 2004
PC Magazine’s Readers’ Choice Award for both high-speed and dial-up services - Sept. 2003
Breaking the EarthLink Logjam
3 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Sometimes A Great Notion
Who is Glen West? Software Architect, EarthLink (Aug
2004) Worked with Progress since 1989,
V5 Old Movie Buff
Why “Sometimes A Great Notion”?
Imagery Logging Metaphor
Breaking the EarthLink Logjam
4 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
5 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
6 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Why Are We Here?
Updating antique applications Improving infrastructure Providing developer tools Positive encouragement for standardization WITHOUT upgrading everything to the
bleeding edge
What’s in it for ME?
7 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
8 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The EarthLink Environment True 24x7 (with a maintenance window) Production in Atlanta, DR in Pasadena Points of Presence (POP) everywhereeverywhere Employees and contractors worldwide
(8000 support calls per hour, 2 per second) 1600 servers, 6M customers, 1.6T of data Progress, Oracle, SQL Server, WebSpeed,
Java, JBOSS, Netcool, Foglight, ChUI, Web 50 developers writing code daily Frequent acquisitions, 10G of ASCII logs per DAY10G of ASCII logs per DAY
9 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
10 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Scope of the Problem 50 developers. How many ways of writing
log file information? 200+ No one can make use of 10G of log files a
DAY Heisenburg principle. Our logging is
choking the disks, 100% CPU and disk IO Log information comes from all over,
different systems and processes All of the applications have their own
logging locations and methods
11 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
12 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Objectives If it isn’t simple, they won’t come Has to support multiple destinations and
types (distribute/consolidate) 80/20 – go for the quick win, then extend Manageable at runtime
(destination/activation) Standardized Efficient Scaleable
13 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
14 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Architecture Persistent Session-Services procedure Simple Publish/Run commands AppServer™ for consolidation/distribution Database tables to control destination
16 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Destinations
App Procedure
Session Services
Procedure
AppServer Log Manager
Database
TablesEmail
Netcool
Foglight
Session Log
Central Logsyslogd
17 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
18 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Tools Foglight® (Quest) – monitors ASCII log files
http://www.quest.com/
Netcool® (Micromuse) – records system eventshttp://www.micromuse.com/products_sols/
Syslogd (unix std) logs to OS logs Progress® AppServer eMail
19 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
20 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Construction Persistent Session-Services procedure AppServer Log Event Manager WebSpeed® Plugin
– Web context management The Slight-of-Hand naming trick
21 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Session Services Procedure Cache Destinations Cache status of/connection to AppServer Handle independent Public vs. Private procedures Difference between PUBLISH/SUBSCRIBE
and running a SUPER-PROCEDURE. Fault tolerant – AppServer may not be
available Structure to provide additional services to
developers
22 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
How Does Publish/Subscribe Work? I
Progress Session (“Agent”)
Program A(Session Svcs)
Program B(Entry)
Program C(Dialog Box)
Event Name Subscriber Target
Named Event Table
23 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
How Does Publish/Subscribe Work? II
Progress Session (“Agent”)
Program A(Session Svcs)
Program B(Entry)
Program C(Dialog Box)
Event Name Subscriber Target
Named Event Table
Launch Persistent
24 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
How Does Publish/Subscribe Work? III
Progress Session (“Agent”)
Event Name Subscriber Target
Get Preference Session Svc Anywhere
Set Preference Session Svc Anywhere
Named Event Table
Program A(Session Svcs)
Program B(Entry)
Program C(Dialog Box)
Publish “Get-Preference” (“background color”, output w-bg-color, output w-success).
Run get-preference in hSessionSvc (“background color”, output w-bg-color, output w-success).
Run
Pro
xy
Publish
Run ProxyP
ublish “Get P
reference”
25 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
How Do Super Procedures Work?
Propath
Progress Session
Procedure xxx:
End.
Procedure xxx:
End.
Procedure xxx:
End.
xxx.p
App.pApp-svc.p
Sess-Svc.p
Run App-Svc.p persistent set hAppSvc.
this-procedure:add-super-procedure(hAppSvc).
RUN xxx.
Session:add-super-procedure()
26 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Super-Procedure - The Little Trick When you say “run xxx” where does Progress
look for x?– Internal procedure xxx IN THIS-PROCEDURE– Internal procedure xxx IN THIS-PROCEDURE:SUPER-
PROCEDURES– Internal procedure xxx in SESSION:SUPER-PROCEDURES– xxx.r in PROPATH.
Run log_manager (message)…– log_manager.p runs sessionsvc.p persistent– Sessionsvc.p defines internal procedure log_manager– Sessionsvc.p makes itself a session:super-procedure
The next Run log_manager will run the super-procedure
27 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Pub/Sub vs. Super Procedure
Publish/Subscribe Pros
– Handle-less– Fire and Forget– Multiple Listeners– No errors in handoff
Cons– Multiple Listeners– No clue no-one is
listening See extras for 2003
Pub/Sub presentation
Super-procedures Pros
– Handle-less– Only one listener– Order of succession
Cons– Throws errors if no
listener– Can be “hard-error”
Pros and Cons
28 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Other Tricks How do you ensure only one services
procedure is running?– PUBLISH/SUBSCRIBE heartbeat
PUBLISH returning output parameters
29 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The AppServer Manager Why use the AppServer at all? It’s all about EfficiencyEfficiency Database Logging – Outside the
transaction Email – Asynchronous Cache messages bound for ASCII files Gatekeeper to ensure integrity of writes Security issues Stateless Future: multi-threaded TCP socket listener
30 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
WebSpeed Plugin Part of the startup of the agents:
pros and cons Why it is even more necessary in
WebSpeed? How do you keep messages separated by
end-user (context management)?
31 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
32 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Result ASCII log volume reduced by x%,
continuing to decrease daily Foglight enabled on critical logs to alert on
critical events Netcool Dashboards now receiving real-
time data on vendor polling results. Disk IO reduced x% WebHitLog can now be relied upon
Statistics not available at press time, will be provided at conference
33 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
Agenda Why Are We Here? The EarthLink Environment The Scope of the Problem The Objectives The Architecture The Tools The Construction The Result The Lessons Learned The Questions
34 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
The Lessons Learned Have accurate numbers on your
environment before starting When to get visibility Give them something to ‘fix’ Incrementalism is both good and bad Everyone wants the process improved, but
no-one has the time to improve it Master of the obvious: plan for the future
35 DB-14: Sometimes A Great Notion© 2005 Progress Software Corporation
In Summary It IS possible to
change the infrastructure of an aging application without a rewrite
Large gains can be made with small, incremental efforts as long as you have a plan
Things flow better with a little bit of organization