ecotools case study : database monitoring at bnp paribas
DESCRIPTION
Ecotools Case Study : Database Monitoring at BNP Paribas. Dennis Adams BNP Paribas. What’s in a name : Paribas Capital Markets. Now part of BNP Paribas Investment Banking € Euro-Denominated Bonds. Equities, Bonds and Derivatives Trading Systems in most major financial centres - PowerPoint PPT PresentationTRANSCRIPT
Ecotools Case Study : Database Monitoring at
BNP ParibasDennis Adams
BNP Paribas
What’s in a name : Paribas Capital Markets
Now part of BNP Paribas Investment Banking € Euro-Denominated Bonds.Equities, Bonds and Derivatives Trading
Systems in most major financial centres Infrastructure in territories and central
London Data Centre.Finance and Reporting in Paris.
Databases and OS Types in Production
HP/UX DecUNIX Solaris NT 4 TOTALSybase 8 16 8 5 37Ingres 2 39 0 0 41TOTALS 10 55 8 5 78
Tokyo, Singapore, Hong Kong, Paris, London, New York
Our Responsibility: Data Management Group
Managing DBMS ServersBased in London, remote Support to
territories (out-of-hours callouts !)Liaise with London Operations 24h/dayOther Teams
Systems Management - hardware & OSNetworks - LAN & WANApplication Teams - Application Support
Objectives when purchasing Ecotools
Monitoring of Sybase DBMS detail down to SQL statementIngres as an additional requirement
ReliableWAN based Event AlertingLinks to TNG for Central AlertingExtract trend data for capacity planning
How we use Ecotools
Command Line Control Files for Scenarios
GUI
Monitored Agent Machines
Ecotools Repository
Alerts
How we use Ecotools
Single Central Solaris Console (V6.2.1)Continuous running 24 X 7Minimal use of the Ecotools GUIScenarios controlled from ECOCLI
start/stop UNIX shell scripts.
Alerts picked up by “tail” of ecotools logsummary on daily in-house web page
Configuring Domains
Domain = Logical grouping of ServersUnit of “discovery” within EcotoolsMonitor “All Servers” together
Configuring Domains
Group by Business Unit / DBMS Type / Territory ?
Lots of small Domains ?speed up “discovery”
Few no of large Domains ?easier to get to individual machines
COMPROMISE: DOMAINS OF MAX 25ORGANISED BY APPLICATION TYPE
(ALMOST)
Creating Scenarios
Scenario = Basic Collection Task One or more agents (cache hit, log size)One or more machines (“All Servers”)Time Interval (“10 Minutes”)
Either using the GUI or from .CTL fileCREATE USING GUI, SAVE AS A CTL
TEMPLATE... SED/AWK START FROM UNIX SHELL
Managing Scenario Control Files
Lots of Small Control files ?One UNIX impf process per scenarioa single machine can have 15 “request”s80 *15 impf processes on console (!!!)ps -ef | more
Few no of Large Control files ?Unix VI ctl file when new machine added.Lots of machines per ctl file makes restart a
problem
Managing Control Files
EVERY MACHINE HAS IT’S OWN CONTROL FILES (MAXIMUM OF 3)Use SED string substitution to created from
standard TemplatesEcotools Startup logic...
For file in *.ctldo
ecoclieco: run $file
done
GUI Stability Issues
GUI “hangs” when Alerts arrive.Determined by ECOCLI_BLINKALARMECOCLI_ALARMINTERVAL =1200 default =
2*CliLogInterval
The GUI Crashes !Corrupts master imdb - lose all dataBetter in version 6.2.1 (latest patches)
SWITCH OFF ECOCLI_BLINKALARM, USE UNIX SCRIPT TO TAIL LOG FILE
Managing Ecotools “Views”
View = Repository for performance dataconsists of indexed/ flat file directory
Need X months data - capacity planningCan keep views going for several weeks
but “imdb” = 12MB, .datalog = 1.8GB.
CREATE NEW VIEW EVERY MONTHSET “NO OF DATA POINTS=1500” IN
CONTROL FILES (OR LOSE DATA)
Trend Data for Capacity Planning
Several Month’s data requiredStart a new GUI for every single view ??SOLUTION : extract data to CSV files for
reading into databaseThere is no supported utility to do this.SHELL SCRIPTS - GREP / SED / AWK
UNSUPPORTED HACK !
Unicenter/TNG Integration
In-house implementation calling KSH script from Ecotools Scenario Language
Can be implemented as “MgrAction” or “AgtAction”
MgrAction = run on Solaris ConsoleGMRPY_CDBSTOP errors - not scalable
AgtAction = run on Agent MachineNOT YET PROVEN
Summary : Ecotools Experience so far
Evaluation of version 4.n in 1997Monitoring of Euro changeover in 1998Y2K tracking and Reporting 1999Version 6.1 chosen as Y2K versionSoon replaced by 6.2.1 with patches.Currently Monitoring nearly 80
Production Systems from London.
Looking at Ecotools V.7
Looks impressive on paperSQL server storing collected dataGenuine 3-layer architectureBetter User interface
Concerns:Support for Legacy Operating SystemsDec UNIX versionsAbility to create own “shell script” agents.Earlier Versions of Sybase
Ecotools Future at BNP Paribas
Evaluate V7 for monitoring NT/Sybase Production Systems.
Need UNIX shell agents for NT console Objective: move to centralised NT
console - eventuallyKeyword: STABILITY
Success Stories
Sybase Settlement System - TokyoReserved log space low
caused by backup server failing and transaction logs not being properly dumped. Restarted backup server
Potential system hang avertedDelayed Settlement = We get Fined
Success Stories
Ingres Equity Derivative Trading - New York
Large Table Approaching Ingres Architectural Limit of 2GBPlanned outage to remodify table into
multiple locations to allow it to grow
Equity Trading Outage avertedKeep the Traders Trading
Success Stories
Ingres Forex Trading - ParisIngres Error Log reported potential
UNIX Filesystem Error one eveningAlerted Paris Unix Team, who decided
to ignore the warningSystem Crashed overnightBut at least we could say “I told you so”
Final Proof
Accepted within DMG GroupIntegrated with our way of workingWe are now a more Pro-Active TeamLooks good on the CV, provided you
call it... E-COTOOLSLunch Time ?