database on demand service (dbod)
DESCRIPTION
Database On Demand Service (DBOD). Daniel Gomez Ruben Gaspar Ignacio Coterillo * Dawid Wojcik *CERN/CSIC funded by Spanish Ministry of Research EIC 2010 program. Researcher code SEII-2010-00237 HEPiX Spring 2012 Workshop. Agenda. Objectives Main functionality Architecture - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/1.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Daniel GomezRuben Gaspar
Ignacio Coterillo*
Dawid Wojcik
*CERN/CSIC funded by Spanish Ministry of Research EIC 2010 program. Researcher code SEII-2010-00237
HEPiX Spring 2012 Workshop
Database On Demand Service(DBOD)
![Page 2: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/2.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Agenda
• Objectives• Main functionality• Architecture• Implementation• Current status
2
![Page 3: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/3.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Objectives
• Allow the CERN user community to request and manage database instances
• Give users full DBA privileges on their instances• Offer different DBMS vendors
– MySQL and Oracle are our starters, but the service is designed to scale easily
• The DBOD service takes care of system administration operations and providing upgrade functionality for DBMS
• The DBOD service does not provide DBA or application support
• Users are entirely responsible for ensuring that their systems, and the use of their systems, are fully compliant with the Rules for the use of CERN Computing Facilities.
3
![Page 4: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/4.jpg)
Home page
4
![Page 5: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/5.jpg)
Instance view
5
Star up
Shut down
File mngmt
Backup mngmt
Restore
Upgrade
Monitoring
![Page 6: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/6.jpg)
Backup management
6
![Page 7: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/7.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Backup of a DBOD instance• Based on snapshots.• Full database backup in few seconds.
• How many snapshots are stored? For how long?– Extra 20% space provided for snapshots (e.g.: active file system
100GB, snapshot area 20 GB → total: 120GB)– Depends on DBOD instance activity (e.g.: changing 2% active file
system will allow to have 10 days of snapshots)• Possibility to send to a backup to tape once a week• For MySQL: InnoDB storage engine recommended
7
Snapshot takenB block is changed → B1Snap1 offers a historical view Snapshot taken
After few hours
![Page 8: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/8.jpg)
Restore database
8
![Page 9: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/9.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Restore of a DBOD instance
9
• Based on snapshots– Restore time reduced to a few seconds
• Full restore of whole database• DBOD instances configured with binary logs
– Point-In-Time Recovery
Binary logs
TIME
Active File System
Scheduled snapshots
Manual snapshot
PITR NOW
X
Snapshot space re-gained by the snap reservation area
![Page 10: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/10.jpg)
Monitor
10
![Page 11: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/11.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Monitoring of a DBOD instance
• Service Level Status (SLS)• LEMON• RACMon
• In-house developed system• Easy to adapt to different use cases• Weekly notifications (to be implemented)
• Job monitoring• Queries jobs table every 5 minutes• Informs admins in case of:
• Pending jobs not executed• Timed out jobs• Failed jobs
11
![Page 12: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/12.jpg)
Admin view
12
![Page 13: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/13.jpg)
Architecture
13
https://cern.ch/dbondemand
Web Serviceshttps://syscontrol/dod/ws
Virtual Machines
MySQL
Oracle
…?
Storage network
User/Admin Monitoring
Mgmt Node
Account Management (FIM)https://cern.ch/account
Mgmt DB
Active File System
Binarylogs
Web Server
FIM DB
RACMON DB
ITCORE
![Page 14: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/14.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Web application
• Implementation – J2EE Web Application running on CERN central web servers
• ZK Framework 5.0.10 (Ajax based)• SSO (Single Sign On) + SSL for authentication/authorisation• JDBC + Apache BDCP connection pooling via JNDI• Apache Axis 1.4 for SOAP Web Services• Google Visualization API for graphs
• User operations – Request for a new instance (via FIM)– Instance administration: start/stop, backup/restore, upgrade, configure…– Documentation for users (integrated with TWiki)
• Service Administration operations– Approval of requests (via FIM)– Special collective actions– System administration maintenance, manage upgrades, edit info…– Documentation for admins (integrated with TWiki)
14
![Page 15: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/15.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Implementation
• Perl + IT-DB framework– Main process (daemon):
• Queries jobs table every 5 seconds• FIFO queue per instance• Launches script on instance• Captures output and updates instance
– Worker processes:• Executes a job in a certain instance
– Operations implemented:• ALL: common to any instance• MYSQL: specific to MySQL instances• ORACLE: under development
Basic DBOD core functionality
15
![Page 16: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/16.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Virtualization Platform
• Based on Oracle VM, a Xen-based hypervisor– Required for support of Oracle DBOD instances– Easy to integrate with our IT-DB infrastructure– Keeping track of CERN Agile Infrastructure project
• Each DBOD instance is delivered in a preconfigured virtual machine– 2 cores, 16 GB memory, 64-bit platform
• Comparable to Amazon EC2 Extra Large Instance• Possibility to upgrade depending on user needs
16
![Page 17: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/17.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Current status
• Production services already testing the system and giving feedback– PH-LBC– IT (PES, ES, CIS, OIS)
• More coming– Other DBMS (Oracle in the works)– On-the-fly VM creation
• Possible future developments– High availability with master/slave replication– Multi-instance entities
17
![Page 18: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/18.jpg)
CERN IT Department
CH-1211 Geneva 23
Switzerlandwww.cern.ch/
it
Questions
18
![Page 19: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/19.jpg)
Account management (FIM)
19
![Page 20: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/20.jpg)
File management
20
![Page 21: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/21.jpg)
Upgrade
21
![Page 22: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/22.jpg)
Help page
22
![Page 23: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/23.jpg)
Admin help page
23
![Page 24: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/24.jpg)
MySQL Workbench
24
![Page 25: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/25.jpg)
DB On Demand schema
25
Authentication & Authorization based on requestor principal + e-group
DBOD instance registered in FIM as an application resource. Transfer of ownership also achieved via FIM.
![Page 26: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/26.jpg)
Monitor
26
![Page 27: Database On Demand Service (DBOD)](https://reader035.vdocuments.us/reader035/viewer/2022062501/5681617a550346895dd10a3a/html5/thumbnails/27.jpg)
Restore of a DBOD instance
27
• Based on snapshots– Restore time reduced to a few seconds
• Full restore of whole database• DBOD instances configured with binary logs
– Point-In-Time Recovery
Active File System
Binary logs
DBOD instances
TIME
Active File System
Scheduled snapshots
Manual snapshot
PITR NOW
X
Snapshot space re-gained by the snap reservation area