www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Delivering Oracle’s High Availability Database Services in minutes, a case study on OEM 11G
• Single Server imaging • Gold Image development and deployments • Data Guard services with Observers • Automated Patch Management
MarCn Power, Oracle ACE
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study
• Large Public Sector Organiza=on • Employs 60,000 plus staff • Tradi=onally used Sun E series servers with Solaris
• Goal – To move to a horizontally scalable High Availability database environment
• Database Systems are required to move to 99.999% availability across 24 x 7 x 365 operaCng window with 2 scheduled outage per year, not exceeding 4 hours in total
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study
What was used to achieve these requirements? • Restricted list of SOE Hardware, Intel x86 – 64 bit servers. • OEL 5.5 with hardening • 11.2.0.2 Grid Infrastructure • 11.2.0.2 RAC Database
• OEM 11g Grid Control with the Provisioning Pack
• Bare Metal and Single Server Image provisioning • Gold Image Deployments
• OEM Patch Plans and Patch Management
• Data Guard Automated Failover and Fail back
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Steps 1. Build a single server with OS, Hardening and RPMs and Kernel
Configura=on
2. Build a Bare Metal / Single Server image in OEM. 3. Iden=fy the servers to Deploy to, PXE boot them from the
network to install the Bare Metal Image
4. Deploy the Single server image to the servers
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
!
Bare Metal and Single Server Imaging
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
!
In OEM, Select, 1. Deployments 2. Images 3. Create Image
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
Select 1. OVS Default Image
Enter details; Eg. Name: OEL5_5_x86_64_v1 Descrip=on: Prototype Linux Image Hardened. Product name/Patch number: OEL 5.5 Product version: 5.5
Vendor: Oracle CorporaCon
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
Select the Target Host – e.g grid11g-‐n1.localdomain
! !
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
Configure the remaining Image op=ons RPM Repository: select local-‐repo Root Password/Confirm Root Password Select =mezone: Australia/Brisbane
Deselect Use UTC Clock AddiConally you will set • Username • Group • Password • And enable SUDO Privileges
!
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
!
On the Customize screen, accept the defaults and click Next
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging
!
Finally OEM submits a job to clone Linux OS DONE!
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
Deployment 1. PXE boot the new server(s) 2. Deploy the Bare Metal Image 3. Deploy the Single Server Image
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
PXE Boot the servers
!
Update the /etc/dhcp.conf file on the PXE boot server with the server MAC address host llama0 { hardware ethernet 04:4B:80:80:80:03; fixed-‐address 192.168.0.254; op=on host-‐name "llama0"; filename "pxelinux.0"; }
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
A Single Server Image contains an operaCng system based on the Default Image, together with tailored disk layout, profile and network detail components,
!!
Set Host Preferred CredenCals
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
!
!
!
In OEM Grid Control, click through to the Deployments > Provisioning > Assignments
Create Image Assignment
Single Server Image
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
!
!
1. Select buion beside the New Network
2. Choose Network screen, select the required Network Component
3. Edit Advanced ProperCes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
On the Edit Advanced ProperCes screen, enter the required custom details: Agent Base Installa=on Directory: /opt/oracle Oracle Central Inventory: /opt/oracle/oraInventory Enable SELinux: Permissive Kernel Parameter Sekngs: change the following line from: net.ipv4.conf.all.log_marCans = 1 …to: net.ipv4.conf.all.log_marCans = 0 (This is a bug that will leads to broadcasCng of errors by the network cards) Post Sec=on Scripts: add the following commands: mkdir /nfs mount lgest:/orastage /nfs -‐o nolock cat /nfs/scripts/post-‐install.sh > /etc/init.d/S90post_install cp /nfs/scripts/conf-‐files.tar /opt/oracle/ chmod 777 /etc/init.d/S90post_install ln -‐s /etc/init.d/S90post_install /etc/rc.d/rc3.d/ umount /nfs
!
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
!
!
select the required Hostname
Click the Add buion
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
!
ConfirmaCon screen is shown, indicaCng the image deployment is ready
Now we return to the server and reboot to pick up the Single Server Image.
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
Select OEM Grid Control under Deployments > Provisioning > Assignments; click the Provision
!On the Provision Assignments page:
Set the Schedule Type: One Time (Immediately) Enter (and confirm) the Agent RegistraCon Password
Select the (previously) Assigned Image
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
On the Assignment Status page, click the Schedule’s Job ID
!
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
!
!
!
Press the refresh bugon to watch the Single Server Image provisioning process execute.
Successful comple=on
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
Symptoms: NB: The following issue must be handled immediately once the Provisioning process has been started and must be completed before the server reboots. The single-‐server image tar and directories created by the assignment staging process are located on OMS in the /oracle/stage/Images directory; and comprise an oms.proper=es file, together with a tar file and corresponding sub-‐directories that match the provisioned single-‐server images. These single-‐server image tar/directories are named according to the NIC MAC address associated with the assignment. An example is: oms.proper=es File : 9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308.tar File 9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308 Directory 9DB50A0232DA68E3E04400144FD42308 Directory : An Oracle Service Request (SR) has been raised to resolve an issue with the above oms.properCes file, which is created with missing entries.
Issues
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Bare Metal and Single Server Imaging Deployment
Cause: Fault/bug in the provisioning process. Solu=on: The oms.proper=es file has to be manually updated with the missing details; namely the OMS Host and Port details. As an example, the original file contains: s_encrSecurePwd="05a61eae5282b1219ee52898b7bd189cd077b40996815fde61" …and should be updated (as the root user) to: s_encrSecurePwd="05a61eae5282b1219ee52898b7bd189cd077b40996815fde61" s_OMSHost="qhsz007p.LT-‐test.logicaltech.com.au" s_OMSPort="1159” Once updated, the oms.proper=es file needs to be copied into the single-‐server image’s subdirectory; for example: 9B1CA81DDA4535B2E04400144FD42308_0.2_97D1A147B2AC1E63E04400144FD42308 …overwri=ng the default one created therein by the assignment process. ised to resolve an issue with the above oms.properCes file, which is created with missing entries.
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Gold Image Deployment Sequencing 1. Install 11gR2 Grid Infrastructure and Database as a Single
Node Cluster.
2. Create a Component in OEM (Image of the server)
3. IdenCfy the servers to Deploy to.
4. Deploy the Gold Image to the servers
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
!
Create the Gold Image
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Create the Gold Image
!
IdenCfy the • Host • Grid Home • Grid User & Group • Etc.
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Create the Gold Image
!
!
Job is Created and run
Confirms compleCon
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Deploying the Gold Image
!
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Deploying the Gold Image
!
Select Image and schedule deployment
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Deploying the Gold Image
!
Set the • Cluster Name, • Hosts • Scan DNS Name • Scan Port • Network Bonding Details
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Deploying the Gold Image
!
Set the • Discovery Path • ASM Disk Details • OCR Disk Group
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Deploying the Gold Image
!
A job is scheduled • View the job process
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Comparison of 2 build methods • Build a 3 Node Linux Cluster, 2 to 3 days
• Deploy 3 Node Linux Cluster with Single server Images, 6 – 10 minutes
• Build a 3 Node Grid Infrastructure & Database Cluster, 1 day • Deploy a 3 Node RAC Cluster, 10 minutes
All nodes are idenCcal…. Use DBCA to create RAC databases
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Data Guard Deployment with Observer
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Data Guard Deployment with Observer
!
!
!
From OEM select, • RAC Database • Select Availability tab • Select the Add Standby Database OEM Provisioning runs through all the step just like the Gold Image Deployment
High Availability in Minutes
In 5 minute you will have a single node standby database
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Data Guard Deployment with Observer
!
Convert to Cluster Database: Cluster CredenCals page, enter/select: In the Cluster CredenCals sec=on:
Oracle Home: for example /u02/app/oracle/product/11.2.0.2/db_1 Username (of the o/s Oracle so@ware owner): oracle Password: (as required)
In the ASM CredenCals sec=on: Username (of the SYSDBA user running the ASM instance): sys Password: (as required) Connect As: SYSDBA Tick the Save as Preferred CredenCals checkbox
!
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Data Guard Deployment with Observer
dgmgrl sys/[password]
show database verbose tadbsb;
Database -‐ tadbsb Role: PHYSICAL STANDBY Intended State: APPLY-‐ON Transport Lag: 0 seconds Apply Lag: 0 seconds Real Time Query: OFF Instance(s): tadbsb_1 (apply instance) tadbsb_2 Proper=es: DGConnectIdenCfier = '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = qhrac2-‐002t-‐v)(PORT = 1521))(CONNECT_DATA = (SERVICE_NAME = tadbsb)))' ObserverConnectIden=fier = '' LogXptMode = 'ASYNC' DelayMins = '0'
Issues
Host address is wrong It should be the SCAN address
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Patching (Out of Place Patching)
High Availability in Minutes
Clone the Oracle Home
tar -‐cf -‐ * -‐-‐exclude=*.aud | (cd ../db_2;tar -‐xvf -‐ ) Update the OraInventory ./runInstaller -‐silent -‐clone ORACLE_HOME="/u02/app/oracle/product/11.2.0.2/db_2" ORACLE_HOME_NAME="OraDb11g_home2" ORACLE_BASE="/u02/app/oracle" Amend the database home details held in cluster services srvctl stop database -‐d tadbnew srvctl modify database -‐d tadbnew -‐o /u02/app/oracle/product/11.2.0.2/db_2 srvctl start database -‐d tadbnew
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Patching (Out of Place Patching)
High Availability in Minutes
Create Patch Plans
Patch Cache Maximum Size (MB): 700 Use Dump Directory Radio check buion set to Yes /var/tmp/
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Patching (Out of Place Patching)
High Availability in Minutes
Awer selec=ng the patches press the upload buion The patches are then staged and a success message returned Now click the apply to schedule the job that runs the patching process
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Patching (Out of Place Patching)
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
High Availability in Minutes
Schedule the applicaCon of the patch set.
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
High Availability in Minutes
Schedule the applicaCon of the patch set.
• Set the details for the patch type • Select the patch sets
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
High Availability in Minutes
Schedule the applicaCon of the patch set.
• IdenCfy the databases to patch • Watch the job execute
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study Patching
• Based on Agreed Patch Plan Standards • Rolling Hot Patch ApplicaCon • 3 Data Centers
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Case Study 1 Patching Sequence
Primary Database Standby Database
Normal Opera=on
Primary Database
Create a third at the primary data center
(Alternate Servers) 1
2
3
Standby Database
Switch to DR site 4 Patch the databases cold
Standby synchronizes
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Primary Database Standby Database
Normal Opera=on
Primary Database Create a third Standby
at the primary data center
(Alternate Servers)
6
5
7
Standby Database
Switch back to original Primary
9
Patch the databases cold
Standby synchronizes
8
Drop the temporary Standby database
11 Drop the temporary Standby database
10 Resynchronize the standby
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Summary
Case Study Benefits • Build 3 Linux Cluster Nodes, 2 to 3 days ($2500 -‐ $4000)
• Deploy 3 Linux Nodes with Single server Images, 6 – 10 minutes ($25.00)
• Build a 3 Grid Infrastructure & Database Cluster, 1 day ($1400) • Deploy a 3 Node RAC Cluster, 10 minutes ($25.00)
• The hardware and licensing costs delivered a 30% to 50% saving on tradiConal costs with 3 5 x 9’s of availability
High Availability in Minutes
www.logicaltech.com.au
1300 858 117
Expert@Oracle www.logicaltech.com.au
1300 858 117
Business Solu=ons Guaranteed Service Availability Fixed Monthly Cost Scales with you business
www.logicaltech.com.au
1300 858 117
www.logicaltech.com.au
1300 858 117