building an oracle grid with
TRANSCRIPT
-
8/3/2019 Building an Oracle Grid With
1/44
-
8/3/2019 Building an Oracle Grid With
2/44
ABOUT THE AUTHORS
2
Kai Yu is a Senior System Engineer and works within the Dell OracleSolutions Lab. He has over 14 years Oracle DBA and SolutionsEngineering experience. His specialties include Oracle RAC, Oracle EBSand OVM. He is an avid author of various Oracle technology articles and
frequent presenter at Oracle Open World and Collaborate. He is also theOracle RAC SIG US Event Chair.
David Mar is an Engineering Manager for the Dell | Oracle Solutions group.He directs a team across the globe to create Oracle RAC referenceconfigurations and build best practices based upon tested and validated
configurations. He has worked for Dell since 2000 and has a MS inComputer Science and BS in Computer Engineering from Texas A&MUniversity.
-
8/3/2019 Building an Oracle Grid With
3/44
3
COMPONENTS WITHIN THE GRID
-
8/3/2019 Building an Oracle Grid With
4/44
PhysicalView
Logical View
-
8/3/2019 Building an Oracle Grid With
5/44
PhysicalView
Scalability
dom1
dom2
dom3 dom4
OVMServers
Scale outblades
Database 1
Database 2
Database 3
Database 4
Volume 1
Volume 2
Volume 3
-
8/3/2019 Building an Oracle Grid With
6/44
OS
Compute Virtualization Roles / Components
Oracle VM Partitioning single server Consolidation
Oracle RAC Scale-up through
scale-out of hw DB Scalability only
OS
OS OS OS
OS OS
DB /app
OS
DB /app
Virtual Grid Physical Grid
VM Server Pool
OS
OS
DB 1
DB 2
DB 3
RAC DB
-
8/3/2019 Building an Oracle Grid With
7/44
Testing Oracle RAC infrastructure with minimalhardware
Testing Oracle RAC pre-production areas
App Servers testing and production
Good to maximize server utilization rates Consolidation for low utilization servers
Partitioning one application from another
OVM | VIRTUALIZATIONS ROLE
-
8/3/2019 Building an Oracle Grid With
8/44
OVERVIEW OF EQUALLOGIC
4 Gigabit network ports per controller
2 GB cache per controller
Intelligent automated management tools
Self Managing Array
Linear Scalability
Data Protection Software
Volume 2
Volume 3
Volume 1
Storage Pool
-
8/3/2019 Building an Oracle Grid With
9/44
STORAGE LAYER ON EQUALLOGIC
EqualLogic and ASM work well togetherhttp://www.dell.com/oracle
Performance Implications of running Oracle ASM with Dell EqualLogicTM PS series storage
EqualLogic ASM Best Practices for Oracle
http://www.dell.com/oracleBest Practices for Deployment of Oracle ASM with DellTM EqualLogicTM PS iSCSI Storage System
Peer Storage architecture
Setup Quickly
Linear Performance Improvements
Volume 2
Volume 3
Volume 1
Storage Pool
http://www.dell.com/oraclehttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/oraclehttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/oraclehttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Performance_Implications_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/oracle -
8/3/2019 Building an Oracle Grid With
10/44
SERVER LAYER ON BLADES
Oracle
RACServers
Front View Back View
Energy efficient PowerEge M100e enclosure
11th
generation blade servers availableIntel Nehalem processorsIntel QuickPath memory technology
Three redundant fabrics16 half height blade servers
-
8/3/2019 Building an Oracle Grid With
11/4411
GRID ReferenceConfiguration POC Project
-
8/3/2019 Building an Oracle Grid With
12/44
1 PHYSICAL GRID
-
8/3/2019 Building an Oracle Grid With
13/44
2 THE VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
14/44
3 MANAGEMENT
Oracle Grid Control to manage bothphysical Grid and virtual Grid
Consider HA for production management Manage externally out of the Blade Cluster,
to avoid single point of failure Dell Servers enable health monitoring from
EM
-
8/3/2019 Building an Oracle Grid With
15/44
4 - STORAGE
-
8/3/2019 Building an Oracle Grid With
16/4416
GRID IMPLEMENTATION
-
8/3/2019 Building an Oracle Grid With
17/44
-
8/3/2019 Building an Oracle Grid With
18/44
GRID IMPLEMENTATION: EQUALLOGICSTORAGE
18
EqualLogic shared storage configurationStorage volumes for the physical Grid
Storage volumes for the virtual Grid
Volume Size Raid Used for OS Mapping
blade_crs 2GB 10 OCR/Votingdisk Yes
blade_data1 100GB 10 Data for DB1 ASM diskgroup1
blade_data2 100GB 10 Data for DB2 ASM diskgroup2
blade_data3 150GB 10 Data for DB3 ASM diskgroup3
blade_data5 150GB 10 Data for DB4 ASM diskgroup5
Volume Size Raid Used for OS Mapping
blade_data4 400GB 10 VM Repositories /OVS
blade_data6 500GB 10 VM Repositories/OVS/9A87460A7EDE43EE92201B8B7989DBA6
vmcor1-5 1GB each 10 OCR/vogtingdisk 2XOCRs/3Xvotingdisks
vmracdb1 50GB 10 Data for RAC DB ASM diskgroup1
vmracdb2 50GB 10 Data for RAC DB ASM diskgroup2
-
8/3/2019 Building an Oracle Grid With
19/44
GRID IMPLEMENTATION: EQUALLOGICSTORAGE
19
EqualLogic storage management console
-
8/3/2019 Building an Oracle Grid With
20/44
GRID IMPLEMENTATION: PHYSICALGRID
20
Physical Grid Configuration : 8-node 11g RAC
Network configuration
iSCSI storage connections
Open-iSCSI administration utility to configure host access to storage
volumes: eth2-iface on eth2, eth3-iface on eth3
Linux Device Mapper to establish mutipath devices to storage alias:/etc/multipath:multipaths {
multipath { wwid 36090a028e093dc7c6099140639aae1c7
alias ocr-crs
}
Check multipath devices in /dev/mapper: /dev/mapper/ocr-crsp1
Network Interface IO modules Connections IP address
eth0 A1 Public Network 155.16.9.71-78
eth2 B1 iSCSI connection 10.16.7.241-255(odd)
eth3 B2 iSCSI connection 10.16.7.240-254(odd)
eth4,eth5 C1,C2 Bonded to bond0 192.168.9.71-78
VIP Virtual IP for 11g RAC 155.16.9.171-178
-
8/3/2019 Building an Oracle Grid With
21/44
GRID IMPLEMENTATION: PHYSICALGRID
21
Use block devices for 11g Clusterware and RAC databases OEL 5 no longer supports raw devices,
Use block devices for 11g clusterware/RAC databases
/etc/rc.local to set proper ownerships and permissions
Configure 11g RAC database Infrastructure
11g RAC clusterware configuration:private interconnect : bond0: 192.168.9.71-78
2 x OCRs and 3 X votingdisks on mutipath devices
11g ASM configuration:
ORA_ASM_HOME =/opt/oracle/product/11.1.0
ASM instances provide the storage virtualization 11g RAC software to provide the database service
ORA_HOME=/opt/oracle/product/11.1.0/db_1
Grid control agent 10.2.0.5 to connect to OMS server
Consolidate multiple databases
-
8/3/2019 Building an Oracle Grid With
22/44
GRID IMPLEMENTATION: PHYSICALGRID
22
Sizing the DB requirement: CPU, IO and memory Determine storage needs and # of DB instances
Determine which nodes to run the database
Provision the storage volumes and create the ASM diskgroup
User DBCA to create the database using the ASM diskgroups
For some ERP applications, convert the installed DB to RAC Examples of Pre-created Databases
-
8/3/2019 Building an Oracle Grid With
23/44
GRID IMPLEMENTATION: PHYSICALGRID
23
-
8/3/2019 Building an Oracle Grid With
24/44
GRID IMPLEMENTATION: PHYSICALGRID
24
Scale out the physical Grid Infrastructure
Consolidate more databases to the Grid
Increase capability of the existing databases
Scale out the storage:
a. add additional Equallogic array to the storage group
b. expand existing or create new volumes
c. make new volume accessible to the serversd. create ASM diskgroup or add partition to the existing
diskgroup or resize the partition of an diskgroup [1]
Scale the Grid by adding servers: Use 11g RAC add nodeprocedure to add new node to the cluster and add the
Expand the database to additional RAC nodes:
Use the adding instance procedure of enterprise manager toadd new database instance to the database [2]
Dynamically move the database to less busy node:
User the adding instance and removing instance of enterprise
manager to add new database instance to the database [2]
-
8/3/2019 Building an Oracle Grid With
25/44
GRID IMPLEMENTATION: VIRTUAL GRID
25
Implementation Tasks Virtual servers Installation
Virtual server network and stroage configuration.
Connect VM servers to the Grid Control
Management VM infrastructure through Grid control
Create guest VM using VM template Manage the resources of guest VMs.
Virtual servers Installation
Prepare local disk and enable virtualization on BIOs
Install Oracle VM server 2.1.2
Change Dom0 memory : /boot/grub/menu.lst:
edit line: kernel /xen-64bit.gz dom0_mem=1024M
Ensure VM agent working: #service ovs-agent status
O C VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
26/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
26
Oracle VM server network configuration
eth0 : public in dom0, presented to domU through xenbr0
eth2, eth3: iSCSI connections
eh4 and eth4 bonded to bond0 for the private interconnectbetween VMs for 11g RAC configuration through xenbr1
Secure OVM agent is running: service ovs-agent status
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
27/44
GRID IMPLEMENTATION: VIRTUAL GRID
27
Customize the default Xen bridge configuration
a. Stop the default bridges/etc/xen/scripts/network-bridges stop
b. Edit /etc/xen/xend-config.sxp:
replace line: (network-script network-bridges) with
(network-script network-bridges-dummy)a. Edit /etc/xen/scripts/network-bridges-dummy : /bin/true
b. Create infcg-bond0, ifcfg-xenbr0, ifcfg-xenbr1 scripts in
/etc/sysconfig/network-scripts
dom0 IO modules connection IP domU
eth0 A1 Public Network 155.16.9.71-78 eth0
eth2,eth3
B1, B2 iSCSI connections 10.16.7.238-235
eth4,eth
5C1,C2 Bonded to bond0 192.168.9.82-85 eth1
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
28/44
GRID IMPLEMENTATION: VIRTUAL GRID
28
ifcfg-eth0: ifcfg-eth4 ifcfg-bond0 ifcfg-xenbr0 ifcfg-xenbr1DEVICE=eth0 DEVICE=eth4 DEVICE=bond0 DEVICE=xenbr0 DEVICE=xenbr1
ONBOOT=yes TYPE=Ethernet ONBOOT=yes ONBOOT=yes ONBOOT=yes
BOOTPROTO=none MASTER=bond0 BOOTPROTO=none TYPE=Bridge TYPE=Bridge
BRIDGE=xenbr0 USERCTL=no BRIDGE=xenbr1 IPADDR=155.16.9.82 TYPE=Bridge
ONBOOT=yes NETMASK=255.255.0.0 NETMASK=255.255.255.0
restart network services: #service network restartcheck Xen bridges: [root@kblade9 scripts]# brctl showbridge name bridge id STP enabled interfaces
xenbr0 8000.002219d1ded0 no eth0
xenbr1 8000.002219d1ded2 no bond0
Configure shared storages on dom0 for VM servers For OVM repositories : blade_data4: 400GB, Blade_data6_ 500GB
For 11g RAC shared disks: OCRs, votingdisks, ASM diskgroups
Configure iSCSI and multipath devices on dom0
Use iSCSI admin tool and Linux DM as the physical serverCheck the multipath devices : ls /dev/mapper/*
/dev/mapper/blade-data6 /dev/mapper/mpath5 /dev/mapper/ovs_data4p1
/dev/mapper/blade-data6p1 /dev/mapper/vmocr-css1 /dev/mapper/vmocr-css1p1
/dev/mapper/vmracdb1 /dev/mapper/vmracdb1p1 /dev/mapper/ovs_data4
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
29/44
GRID IMPLEMENTATION: VIRTUAL GRID
29
Convert OVM repository to the shared disks: configure OCFS2
a) Edit /etc/ocfs2/clutser.conf b) service o2cb stopC)
Fcb
d) # mkfs.ocfs2 -b 4k -C 64k -L ovs /dev/mapper/ovs_data4p1e) umount /OVS
f) edit /etc/fstab:#/dev/sda3 /OVS ocfs2 defaults 1,0
/dev/mapper/ovs_data4p1 /OVS ocfs2 _netdev,datavolume,nointr 0 0
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
30/44
GRID IMPLEMENTATION: VIRTUAL GRID
30
g) mount ocfs2 partitions: mount -a -t ocfs2h) create directories under /OVS: running_pool, seed_pool,sharedDisk
i) repeat steps a-h on all the VM servers
Add additional volume to the OVS repositories
Connect the VM servers to Enterprise Manager Grid Control
Meet pre-requisites:a. Oracle user with oinstall groupb. Oracle users ssh user equivalence between dom0 & OMS
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
31/44
GRID IMPLEMENTATION: VIRTUAL GRID
31
c. Create /OVS/provxy
d. Oracle user sudo privileges: visudo /etc/sudoersAdd line: oracle ALL=(ALL) NOPASSWD: ALLComment out line: Defaults requiretty
Create VM server Pool: login to Grid control
GRID IMPLEMENTATION VIRTUAL GRID
-
8/3/2019 Building an Oracle Grid With
32/44
GRID IMPLEMENTATION: VIRTUAL GRID
32
Add additional VM servers to the VM server pool
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
33/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
33
Create Guest VMs using VM template
Virtual Machine as node of Virtual Grid
Methods of guest VMs: VM template; install media; PXE boot
Import a VM template
Download OVM_EL5U2_X86_64_11GRAC_PVM.gz to repository
Discover VM template from Grid control Virtual Central
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
34/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
34
Create VM using the template
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
35/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
35
Add additional disk to the VM
Attach shared disk to the VM
s
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
36/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
36
Images files in repository:Local disk:
Shared disk
Disks as resources for VM:
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
37/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
37
Disks shown in VM as the virtual disk partitions:
Attached the storage to to the guest VMa) vm.cfg: disk = [''file:/OVS/sharedDisk/racdb.img,xvdc,w!',b) vm.cfg: disk = ['phy: /dev/mapper/vmracdbp1, xvdc,w!',
Exposed Xen bridge for virtual network interfacevm.cfg:vif = ['bridge=xenbr0,mac=00:16:3E:11:8E:CE,type=netfront','bridge=xenbr1,mac=00:16:3E:50:63:25,type=netfront',]xenbr0 - eth0 in the guest VMxenbr1 eth1 in the guest VM
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
38/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
38
Storage/Network configuration for 11g RAC on VMs:
ORACLE VIRTUALIZATION
-
8/3/2019 Building an Oracle Grid With
39/44
ORACLE VIRTUALIZATIONINFRASTRUCTURE IMPLEMENTATION
39
Consolidate enterprise applications on the Grid Applications and middleware on the virtual Grid
Create a guest VM using Oracle OEL 5.2 template
Deploy application on the guest VM
Build the VM template of the VM
Create new guest VMs based on the VM template
Deploy database service on the physical Grid
Provision adequate size of storage volume from SAN
Make the volume accessible to all the physical Grid Nodes
Create the ASM diskgroup
Create database service on the ASM diskgroup
Create application database schema on the database Establish the application database connections
Deploy DEV/Test Application suite on the virtual Grid
Multi-tier nodes are on the VMs
Fast deployment based on templates
CASE STUDIES OF GRID HOSTING
-
8/3/2019 Building an Oracle Grid With
40/44
CASE STUDIES OF GRID HOSTINGAPPLICATIONS
40
Grid Architecture to host applications Oracle Applications E-Business suites on RAC/VM
Three Applications Tier nodes on three VMs
Two Database Tier nodes on Two node RAC.
Oracle E-Business Suite single node on Virtual Grid
Both Applications tier and Database tier node on VMs
Banner applications on physical/virtual Grid
Two Applications Tier nodes on two VMs
Two Database Tier nodes on Two node RAC
Provisioning Oracle 11g RAC on Virtual Grid
Two Database RAC nodes on two VMs
11g RAC provisioning by Grid control Provisioning Pack
11g RAC provisionging procedure
Add additonal RAC node on new VM Grid control ProvisioningPack Add node Procedure
CASE STUDIES OF GRID HOSTING
-
8/3/2019 Building an Oracle Grid With
41/44
CASE STUDIES OF GRID HOSTINGAPPLICATIONS
41
Grid Architecture to host applications
CASE STUDIES OF GRID HOSTING
-
8/3/2019 Building an Oracle Grid With
42/44
CASE STUDIES OF GRID HOSTINGAPPLICATIONS
42
Virtual Grid shown on Grid Control Virtual Central
-
8/3/2019 Building an Oracle Grid With
43/44
SUMMARY
43
Dell Grid POC Project: Pre-built Grid with physical and virtual grids
Grid control as the unified management solution for the Grid
Dell blades as the platform of the Grid: RAC node and VM servers
EqualLogic provides the shared storage for the physical and virtualGrid
The Grid provides the infrastructure to consolidate the enterpriseapplications as well as the RAC databases
Acknowledge the support of Oracle engineers: Akanksha Sheoran,Rajat Nigam, Daniel Dibbets, Kurt Hackel, Channabasappa Nekar,Premjith Rayaroth, and Dell Engineer: Roger Lopez
Related OpenWorld Presentations:
ID#: S308185, Provisioning Oracle RAC in a Virtualized Environment,Using Oracle Enterprise Manager, 10/11/09 13:00-14:00, Kai Yu & RajatNigam
ID#: S310132, Oracle E-Business Suite on Oracle RAC and Oracle VM:Architecture and Implementation, 10/14/09 10:15 -11:15 Kai Yu and JohnTao
-
8/3/2019 Building an Oracle Grid With
44/44
References:
1. Best Practices of Deployment of Oracle ASM with Dell EqualLogic PSiSCSI Storage System, Dell White Paper
2 . Oracle Press Release: Oracle Enterprise Manager Extends Management
to Oracle VM Server Virtualization3. Oracle Enterprise Manager Concepts 10gRelease 5 (10.2.0.5)
Part Number B31949-10
4. Oracle Enterprise Manager Grid Control ReadMe for Linux x86-6410g Release 5 (10.2.0.5) April 2009
5. How to Enable Oracle VM Management Pack in EM Grid Control 10.2.0.5,Metalink Note #: 781879.1
6. Oracle VM: Converting from a Local to Shared OVS Repository, Metalinknote # 756968.1
7. How to Add Shared Repositories to Oracle VM Pools with Multiple Servers,Metalink Note # 869430.1
8. Implementing Oracle Grid: A successful Customer Case Study, Kai Yu, DanBrint, IOUG Select Journal Volume 16, Number 2, Second Quarter 2009
9. Deploying Oracle VM Release 2.1 on Dell PowewrEdge servers and
Dell/EMC storage Dell white paper10. Dell Reference Configuration Deploying Oracle Database on Dell
EqualLogic PS5000XV iSCSI Storage A Dell Technical White Paper
11. Technical Best Practices for Virtualization & RAC Oracle RAC SIGwebseminar slides, Michael Timpanaro-Perota & Daniel Dibbets
http://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.oracle.com/us/corporate/press/017955_ENhttp://www.oracle.com/us/corporate/press/017955_ENhttp://www.oracle.com/us/corporate/press/017955_ENhttp://www.dell.com/downloads/global/solutions/ovm_ref_config_final.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/ovm_ref_config_final.pdf?c=us&cs=555&l=en&s=bizhttp://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002855&p_id=1002855&p_cat=documents_new&p_user=nobody&p_company=994323795175833http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002855&p_id=1002855&p_cat=documents_new&p_user=nobody&p_company=994323795175833http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002855&p_id=1002855&p_cat=documents_new&p_user=nobody&p_company=994323795175833http://www.oracleracsig.org/pls/apex/Z?p_url=RAC_SIG.download_my_file?p_file=1002855&p_id=1002855&p_cat=documents_new&p_user=nobody&p_company=994323795175833http://www.dell.com/downloads/global/solutions/ovm_ref_config_final.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/ovm_ref_config_final.pdf?c=us&cs=555&l=en&s=bizhttp://www.oracle.com/us/corporate/press/017955_ENhttp://www.oracle.com/us/corporate/press/017955_ENhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=bizhttp://www.dell.com/downloads/global/solutions/Best_Practices_Oracle_ASM.pdf?c=us&cs=555&l=en&s=biz