21 oracle rac with asm on linux

8/8/2019 21 Oracle Rac With Asm on Linux

1/28


2/28


3/28


4/28

RAC with ASM on

Linux x86-64Experience with an

extended cluster(stretched cluster)


5/28

The Issue

Issue

Overloaded server Hundreds of applications running against a single

database (Oracle 9.2.0.6 on Sun Solaris).

Protected by a failover cluster (Veritas) Decision: New DWH apps on new DB server

Requirements

Meet cost, stability, availability and performancecriteria

Do not break internal standards (e.g. multipathing)


6/28

The Requirements

Meet cost criteria

RAC on Linux with ASM is an option. Meet availability criteria

Use cluster technology.

Test in a PoC Meet stability and performance criteria

Customer provided data and queries of their internal systemsfor a DWH benchmark.

Benchmark: Oracle, , .

Test in a PoC


7/28

The Benchmark

1 week at the HP-center in Bblingen

Hardware: 4 servers with 4 XEON 64Bit CPUs, 16G RAM.

EVA 8000 storage with 156 disks.

Result:

1. Oracle/

2.

Why Oracle?


8/28

Proof of Concept (PoC)

Test extended RAC cluster configuration

Installation High Availability

Performance

Workload Management

Backup/Recovery

Maintenance Job Scheduling

Monitoring


9/28

Benefits of an extended

cluster Full utilization of resources no matter where

they are located

Site A

All Work gets Distributed to All Nodes

One PhysicalDatabase

Site B


10/28

Benefits of an extended

cluster Faster recovery from site failure than any

other technology in the market

Work Continues on Remaining Site

Site AOne Physical

Database

Site B


11/28

PoC: Design Considerations

Connectivity: Redundant connections for

public, interconnect and IO traffic

Site A Site BDual SAN Connections

Dual Private Interconnects

Dual Public Connections


12/28

PoC: Design Considerations

Disk (or better SAN-) mirroring: Host based

mirroring with ASM


13/28

ASM setupSAN 1:

USP 600SAN 2:

USP 600

Diskgroup+DATA01

SPFILE; CONTROL_1DATAFILES, REDO_1

Diskgroup +FLASH01CONTROL_2

REDO_2ARCH REDO

BACKUPS

ASMFailuregroup

D01_02

ASMFailuregroupFlash01_02

ASMFailuregroup

D01_01

ASMFailuregroupFlash01_01


14/28

PoC: Architecture

+asm2

CRS

ORA2

RAID1

SCRATCH

Raid 1

OS

ORA_HOME

ORACLE LOGS

+asm1

CRS

ORA1

RAID 1

SCRATCH

Raid 1

OS

ORA_HOME

ORACLE LOGS

+asm4

CRS

ORA4

RAID 1

SCRATCH

Raid 1

OS

ORA_HOME

ORACLE LOGS

Data

Center 1

SAN1OCR Standby/Backup

VOTING DISK Standby /Backup

DATAFILES

REDOLOGS

ARCH REDO

SPFILE

CONTROL FILE

SAN2OCR Active

VOTING DISK Active

DATAFILES

REDOLOGS

ARCH REDO

SPFILE

CONTROL FILE

PUBLIC

Network

PUBLIC

Network

PRIVATE

INTERCONNECT

PRIVATE

INTERCONNECT

Data

Center 2

+asm3

CRS

ORA3

RAID 1

SCRATCH

Raid 1

OS

ORA_HOME

ORACLE LOGS


15/28

PoC: Network Layout

Data Center 1 Data Center 2

Node1

Node2

Node3

Node4

Router DC1/1

Router DC1/2

Router DC2/1

Router DC2/2

VLAN forPrivat NWwith dedicatedlines

PublicPrivat

ActivePassive


16/28

Used Software

Linux SLES-9 SP2

Native Multipathing: Device Mapper 32Bit or 64Bit?

32Bit: Availability of versions and patches

64Bit: Addressable memory, no midterm-migration

Oracle for x86-64:

10.1.0.3 10.1.0.4

10.2.0.1


17/28

Used Hardware

HP Proliant DL585 with 4 AMD Opteron 800

series processor running at 2.2 GHz, 32GBRAM.

2 Fibre-channels per server

Storage: 2 Hitachi USP 600


18/28

Tests

Installation tests

High Availability Performance

Workload Management

Backup/Recovery

Maintenance

Job Scheduling Monitoring


19/28

High Availability Tests


20/28


Bonding

NIC failover tests successful. Multipathing

Device mapper available in Linux 2.6 (SLES9 SP2).

Multiple paths to the same drive are auto-detected, based ondrives WWID.

We successfully tested failover-policy (fault tolerance) andmultibus-policy (fault tolerance and throughput).

Check with your storage vendor on which multipath software(in our case Device Mapper or HDLM) and policy to use.


21/28

High Availability Tests 10gR1

Crash Test Impact Service Main component BehaviorPower off Server TAF, CRS second instance is up and running

Disk failure - LUN_data RAID/ASM instances up and running

Disk failure - LUN_voting RAID instances up and running

Disk failure - LUN_ocr RAID instances up and running

Unplug fiber attachment Multipathing instances up and running

Kill Oracle Instance TAF, CRS second instance is up and running

Kill ASM Instance TAF, CRS second instance is up and running

Halt q on Linux TAF, CRS second instance is up and running

Unplug Interconnect cable Bonding instances up and running

Unplug all Interconnect cables TAF,CRS/Voting second instance is up and running

Power off - SAN (NoVoting) ASM instances up and running

Power off - SAN ( voting ) cluster down


22/28

HA Design Considerations What about quorum?

What happens if the site with the quorum fails or allcommunications between sites are lost?

ThirdSite (10gR1)

Quorum

Quorum

Quorum

10gR2


23/28

High Availability Tests 10gR2

Crash Test Impact Service Main component BehaviorPower off Server TAF, CRS second instance is up and running

Disk failure - LUN_data RAID/ASM instances up and running

Disk failure - LUN_voting RAID instances up and running

Disk failure - LUN_ocr RAID instances up and running

Unplug fiber attachment Multipathing instances up and running

Kill Oracle Instance TAF, CRS second instance is up and running

Kill ASM Instance TAF, CRS second instance is up and running

Halt q on Linux TAF, CRS second instance is up and running

Unplug Interconnect cable Bonding instances up and running

Unplug all Interconnect cables TAF,CRS/Voting second instance is up and running

Power off - SAN ( voting ) ASM, 3rd voting instances up and running

Power off - 3rd voting disk 2 remaining votes instances up and running


24/28


Switch off SAN (i.e. switch off one HDS USP 600) Disk mount status changes from CACHED to MISSING in

ASM (v$asm_disk)

Processing continues without interruption

Switch on SAN again

Disks are visible twice: Mount status MISSING and CLOSED. Add disks again

alter diskgroup failgroup disk name force rebalance power 10;

Disks with status MISSING disappear. Disks with previousstatus CLOSED have status CACHED now.

Result: No downtime even with a SAN unavailability.


25/28

Performance considerations

Need to minimize latency

Direct effect on ASM mirroring and cache fusionoperations

Its better to use direct connections. Additional

routers, hubs or extra switches add latency.


26/28

Extended Cluster versus MAA

Extended RAC RAC + DG

Minimum Nodes Needed 2 3

Active Nodes All One Side Only

Recovery from Site Failure Seconds, No InterventionRequired

Manual Restart UsuallyRequired

Network Requirements High cost direct dedicatednetwork w/ lowest latency.Much greater networkbandwidth

Shared commercially availablenetwork. Does not have lowlatency requirements

Effective Distance Campus & Metro Continent

Disaster Protection Minor Site or LocalizedFailuresUser Errors

Database Corruptions

Wider DisastersUser Errors


27/28

Summary

RAC on Extended Cluster with ASM

It works!

Good design is key!

Data Guard offers additional benefits


28/28

21 oracle rac with asm on linux

Documents