rac attack

8/12/2019 Rac Attack

1/246

RAC Attack - Oracle Cluster Database at Home

Edited by Jeremy Schneider

Wikibooks.org

Sponsored by Pythian


2/246

April 5, 2013

On the 28th of April 2012 the contents of the English as well as German Wikibooks and Wikipedia projects were

licensed under Creative Commons Attribution-ShareAlike 3.0 Unported license. An URI to this license is given

in the list of figures on page229.If this document is a derived work from the contents of one of these projects

and the content was still licensed by the project under this license at the time of derivation this document has to

be licensed under the same, a similar or a compatible license, as stated in section 4b of the license. The list of

contributors is included in chapter Contributors on page227.The licenses GPL, LGPL and GFDL are included in

chapter Licenses on page239,since this book and/or parts of it may or may not be licensed under one or more of

these licenses, and thus require inclusion of these licenses. The licenses of the figures are given in the list of figures

on page229.This PDF was generated by the LATEX typesetting software. The LATEX source itself was generated

by a program written by Dirk Hnniger with modifications by Jeremy Schneider. Dirks original programis freely available under an open source license from http://de.wikibooks.org/wiki/Benutzer:

Dirk_Huenniger/wb2pdf. This distribution also contains a configured version of the pdflatexcompiler

with all necessary packages and fonts needed to compile the LATEX source included in this PDF file. Full

instructions (including Jeremys modifications) for dynamically generating this print book using the RAC Attack

wikibook contents are available at http://racattack.org/book.
http://de.wikibooks.org/wiki/Benutzer:Dirk_Huenniger/wb2pdfhttp://de.wikibooks.org/wiki/Benutzer:Dirk_Huenniger/wb2pdfhttp://de.wikibooks.org/wiki/Benutzer:Dirk_Huenniger/wb2pdfhttp://racattack.org/bookhttp://racattack.org/bookhttp://racattack.org/bookhttp://de.wikibooks.org/wiki/Benutzer:Dirk_Huenniger/wb2pdfhttp://de.wikibooks.org/wiki/Benutzer:Dirk_Huenniger/wb2pdf


3/246


4/246

Contents

1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1. Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2. Making This Lab Successful. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3. Lab Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2. Planning Your Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.1. Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2. Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

I. Setting Up RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3. Hardware and Windows Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.1. Hardware and Windows Minimum Requirements. . . . . . . . . . . . . . . . . . . . 13

3.2. Install VMware Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3. Setup Virtual Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.4. Setup Virtual Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3.5. Download Oracle Enterprise Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4. Linux Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.1. Create VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4.2. Prep for OS Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4.3. OS Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

4.4. Wrap-up OS Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.5. Create RAC Attack DVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.6. Prep for Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

5. Create Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.1. Create Interconnect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

5.2. Create Shared Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5.3. Copy VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

5.4. Configure Disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

5.5. Configure Node 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5.6. Configure Node 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.7. SSH and CVU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

6. Grid Install (ASM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

6.1. Setup ASMLIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

6.2. Cluster Verification Utility (ASM) . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

6.3. Install Grid Infrastructure (ASM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

6.4. Increase CRS Fencing Timeout (ASM) . . . . . . . . . . . . . . . . . . . . . . . . . 116

6.5. Setup ASM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

7. Grid Install (Shared Filesystem) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

7.1. Setup OCFS2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

7.2. Cluster Verification Utility (Shared Filesystem). . . . . . . . . . . . . . . . . . . . . 133

IV


5/246

Contents

7.3. 11gR2 Bug Workaround . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

7.4. Install Grid Infrastructure (Shared Filesystem) . . . . . . . . . . . . . . . . . . . . . 137

7.5. Increase CRS Fencing Timeout (Shared Filesystem) . . . . . . . . . . . . . . . . . . 152

8. RAC Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

8.1. Install Database Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

9. Create Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

9.1. Create DB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16510. Rolling Patches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

10.1. Patching Grid and Database Software . . . . . . . . . . . . . . . . . . . . . . . . . . 177

II. Exploring RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

11. Clusterware Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

11.1. Clusterware and Fencing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

11.2. Clusterware Callouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

12. Services, Failover and Load Balancing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

12.1. Install Instant Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

12.2. Service Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

12.3. Connection Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

12.4. Runtime Failover. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202

12.5. Client Load Balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

12.6. Server Load Balancing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

13. RAC SQL and PLSQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

13.1. Install Runstats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

13.2. Sequence Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

13.3. Parallel Query Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

13.4. Scheduler Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

13.5. File Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

14. RAC Backups and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

14.1. Setup Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

14.2. Flashback Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

14.3. Block Change Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22214.4. Archived Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222

14.5. Database Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

14.6. Database Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226

15. Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

16. Licenses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

16.1. Fair Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

16.2. Copyrighted Web Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

16.3. Use of Microsoft Copyrighted Content . . . . . . . . . . . . . . . . . . . . . . . . . 239

16.4. GNU GENERAL PUBLIC LICENSE. . . . . . . . . . . . . . . . . . . . . . . . . . 239

16.5. GNU Free Documentation License . . . . . . . . . . . . . . . . . . . . . . . . . . . 24116.6. GNU Lesser General Public License . . . . . . . . . . . . . . . . . . . . . . . . . . 242

1


6/246


7/246

1. Overview

RAC Attack is a free curriculum and platform for hands-on learning labs related to Oracle RAC (cluster

database). We believe that the best way to learn about RAC is with a lot of hands-on experience. This

curriculum has been used by individuals at home and by instructors in classes since 2008.

The original contributors were Jeremy Schneider, Dan Norris and Parto Jalili. The handbook was

published at http://www.ardentperf.com for several years before its migration to this wikibook.

All RAC Attack content was released under the CC-BY-SA license in May 2011 when this project was

initiated.

To learn about upcoming RAC Attack events or to organize one yourself, visit the Events page. You

can use the shortcut http://racattack.org/eventsto access this page at any time.

The goal of this workbook is to help students learn about Oracle RAC cluster databases through guided

examples. (Specifically, 11gR2 RAC on VMware Server with ASM or Shared Filesystem and Oracle

Enterprise Linux 5.) It can be used by organizers of events, by instructors in classes or by individuals at

home.

RAC Attack differs in depth from other tutorials currently available.

Every keystroke and mouse click is carefully documented here.

The process is covered from the very beginning to the very end - from the very first installation of

VMware on your laptop to various experiments on your running cluster database... with everything in

between.

The labs in the main workbook have been tested thoroughly and repeatedly.

1.1. Prerequisites

Students should be able to navigate in Unix - for example, listing files with "ls".

1.1.1. Hardware Minimum Requirements

Most modern laptop and desktop computers should be powerful enough to run a two-node virtual RAC

cluster. In a nutshell, these are the recommended minimums:

Dual-core 2GHz 32-bit processor(it's been done with single-core)

4GB memory(it's been done with 3GB)

Two physical hard disks - not partitions(it's been done with one)

External HD for laptops(it's been done with certain USB flash memory sticks)

50 GB + 10.5 GB free space(it's been done with slightly less)

3
http://www.ardentperf.com/http://racattack.org/eventshttp://racattack.org/eventshttp://www.ardentperf.com/


8/246

Overview

Windows XP or Vista(linux & mac are not covered in these instructions)

1.2. Making This Lab Successful

Read about: Planning Your Time

Focus on what you can learn.

Choose as many specific learning goals as possible and take your time to investigate them.

Be creative and experiment.

Take risks and don't be afraid to break things.

Take advantage of the classroom or event setting, if you're in one

You can "jumpstart" back to the beginning of a lab with one click.

Record discoveries and questions to share with others.

Help each other out. There are more participants than instructors!

When possible, cut-and-paste steps directly into a PuTTY SSH terminal session.

1.3. Lab Tips

All passwords are racattack

Always choose "I moved this VM" when asked, unless instructions specifically say to choose "copied".

If using your own laptop, start the VMs one after another. That is, wait until the first VM completes

bootup - including clusterware and database - before starting the second.

Classroom specific tips:

Common login account for Workstation and VMware console: "admin"

Common changes from the lab handbook:

Use 5GB shared disks not 3.25GB

Use 900MB memory not 760MB

Jumpstarts could take longer than you think... read: Planning Your Time

"RAC11g" directory is often at C:\RAC11g

Handout is available with stretch goals for advanced participants

1.3.1. Storage Overview

ASM Shared FS

/dev/sdb DATA /u51 (/u61)

/dev/sdc BACKUP /u52

1.3.2. Networking Overview

collabn1 collabn2

4


9/246

Lab Tips

collabn1 collabn2

Interconnect 172.16.100.51 172.16.100.52

Administration 192.168.78.51 192.168.78.52

VIP 192.168.78.61 192.168.78.62

SCAN 192.168.78.250

5


10/246


11/246

2. Planning Your Time

For the most benefit, you must plan your time carefully. There will not be enough time to complete all

of the labs - so choose the ones which most interest you.

Information

If you are using your own computer athomeor at an event, then you always need to complete the

first lab (Hardware and Windows Preparation1) before you can jumpstart to any following labs. If

you are in aclassthen the instructor has probably completed the first lab for you, and you can

begin with a jumpstart.

2.1. Overview

Figure 1 Overview of Part I with Jumpstarts

7


12/246

Planning Your Time

2.2. Time

These times were gathered with a laptop just meeting the recommended minimum requirements2.In

addition to the wait times listed below, we suggest that you reserve about 40 minutes of work time to

complete any given lab.

InformationDownloadsonly apply to home users. If you are at an event or a class then the organizers have

already downloaded the software for you.

2 Chapter3.1on page13

8


13/246

Time

Jump

start

Size

NextLab

JumpstartTime

WaitTimesinthe

Lab

BuildTime

Hardwarea

ndWin-

dowsPrepar

ation3

Download:VMware

Server(500MB)

Download:OEL

(3GB)

0

CreateVM4

15min:LinuxInstall

Download:Oracle

Cluster&Database

(4.5

GB)

1

3GB

CreateClus

ter5

4min:jumpstart

5min:CopyVM

3min:startupcol-

labn1

3min:startupcol-

labn2

6min

2

13GB

GridInstall

(ASM6or

SharedFS7)

11min:jumpstart

3min:startupnodes

25min:GIInstall

10min:root.shcol-

labn1

8min:root.shcol-

labn2

17m

in

3

20GB

RACInstall8

18min:jumpstart

4min:startupcol-

labn1

4min:startupcol-

labn2

42min:DBInstall

30m

in

3

Chapter3.1on

page13

4

http://en.

wikibooks.org/wiki/..%2FLinux%20Install

5

Chapter5.1on

page71

6

Chapter6.1on

page95

7

Chapter7.1on

page123

8

Chapter8.1on

page153

9
http://en.wikibooks.org/wiki/..%2FLinux%20Install


14/246

Planning Your Time

Jump

start

Size

NextLab

JumpstartTime

WaitTimesinthe

Lab

BuildTime

4

27GB

CreateData

base9

25min:jumpstart

4min:startupcol-

labn1

4min:startupcol-

labn2

22min:DBCreate

45m

in

5

28GB

RollingPatches10

PartII:Exploring

RAC11(w/o

PSU)

25min:jumpstart

6min:startupcol-

labn1

6min:startupcol-

labn2

22min:GIPatch

collabn1

20min:DBPatch

collabn1

22min:GIPatchcol-

labn2

20min:DBPatch

collabn2

5min:catbundle

45m

in

6

37GB

PartII:Exp

loring

RAC12(w/P

SU)

35min:jumpstart

6min:startupcol-

labn1

6min:startupcol-

labn2

60m

in

9

Chapter9.1on

page165

10

Chapter10.1o

npage177

11

Chapter11.1o

npage189

12

Chapter11.1o

npage189

10


15/246

Part I.

Setting Up RAC

11


16/246


17/246

3. Hardware and Windows Preparation

3.1. Hardware and Windows Minimum Requirements

This handbook will walk you through the process of creating a two-node Oracle RAC cluster on your

own laptop or desktop computer.

A detailed explanation of virtualization is beyond the scope of this lab but here is a simple overview of

what we are building:

Figure 2

3.1.1. Hardware Minimum Requirements

Most modern laptop and desktop computers should be powerful enough to run a two-node virtual RAC

cluster. In a nutshell, these are the recommended minimums:

Dual-core 2GHz 32-bit processor(it's been done with single-core)

13


18/246

Hardware and Windows Preparation

4GB memory(it's been done with 3GB)

Two physical hard disks - not partitions(it's been done with one)

External HD for laptops(it's been done with certain USB flash memory sticks)

50 GB + 10.5 GB free space(it's been done with slightly less)

Support files larger than 2GB, e.g. NTFS(it's been done without 2GB file support, on FAT32)

Windows XP or Vista(Linux & Mac are not directly covered in these instructions, but a supplementary

PDF covering VirtualBox1

can be downloaded)

If your laptop or desktop does not meet these minimum requirements then it is not recommended

to try completing the RAC Attack labs. Although it is possible to complete these labs with smaller

configurations, there are many potential problems.

Information

Although we recommend against trying, RAC Attack has been done with: single-core, 3GB

memory, one physical hard drive, certain USB flash memory sticks, and less than 60GB of free

space.

3.1.2. Windows Preparation

1. Reboot windows. After this clean boot-up, don't start any unneeded programs - for example email

or instant messenger.

2. If possible, disable virus scanning (so that your antivirus software doesn't try to scan I/O on the

virtual machine disks).

3. Terminate any memory resident programs which are running, especially programs that help big

applications "quick-start" (these often use up a lot of memory).

1 http://www.orasavon.com/files/rac-attack-using-virtualbox-v0.4.pdf

14
http://www.orasavon.com/files/rac-attack-using-virtualbox-v0.4.pdfhttp://www.orasavon.com/files/rac-attack-using-virtualbox-v0.4.pdf


19/246

Hardware and Windows Minimum Requirements

3.1.3. Hardware Verificaton

Processor

1. From theStartmenu, choose or type RUN. In the dialog box that appears, typemsinfo32.

Figure 3

2. SelectSystem Summaryin the left pane. Scroll down to Processorin the right pane. Verify that

you have at least2 coresand that the speed is at least 2000 Mhz.

Figure 4

Memory

1. Scroll down to Memoryin the right pane. Verify thatInstalled Physical Memory is at least

4GB. Also, verify thatAvailable Memory is at least 1.4GB. You can terminate programs which

run in the foreground and background to increase the Available Memory.

Figure 5

15


20/246


Hard Disks

1. In the left pane, choose Components -> Storage -> Disks. Count the number of Disk Drive

entries and verify that there are at least two.

Figure 6

Second Hard Disk

Connection Storage Type

Most Preferred

Least Preferred

Inside Computer (SATA)

- or -USB 3.0- or -GigaBit

Ethernet Direct

no network, not shared - di-

rect to dedicated External

Hard Disk

Hard Disknot shared with anything

else

USB 2.0 Thumb Drive Flash Mem-

ory

advertised / reviewed / tested

at least 50 MB/s

16


21/246

Hardware and Windows Minimum Requirements

Connection Storage Type

GigaBit Ethernet to Net-

work - or -100 MegaBit Eth-

ernet Direct

no network, not shared -

direct to dedicated External

Hard Disk

Thumb Drive Flash Mem-

ory

advertised / reviewed /

tested at least 15 MB/s

Shared Hard Disk

shared with other programs

Information

A single hard disk can max out as low as 45 MB/s. (This has been observed during RAC Attack

testing.) Typical USB Flash Thumb Drives get very, very poor performance and should not be

used. Some USB Flash Thumb Drives are marketed for performance; these typically get a

maximum around 30 MB/s. In tests for RAC Attack, USB drives worked well for storing ISO

images but somewhat poorly for storing virtual machine files.

For a detailed comparison of different connection types, refer to:

http://www.pixelbeat.org/speeds.html

Free Space Requirements

RAC Attack is carefully designed to use three directories and spread out I/O for the best possible

responsiveness during labs. You can choose how to spread the directories across your hard disks, and

the best configuration may vary depending on your connection and storage type.

Directory Name Description Free Space Suggested LocationRAC11g Operating System

Oracle RAC Software

50 GB Second Hard Disk(not

flash)

RAC11g-shared Oracle RAC Data 7.5 GB Windows Hard Disk*

RAC11g-iso OEL Installation DVD

(read-only)

3 GB Windows Hard Disk*

*page file is usually on Windows Hard Disk

Note: do notcreate the RAC11g directory (with OS and Oracle Software) on a Flash Thumb Drive.

Information

We worked hard to reduce the footprint of RAC Attack, however with 11gR2 it's very difficult to

reduce it beyond this.

17
http://www.pixelbeat.org/speeds.htmlhttp://www.pixelbeat.org/speeds.html


22/246


3.1.4. Login Accounts

RAC Attack requires alocal windows user accountwith apasswordand withadministrative privi-

leges. You may login using a network or password-free account only if the login account has admin

privileges and you know the password for a local account which also has admin privileges (and not an

empty password).

If your account is not local, or if your account does not have local admin privileges then you can create

an admin account by following the directions here.

Creating a Local Admin Account

1. From the Start menu, choose RUN. In the dialog box that appears, type cmd to launch a command

prompt.

On Windows 7 find the "search programs" field at the bottom of the Start menu. Type cmdin but

don't run it. Right click on "cmd" then choose to"Run as administrator".

After you have opened the command prompt as an admin user, run the following two commands:

net user admin racattack /add

net localgroup administrators admin /add

Login: admin

Password: racattack

Verifying the Login Account

1. Type net user %username% (if you're using a network or password-free login account then

replace %username% with the local password-ed admin account).

VERIFY theusername, VERIFY thatpassword required is yes, and VERIFY that local group

memberships include Administrators.

Figure 8

18


23/246

Install VMware Server

3.2. Install VMware Server

1. These labs have been tested with version 2.0.1 of VMware Server. Go to the VMware Server

website at http://www.vmware.com/go/getserver

Figure 9

2. Register for an account if you don't have one already.

a) Write down your license number.

b) If you are at home, then download VMware Server. If you are at a RAC Attack event then

the instructor-provided Jumpstart Drive contains a copy of VMware Server, so that you don 't

need to download it. (However you still need a license number from the VMware website.)

Information

These labs have been tested with version 2.0.1 of VMware Server.

Figure 10

19
http://www.vmware.com/go/getserverhttp://www.vmware.com/go/getserver


24/246


3. Run the VMware Installer

Figure 11

4. Accept the license agreement and all default options during the installation process.

Figure 12

20


25/246

Install VMware Server

Figure 13

Figure 14

21


26/246


5. Enter your license information, which is visible at the VMware website on the same page where

you downloaded the software.

Figure 15

6. Reboot your computer if you are asked by the VMware installer.

3.3. Setup Virtual Networks

1. Choose Manage Virtual Networksfrom the start menu. After the program starts, make sure

that you see an "Apply" button at the bottom. If you do not see an "Apply" button then close the

program and re-start it by right-clicking and choosing to "run as administrator" (this must be done

on Windows 7 normally).

Figure 16

22


27/246

Setup Virtual Networks

2. Click theHost Virtual Network MappingTab and then click the Right Arrow Button next to

VMnet1. ChooseSubnetfrom the submenu.

Figure 17

3. Set the IP address to172.16.100.0and clickOK.

Figure 18

23


28/246


4. Click the Right Arrow Button next toVmnet8and chooseSubnetfrom the submenu.

Figure 19

5. Set the IP address to192.168.78.0and clickOK.

Figure 20

6. Click theAPPLYbutton.

7. Return to theSummarytab and VALIDATE:

VMnet1 has subnet172.16.100.0

24


29/246

Setup Virtual Networks

VMnet8 has subnet192.168.78.0

Figure 21

8. Go to the NAT tab and VALIDATE that the VMnet host is VMnet8 and Gateway IP is

192.168.78.2

Figure 22

25


30/246


3.4. Setup Virtual Storage

If you are at an event, then the event organizers might provide a special DEMOoption - where you can

run a pre-configured RAC cluster on your own laptop. In order to use this DEMO option, follow this

lab but use the directories on the event-provided external hard drive.

1. RAC Attack is carefully designed to use three directories and spread out I/O for the best possibleresponsiveness during labs. Create these three directories in the destinations that you chose

in Hardware and Windows Minimum Requirements , taking the guidelines into

consideration.

mkdir C:\RAC11g

mkdir D:\RAC11g-shared

mkdir D:\RAC11g-iso

In the RAC11g directory, make sure that collabn1 and collabn2 subdirectories don'texist.

rmdir C:\RAC11g\collabn1

rmdir C:\RAC11g\collabn2

2. The VMware Server management interface is web-based, and some new web browsers are not

compatible with it. There are two ways to open this management interface:

Open a non-default web browser and go to the address https://localhost:8333/

If you are at anevent, then the event organizers might have provided Firefox 2.0.0.20which

has been tested with RAC Attack. You can run this browser directly from the Jumpstart Drive

without installing it on your PC. This version of firefox can also be downloaded from the

internet.

Launch VMware Server Home Page from the start menu. This will use your default web

browser.

Figure 23

3. Depending on what web browser you use, you might receive security-related warnings. Proceed

through all of these warnings and choose to view the web page.

26
https://localhost:8333/https://localhost:8333/


31/246

Setup Virtual Storage

Figure 24 The warning in

Mozilla Firefox

Figure 25 The same alert inInternet Explorer 6

4. Login to the VMware console with the local windows admin account username and password.

Figure 26

5. On the main screen (Summarytab), find theCommandsbox and chooseAdd Datastore.

Figure 27

27


32/246


6. Repeat this step three times. Set the datastore names toRAC11g,RAC11g-sharedandRAC11g-

iso. ChooseLocal Datastoreand use the directory path which you previously chose and created.

Figure 28

7. VERIFY that the three new datastores exist in the Summary screen named RAC11g and

RAC11g-isoand RAC11g-shared. Also VERIFY that the two networks vmnet1and vmnet8

are available asHostOnlyand NAT respectively.

Figure 29

3.5. Download Oracle Enterprise Linux

If you are at anevent, then the event organizers have already downloaded the software and copied it to

the Jumpstart Drive.

Skip this lab.

If you are at home, then follow these instructions to download Oracle Enterprise Linux.

Continue below.

28


33/246

Download Oracle Enterprise Linux

1. Visit http://edelivery.oracle.com/linuxand chooseContinue.

Figure 30

2. Enter your Name, Company, Email and Country and review/accept the license and export

restrictions before clickingContinue. If you have visited Oracle EDelivery before then make

sure to enter your information exactly the same.

If this is the first time you've downloaded software from Oracle, then you might have to wait a

few days until you receive an email from Oracle granting you permission to continue.

Figure 31

3. Search forOracle Linuxon thex86-32-bitplatform. Choose Oracle Linux Release 5 Update 6.

Figure 32

29
http://edelivery.oracle.com/linuxhttp://edelivery.oracle.com/linux


34/246


4. Download the file forx86 (32 bit) not the source.

Figure 33

5. This zip file will contain a single file namedEnterprise-R5-U6-Server-i386-dvd.iso extract

this file into theRAC11g-isofolder and then delete the original zip file.

30


35/246

4. Linux Install

4.1. Create VM

1. From the SUMMARYscreen, choose Create Virtual Machine. Name the new machine col-

labn1and select theRAC11gdatastore.

Figure 34

2. SelectLinux Operating Systemand chooseRed Hat Enterprise Linux 5 (32-bit).

Figure 35

31


36/246

Linux Install

3. Allocate760Mof memory for the virtual machine and choose1 processor.

Figure 36

4. Choose toCreate a New Virtual Disk.

Figure 37

32


37/246

Create VM

5. Set the disk size to 30G and name the file [RAC11g] collabn1/system.vmdk leave all other

options at their defaults and clickNext.

Figure 38

6. Choose toAdd a Network Adapter.

Figure 39

33


38/246

Linux Install

7. Choose to create aNATnetwork connection.

Figure 40

8. ChooseDon't Add a CD/DVD Drive.

Figure 41

34


39/246

Create VM

9. ChooseDon't Add a Floppy Drive.

Figure 42

10. ChooseDon't Add a USB Controller.

Figure 43

35


40/246

Linux Install

11. Review the configuration and click Finish. Do not power on the virtual machine yet.

Figure 44

4.2. Prep for OS Installation

VIRTUAL DVD CONFIGURATION:

Virtual Device Node (Adapter/Device) Path (Datastore/Directory/Filename)

IDE 0:0 [RAC11g-iso]Enterprise-R5-U6-Server-i386-

dvd.iso

IDE 0:1*** [RAC11g-iso] RAC11gR2.iso***

***INSTRUCTOR-LED CLASS ONLY

1. In theInventorytab at the left, selectcollabn1(the virtual machine we just created).

Figure 45

36


41/246

Prep for OS Installation

2. From the Commandsbox, clickAdd Hardware. In the window that appears, clickCD/DVD

Drive.

Figure 46

3. Choose toUse an ISO Image.

Figure 47

37


42/246

Linux Install

4. ClickBrowseand locate the file[RAC11g-iso] Enterprise-R5-U6-Server-i386-dvd.iso.

Figure 48

5. Open the section called Virtual Device Node and choose IDE 0:0. Then click

Next.

Warning

Carefully follow this stepbecause it's easy to miss.

Figure 49

38


43/246

Prep for OS Installation

6. ClickFinishto add the device. Don't power on the virtual machine yet.

Figure 50

7. If you are in a class, then the instructor may have provided a second virtual DVD named

RAC11gR2.isoto save some class time. It contains all additional software downloads. Repeat all previous stepsfrom this lab to add the second DVD using RAC11gR2.iso image

and choosingIDE 0:1.

If you are not in a class, then you will later download all needed software and build the second

DVD yourself.

Continue below.

8. Scroll down to the Hardware box and confirm the Virtual Machine settings. They should match

this picture (except that you should only see the second DVD if you are in a class and it was

provided by the instructor):

Figure 51

39


44/246

Linux Install

4.3. OS Installation

1. Click theConsoletab. You might see a message saying that the Remote Console Plug-in is not

installed. If you see this message then clickInstall plug-inand follow the directions before

continuing. (Note: you may be asked to restart your computer during this process.)

Figure 52

2. When the plugin is installed, you should see a large play button in the center of the console.

Click on the play button to start the VM.

Figure 53

40


45/246

OS Installation

3. When you see the square boxes, click anywhere to open a console window.

Figure 54

4. A new window will now open - outside of your web browser. If you opened this window soon

after starting the Virtual Machine, then you will see the boot screen of the Oracle Enterprise

Linux installer.

Figure 55

At first, this new console window will ignore your keyboard and mouse. Click inside the new

console windowand it will begin accepting your keyboard and mouse.

41


46/246

Linux Install

Information

Anytime your keyboard and mouse are stuck in the VMware Virtual Machine, you can press

CTRL and ALT togetherto move them outside the VM.

If you still see the boot screen then you may press enter to continue, or just wait for it to

automatically continue.

5. Choose to SKIP the media test.

Figure 56

6. Choose NEXT when the first installer screen comes up.

Figure 57

42


47/246

OS Installation

7. Accept the defaultEnglishlanguage and chooseNext.

Figure 58

8. ChooseUS Englishkeyboard layout and clickNext.

Figure 59

43


48/246


49/246

OS Installation

11. ChooseYESto remove all partitions.

Figure 62

12. Set the hostname to collabn1.vm.ardentperf.comand leave DHCP enabled before choosing

NEXT.

Figure 63

45


50/246

Linux Install

13. Choose the timezone where you are located! Let the system clock run on UTC though.

Figure 64

14. Set the root password toracattack

Figure 65

46


51/246

OS Installation

15. ChooseCustomize Now but don't choose any "additional tasks". Then clickNEXT.

Figure 66

16. Selectonly these package groups, then clickNEXTto continue:

Category Selections

Desktop Environments Gnome Desktop Environment

Applications Editors

Graphical Internet

Text-based Internet

Development Development Libraries

Development ToolsServers Server Configuration Tools

Base System Administration Tools

Base

System Tools

X Window System

Warning

Do notchooseCluster Storageor Clustering.

47


52/246

Linux Install

Figure 67

17. ChooseNEXTto start the installation.

Figure 68

48


53/246

OS Installation

Figure 69

18. ChooseREBOOTwhen the installation is complete.

Figure 70

49


54/246

Linux Install

19. After the machine reboots when you wee the Welcome screen chooseFORWARD.

Figure 71

20. ACCEPTthe license and choose FORWARD.

Figure 72

50


55/246

OS Installation

21. DISABLEthe firewall and chooseFORWARD. Confirm by clickingYES.

Figure 73

22. DISABLESELinux before choosing FORWARD. Confirm withYES.

Figure 74

51


56/246

Linux Install

23. Leave Kdump disabled and chooseFORWARD.

Figure 75

24. Leave the clock alone (with the wrong time) and clickFORWARD.

Figure 76

52


57/246

OS Installation

25. Don't create a user; clickFORWARD. ChooseCONTINUEto confirm.

Figure 77

26. ChooseFORWARDto skip sound card config.

Figure 78

53


58/246

Linux Install

27. ChooseFINISHto close the installer. ClickOKto reboot.

Figure 79

28. After reboot you will see a login screen.

Figure 80

54


59/246


60/246

Linux Install

2. GNOME is the the graphical window environment installed by default in OEL. First, disable

GNOME CD automount. Go to the menuSystem >> Preferences >> Removable Drives and

Media.

Figure 82

3. Uncheck all of the options underRemovable Storageand clickClose.

Figure 83

56


61/246

Wrap-up OS Installation

4. Open a terminal window from the menuApplications >> Accessories >> Terminal.

Figure 84

5. From the menus, openEdit >> Current Profile.

Figure 85

57


62/246

Linux Install

6. In theTitle and Commandtab, check the box for Run command as a login shell, then close

the dialog.

Figure 86

7. The editor "gedit" is a simple graphical editor similar to notepad and it can be used to edit

files on Linux. If you are going to use gedit, then it is helpful if you openEdit > Preferences to

disabletext wrapping andenableline numbers.

Figure 87

8. In a terminal window as therootuser, shutdown and disable anacron then run it manually with

no delay.

58


63/246


[root@collabn1 ]# service anacron stop

Stopping anacron: [ OK ]

[root@collabn1 ]# chkconfig anacron off

[root@collabn1 ]# chkconfig --list anacron

anacron 0:off 1:off 2:off 3:off 4:off 5:off

6:off

[root@collabn1 ]# anacron -n

Information

It should not cause any problems for you, but be aware that several CPU and I/O intensive jobs

will run in the background for about 10 minutes while you continue with this lab (e.g. updatedb

and makewhatis). You might notice some slight system performance degradation. You can always

use the programtopto see what is currently running.

9. In a terminal window as therootuser, shutdown and disable the automounter.

[root@collabn1 ]# service autofs stop

Stopping automount: [ OK ]

[root@collabn1 ]# umount /media/*[root@collabn1 ]# chkconfig autofs off

[root@collabn1 ]# chkconfig --list autofs

autofs 0:off 1:off 2:off 3:off 4:off 5:off

6:off

10. In a terminal window as therootuser, shutdown and disable the apple zeroconf service (avahi).

[root@collabn1 ]# service avahi-daemon stop

Shutting down Avahi daemon: [ OK ]

[root@collabn1 ]# chkconfig avahi-daemon off

[root@collabn1 ]# chkconfig --list avahi-daemon

avahi-daemon 0:off 1:off 2:off 3:off 4:off 5:off

6:off

11. If any of the small CD images in the status bar do not have a green dot, then click on the CD

image and choose "Connect to [RAC11g]iso/... on Server". If a window opens showing the CD

contents then make sure toclosethe window.

Figure 88

12. Create two CDROM directories namedcdromand cdrom5.

Warning

Make sure to use these names because many later steps in this handbook will reference them!

59


64/246

Linux Install

[root@collabn1 ]# cd /mnt

[root@collabn1 mnt]# mkdir cdrom

[root@collabn1 mnt]# mkdir cdrom5

13. Add entries to /etc/fstab for all CD's and then mount them. If you are in a class then you will

probably have two CD's. If you are not in a class then you will probably have only one.

[root@collabn1 ]# ls /dev/cdrom-*/dev/cdrom-hda /dev/cdrom-hdb

[root@collabn1 ]# gedit /etc/fstab

/de v/cd rom- hda /mnt /cdr om i so96 60 d efau lts 0 0

/dev/cdrom-hdb /mnt/cdrom5 iso9660 defaults 0 0

[root@collabn1 mnt]# mount cdrom

[root@collabn1 mnt]# mount cdrom5

14. Install the additional required RPM's

cd /mnt

# From Enterprise Linux 5 CDROM 2rpm -Uvh */*/compat-libstdc++-33*rpm -Uvh */*/libaio-devel-0.*rpm -Uvh */*/unixODBC-2.*rpm -Uvh */*/unixODBC-devel-2.*# From Enterprise Linux 5 CDROM 3

rpm -Uvh */*/sysstat-7.*

# Additional required packages which are already installed:

# From Enterprise Linux 5 CDROM 1

rpm -Uvh */*/binutils-2.*rpm -Uvh */*/elfutils-libelf-0.*rpm -Uvh */*/glibc-2.*i686*rpm -Uvh */*/glibc-common-2.*rpm -Uvh */*/libaio-0.*rpm -Uvh */*/libgcc-4.*

rpm -Uvh */*/libstdc++-4.*rpm -Uvh */*/make-3.*# From Enterprise Linux 5 CDROM 2

rpm -Uvh */*/elfutils-libelf-devel-*rpm -Uvh */*/glibc-headers*rpm -Uvh */*/glibc-devel-2.*rpm -Uvh */*/libgomp*rpm -Uvh */*/gcc-4.*rpm -Uvh */*/gcc-c++-4.*rpm -Uvh */*/libstdc++-devel-4.*

15. Eject the cdrom

[root@collabn1 mnt]# eject /mnt/cdrom

60


65/246


16. Return to the Summary tab in the VMware console. From theStatusbox, choose to Install

VMware Tools. Click theInstallbutton to begin.

Figure 89

17. Install VMware client tools and run configuration tool.

Information

You must perform this step in the VMware Console; do not use PuTTY or any other terminal

program.

[root@collabn1 mnt]# mount /mnt/cdrom

mount: block device /dev/cdrom-hda is write-protected, mounting

read-only

[root@collabn1 mnt]# rpm -ivh /mnt/cdrom/VMwareTools-7.7.5-156745.i386.rpm

Preparing...

########################################### [100%]

1:VMwareTools

########################################### [100%]

The installation of VMware Tools 7.7.5 for Linux completed

successfully.

You can decide to remove this software from your system at any time

by

invoking the following command: "rpm -e VMwareTools".

Before running VMware Tools for the first time, you need to

configure it for your running kernel by invoking the

following command: "/usr/bin/vmware-config-tools.pl".

Enjoy,

--the VMware team

[root@collabn1 cdrom]# vmware-config-tools.pl

...

ChooseNOto skip the VMware FileSystem Sync Driver (vmsync)

61


66/246

Linux Install

Choose display size[12] 1024x768

Mounting HGFS shares will probably FAIL, but this is ok.

Figure 90

18. Run the network commands. (You can cut and paste the commands into the terminal.) Next, run

vmware-toolboxand enable clock synchronization.

Figure 91

62


67/246

Create RAC Attack DVD

19. Logout from your session.

Figure 92

20. Unmount the VMware Tools image:

umount /mnt/cdrom

Using VMware Infrastructure web interface, mount Oracle Enterprise Linux from[RAC11g-iso]

data store. Mount it:

mount /mnt/cdrom

4.5. Create RAC Attack DVD

If you are in a classor at anevent, then the instructor may have provided a second virtual DVD named

RAC11gR2.isoto save some class time. It contains all additional software downloads.

Skip this laband continue to the next one.

If you are at home, then follow these instructions to download all needed software and build the second

DVD yourself.

Continue below.

1. While logged in as root, download the latest release of the RAC Attack supporting code (GPL).

63


68/246

Linux Install

# cd

# curl -kL github.com/ardentperf/racattack/tarball/master | tar xz

2. You can view the master list of downloads here: https://github.com/ardentperf/

racattack/blob/master/makeDVD/auto.sh

If you have already downloaded any of these files, you may optionally copy them to the /tmp

directory in your virtual machine. When you create the DVD, any remaining files will beautomatically downloaded.

3. Create the DVD by running the automatic build script. You will be prompted for your Oracle

SSO login and password.

Warning

If your account is not authorized for Oracle Support then patch downloads will fail.

[root@collabn1 ]# sh ardentperf-racattack-*/makeDVD/auto.sh /mnt/cdrom5

oracle-profile' -> /mnt/cdrom5/oracle-profile'

root/

root/fix_cssd/

root/fix_cssd/fix_cssd.shOracle SSO Username:

Oracle SSO Password:

LOGGING IN TO ORACLE SSO

% Total % Received % Xferd Average Speed Time Time

Time Current

Dload Upload Total Spent

Left Speed

414 733 414 2977 0 0 1621 0 --:--:-- 0:00:01

--:--:-- 22383

DOWNLOADING: /tmp/oracleasmlib-2.0.4 -1.el5.i386.rpm

% Total % Received % Xferd Average Speed Time Time

Time Current

Dload Upload Total Spent

Left Speed

100 13929 100 13929 0 0 23118 0 --:--:-- --:--:--

--:--:-- 2720k/tmp/oracleasmlib-2.0.4-1.el5.i386.rpm' ->

./oracleasmlib-2.0.4-1.el5.i386.rpm'

DOWNLOADING: /tmp/linux_11gR2_database_1of2.zip

( ... )

FINISHED BUILDING RAC ATTACK DVD

4.6. Prep for Oracle

1. Edit/etc/sysctl.conf

[root@collabn1 ]# gedit /etc/sysctl.conf

kernel.shmmni = 4096

# semaphores: semmsl, semmns, semopm, semmni

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 1024 65000

64
https://github.com/ardentperf/racattack/blob/master/makeDVD/auto.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/auto.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/auto.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/auto.sh


69/246

Prep for Oracle

net.core.rmem_default=4194304

net.core.rmem_max=4194304

net.core.wmem_default=262144

net.core.wmem_max=262144

[root@collabn1 ]# sysctl -p

2. Edit/etc/security/limits.conf

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

3. Edit/etc/pam.d/loginand insert the following lines BEFOREthe "selinux open" line.

session required /lib/security/pam_limits.so

se ssion requ ired pam _limi ts.s o

4. Create groups and users. Make the oracle passwordracattack

[root@collabn1 ]# groupadd oinstall

[root@collabn1 ]# groupadd dba

[root@collabn1 ]# groupadd oper

[root@collabn1 ]# groupadd asmdba

[root@collabn1 ]# groupadd asmoper

[root@collabn1 ]# groupadd asmadmin

[root@collabn1 ]#

[root@collabn1 ]# useradd -u 500 -g oinstall -G dba,oper,asmdba,asmoper,asmadmin

oracle

[root@collabn1 ]# passwd oracle

Changing password for user oracle.

New UNIX password: racattack

BAD PASSWORD: it is based on a dictionary word

Retype new UNIX password: racattack

passwd: all authentication tokens updated successfully.

5. Create directories.

[root@collabn1 ]# mkdir -p /u01/grid/oracle/product/11.2.0/grid_1

[root@collabn1 ]# mkdir -p /u01/app/oracle/product/11.2.0/db_1

[root@collabn1 ]# chown -R oracle:oinstall /u01

65


70/246

Linux Install

6. Logout of your session and login as theoracleuser.

Figure 93

7. Disable GNOME CD automount for the oracle user. Go to the menu System >> Preferences >>

Removable Drives and Media.

Figure 94

66


71/246

Prep for Oracle

8. Uncheck all of the options underRemovable Storageand clickClose.

Figure 95

9. Open a terminal window.

Figure 96

67


72/246

Linux Install

10. From the menus, openEdit >> Current Profile.

Figure 97

11. In theTitle and Commandtab, check the box for Run command as a login shell, then close

the dialog.

Figure 98

68


73/246

Prep for Oracle

12. If you are using gedit, then openEdit > Preferences to disabletext wrapping andenableline

numbers.

Figure 99

13. Setup ssh equivalence for oracle user.

[oracle@collabn1 ]$ ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be

established.

RSA key fingerprint is

3f:65:e6:77:af:11:84:56:a4:b9:0f:8b:41:b4:2f:8a.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'localhost' (RSA) to the list of known

hosts.oracle@localhost's password: C

[oracle@collabn1 ]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa): M [default]

Enter passphrase (empty for no passphrase): M [no password]

Enter same passphrase again: M [no password]

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

0a:33:74:d4:7a:c0:c6:d0:56:33:10:4e:71:63:4c:e8

[email protected]

[oracle@collabn1 ]$ cat /home/oracle/.ssh/id_rsa.pub >> /home/oracle/.ssh/authorized_-

keys

14. Setup oracle and root users' profiles.

[oracle@collabn1 ]$ su - root

[root@collabn1 ]# gedit /etc/oratab

69


74/246

Linux Install

grid:/u01/grid/oracle/product/11.2.0/grid_1:N

[root@collabn1 ]# chown oracle:dba /etc/oratab

[root@collabn1 ]# gedit /root/.bashrc

ORAENV_ASK=NO

ORACLE_SID=grid

. oraenv >/dev/null

unset ORAENV_ASK

[root@collabn1 ]# su - oracle

[oracle@collabn1 ]$ cd /mnt

[oracle@collabn1 ]$ cat */oracle-profile >>/home/oracle/.bash_profile

Information

The source code for this file can be found at https://github.com/ardentperf/

racattack/blob/master/makeDVD/oracle-profile

15. Closeandre-openyour terminal sessions so that the new profiles take effect.

16. Installfix_cssdscript.

Information

In VMware test environments you usually have a very small amount of memory. Oracle CSS

processes can take up a *LOT* of the memory (over 50% in this lab) because it locks several

hundred MB in physical memory. In VMware (for both ASM and RAC environments) this may be

undesirable. This low-level hack will make the memory swappable at runtime.

NEVER, EVER, EVER EVEN IN YOUR WILDEST DREAMS THINK ABOUT TRYING

THIS ON ANYTHING CLOSE TO A PRODUCTION SYSTEM.

The source code for this file can be found at https://github.com/ardentperf/

racattack/blob/master/makeDVD/root/fix_cssd/fix_cssd.sh

[oracle@collabn1 ]$ su - root[root@collabn1 ]# cd /

[root@collabn1 ]# tar xvf mnt/*/fix_cssd.tar

root/fix_cssd/fix_cssd.sh

[root@collabn1 ]# chmod 774 /root/fix_cssd/fix_cssd.sh

[root@collabn1 ]# gedit /etc/rc.d/rc.local

cd /root/fix_cssd

nohup nice -n -20 ./fix_cssd.sh 2>&1 &

[root@collabn1 ]# /etc/rc.d/rc.local

nohup: appending output to nohup.out'

70
https://github.com/ardentperf/racattack/blob/master/makeDVD/oracle-profilehttps://github.com/ardentperf/racattack/blob/master/makeDVD/oracle-profilehttps://github.com/ardentperf/racattack/blob/master/makeDVD/root/fix_cssd/fix_cssd.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/root/fix_cssd/fix_cssd.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/root/fix_cssd/fix_cssd.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/root/fix_cssd/fix_cssd.shhttps://github.com/ardentperf/racattack/blob/master/makeDVD/oracle-profilehttps://github.com/ardentperf/racattack/blob/master/makeDVD/oracle-profile


75/246

5. Create Cluster

5.1. Create Interconnect

1. If the machine is running thenlogout and shutdown. The machine needs to be powered off.


Figure 100

3. Scroll down to the Hardware box and CONFIRM that there is one Network Adapter of type

NAT.

Figure 101

71


76/246

Create Cluster

4. From the Commandsbox, clickAdd Hardware. In the window that appears, clickNetwork

Adapter.

Figure 102

5. Choose to create aHostOnlynetwork connection. This will be used for the interconnect. ThenclickNext.

Figure 103

72


77/246

Create Shared Disks

6. ClickFINISHto create the network adapter.

Figure 104

5.2. Create Shared Disks

REPEAT STEPS 1-6 FOR BOTH OF THESE DEVICES:

Virtual Device Node

(Adapter/Device)

Size Path (Datastore/Directory/Filename)

SCSI 1:0 3.25 GB [RAC11g-shared] data.vmdk

SCSI 1:1 3.25 GB [RAC11g-shared] backup.vmdk


Figure 105

73


78/246

Create Cluster

2. From theCommandsbox, clickAdd Hardware. In the window that appears, clickHard Disk.

Figure 106

3. Choose toCreate a New Virtual Diskand clickNext.

Figure 107

4. Enter a capacity of3.25 GBand type the name[RAC11g-shared] data.vmdk.

ChooseFile Options Allocate all disk space now.

ChooseDisk Mode Independentand Persistent.

ChooseVirtual Device Node SCSI 1:0. ClickNextto continue.

74


79/246


80/246


81/246

Copy VM

Name Value

diskLib.dataCacheMaxSize 0

diskLib.maxUnsyncedWrites 0

mainMem.useNamedFile false

Figure 112

Information

I have found the following three websites among the most useful while creating custom VMware

configurations. They show how powerful and versatile VMware is even the free VMware Server

product.

http://sanbarrow.com/vmx/vmx-advanced.html

http://sanbarrow.com/vmx/vmx-config-ini.html

http://vmfaq.com/?View=entry&EntryID=25

5.3. Copy VM

1. InWindows Explorer, browse to the folder [RAC11g] \collabn1. Copy all of the files to the

folder[RAC11g] \collabn2.

The location of[RAC11g]was determined in the first lab. In a class, the instructor may provide

the location.

If thecollabn2folder does not exist, then create it.

77
http://sanbarrow.com/vmx/vmx-advanced.htmlhttp://sanbarrow.com/vmx/vmx-config-ini.htmlhttp://vmfaq.com/?View=entry&EntryID=25http://vmfaq.com/?View=entry&EntryID=25http://sanbarrow.com/vmx/vmx-config-ini.htmlhttp://sanbarrow.com/vmx/vmx-advanced.html


82/246

Create Cluster

Make sure that you copy - not move - the files. Hold down CTRL to copy with drag-and-drop in

windows.

Figure 113

Figure 114

78


83/246

Copy VM

2. Browse to[RAC11g] \collabn2. Edit the file collabn1.vmx (the VMware Configuration File).

You can use notepad or wordpad to edit the file.

Figure 115

3. Find the linedisplayNameand change it to collabn2(the new node name), then save and close

the file.

displayName = "collabn2"

Figure 116

79


84/246

Create Cluster

5.4. Configure Disks

1. In theInventorytab at the left, selectcollabn1.

Figure 117

2. Choose theConsoletab. Click on the play button to start the VM. When you see the square boxes,

click anywhere to open a console window.

Figure 118

80


85/246

Configure Disks

3. Login as theoracleuser with the passwordracattack.

Figure 119

4. Open a terminal window.

Figure 120

5. Switch to the root user with the "su -" command. (The password isracattack.) CONFIRM

that two new disks exist and that they have the appropriate sizes. These two lines should exactly

match!!

81


86/246

Create Cluster

[root@collabn1 ]# cat /proc/partitions

major minor #blocks name

...

8 16 3407872 sdb

8 32 3407872 sdc

6. Create partitions on all of the newly created disks withfdisk.

a) run fdisk /dev/sdb You should see the message "Device contains neither a valid DOSpartition table, nor Sun, SGI or OSF disklabel"

b) type"n"to create a new partition.

c) type"p"for a primary partition.

d) type partition number1.

e) press enter twice to accept the default first/last cylinders.

f) type"t"to set the partition type.

g) enter partition typeda (Non-FS data).

h) type"w"to write the partition table to disk.

[root@collabn1 ]# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or

OSF disklabel

Building a new DOS disklabel. Changes will remain in memory only,until you decide to write them. After that, of course, the previous

content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected

by w(rite)

Command (m for help): n

Command action

e exte nded

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-652, default 1): M [default]

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-652, default 652): M [default]

Using default value 652

Command (m for help): t

Selected partition 1

Hex code (type L to list codes): da

Changed system type of partition 1 to da (Non-FS data)

Command (m for help): w

The partition table has been altered!

WARNING: Re-reading the partition table failed with error 16: Device

or resource busy.

The kernel still uses the old table.

Syncing disks.

7. Repeat the previous step for disk/dev/sdc

82


87/246

Configure Node 1

5.5. Configure Node 1

After completing the previous lab, you should already be logged in as the oracleuser on collabn1. You

should already have a terminal open, where you are currently switched to the rootuser.

1. As the rootuser,CONFIRMthat the IP address ifeth0starts with192.168.78and that the IP

address ofeth1starts with172.16.100.

[root@collabn1 ]# ifconfig|grep net

eth0 Link encap:Ethernet HWaddr 00:0C:29:07:F5:C0

inet addr:192.168.78.128 Bcast:192.168.78.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe07:f5c0/64 Scope:Link

eth1 Link encap:Ethernet HWaddr 00:0C:29:07:F5:CA


inet6 addr: fe80::20c:29ff:fe07:f5ca/64 Scope:Link

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

2. Update the IP addresses by directly editing the linux network configuration files. Update the

BOOTPROTO line and add the remaining lines.

[root@collabn1 ]# cd /etc/sysconfig/network-scripts

[root@collabn1 network-scripts]# gedit ifcfg-eth0

BOOTPROTO=none

IPADDR=192.168.78.51

NETMASK=255.255.255.0

GATEWAY=192.168.78.2


BOOTPROTO=none

IPADDR=172.16.100.51

NETMASK=255.255.255.0

3. Update the DNS search domain

[root@collabn1 network-scripts]# gedit /etc/resolv.conf

search vm.ardentperf.com

nameserver 192.168.78.2

4. Asroot, restart the network services by typing service network restart. Then confirm the new

ip addresses withifconfig. Also confirm the search domain by inspecting /etc/resolv.conf if

the file has reverted then edit it again. (When I wrote this lab, the change stuck after the second

time I edited the file.)

Warning

You must perform this step in VMware; do not use PuTTY.

83


88/246

Create Cluster

Figure 121

5. Edit/etc/ hosts. EDITthe line with127.0.0.1and thenADDall of the other lines below:

[root@collabn1 etc]# vi /etc/hosts

192.168.78.51 collabn1 collabn1.vm.ardentperf.com

192.168.78.61 collabn1-vip collabn1-vip.vm.ardentperf.com

172.16.100.51 collabn1-priv collabn1-priv.vm.ardentperf.com




192.168.78.250 collab-scan collab-scan.vm.ardentperf.com

192.168.78.251 collab-gns collab-gns.vm.ardentperf.com

::1 localhost6.localdomain6 localhost6

127.0.0.1 localhost.localdomain localhost

84


89/246

Configure Node 2

5.6. Configure Node 2

1. In the VMware console, go to the Virtual Machinemenu and chooseAdd Virtual Machine to

Inventory.

Figure 122

2. Browse toRAC11g/collabn2and opencollabn1.vmx. Make sure you browse to the right folder!

Figure 123

85


90/246

Create Cluster

3. In theInventorytab at the left, select the new VM collabn2. Choose theConsoletab and click

the play button to start the VM.

Figure 124

4. The VMware console should ask you whether you copied or moved the VM. Answer that you

copiedthe files and clickOK.

Figure 125

86


91/246

Configure Node 2

5. When you see the square boxes, click anywhere to open a console window.

Figure 126

6. Login as theoracleuser with the passwordracattack.

Figure 127

7. Open a terminal ( Applications >> Accessories >> Terminal ) and switch to the root

user.

87


92/246

Create Cluster

Information

When you first open a console on collabn2, you will notice that the hostname still sayscollabn1.

This is because we copied the machine; we will change the hostname shortly.

88


93/246

Configure Node 2

Figure 128

8. CONFIRMthat the IP address ofeth1starts with172.16.100.

[root@collabn1 ]# ifconfig|grep net

eth1 Link encap:Ethernet HWaddr 00:0C:29:C9:3E:C8


inet addr:127.0.0.1 Mask:255.0.0.0

9. Update the IP addresses by directly editing the linux network configuration files. Update the

BOOTPROTO line and add the remaining lines.COMMENT or DELETEthe line which begins with HWADDR.

[root@collabn1 ]# cd /etc/sysconfig/network-scripts


BOOTPROTO=none

#HWADDR=...

IPADDR=192.168.78.52

NETMASK=255.255.255.0

GATEWAY=192.168.78.2


BOOTPROTO=none

#HWADDR=...

IPADDR=172.16.100.52NETMASK=255.255.255.0

10. Update the DNS search domain

89


94/246

Create Cluster

[root@collabn1 network-scripts]# gedit /etc/resolv.conf

search vm.ardentperf.com

nameserver 192.168.78.2

11. Update the HOSTNAME by editing the linux system configuration files. Set it to col-

labn2.vm.ardentperf.com. Then change the active system hostname with the hostname

command.

[root@collabn1 network-scripts]# gedit /etc/sysconfig/network

HOSTNAME=collabn2.vm.ardentperf.com

[root@collabn1 network-scripts]# hostname collabn2.vm.ardentperf.com

12. Asroot, restart the network services by typing service network restart. Then confirm the new

ip addresses withifconfig. Confirm that search domain by inspecting /etc/resolv.conf if the file

has reverted then edit it again. (The change stuck after the second time I edited the file while

walking through this lab.) Also confirm the new hostname with hostname.

Warning

You must perform this step in VMware; do not use PuTTY.

Figure 129

13. Exit your terminal session and start a new one so that you can see the updated hostname in the

prompt.14. Edit/etc/hosts. EDITthe line with127.0.0.1and thenADDall of the other lines below:

[root@collabn2 etc]# vi /etc/hosts


90


95/246

SSH and CVU






192.168.78.250 collab-scan collab-scan.vm.ardentperf.com

192.168.78.251 collab-gns collab-gns.vm.ardentperf.com

127.0.0.1 localhost.localdomain localhost

::1 localhost6.localdomain6 localhost6

5.7. SSH and CVU

1. Login to node collabn2. As theoracleuser, make sure that you can ping both nodes on the

interconnect and public network.

[oracle@collabn2 ]$ ping -c 1 collabn1

PING collabn1 (192.168.78.51) 56(84) bytes of data.

64 bytes from collabn1 (192.168.78.51): icmp_seq=1 ttl=64 time=0.793 ms




[oracle@collabn2 ]$ ping -c 1 collabn1-priv

PING collabn1-priv (172.16.100.51) 56(84) bytes of data.

64 bytes from collabn1-priv (172.16.100.51): icmp_seq=1 ttl=128 time=0.901 ms




2. Store the public keys for the hosts. You can do this with a tool calledssh-keyscan.

[oracle@collabn2 .ssh]$ ssh-keyscan -t rsa collabn1 collabn1-priv collabn2

collabn2-priv >> /.ssh/known_hosts

# collabn1 SSH-2.0-OpenSSH_4.3

# collabn1-priv SSH-2.0-OpenSSH_4.3

# collabn2 SSH-2.0-OpenSSH_4.3

# collabn2-priv SSH-2.0-OpenSSH_4.3

3. Test ssh equivilance. It is important that there is no prompt and you are automatically logged in.

[oracle@collabn2 .ssh]$ ssh collabn1

Warning: Permanently added the RSA host key for IP address

'192.168.78.51' to the list of known hosts.

Last login: Tue Apr 15 16:59:36 2008 from collabn2

[oracle@collabn1 ]$ exit

logout

Connection to collabn1 closed.

91


96/246

Create Cluster

4. Switch to nodecollabn1. As the oracle user, confirm that you can ping both servers. Then copy

theknown_hostsfile from collabn2.




[oracle@collabn1 ]$ ping -c 1 collabn2PING collabn2 (192.168.78.52) 56(84) bytes of data.








[oracle@collabn1 ]$ scp collabn2:.ssh/known_hosts /.ssh/known_hosts

The authenticity of host 'collabn2 (192.168.78.52)' can't be

established.

RSA key fingerprint is

3f:65:e6:77:af:11:84:56:a4:b9:0f:8b:41:b4:2f:8a.Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'collabn2,192.168.78.52' (RSA) to the

list of known hosts.

known_hosts 100% 3146 3.1KB/s

00:00

5. Test ssh equivilance.

[oracle@collabn1 ]$ ssh collabn2



logout


6. You need to be able to login to yourself too; test this.

[oracle@collabn1 ]$ ssh collabn1



logout


7. RunCLUVFYto validate your hardware and OS installation! The shared disk test will fail; this

is OK. Ignore that error.

collabn1:/home/oracle[RAC1]$ /mnt/cdrom*/grid/runcluvfy.sh stage -post hwos -n

collabn1,collabn2

Performing post-checks for hardware and operating system setup

Checking node reachability...

92


97/246

SSH and CVU

Node reachability check passed from node "collabn1"

Checking user equivalence...

User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.78.0" with node(s)

collabn2,collabn1

TCP connectivity check passed for subnet "192.168.78.0"


collabn2,collabn1


Interfaces found on subnet "192.168.78.0" that are likely candidates

for VIP are:

collabn2 eth0:192.168.78.52

collabn1 eth0:192.168.78.51

Interfaces found on subnet "172.16.100.0" that are likely candidatesfor a private interconnect are:

collabn2 eth1:172.16.100.52

collabn1 eth1:172.16.100.51

Node connectivity check passed

Check for multiple users with UID value 0 passed

Post-check for hardware and operating system setup was successful.

93


98/246


99/246

6. Grid Install (ASM)

6.1. Setup ASMLIB

1. Login tocollabn1as oracle and open a teminal. Switch to therootuser.

Figure 130

2. Install ASMLib on both nodes and initialize the disks. mount command may fail (mount:

/dev/cdrom-hda already mounted or /mnt/cdrom busy) if you already have cdrom mounted from

previous steps.


[root@collabn1 mnt]# mount /mnt/cdrom

mount: block device /dev/cdrom-hda is write-protected, mounting

read-only[root@collabn1 mnt]# mount /mnt/cdrom5

mount: block device /dev/cdrom-hdb is write-protected, mounting

read-only

95


100/246

Grid Install (ASM)

# OracleASM

rpm -Uvh */*/oracleasm-support*rpm -Uvh */*/oracleasm*el5-2* # (Note - that's lowercase EL5-2.)

# Missing from the install media; copied to RAC Attack DVD.

rpm -Uvh */oracleasmlib*

[root@collabn1 mnt]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver

is

loaded on boot and what permissions it will have. The current

values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

Information

The disks are shared so you only need to initialize them once.

[root@collabn1 ]# /etc/init.d/oracleasm createdisk data /dev/sdb1

Marking disk "data" as an ASM disk: [ OK ]

[root@collabn1 ]# /etc/init.d/oracleasm createdisk backup /dev/sdc1

Marking disk "backup" as an ASM disk: [ OK ]

[root@collabn1 ]# /etc/init.d/oracleasm listdisks

BACKUP

DATA

Now repeat some of the steps on the second virtual machine:

[root@collabn1 ]# ssh collabn2


# OracleASM

rpm -Uvh */*/oracleasm-support*rpm -Uvh */*/oracleasm*el5-2*# Missing from the install media; copied to RAC Attack DVD.

rpm -Uvh */oracleasmlib*

[root@collabn2 ]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver

is

loaded on boot and what permissions it will have. The current

96


101/246

Cluster Verification Utility (ASM)

values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: doneInitializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@collabn2 ]# /etc/init.d/oracleasm listdisks

BACKUP

DATA

6.2. Cluster Verification Utility (ASM)

1. Login tocollabn1as theoracleuser and open a terminal.

Figure 131

2. RunCVU to validate that you're ready to install CRS. Use the new fixup feature of 11gR2

CVU to create a script that can fix missing parameters.

[oracle@collabn1 ]$ /mnt/cdrom*/grid/runcluvfy.sh stage -pre crsinst -n

collabn1,collabn2 -fixup -r 11gR2

97


102/246

Grid Install (ASM)

Performing pre-checks for cluster services setup

Checking node reachability...

Node reachability check passed from node "collabn1"

Checking user equivalence...

User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful


collabn2,collabn1



collabn2,collabn1



for VIP are:

collabn2 eth0:192.168.78.52collabn1 eth0:192.168.78.51


for a private interconnect are:

collabn2 eth1:172.16.100.52

collabn1 eth1:172.16.100.51

Node connectivity check passed

Total memory check failed

Check failed on nodes:

collabn2,collabn1

Available memory check passed

Swap space check failed

Free disk space check passed for "collabn2:/tmp"

Free disk space check passed for "collabn1:/tmp"User existence check passed for "oracle"

Group existence check passed for "oinstall"

Group existence check passed for "dba"

Membership check for user "oracle" in group "oinstall" [as Primary]

passed

Membership check for user "oracle" in group "dba" passed

Run level check passed

Hard limits check passed for "maximum open file descriptors"

Soft limits check passed for "maximum open file descriptors"

Hard limits check passed for "maximum user processes"

Soft limits check passed for "maximum user processes"

System architecture check passed

Kernel version check passed

Kernel parameter check passed for "semmsl"

Kernel parameter check passed for "semmns"

Kernel parameter check passed for "semopm"Kernel parameter check passed for "semmni"

Kernel parameter check passed for "shmmax"

Kernel parameter check passed for "shmmni"

Kernel parameter check passed for "shmall"

Kernel parameter check failed for "file-max"


98


103/246

Cluster Verification Utility (ASM)

collabn2,collabn1

Kernel parameter check passed for "ip_local_port_range"

Kernel parameter check passed for "rmem_default"

Kernel parameter check passed for "rmem_max"

Kernel parameter check passed for "wmem_default"

Kernel parameter check failed for "wmem_max"


collabn2,collabn1

Kernel parameter check failed for "aio-max-nr"Check failed on nodes:

collabn2,collabn1

Package existence check passed for "make-3.81"

Package existence check passed for "binutils-2.17.50.0.6"

Package existence check passed for "gcc-4.1.2"

Package existence check passed for "gcc-c++-4.1.2"

Package existence check passed for "libgomp-4.1.2"

Package existence check passed for "libaio-0.3.106"

Package existence check passed for "glibc-2.5-24"

Package existence check passed for "compat-libstdc++-33-3.2.3"

Package existence check passed for "elfutils-libelf-0.125"

Package existence check passed for "elfutils-libelf-devel-0.125"

Package existence check passed for "glibc-common-2.5"

Package existence check passed for "glibc-devel-2.5"

Package existence check passed for "glibc-headers-2.5"

Package existence check passed for "libaio-devel-0.3.106"Package existence check passed for "libgcc-4.1.2"

Package existence check passed for "libstdc++-4.1.2"

Package existence check passed for "libstdc++-devel-4.1.2"

Package existence check passed for "sysstat-7.0.2"

Package existence check passed for "unixODBC-2.2.11"

Package existence check passed for "unixODBC-devel-2.2.11"

Package existence check passed for "ksh-20060214"

Check for multiple users with UID value 0 passed

Current group ID check passed

Core file name pattern consistency check passed.

User "oracle" is not part of "root" group. Check passed

Default user file creation mask check passed

Starting Clock synchronization checks using Network Time

Protocol(NTP)...

NTP Configuration file check started...

NTP Configuration file check passed

Checking daemon liveness...

Liveness check failed for "ntpd"


collabn2,collabn1

PRVF-5415 : Check to see if NTP daemon is running failed

Clock synchronization check using Network Time Protocol(NTP) failed

Fixup information has been generated for following node(s):

collabn2,collabn1

Please run the following script on each node as "root" user to execute the

fixups:

'/tmp/CVU_11.2.0.1.0_oracle/runfixup.sh '

Pre-check for cluster services setup was unsuccessful on all the

nodes.

3. Switch to therootuser and run the fixup script on both nodes.

99


104/246

Grid Install (ASM)

collabn1:/home/oracle[RAC1]$ su -

Password:

-bash: oraenv: No such file or directory

[root@collabn1 ]# /tmp/CVU_11.2.0.1.0_oracle/runfixup.sh

Response file being used is

:/tmp/CVU_11.2.0.1.0_oracle/fixup.response

Enable file being used is :/tmp/CVU_11.2.0.1.0_oracle/fixup.enableLog file location: /tmp/CVU_11.2.0.1.0_oracle/orarun.log

Setting Ke

rac attack

Documents