uthoc2 - under the hood of oracle clusterware 2.0 - grid infrastructure by alex gorbachev pythian
DESCRIPTION
Under the Hoold of Oracle Clusterware 2.0 - updated version of the slides covering Oracle Database 11.2TRANSCRIPT
![Page 1: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/1.jpg)
Under the Hoodof Oracle Clusterware 2.0:Grid InfrastructureAlex Gorbachev
29 August, 2012
![Page 2: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/2.jpg)
© 2009/2010 Pythian
Agenda
• Place of Grid Infrastructure in Oracle RAC
• Node membership and evictions
• Clusterware architecture & startup sequence
• Resources Management and troubleshooting
2
![Page 3: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/3.jpg)
© 2009/2010 Pythian
Agenda
2
Nee
d to
mem
oriz
e
Understanding
Low
Hig
h
Shallow In-depth
The more you understand,
the less you need to memorize
![Page 4: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/4.jpg)
© 2012 Pythian
Single Instance Oracle Database
3
DatafilesControlfilesredo logsflashback logs, change tracking and etc...Database
SERVER
INSTANCE
Memory (SGA, PGA)Processes (PMON, SMON, LGWR and etc. + multiple shadow processes)
Read/write
APP
Query/DML/DDL
![Page 5: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/5.jpg)
© 2012 Pythian
Single Instance Oracle Database
4
Database
SERVER
INSTANCE
APP
![Page 6: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/6.jpg)
© 2012 Pythian
Oracle RAC Database
5
Database
SERVER 1
INSTANCE 1
APP
![Page 7: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/7.jpg)
© 2012 Pythian
Oracle RAC Database
5
Database
SERVER 1
INSTANCE 1
APP
SERVER 2
INSTANCE 2
![Page 8: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/8.jpg)
© 2012 Pythian
Oracle RAC Database
5
Database
SERVER 1
INSTANCE 1
APP
SERVER 2
INSTANCE 2
SERVER 3
INSTANCE 3
![Page 9: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/9.jpg)
© 2012 Pythian
Oracle RAC Database
5
Database
SERVER 1
INSTANCE 1
APP
SERVER 2
INSTANCE 2
SERVER 3
INSTANCE 3
![Page 10: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/10.jpg)
© 2012 Pythian
Oracle RAC Database
5
Database
SERVER 1
INSTANCE 1
APP
SERVER 2
INSTANCE 2
SERVER 3
INSTANCE 3
![Page 11: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/11.jpg)
© 2012 Pythian6
RAC looks simple. Eh?
![Page 12: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/12.jpg)
© 2009/2010 Pythian
Role of Grid Infrastructure
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
Shared storage
OCR Votingdisk
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
interconnectstorage access
7
![Page 13: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/13.jpg)
© 2009/2010 Pythian
Role of Grid Infrastructure
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
Shared storage
OCR Votingdisk
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
OS
Grid Infrastr.
Instance
ASM
VIPListener
Service
interconnectstorage access
7
![Page 14: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/14.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
8
![Page 15: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/15.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
8
![Page 16: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/16.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
8
![Page 17: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/17.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
9
![Page 18: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/18.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
9
![Page 19: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/19.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
ShootTheOtherNodeInTheHead
9
![Page 20: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/20.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
CRSDEVM
D
OPROCD
RACG
VIP
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
10
![Page 21: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/21.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
11
![Page 22: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/22.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
11
![Page 23: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/23.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
11
![Page 24: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/24.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
11
![Page 25: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/25.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
AskTheOtherNodeToRebootItself
(c) known quote
11
![Page 26: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/26.jpg)
© 2009/2010 Pythian12
11gR2 Grid Infrastructure:CSSD attempts graceful
shutdown
![Page 27: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/27.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
Votingdisk
OS
Clusterware
OPROCD
CSSD
interconnect
13
![Page 28: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/28.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
Votingdisk
OS
Clusterware
OPROCD
CSSD
interconnect
CSSD Monitor/Agent
13
![Page 29: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/29.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
Votingdisk
interconnect
CSSD Monitor/Agent
13
![Page 30: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/30.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
14
![Page 31: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/31.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
14
![Page 32: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/32.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
14
![Page 33: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/33.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
OPROCD/CSSD Mon
14
![Page 34: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/34.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
OPROCD/CSSD Mon
14
![Page 35: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/35.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
15
![Page 36: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/36.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
15
![Page 37: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/37.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
15
![Page 38: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/38.jpg)
© 2009/2010 Pythian
CSSD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
15
![Page 39: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/39.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
OPROCD
CSSDinterconnect
Votingdisk
16
![Page 40: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/40.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
OPROCD
CSSDinterconnect
16
![Page 41: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/41.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
CSSD
OS
Clusterware
OPROCD
CSSDinterconnect
16
![Page 42: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/42.jpg)
© 2009/2010 Pythian
CSSD CSSDinterconnect
16
![Page 43: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/43.jpg)
© 2009/2010 Pythian17
11gR2 Grid Infrastructure:CSSD attempts graceful
shutdown
![Page 44: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/44.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
![Page 45: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/45.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
member kill
![Page 46: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/46.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
member kill
![Page 47: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/47.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
member kill
![Page 48: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/48.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
member kill
![Page 49: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/49.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
Instance
member kill
Eviction by escalation of a member kill
![Page 50: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/50.jpg)
© 2009/2010 Pythian
OS
Clusterware
CSSD
OPROCDinterconnect
18
InstanceLMON
member kill
Eviction by escalation of a member kill
![Page 51: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/51.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
19
![Page 52: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/52.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
19
11gR2 Intelligent Platform Management Interface
![Page 53: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/53.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
19
11gR2 Intelligent Platform Management Interface
![Page 54: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/54.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
19
11gR2 Intelligent Platform Management Interface
![Page 55: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/55.jpg)
© 2009/2010 Pythian
OS
Clusterware
OPROCD
OS
Clusterware
CSSD
OPROCDinterconnect
Votingdisk
CSSD
20
Exadata Fencing
![Page 56: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/56.jpg)
© 2009/2010 Pythian
10g / 11gR1• Linux & UNIX inittab
init.cssd
init.evmd
init.crsd
• Linux & UNIX init.dinit.crs
• Windows Services
Grid Infrastructure Startup
21
11gR2• Linux & UNIX inittab
init.ohasd run
• Linux & UNIX init.dohasd start
• Windows Services
![Page 57: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/57.jpg)
© 2009/2010 Pythian
Startup in Linux & Unix
[root@cheese2 ~]# ps -fe | grep 'init\.' | grep -v grep
root 4283 1 0 02:52 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
[root@cheese2 ~]# tail -1 /etc/inittab
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
[root@cheese2 ~]# ls -l /etc/rc3.d/*ohasd*
lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/K15ohasd -> /etc/init.d/ohasd
lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/S96ohasd -> /etc/init.d/ohasd
22
![Page 58: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/58.jpg)
© 2009/2010 Pythian
init.crsd run
init.evmd run
init.cssd fatal
init.crs start
init.cssd autostart
init.cssd oprodc
init.cssd oclsomon
init.cssd daemon
init.cssd oclsvmon
oprocd
oclsomon.bin
ocssd.bin
oclsvmon.bin
evmd.bin
t
/etc/oracle/scls_scr/{host}/root/cssrun
/etc/oracle/scls_scr/{host}/root/crsstart• enable• disable
crsd.bin
Pre 11gR2 Clusterware Startup
23
![Page 59: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/59.jpg)
© 2009/2010 Pythian
t
24
Grid Infrastructure Startup
![Page 60: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/60.jpg)
© 2009/2010 Pythian
t
24
init.ohasd run
Grid Infrastructure Startup
![Page 61: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/61.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
Grid Infrastructure Startup
![Page 62: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/62.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
Grid Infrastructure Startup
![Page 63: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/63.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
/etc/init.d/ohasd
Grid Infrastructure Startup
![Page 64: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/64.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
/etc/oracle/scls_scr/{host}/root/ohasdstr• enable• disable
24
init.ohasd run
/etc/init.d/ohasd
Grid Infrastructure Startup
![Page 65: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/65.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
/etc/oracle/scls_scr/{host}/root/ohasdstr• enable• disable
24
init.ohasd run
/etc/init.d/ohasd
Grid Infrastructure Startup
![Page 66: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/66.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
Grid Infrastructure Startup
![Page 67: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/67.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS
Grid Infrastructure Startup
![Page 68: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/68.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS
Ora Agent
Grid Infrastructure Startup
![Page 69: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/69.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
Grid Infrastructure Startup
![Page 70: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/70.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
Grid Infrastructure Startup
![Page 71: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/71.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
Grid Infrastructure Startup
![Page 72: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/72.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
Grid Infrastructure Startup
![Page 73: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/73.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
Grid Infrastructure Startup
![Page 74: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/74.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
MDNS
GIPC
GPnP
EVM
ASM
Grid Infrastructure Startup
![Page 75: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/75.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
MDNS
GIPC
GPnP
EVM
CRS Root Agent
•VIP•SCAN IP•Network•GNS•ACFS Registry
ASM
Grid Infrastructure Startup
![Page 76: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/76.jpg)
© 2009/2010 Pythian
t
/etc/oracle/scls_scr/{host}/root/ohasdrun
24
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
MDNS
GIPC
GPnP
EVM
CRS Root Agent
CRS Ora Agent
•VIP•SCAN IP•Network•GNS•ACFS Registry
•Database•Instance•Listener•Services•Diskgroups•ONS•eONS•SCAN Listener
ASM
Grid Infrastructure Startup
![Page 77: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/77.jpg)
© 2009/2010 Pythian
t
25
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
MDNS
GIPC
GPnP
EVM
CRS Root Agent
CRS Ora Agent
•VIP•SCAN IP•Network•GNS•ACFS Registry
•Database•Instance•Listener•Services•Diskgroups•ONS•eONS•SCAN Listener
ASM
Grid Infrastructure Startup (static slide)
![Page 78: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/78.jpg)
© 2009/2010 Pythian
Grid Infrastructure Log Files
• alert<host>.log• ohasd• crsd• cssd• agent/ohasd/oraagent_oracle• agent/ohasd/oracssdagent_root• agent/ohasd/oracssdmonitor_root• agent/ohasd/orarootagent_root• agent/crsd/oraagent_oracle• agent/crsd/orarootagent_root
• ctssd• diskmon• gipcd• gnsd• gpnpd• mdnsd• racg
26
$GRID_HOME/log/{hostname}/
![Page 79: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/79.jpg)
© 2009/2010 Pythian
Oracle Cluster Registry
• Repository for all shared configuration data
• Except OCR location itself
• OCR is accessed mostly read-only• Every component reads OCR
• OCR is written only by CRS• only from a single OCR master node
• 11gR2 - Oracle Local Registry (OLR)• managed by ohasd
27
![Page 80: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/80.jpg)
© 2009/2010 Pythian
DEMOInterconnect Failure
• Simulate with “ifconfig eth1 down” on node 2• Both nodes notice the loss• Racing to evict each other
• from voting disk => 2 equal sub-clusters (cohorts)
• survives the one with the lowest leader #
• leader is the node with lowest # in sub-cluster
• Winner evicts another node• Setting kill-block in voting disk
• 11gR2 new feature: CSSD does clean restart
28
![Page 81: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/81.jpg)
© 2009/2010 Pythian
DEMOVoting Disk Failure
• Simulate with storage interface down or on NFS server
• CSSD detects voting disk IO stale• disktimeout setting - 200 seconds default
• CSSD starts eviction • 11gR2 new feature: CSSD does clean restart
29
![Page 82: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/82.jpg)
© 2009/2010 Pythian
DEMOCSSD is not healthy
• Simulate using kill -STOP <cssd.bin pid> (and try kill -9)• Another node observes NHB loss
• After misscount seconds => attempt eviction
• but CSSD is frozen and can’t commit suicide
• CSSD Monitor detects CSSD timeout• Commit suicide
30
![Page 83: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/83.jpg)
© 2009/2010 Pythian
HomeworkHost sick - CPU stalled
• Used to simulate by pausing OPROCD• kill -STOP <oprocd pid>
• sleep 1 or 2
• kill -CONT <oprocd pid>
• OPROCD is now a thread in CSSD Monitor• kill -STOP {cssdmonitor.bin} ; sleep 1 ; kill -CONT {cssdmonitor.bin}
• Doesn’t produce any visible results! Is there still OPROCD?
31
![Page 84: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/84.jpg)
© 2009/2010 Pythian
DEMOStartup troubleshooting
• Break before starting up• Interconnect, voting disk, Grid Home missing
• Check processes using “ps -fe | grep init”• Check syslog (/var/log/messages)• boot sequence• Clusterware log files
• if *.bin processes are running already
• crsctl check crs• crsctl status resource -t -init
32
![Page 85: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/85.jpg)
© 2009/2010 Pythian
11gR2 Cluster Resources
• Introduced resource type• APPLICATION was the only supported resource type in 11gR1
• Resource types using resource agents
• Cluster and local resources• Sophisticated resources dependencies (stop & start)
• hard
• weak
• attraction
• pullup
• dispersion
33
![Page 86: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/86.jpg)
© 2009/2010 Pythian
t
34
init.ohasd run
OHAS Root Agent
Ora Agent
CSSD Agent
CSSD Monitor
CSS
CRS
Disk Monitor
CTSS
ACFS Drivers
MDNS
GIPC
GPnP
EVM
CRS Root Agent
CRS Ora Agent
• VIP• SCAN IP• Network• GNS• ACFS Registry
• Database• Instance• Listener• Services• Diskgroups• ONS• eONS• SCAN Listener
ASM
Grid Infrastructure Processes• ora.cluster_interconnect.haip
OHAS Internal Resources
CRS Managed Resources
![Page 87: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/87.jpg)
© 2009/2010 Pythian
Troubleshooting Something Down
• OHASD up?• OHASD Agents up? • Internal resources up?• CRSD Agents up?• Managed resources up?
• Listener & VIP
• Database & ASM instance
• Services
• Have the nodes rebooted?• Have resources re-started?
35
![Page 88: UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian](https://reader031.vdocuments.us/reader031/viewer/2022020122/554fcc82b4c9050e7d8b5469/html5/thumbnails/88.jpg)
© 2009/2010 Pythian
11gR2 Grid Infrastructure References
• Oracle Clusterware Administration and Deployment Guide
• MOS 1053147.1
• 11gR2 Clusterware and Grid Home - What You Need to Know
• MOS 1050908.1
• How to Troubleshoot Grid Infrastructure Startup Issues
• MOS 1053970.1
• Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues
• MOS 1050693.1
• Troubleshooting 11.2 Clusterware Node Evictions (Reboots)
• MOS 942166.1
• How to Proceed from Failed 11gR2 Grid Infrastructure Installation
36