discussion sap on exadata 20140618
DESCRIPTION
Discussion SAP on Exadata 20140618TRANSCRIPT
RAC
RAC
What is a Exadata for SAP?
• GRID Cluster of database nodes
• Cluster Manager CRS
• „Volume Manager“ ASM
• Operating RAC Database
• Storage in Exadata Storage Cells
• Storage cell software on Cells
• Optionally hosting SAP instances
• Controlled by CRS
• sapctl
Database Database Node Node
RAC RAC Instance Instance
SAP SAP
GRID GRID
ASM ASM
CRS CRS
Storage Cell Storage Cell
Storage Cell Storage Cell
Exadata for SAP facts 1
•
•
•
GRID cluster, RAC databases
Database files on ASM only
no shared file system offered by Exadata
(DBFS is not supported in SAP environment)
•Local Oracle Homes for GRID and for RDBMS
(exception standard SAP RAC setup)
• all Oracle software installed on Exadata using Exadata standards
• all Oracle software installed under OS environment ‘oracle’
• ASM using standard Exadata setup (different to SAP ASM setups)
• ASM in cluster mode
Exadata for SAP facts 2
• SAP instances cannot run on Exadata (Exceptions: ASCS, SCS,ERS)
• SAP central services on the database nodes require an SAP Unicode Kernel
•
•
BR*Tools can be installed on the database nodes
If non-SAP and SAP databases are operated on the same
Exadata, refer to SAP OSS note 1677978 "Administration of mixed SAP/Non-SAP Environments on Exadata“
•OSS note 1634767 “Support for SAP software in Linux / Oracle
ASM/ACFS Cluster“.
Exadata for SAP facts 3
• OSS note 1590515 "SAP Software and Oracle Exadata“
• Exadata models: X2-2, X2-8, X3-2, X3-8, X4-2 with Linux 5
• RDBMS 11.2.0.2 Patch 1 (OSS 1591389) or 11.2.0.3 Patch 2 (OSS 1656654)
•
•
•
•
•
•
Exadata SW version 11.2.2.3.2
No Hybrid Columnar Compression (HCC)
No SAP instance (except ASCS, SCS, ERS)
Unicode kernels only
SAP Kernel on Exadata 7.20 64-Bit, don’t use EXT kernel
Brtools supported (OSS 1627541)
Exadata for SAP facts 4
• fast recovery area (FRA) may be used for flashback logs
(flashback database). No support by brtools for backups!!
Exadata for SAP prereqs 1
• Again important:
• SAP OSS note 1590515 "SAP Software and Oracle Exadata",
• SAP OSS note 1431798
"Oracle 11.2.0: Database Parameter Settings“
• All Oracle software installed on Exadata using Exadata standards, SAP general standards don’t apply on this platform
• All Oracle software installed under OS environment ‘oracle’ SAP Kernel CD (minimum 7.00 EP 3 or 7.20 EP 1 and higher)
Central services on Exadata: kernel 7.20 64bit unicode non EXT
BR*Tools software (minimum 7.20 patch 18 and higher)
Exadata for SAP prereqs 2
•
•
•
•
log_buffer not less than 128MB
Disk_asynch_io = true
Filesystemio_options = setall (even if on ASM)
ASM diskgroups +DATA (high) +RECO (normal)
RAC: The Cluster Database
Network
Users Centralized Management
Console
Interconnect No Single Point Of Failure High Speed
Switch or
Interconnect
Clustered Database Servers
Hub or Switch
Fabric Storage Area Network
Drive and Exploit Industry Advances in
Clustering
Mirrored Disk Subsystem
Oracle RAC Architecture
public network
VIP1 VIPn
Node n Node1
Oracle Clusterware interconnect
shared storage
Redo / Archive logs all instances
Database / Control files Managed by ASM
OCR and Voting Disks
Oracle Clusterwa
Services
cluster
Services
Listener Listener
instance 1
instance n
ASM ASM
Oracle Clusterware
Oracle Clusterwa
Operating Sys Operating Sys
What does RAC mean?
Instance
Instance
Instance
Oracle
Oracle
Oracle
SAP Instance
SAP Instance
SAP Instance
Building a HA Solution with Oracle
• •
RAC only protects the Oracle Instance
For a complete HA solution we need to protect also
• •
•
•
•
•
•
•
•
•
HW: Multiple Servers Network: Bonding
Physical Database: ASM, Mirroring, DG
SAP Enqueue server: sapctl
Application Level : DG
Rolling Upgrades and Patches
Maintenance Procedure: Online Reorg. etc
BRR: RMAN, Flashback
Migration Procedures: O2O/OOO
Planned/Unplanned Downtimes
HA
•
•
•
•
•
Oracle – RAC
SAP – Enqueue Replication
Disks – Mirroring / Standby Database
Network – Bonding, redundant networks
Logical- / User errors – Standby Database,
Flashback database Site failures – streched clusters, Standby
Database
•
Standby - Dataguard
• physical standby
• logical standby ( Not in SAP per 105047) • DG on RAC needs db_unique_name in srvctl • commands: srvctl ... –d <db_unique_name> ...
• ...
Still 2 tools: CRSCTL & SRVCTL
1) CRSCTL manages all cluster(ware) related operations
Usage: crsctl <command> <object> [<options>] command:
enable|disable|config|start|stop|relocate|replace|stat|add|delete|modify |getperm|setperm|check|set|get|unset|debug|lsmodules|query|pin|unpin
2) SRVCTL manages all Oracle resources related operations
Usage: … objects:
database|instance|service|nodeapps|vip|asm|diskgroup|listener|srvpool|se rver|scan|scan_listener|oc4j|home|filesystem|gns
New
Single
Components in 11.2
Client Access Name (SCAN)
• Used by clients to connect to any database in the cluster
• Removes the requirement to change the client connection if cluster changes
• Load balances across the instances providing a service
• Provides failover between “moved instances”
FR
EE
Ora
cle
Gri
d In
frastr
uctu
re
Clu
ste
rSC
AN
na
me
Siebel
PSFT
RAC
DB1
RAC
DB2
RAC
One
2 Listener level providing SCAN
Application
Server
Oracle RAC Database Cluster
SCAN
Listeners
Clients
Local Listeners
2 Listener level providing SCAN
Application
Server
Oracle RAC Database Cluster SCAN
Listeners Listeners
Clients
1. client asks the scan-listener adressed by one of the 3 scan ip‘s for TNS service
scan listener responds to client with right node-listener address
client asks node listener for service
2.
3.
1
2
3
Local L
11.2 network
3
DNS connect
1
DNS 2 4
client
SCAN address
Resolved by 3 fix IP’s returned in a round robin fashion Oracle RAC cluster - GRIDA
Local listeners
SCAN listeners
corporate DNS domain
TNS-Setup / Services (11g)
• Connect to a TNS-Service via SCAN listeners which is also a CRS-Resource
• EZCONNECT
• Advantage: Services can be dynamically relocated and switched with CRS-commands on SAP instance level
• No more tnsnames.ora entries
listener.ora in GRID-Home
1) SQL*net EZconnect
New naming convention without need for tnsname.ora file
Syntax: <user>/<password>@//<IP-name>:<port>/<service> Or short: //<IP-name>:<port>/<service>
To be used in spfile: local_listener, Remote_listener
Environment : dbs_ora_tnsname
SAP profiles: dbs_ora_tnsname, dbs/ora/tnsname
IP-Name: SCAN-VIP is used only!!
IP addresses
Static (DNS) setup needs
One hostname, ora-vip per host One abap-vip,one java-vip One asinst-vip per asinst 3 Scan IP’s (one name and 3 IPs)
Single Client Access Name Network Configuration for SCAN - DNS
•
•
Requires a DNS adress entry to be used
In DNS, SCAN is a single name defined to resolve to 3 IP-addresses:
• Each cluster will have 3 SCAN-Listeners, combined with a SCAN-VIP defined as cluster resources
The SCAN VIP/LISTENER combination will failover
to another node in the cluster, if the current node fails •
Cluster Resources -------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE dherac1
ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE dherac2
ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE dherac1
Scan.saptest.pub
IN A 133.22.67.194
IN A 133.22.67.193
IN A 133.22.67.192
2) Tnsnames.ora
If there is any requirement for using tnsnames.ora, it‘s still possible
Must reside in $GRID_HOME/network/admin and RDBMS home/network/admin or $TNS_ADMIN
Remove TNS aliases used for 10g local_listener and remote_listener DB parameters from tnsnames.ora
TNS name resolution
Init.ora
Db_domain=‘world‘
Init.ora
Service_names=‘...‘,‘RAC001‘,‘...‘
Sqlnet.ora
NAMES.DEFAULT_DOMAIN=de.oracle.com
Connect sapr3/sap@rac001
TNS-Setup (SAP) with 11g
• Sqlnet.ora: NAMES.DIRECTORY_PATH=(EZCONNECT)
• SAP-Instance connects with sqlplus sapr3/sap@//<scan-vip>:<port>/<service>
sqlplus sapr3/sap@//saprac-scan/D04 (default port#)
• SAP-Startprofile
EZCONNECT = //saprac-scan.de.oracle.com/EDU_D04.world
SETENV_03 = dbs_ora_tnsname=$(EZCONNECT)
( .world comming from db_domain!!)
• SAP-Instanceprofile
//saprac-scan.de.oracle.com/EDU_D04.world
TNS-Setup (Listener)
• lsnrctl status:
Service "EDU_DVEBMGS00_saprac1" has 1 instance(s).
Instance "EDU001", status READY, has 1 handler(s) for this service...
• lsnrctl needs GRID environment for all listeners (node and scan)
TNS-Setup R3trans
• R3trans is needed during SAP startup to DB, i.e. at least one instance is up
It also connects with
check if
• • connect sap<sid>/pwd@<service_name>
•
•
•
•
Service to check all instances
Uses dbs_ora_tnsname
service_names=‘EDU, ...‘
R3trans -x
TNS-Setup (tnsnames.ora) – 11g
EDU.WORLD =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan-vip)(PORT = 1527))
(CONNECT_DATA =
(SERVICE_NAME = EDU.WORLD)
(GLOBAL_NAME = EDU.WORLD) - ERASE!!!!
) )
•
•
• •
No address list
Scan address used only
No TAF or failover entries, provided in CRS – TNS service definition srvctl add service -d <db_unique_name> -s <service_name> -r
"<preferred_instance>" -a "<failover_instances>" -P BASIC -k <net_num> -l PRIMARY -e SELECT -z <failover_retries> -w <failover_delay>
TNS for brtools •
•
•
•
Init.sap: parallel_instances
Init.sap: db_services
password files plus sysdba to system
Tnsnames entries for dedicated single instances EDU001.WORLD =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = saprac1-vip)(PORT = 1527))
)
(CONNECT_DATA =
(SERVICE_NAME = EDU001.WORLD)
)
)
• Static instance definition in listener.ora SID_LIST_LISTENER =
(SID_LIST=(SID_DESC=(SID_NAME=<SID>001)
(ORACLE_HOME=/oracle/<SID>/11202)))
TNS-Settings for 11g
Instance Profile
SAPSYSTEMNAME=EDU
SAPDBHOST=<node vip>
INSTANCE_NAME=D04
dbs/ora/tnsname=//saprac-scan.de.oracle.com/EDU_D04.world
Start Profile
EZCONNECT = //saprac-scan.de.oracle.com/EDU_D04.world
SETENV_03 = dbs_ora_tnsname=$(EZCONNECT)
Parameter Name Value
local_listener //<node vip.domain>:<port>
remote_listener //<SCAN-vip.domain>:<port>
<ORACLE_SID>.service_names (<DBSID>,<ORACLE_SID>)
db_domain World (SAP default, but can also be
removed …)
listener_networks ((name=<network-name>)
(local_listener= "//<second-ora-vip-
of-local-node>:<port>"))
SAP instance startup and TNS 1 SAP instance profile
• SAP instance profile defines settings for particular SAP on instance startup
• SAP instance requires instance specific TNS service
• SAP instance profile parameter :
dbs/ora/tnsname=<scan-ip-name>:<port>/<SAP-instance-TNS-service>
• Dbs/ora/tnsname=xd01-scan.th.oracle.com:1527/E11_D04
• Dbs/ora/tnsname=E11_D04.world (tnsnames.ora)
instance
SAP instance startup and TNS 2 OS environment
• R3trans checks database connection on SAP instance start
• only needs to know if database available, don‘t cares for DB instances
• uses static common service (<SID>.service_names) offered all DB instances
• R3trans uses OS environment variable dbs_ora_tnsnames
by
dbs_ora_tnsname=<scan-ip-name>:<port>/<DBSID-service>
•
•
•
Dbs_ora_tnsname=xd01-scan.th.oracle.com:1527/E11
Dbs_ora_tnsname=E11 (tnsnames.ora)
Problem: OS environment overwrites SAP instance profile setting dbs/ora/tnsnames, solution next slide
SAP instance startup and TNS 3 SAP start profile
• SAP start profile is able to define environment settings particular SAP instance startup
for
•
•
•
overwrites OS environment variables
requires instance specific TNS service
SETENV_03 = dbs_ora_tnsname=<scan-ip-name>:<port>/<SAP-
instance-TNS-service>
• SETENV_03=dbs_ora_tnsname=xd01- scan.th.oracle.com:1527/E11_D04
• SETENV_03=dbs_ora_tnsname=E11_D04.world (tnsnames.ora)
SAPCTL Agenda
•
•
•
•
•
SAP Enqueue Replication
Control Flow SAPCTL
Oracle Clusterware Framework
SAPCTL Commandline Interface
Changes in SAP Profiles / Scripts
Concept of SAP Enqueue Replication
SAP Business Transaction Oracle database transaction
The SAP Enqueue Service provides distributed lock management for SAP R/3 instances.
SAP uses an active/passive approach to enable high availability for the Enqueue Service.
There is exactly one Enqueue Service running in the SAP system. The Replication Service
run on the node on which the Enqueue Service runs in the event of a failure or planned
outage.
The Replication Service maintains a copy of the Enqueue Service´s lock table; this copy is kept in a shared memory segment during runtime.
The node that supports the Enqueue Service also hosts a virtual IP address (VIP) for communication. Whenever the active node fails or is shut down, the VIP and the Enqueue Service fail over to the next node.
Control flow within SAPCTL to
support
Host 1
SAP Enqueue Replication
Host 2
TCP Connection Lock
Table
Enqueue
Server
Lock Replica
Table
Lock State
Transfer
Replication Server
VIP
0. Normal Operation, CRS checks healthiness of resources on all hosts in the cluster
Control flow within SAPCTL to
support
Host 1
SAP Enqueue Replication
Host 2
Crash Lock
Table
Enqueue
Server
Lock Replica
Table
Replication Server
VIP
1. CRS detects a failure of the node or in the Enqueue Server
Control flow within SAPCTL to
support SAP Enqueue Replication
Host 2
Lock Replica
Table
Replication Server
VIP
2. CRS switches the VIP to the node where Replication Server is running
Control flow within SAPCTL to
support SAP Enqueue Replication
Host 2
Enqueue
Server
Lock Replica
Table
Replication Server
3. CRS starts a new Enqueue Server on the node supporting the VIP attaching to
Memory segment of replicator VIP
Control flow within SAPCTL to
support SAP Enqueue Replication
Host 2
Enqueue
Server
Lock Table Replication
Server
4. by attaching to the shared memory
segment hold by replicator, new enqueue
server inherits lock table replica, which
becomes the new lock table
VIP
Control flow within SAPCTL to
support SAP Enqueue Replication
Host 2
Enqueue
Server
Lock Table
5. After new enqueue server took over
lock table, replicator is stopped and
memory is now owned by new enqueue
server; SAP transactions can continue
VIP
Control flow within SAPCTL to
support SAP Enqueue Replication
Host N Host 2
Lock
Table
Enqueue
Server
Lock Replica
Table
Replication Server
VIP
6. CRS starts Replication Server on another available node in the cluster
Control flow within SAPCTL to
support SAP Enqueue Replication
Host N Host 2
Connect Lock
Table
Enqueue
Server
Lock State
Transfer Lock Replica
Table
Replication Server
VIP
7. Back to normal operation, CRS checks healthiness of resources on all hosts in the cluster
Possible Configuration
SAP Enqueue Repl. SAP Enqueue Server SAP Enqueue Repl. SAP Enqueue Repl.
listener listener
cluster database cluster database
ORACLE_HOME ORACLE_HOME
ORA_CRS_HOME ORA_CRS_HOME ORA_CRS_HOME ORA_CRS_HOME
Shared Storage
• Oracle Clusterware HA Agent for SAP is protecting SAP Enqueue Server and SAP Enqueue Replication Server It is also providing resources to the RAC database running on
nodes 1 & 2
•
The Oracle Clusterware Framework
•
•
Calls SAP HA – agent
agent will be invoked by
of three commands
Oracle Clusterware with one
•
•
•
start
stop
check
• Oracle Cluster Registry
• Stores information relating interacts and manages the
to how Oracle Clusterware application
Oracle Clusterware Framework Architecture
Framework
Please Start
Agent
‘start’
‘check’ How are you ?
Please Stop ‘stop’
How the HA Agent for SAP is called by the Framework The ‘start’ command invokes the SAP tool sapcontrol
command depending on CRS resource type to issue the start
- for startservice process
CRS resource type SAP.<SID>.<INST>.startsrv
sapcontrol -nr <instance-nr> -function StartService <SID>
- for the SAP instance
CRS resource type SAP.<SID>.<INST>.<type>
<type> qualifies instance type, e.g. abapenq, gwinst, webdisp
sapcontrol -nr <instance-nr>
-function Start
How the HA Agent for SAP is called by the Framework The ‘stop’ command invokes the SAP tool sapcontrol
command depending on CRS resource type to issue the stop
- for startservice process
CRS resource type SAP.<SID>.<INST>.startsrv
sapcontrol -nr <instance-nr> -function StopService
- for the SAP instance
CRS resource type SAP.<SID>.<INST>.<type>
<type> qualifies instance type, e.g. abapenq, gwinst,
sapcontrol -nr <instance-nr>
-function Stop
webdisp
How the HA Agent for SAP is called by the Framework
The ‘check’ command invokes the SAP tool
check function
sapcontrol to issue the
- for startservice process and SAP instance
All CRS resource types defined for SAP
sapcontrol -nr <instance-nr>
-function GetProcessList
No specific check for different instance type, e.g. by calls to
ensmon, msprot, dpmon, gwmon. All is covered by sapcontrol.
Overview: SAPCTL commands for with SAP R/3 Enqueue Replication
use
•
•
•
•
•
•
sapctl
sapctl
sapctl
sapctl
sapctl
sapctl
create
status
start
stop
relocate
remove
- registers resources
- shows status of resources
-
-
-
-
starts resources
stops resources
move resource to different
delete configuration
host
SAPCTL command create
• sapctl create
-sapsid EDU
-if eth0
-nm 255.255.254.0
-nodes node1,node2
-abapenq ASCS00
-abaprep ERS01
-abapvip 140.86.242.75
-abapmsport 3601
-javaenq SCS02
-javarep ERS03
-javavip 140.86.242.76
-javamsport 3902
-asinst DVEBMGS04
-asinstvip 140.86.242.77
<- name of the SAP System
<- network interface
<- netmask
<- list of nodes
<- Enqueue Service Identifier for WEB AS ABAP
<- Replication Service Identifier for WEB AS ABAP
<- VIP Address for WEB AS ABAP
<- Portnumber of ABAP message server
<- Enqueue Service Identifier for WEB AS JAVA
<- Replication Service Identifier for WEB AS JAVA
<- VIP Address for WEB AS JAVA
<- Portnumber of JAVA message server
<- Instance name SAP Central Instance
<- VIP Address for SAP Central Instance
SAPCTL command config ABAP
• sapctl config
shows list of protected SAP systems
• sapctl config abapenq –sapsid EDU
shows configuration of Central Instance
• sapctl config abaprep –sapsid EDU
shows configuration of Replication Server
• sapctl config all –sapsid EDU
shows complete configuration for SAP system
SAPCTL command status
• sapctl status abapall –sapsid EDU
shows status of ABAP Enqueue and Replication
• sapctl status javaall –sapsid EDU
shows status of JAVA Enqueue and Replication
• sapctl status all –sapsid EDU
shows status of all SAP resources
SAPCTL command start
• sapctl start abapenq –sapsid EDU
starts SAP ABAP Enqueue instance
• sapctl start abaprep –sapsid EDU
starts SAP ABAP Replication instance
• sapctl start all –sapsid EDU
starts all SAP resources
SAPCTL command stop
• sapctl stop javaenq –sapsid EDU
stops SAP JAVA Enqueue instance
• sapctl stop javarep –sapsid EDU
stops SAP JAVA Replication instance
• sapctl stop all –sapsid EDU
stops all SAP resources
SAPCTL command update
• sapctl update
-sapsid EDU <- name of the SAP System
-abapvip 140.86.242.63
-javapvip 140.86.242.63
<-
<-
<-
IP address VIP
IP address VIP
IP address VIP -asinstvip 140.86.242.63
-a node
-d node
<- node to add
<- node to delete
SAPCTL command relocate
• sapctl relocate
-sapsid EDU <- name of the SAP System
-abapenq ASCS01
-javaenq SCS02
-abaprep ERS11
-javarep ERS12 -asinst DVEBMGS00
<-
<-
<-
<- <-
ABAP Enqueue instance
JAVA Enqueue instance
ABAP Replication instance
JAVA Replication instance SAP CI central instance
-to node <- where to relocate
SAPCTL command remove
• sapctl remove all –sapsid EDU
removal of all SAP resources
- only remove of all SAP resources possible
- `root` permission required
Supported SAP instance types
ASCS
SCS ERS
DVEBMGS
DV
GW
WebDisp
Central Services ABAP
Central Services JAVA
Replication Service ABAP / JAVA
Cental Instance (e.g. For solution
Application Instance
Standalone Gateway
SAP WebDispatcher
manager)
Sapstartsrv SAP Start Service per Instance
SAPCTL installation location
sapctl uses direrctory /usr/sap/sapctl.
Perl scripts in /usr/sap/sapctl/bin
Logs in /usr/sap/sapctl/log
Copy of Perl interpreter in /usr/sap/sapctl/perl
Symbolic link to crsctl and crsctl.bin in /usr/sap/sapctl/bin
The sapctl package
sapctl
crssapactions.pl
has two perl scripts:
Command line interface
Action script called by CRS
No wrapper, no user environment, no configuration required.
No change in SAP scripts. Instance specific information is stored
In CRS resource.
SAP changes required to use SAP HA
• Instance profile Standalone Enqueue Server (ENQ + MSG)
Instance profile Enqueue Replication Server (ENR)
enque/process_location=REMOTESA
(in DEFAULT.PFL important for all dialog instances)
enque/server/replication=true
enque/server/internal_replication= true
•
•
•
Start profile Standalone Enqueue Server
Start profile Enqueue replication Server
As recommended by SAP HA documentation
Server Pool SAPCTL
SAP_SP_<SID> defined by
• The Server Pool SAP_SP_<SID> contains the list of cluster nodes where SAP resources can run on. All CRS resources for a specific
SAP instance are associated with the Server Pool
• Adding or deleting nodes can be done by modifying the Server Pool attribute SERVER_NAMES
• The Server Pool is owned by root, so root permission is required when changing attribute values
List of CRS Types used for SAPCTL
Defined by SAPCTL create command
•
•
•
•
•
•
sap.abapenq.type
sap.abaprep.type
sap.javaenq.type
sap.javarep.type
sap.startsrv.type
app.appvip.type
Already existing, defined by CRS
•
•
ora.network.type
cluster_resource
List of CRS SAPCTL
Resources used for
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
sap.<SID>.ASCS<NR>.startsrv
sap.<SID>.ASCS<NR>.abapenq
sap.<SID>.SCS<NR>.startsrv
sap.<SID>.SCS<NR>.javaenq
sap.<SID>.ERS<NR>.startsrv
sap.<SID>.ERS<NR>.abaprep
sap.<SID>.abapvip
sap.<SID>.javavip
sap.<SID>.DVEBMGS<NR>.startsrv
sap.<SID>.DVEBMGS<NR>.sapinst
sap.<SID>.DVEBMGS<NR>.instvip
sap.<SID>.DV<NR>.startsrv
sap.<SID>.DV<NR>.sapinst
sap.<SID>.DV<NR>.instvip
sap.<SID>.G<NR>.startsrv
sap.<SID>.G<NR>.sapgw
sap.<SID>.G<NR>.gwvip
sap.<SID>.W<NR>.startsrv
sap.<SID>.W<NR>.wdspinst
sap.<SID>.W<NR>.wdspvip
ora.net<x>.network
Dependencies for SAPCTL
between resources used
(A)SCS ERS
Start
Stop
Attraction
Dispersion
StartService StartService
Application vip
Dependencies for SAPCTL
between resources used
DVEBMGS,DV,GW,WebDisp
Start
Stop
StartService
Application vip
SAPCTL Log Files
• The logfile sapctl.log shows all actions performed by sapctl. All actions as well as output of system commands is recorded here.
Structure of log entries:
<host> date time | pid | trace information of executed program steps
Output of calls to SAP tool sapcontrol is also recorded
Similar log in
${ORA_CRS_HOME}/log/<hostname>/agent/crsd/scriptagent_<sidadm>
•
•
•
SAP Installation on DB nodes 1 Overview
• Recommended: NFS mount /sapmnt of existing SAP system can be mounted on DB nodes as usual
•
•
to provide profiles
No general need for HA NFS
Able to host multiple platform binaries
• 1. DB
Step DB node preparation using sapinst on nodes
all
•
•
•
2.
3.
4.
Step install ASCS on all nodes using sapinst
Step install ERS on all nodes using sapinst
maintain ERS profiles
SAP Installation on DB nodes 2 Overview
•
•
•
5. Step: manually test ASCS and ERS
optional: JAVA installation (SCS + another
6. Step: install sapctl
ERS)
SAP Installation on DB nodes 3 Requirements
•
•
•
SAP Kernel CD (minimum 7.00 EP 3 or 7.20 EP 1 and higher)
Sapinst support starting from version 7.0 EP 3 and 7.3 EP 1
SAP OSS note 1619343 "SAPinst for Oracle Exadata on Oracle Linux and Solaris X86“
• Download Software Provisioning Manager from SWDC Hint:
Install VNCserver on DB nodes to locally run sapinst (no need for JCE, Java Cryptography Extension policy file) Requires one virtual IP for ABAP (<ABAP-VIP>) and one for JAVA (<JAVA-VIP>), if used
•
•
1. Step – prepare node 1
• OSS 1619343 - SAPinst for Oracle Exadata and Solaris X86
• Make use of NFS for /sapmnt
on Oracle Linux
• Run sapinst on all DB nodes for Exadata Node Preparation
1. Step – prepare node 2
•
•
•
•
•
•
start DB node preparation using sapinst to
Create OS users and groups
OS environment, Oracle client
SAP kernel plus brtools
Requires SAP kernel CD and ORACLE client
Must be executed on each node
additionally
2. Step – ASCS installation 1
• run sapinst on node 1 having /sapmnt mounted all DB nodes
• start with NO SAP profile
to share on
• unmount /sapmnt on all other DB nodes before starting sapinst (local /sapmnt files will not be used, just for install)
• use virtual hostname <ABAP-VIP>
• use of environment SAPINST_USE_HOSTNAME = < ABAP-VIP >
• Bind <ABAP-VIP> locally on node for sapinst run crsctl start ip –A <ABAP-VIP>/<netmask>/eth0
crsctl stop ip –A <ABAP-VIP>/<netmask>/eth0
• Run sapinst for ASCS installation on all remaining DB nodes
3. Step – ERS installation 1
• except node 1, all other DB nodes run sapinst /sapmnt NFS mount
• start with NO SAP ERS profile
without
• NO use of environment SAPINST_USE_HOSTNAME
• Run sapinst for ERS installation on all remaining DB nodes
• Merge ERS profiles on „shared“ NFS mounted /sapmnt
• ERS for ABAP and ERS for JAVA must use different instance numbers
4. Step – maintain SAP DEFAULT.PFL
profiles 1
SAPSYSTEMNAME = <SAP-SID>
SAPGLOBALHOST = <ABAP-VIP>
SAPDBHOST = <ABAP-VIP>
rdisp/mshost = <ABAP-VIP> rdisp/msserv_internal = 39<ASCS-Instance#>
rdisp/msserv = sapms<SAP-SID>
enque/process_location = REMOTESA enque/serverhost = <ABAP-VIP>
enque/serverinst = <ASCS-Instance#>
4. Step – maintain DEFAULT.PFL
SAP profiles 2
dbms/type = ora
j2ee/dbtype j2ee/dbname
j2ee/dbhost
= =
=
ora <SAP-SID>
<JAVA-VIP>
= <JAVA-VIP> j2ee/scs/host j2ee/scs/system = <SCS-Instance#> j2ee/ms/port = 39<SCS-Instance#>
rdisp/vb_dispatching = 0
4. Step – maintain SAP profiles AS Instance profiles
SAPSYSTEM = <local-SAP-Instance#>
INSTANCE_NAME = <local-SAP-Instance-name>
SAPLOCALHOST = <local-Hostname>
rdisp/wp_no_vb = <# > 1> rdisp/wp_no_vb2 = <# > 1>
rdisp/vbname = <local-Hostname>
dbs/ora/tnsname = //<SCAN-NAME>[:<PORT>]/<SAP- SID>_<SAP-Instance-name>
5. Step – test ASCS & ERS 1
• Bind <ABAP-VIP> on node 1 crsctl start ip –A <ABAP-VIP>/<netmask>/eth0
•
•
•
Start ASCS on node 1
Check ASCS
On remaining DB nodes one by one
•
•
•
Start ERS
Check ERS
Stop ERS
• Unbind <ABAP-VIP> on node 1 crsctl start ip –A <ABAP-VIP>/<netmask>/eth0
• Repeat on next node
5. Step – test ASCS & ERS 2
• Check ASCS sapcontrol –host <ABAP-VIP> -prot NI_HTTP -nr <#> -function
GetProcessList
25.04.2013 18:13:28 GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
ms.sapE11_ASCS01, , GREEN, Running, 2013 04 25 08:35:33, 9:40:57, 57023
en.sapE11_ASCS01, , GREEN, Running, 2013 04 25 08:35:33, 9:40:57, 57024
5. Step – test ASCS & ERS 3
• Check ERS sapcontrol -host <host> -prot NI_HTTP -nr <#> -function GetProcessList
25.04.2013 18:14:34 GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
enrepserver, EnqueueReplicator, GREEN, Running, 2013 04 25 12:56:25,
5:21:11, 103217
6. Step – Sapctl installation
1. Download sapctl package is attached to OSS note Note 1496927 - Protection of Enqueue Replication through Oracle Clusterware
2.
3.
4.
unpack SAPCTL.tgz into a temporary directory
create 2 symbolic links for crsctl and crsctl.pl
copy perl directory from ORACLE_HOME
•
•
•
•
2 Versions: one for 10g and one for 11g
Use enclosed documentation
New directory structure: /usr/sap/sapctl/bin and
Starting/stopping/control by sapcontrol (11g)
log
Sapctl – Definition of Resources
• Example: sapctl create -sapsid EDU -if eth0 -nm
255.255.255.0 -nodes saprac1,saprac2 -abapvip 192.162.119.145 -abapenq ASCS01 -abaprep
ERS11
Backup, Restore and Recovery in RAC with or without ASM/Exadata
•
•
No differences to Single Instance
Recovery: Could be done in parallel by multiple
instances using parallel recovery using PQX
slaves of mounted instances
Archiving to shared directory or ASM or FRA
For ASM or FRA use RMAN and brtools
ASM: DB consistency check only with RMAN
•
•
•
Brtool ASM/Exadata support
•
•
•
•
•
•
•
•
•
•
One dedicated node, not for Exadata
Shared Oracle Home (init<SID>.sap), not for Exadata
Brtools support Exadata starting 7.20 patch
SYSTEM needs SYSOPER or SYSDBA
Static Instance definition in listener.ora
PWD file
brbackup
brarchive
RMAN, 3rd party backup (backint)
ASM: RMAN plus brtools only
18
Brtool ASM/Exadata support
• brrecover
•
•
•
Do
brrecover
brrecover
brrecover
–u
–u
–u
/
/
/
-t
-t
-t
complete
dbpit
reset
•
•
•
•
not use brrestore with ASM.
OMF format is used for datafile creation.
Can not move/rename OMF file, alias can be changed.
<sid>adm can ssh to <oracle> with no password
within and between all nodes.
Brtool configuration
•
•
•
Support from BR*Tools 7.20 level 18 and up
All GRID/RDBMS installed under oracle user.
Only
•
supported backup types
backup_dev_type = rman_util | rman_disk
rman_stage backup_dev_type =
disk disk_copy_cmd = rman |
rman_set backup_dev_type = tape
| pipe
tape_copy_cmd = rman
|
•
•
• All backup strategies with RMAN are supported
Brtool configuration
• All br* executable has to own by oracle with 6774 brarchive*
brbackup*
brconnect*
brrecover*
brrestore*
-rwsrwsr--
-rwsrwsr--
-rwsrwsr--
-rwsrwsr--
-rwsrwsr--
-rwsrwsr--
-rwsrwsr--
1
1
1
1
1
1
1
oracle
oracle
oracle
oracle
oracle
oracle
oracle
oinstall
oinstall
oinstall
oinstall
oinstall
oinstall
oinstall
7732338
7908129
9970354
8376747
2783544
May
May
May
May
May
31
31
31
31
31
16:30
16:30
16:30
16:31
16:31
10479944 May 31 16:31 brspace*
4103679 May 31 16:31 brtools*
• All directories in SAP data home /oracle/<SID> should owned by oracle:oinstall and have g+w permission.
new sapprof contains init.ora and init.sap for brtools
brtools should initiate from <sid>adm account.
be
•
•
Brtools configuration
• oracle user account should contain brtool environment variables.
OPS$oracle account should be created. • SQL>
SQL>
SQL>
connect / as sysdba
create user ops$oracle identified externally;
grant sapdba to ops$oracle;
Brtools configuration
• Use ASM alias for spfile and controlfile
<asm_disk_group>/<db_name>/spfile<db_name>.ora
<asm_disk_group>/<db_name>/cntrl<db_name>.dbf
Use ASM alias for archive destination
log_archive_dest_1 = "LOCATION=+ARCH/EDU/oraarch“
•
SQL>
SQL>
SQL>
connect / as sysdba
alter diskgroup ARCH add directory '+ARCH/EDU';
alter diskgroup ARCH add directory '+ARCH/EDU/oraarch';
Brtools configuration
• Need
•
•
•
Special init.sap parameters for Exadata:
loc_ora_homes = yes
asm_root_dir = ASM
asm_ora_sid = (<db_inst1>:<asm_inst1>,
<db_inst2>:<asm_inst2>, ...)
asm_ora_home = (<db_inst1>:<asm_home1>,
<db_inst2>:<asm_home2>, ...)
•
Backup/Restore of ASM Data
•
•
•
Normal RMAN backups
Split Mirror backups
Special cases:
• Loss of Oracle cluster registry and
MOS note 1062983.1
voting files
• ASM metadata restore, MOS note 451900.1