using the oracle asm cluster file system (oracle acfs) on linux - (11gr2)

7/29/2019 Using the Oracle ASM Cluster File System (Oracle ACFS) on Linux - (11gR2)

1/43

Contents

Introduction

ACFS Components

ACFS Prerequisites

ASM Configuration Assistant (ASMCA)

Oracle Enterprise Manager (OEM)

Command Line / SQL

ACFS Snapshots

Managing ACFS

About the Author

Introduction

Introduced with Oracle ASM 11g Release 2, Oracle ASM Cluster File System (Oracle ACFS) is a generalpurpose POSIX compliant cluster file system implemented as part of Oracle Automatic StorageManagement (Oracle ASM). Being POSIX compliant, all operating system utilities we use with ext3 and

other file systems can also be used with Oracle ACFS given it belongs to the same family of relatedstandards. Oracle ACFS extends the Oracle ASM architecture and is used to support many types of fileswhich are typically maintained outside of the Oracle database. For example Oracle ACFS can be used tostore BFILEs, database trace files, executables, report files and even general purpose files like image,text, video, and audio files. In addition, Oracle ACFS can be used as a shared file system for Oraclehome binaries.

The features included with Oracle ACFS allow users to create, mount, and manage ACFS using familiarLinux commands. Oracle ACFS provides support for snapshots and the ability to dynamically resizeexisting file systems online using Oracle ASM Dynamic Volume Manager (ADVM).

Oracle ACFS leverages Oracle ASM functionality that enables:

Oracle ACFS dynamic file system resizing

Maximized performance through direct access to Oracle ASM disk group storage

Balanced distribution of Oracle ACFS across Oracle ASM disk group storage for increased I/O

parallelism

Data reliability through Oracle ASM mirroring protection mechanisms

While Oracle ACFS is useful for storing general purpose files, there are certain files that it is not meantfor. For example, Oracle ASM (traditional disk groups) is still the preferred storage manager for alldatabase files because Oracle ACFS does not support directIO for file read and write operations in 11gRelease 2 (11.2). Oracle ASM was specifically designed and optimized to provide the best performancefor database file types. In addition to Oracle database files, Oracle ACFS does not support files for theOracle Grid Infrastructure home. Finally, Oracle ACFS does not support any Oracle files that can be

directly stored in Oracle ASM. For example, SPFILE, flashback log files, control files, archived redo logfiles, the Grid Infrastructure OCR and voting disk, etc should be stored in Oracle ASM disk groups. Thekey point to remember is that Oracle ACFS is the preferred file manager for non-database files and isoptimized for general purpose / customer files which are maintained outside of the Oracle database.

This article describes three ways to create an Oracle ASM Cluster File System in an Oracle 11g Release2 RAC database on the Linux operating environment:

http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Introductionhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Componentshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Prerequisiteshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQLhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Snapshotshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Managing%20ACFShttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#About%20the%20Authorhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20ASM%20Dynamic%20Volume%20Manager%20(ADVM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Componentshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Prerequisiteshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQLhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ACFS%20Snapshotshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Managing%20ACFShttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#About%20the%20Authorhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20ASM%20Dynamic%20Volume%20Manager%20(ADVM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Introduction


2/43


Command Line / SQL

There is actually a fourth method that can be employed to create an Oracle ASM Cluster File Systemwhich is to use the ASMCMD command line interface. Throughout this guide, I'll demonstrate how to usethe ASMCMD command line interface in place of SQL where appropriate.

The Linux distribution used in this guide is CentOS 5.5. CentOS is a free Enterprise-class LinuxDistribution derived from the Red Hat Enterprise Linux (RHEL) source and aims to be 100% binarycompatible. Although Centos 5 is equivalent to RHEL 5, the CentOS operating system is not supported byOracle ASM Cluster File System. Refer to the workaround documented in the prerequisites section of thisarticle if you are using CentOS or a similar Red Hat clone.

It is assumed that an Oracle RAC database isalready installed, configured, and running. Refertothis guidefor instructions on how to build aninexpensive two-node Oracle RAC 11g Release 2database on Linux.

ACFS Components

Before diving into the details on how to create and manage Oracle ASM Cluster File System, it may behelpful to first discuss the major components.

Figure 1 shows the various components that make up Oracle ACFS and provides an illustration of theexample configuration that will be created using this guide.
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQLhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driverhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Figure%201http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQLhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driverhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Figure%201


3/43

Figure 1: Oracle ASM Cluster File System Components

Everything starts with an Oracle ASM disk group. An Oracle ASM disk group is made up of one or moredisks shown in figure 1 as DOCSDG1. The next component is an Oracle ASM volume which is created

within an Oracle ASM disk group. The example configuration illustrated above shows that we will becreating three volumes named docsvol1, docsvol2, and docsvol3 on the new disk group

namedDOCSDG1. Finally, we will be creating a cluster file system for each volume whose mount points will

be /documents1, /documents2, and /documents3 respectively.

With Oracle ACFS, as long as there exists free space within the ASM disk group, any of the volumes canbe dynamically expanded which means the file system gets expanded as a result. As I willdemonstratelater in this article, expanding a volume / file system is an effortless process and can beperformed online without the need to take any type of outage!

Oracle ASM Dynamic Volume Manager (ADVM)

Besides an Oracle ASM disk group, another key component to Oracle ACFS is the new Oracle ASMDynamic Volume Manager (ADVM). ADVM provides volume management services and a standard driverinterface to its client (ACFS, ext3, ext4, reiserfs, OCFS2, etc.). The ADVM services include functionality tocreate, resize, delete, enable, and disable dynamic volumes. An ASM dynamic volume is constructed outof an ASM file with an 'ASMVOL' type attribute to distinguish it from other ASM file types (i.e. DATAFILE,

TEMPFILE, ONLINELOG, etc.):
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Figure%201http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Figure%201http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20System


4/43

ASM File Name / Volume Name / Device Name BytesFile Type--------------------------------------------------------------- ------------------------------------+CRS/racnode-cluster/ASMPARAMETERFILE/REGISTRY.253.734544679 1,536

ASMPARAMETERFILE+CRS/racnode-cluster/OCRFILE/REGISTRY.255.734544681 272,756,736OCRFILE

------------------272,758,272

+DOCSDG1 [DOCSVOL1] /dev/asm/docsvol1-300 34,359,738,368ASMVOL+DOCSDG1 [DOCSVOL2] /dev/asm/docsvol2-300 34,359,738,368ASMVOL+DOCSDG1 [DOCSVOL3] /dev/asm/docsvol3-300 26,843,545,600ASMVOL

------------------95,563,022,336

+FRA/RACDB/ARCHIVELOG/2010_11_08/thread_1_seq_69.264.734565029 42,991,616ARCHIVELOG+FRA/RACDB/ARCHIVELOG/2010_11_08/thread_2_seq_2.266.734565685 41,260,544ARCHIVELOG

< SNIP >

+FRA/RACDB/ONLINELOG/group_3.259.734554873 52,429,312ONLINELOG+FRA/RACDB/ONLINELOG/group_4.260.734554877 52,429,312ONLINELOG

------------------

12,227,537,408

+RACDB_DATA/RACDB/CONTROLFILE/Current.256.734552525 18,890,752CONTROLFILE+RACDB_DATA/RACDB/DATAFILE/EXAMPLE.263.734552611 157,294,592DATAFILE+RACDB_DATA/RACDB/DATAFILE/SYSAUX.260.734552569 1,121,984,512DATAFILE+RACDB_DATA/RACDB/DATAFILE/SYSTEM.259.734552539 744,497,152DATAFILE+RACDB_DATA/RACDB/DATAFILE/UNDOTBS1.261.734552595 791,683,072DATAFILE+RACDB_DATA/RACDB/DATAFILE/UNDOTBS2.264.734552619 209,723,392DATAFILE+RACDB_DATA/RACDB/DATAFILE/USERS.265.734552627 5,251,072DATAFILE+RACDB_DATA/RACDB/ONLINELOG/group_1.257.734552529 52,429,312ONLINELOG+RACDB_DATA/RACDB/ONLINELOG/group_2.258.734552533 52,429,312ONLINELOG+RACDB_DATA/RACDB/ONLINELOG/group_3.266.734554871 52,429,312ONLINELOG+RACDB_DATA/RACDB/ONLINELOG/group_4.267.734554875 52,429,312


5/43

ONLINELOG+RACDB_DATA/RACDB/PARAMETERFILE/spfile.268.734554879 4,608PARAMETERFILE+RACDB_DATA/RACDB/TEMPFILE/TEMP.262.734552605 93,331,456TEMPFILE+RACDB_DATA/RACDB/spfileracdb.ora 4,608

PARAMETERFILE------------------

3,352,382,464

Oracle ACFS and other supported 3rd party file systems can use Oracle ADVM as a volume managementplatform to create and manage file systems while leveraging the full power and functionality of Oracle

ASM features. A volume may be created in its own Oracle ASM disk group or can share space in analready existing disk group. Any number of volumes can be created in an ASM disk group. Creating anew volume in an Oracle ASM disk group can be performed using theASM Configuration Assistant(ASMCA),Oracle Enterprise Manager (OEM), SQL, or ASMCMD. For example:

asmcmd volcreate -G docsdg1 -s 20G docsvol3

Once a new volume is created in Linux, the ADVM device driver automatically creates a volume device onthe OS that is used by clients to access the volume. These volumes may be used as block devices, maycontain a file system such as ext3, ext4, reiserfs, OCFS2 or Oracle ACFS may used (as described in thisguide) in which case the oracleacfs driver is also used for I/O to the file system.

On the Linux platform, Oracle ADVM volume devicesare created as block devices regardless of theconfiguration of the underlying storage in the Oracle

ASM disk group. Do not use raw (8) to map OracleADVM volume block devices into raw volume

devices.

Under Linux, all volume devices are externalized to the OS and appear dynamically as special files inthe /dev/asm directory. In this guide, we will use this OS volume device to create an Oracle ACFS.

$ ls -l /dev/asm/total 0brwxrwx--- 1 root asmadmin 252, 153601 Nov 28 13:49 docsvol1-300brwxrwx--- 1 root asmadmin 252, 153602 Nov 28 13:49 docsvol2-300brwxrwx--- 1 root asmadmin 252, 153603 Nov 28 13:56 docsvol3-300

$ /sbin/mkfs -t acfs -b 4k /dev/asm/docsvol3-300 -n "DOCSVOL3"

Oracle ADVM implements its own extent and striping algorithm to ensure the highest performance forgeneral purpose files. An ADVM volume is four columns of 64MB extents and 128KB stripe width bydefault. ADVM writes data in 128KB stripes in a Round Robin fashion to each column before starting onthe next four column extents. ADVM uses Dirty Region Logging (DRL) for mirror recovery after a node orinstance failure. This DRL scheme requires a DRL file in the ASM disk group to be associated with each

ASM dynamic volume.
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQLhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Command%20Line%20/%20SQL


6/43

ACFS Prerequisites

Install Oracle Grid Infrastructure

Oracle Grid Infrastructure 11g Release 2 (11.2) or higher is required for Oracle ACFS. Oracle GridInfrastructure includes Oracle Clusterware, Oracle ASM, Oracle ACFS, Oracle ADVM, and driver

resources software components, which are installed into the Grid Infrastructure Home using the OracleUniversal Installation (OUI) tool. Refer to this guide for instructions on how to configure Oracle GridInfrastructure as part of an Oracle RAC 11g Release 2 database install on Linux.

Log In as the Grid Infrastructure User

To perform the examples demonstrated in this guide, it is assumed that the Oracle Grid Infrastructureowneris 'grid'. If the Oracle Grid Infrastructure owner is 'oracle', then log in as the oracle account.

Log in as the Oracle Grid Infrastructure owner and switch to the Oracle ASM environment on node 1 ofthe RAC when performing non-root ACFS tasks:

[grid@racnode1 ~]$ iduid=1100(grid) gid=1000(oinstall)groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)

[grid@racnode1 ~]$ . oraenvORACLE_SID = [+ASM1] ? +ASM1The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/grid

[grid@racnode1 ~]$ dbhome/u01/app/11.2.0/grid

[grid@racnode1 ~]$ echo $ORACLE_SID

+ASM1

Verify / Create ASM Disk Group

After validating the Oracle Grid Infrastructure installation and logging in as the Oracle Grid Infrastructureowner (grid), the next step is to decide which Oracle ASM disk group should be used to create the

Oracle ASM dynamic volume(s). The following SQL demonstrates how to search the available ASM diskgroups:

break on inst_id skip 1

column inst_id format 9999999 heading "Instance ID" justify leftcolumn name format a15 heading "Disk Group" justify leftcolumn total_mb format 999,999,999 heading "Total (MB)" justify rightcolumn free_mb format 999,999,999 heading "Free (MB)" justify rightcolumn pct_free format 999.99 heading "% Free" justify right

======================================================================

SQL> select inst_id, name, total_mb, free_mb, round((free_mb/total_mb)*100,2)
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#About%20Oracle%20Grid%20Infrastructure%2011g%20Release%202%20(11.2)http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Create%20Job%20Role%20Separation%20Operating%20System%20Privileges%20Groups,%20Users,%20and%20Directorieshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Create%20Job%20Role%20Separation%20Operating%20System%20Privileges%20Groups,%20Users,%20and%20Directorieshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Create%20Job%20Role%20Separation%20Operating%20System%20Privileges%20Groups,%20Users,%20and%20Directorieshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#About%20Oracle%20Grid%20Infrastructure%2011g%20Release%202%20(11.2)http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtmlhttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Create%20Job%20Role%20Separation%20Operating%20System%20Privileges%20Groups,%20Users,%20and%20Directorieshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Create%20Job%20Role%20Separation%20Operating%20System%20Privileges%20Groups,%20Users,%20and%20Directories


7/43

pct_free2 from gv$asm_diskgroup3 where total_mb != 04 order by inst_id, name;

Instance ID Disk Group Total (MB) Free (MB) % Free

----------- --------------- ------------ ------------ -------1 CRS 2,205 1,809 82.04FRA 33,887 24,802 73.19RACDB_DATA 33,887 30,623 90.37

2 CRS 2,205 1,809 82.04FRA 33,887 24,802 73.19RACDB_DATA 33,887 30,623 90.37

The same task can be accomplished usingthe ASMCMD command-line utility:[grid@racnode1 ~]$ asmcmd lsdg

If you find an existing Oracle ASM disk group that has adequate space, the Oracle ASM dynamicvolume(s) can be created on that free space or a new ASM disk group can be created.

For the purpose of this guide, I will be creating a dedicated Oracle ASM disk groupnamed DOCSDG1 which will be used for all three Oracle ASM dynamic volumes. I already setup a

sharediSCSI volume and provisioned it usingASMLib. The ASMLib shared volume that will be used tocreate the new disk group is named ORCL:ASMDOCSVOL1.

[grid@racnode1 ~]$ sqlplus / as sysasm

SQL> select path, name, header_status, os_mb from v$asm_disk;

PATH NAME HEADER_STATUS OS_MB------------------ --------------- ------------- ----------ORCL:ASMDOCSVOL1 PROVISIONED 98,303ORCL:CRSVOL1 CRSVOL1 MEMBER 2,205ORCL:DATAVOL1 DATAVOL1 MEMBER 33,887ORCL:FRAVOL1 FRAVOL1 MEMBER 33,887

After identifying the ASMLib volume and verifying it is accessible from all Oracle RAC nodes, log in to theOracle ASM instance and create the new disk group from one of the Oracle RAC nodes. After verifyingthe disk group was created, log in to the Oracle ASM instance on all other RAC nodes and mount the new

disk group:


SQL> CREATE DISKGROUP docsdg1 EXTERNAL REDUNDANCY DISK 'ORCL:ASMDOCSVOL1' SIZE 98303

Diskgroup created.
http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Configure%20iSCSI%20Volumes%20on%20Oracle%20RAC%20Nodeshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Configure%20iSCSI%20Volumes%20on%20Oracle%20RAC%20Nodeshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Install%20and%20Configure%20ASMLib%202.0http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Install%20and%20Configure%20ASMLib%202.0http://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Configure%20iSCSI%20Volumes%20on%20Oracle%20RAC%20Nodeshttp://www.idevelopment.info/data/Oracle/DBA_tips/Oracle11gRAC/CLUSTER_12.shtml#Install%20and%20Configure%20ASMLib%202.0


8/43

SQL> @asm_diskgroups

Disk Group Sector Block AllocationName Size Size Unit Size State Type Total Size (MB) Used Size (MB)---------- ------- ------ ----------- -------- ------ --------------- --------------

CRS 512 4,096 1,048,576 MOUNTED EXTERN 2,205 396DOCSDG1 512 4,096 1,048,576 MOUNTED EXTERN 98,303 50FRA 512 4,096 1,048,576 MOUNTED EXTERN 33,887 9,085RACDB_DATA 512 4,096 1,048,576 MOUNTED EXTERN 33,887 3,264

--------------- --------------Grand Total: 168,282 12,795

====================================================================================


SQL>ALTER DISKGROUP docsdg1 MOUNT;

Diskgroup altered.

SQL> @asm_diskgroups

Disk Group Sector Block AllocationName Size Size Unit Size State Type Total Size (MB) Used Size (MB)---------- ------- ------ ----------- -------- ------ --------------- --------------CRS 512 4,096 1,048,576 MOUNTED EXTERN 2,205 396DOCSDG1 512 4,096 1,048,576 MOUNTED EXTERN 98,303 50FRA 512 4,096 1,048,576 MOUNTED EXTERN 33,887 9,085

RACDB_DATA 512 4,096 1,048,576 MOUNTED EXTERN 33,887 3,264--------------- --------------

Grand Total: 168,282 12,795

Verify Oracle ASM Volume Driver

The operating environment used in this guide is CentOS 5.5 x86_64:

[root@racnode1 ~]# uname -aLinux racnode1 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_6GNU/Linux

On supported operating systems, the Oracle ACFS modules will be configured and the Oracle ASMvolume driver started by default after installing Oracle Grid Infrastructure. With CentOS and otherunsupported operating systems, a workaround is required to enable Oracle ACFS. One of the first tasks isto manually start the Oracle ASM volume driver:

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/acfsload start -s


9/43

ADVM/ACFS is not supported on centos-release-5-5.el5.centos

The failed output from the above command should come as no surprise given Oracle ACFS is notsupported on CentOS.

By default, the Oracle ACFS modules do not get installed on CentOS because it is not a supportedoperating environment. This section provides a simple, but unsupported, workaround to get Oracle ACFSworking on CentOS. This workaround includes some of the manual steps that are required to launch theOracle ASM volume driver when installing Oracle ACFS on a non-clustered system.

The steps documented in this section serve as aworkaround in order to setup Oracle ACFS onCentOS and is by no means supported by OracleCorporation. Do not attempt these steps on a criticalproduction environment. You have been warned.

The following steps will need to be run from all nodes in an Oracle RAC database cluster as root.

First, make a copy of the following Perl module:

[root@racnode1 ~]# cd /u01/app/11.2.0/grid/lib[root@racnode1 lib]# cp -p osds_acfslib.pm osds_acfslib.pm.orig

[root@racnode2 ~]# cd /u01/app/11.2.0/grid/lib[root@racnode2 lib]# cp -p osds_acfslib.pm osds_acfslib.pm.orig

Next, edit the osds_acfslib.pm Perl module. Search for the string 'support this release' (which

was line 278 in my case).

Replace

if (($release =~ /enterprise-release-5/) ||($release =~ /redhat-release-5/))

with

if (($release =~ /enterprise-release-5/) ||

($release =~ /redhat-release-5/) ||($release =~ /centos-release-5/))

This will get you past the supported version check; however, if you attempt to load the Oracle ASMvolume driver from either Oracle RAC node, you get the following error:



10/43

acfsload: ACFS-9129: ADVM/ACFS not installed

To install ADVM/ACFS, copy the following kernel modules from the Oracle Grid Infrastructure home to theexpected location:

[root@racnode1 ~]#mkdir /lib/modules/2.6.18-194.el5/extra/usm[root@racnode1 ~]# cd /u01/app/11.2.0/grid/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.el5-x86_64/bin[root@racnode1 bin]# cp *ko /lib/modules/2.6.18-194.el5/extra/usm/

[root@racnode2 ~]#mkdir /lib/modules/2.6.18-194.el5/extra/usm[root@racnode2 ~]# cd /u01/app/11.2.0/grid/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.el5-x86_64/bin[root@racnode2 bin]# cp *ko /lib/modules/2.6.18-194.el5/extra/usm/

Once the kernel modules have been copied, we can verify the ADVM/ACFS installation by running thefollowing from all Oracle RAC nodes:

[root@racnode1 ~]# cd /u01/app/11.2.0/grid/bin[root@racnode1 bin]# ./acfsdriverstate -orahome /u01/app/11.2.0/grid versionACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(x86_64)/090715.1

[root@racnode2 ~]# cd /u01/app/11.2.0/grid/bin[root@racnode2 bin]# ./acfsdriverstate -orahome /u01/app/11.2.0/grid versionACFS-9205: OS/ADVM,ACFS installed version = 2.6.18-8.el5(x86_64)/090715.1

The next step is to record dependencies for the new kernel modules:

[root@racnode1 ~]# depmod

[root@racnode2 ~]# depmod

Next, copy the Oracle ACFS executables to /sbin and set the appropriate permissions. The Oracle

ACFS executables are located intheGRID_HOME/install/usm/EL5////bin directory or in the /u01/app/11.2.0/grid/install/usm/cmds/bin directory (12 files)

and include any file without the *.ko extension:

[root@racnode1 ~]# cd /u01/app/11.2.0/grid/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.[root@racnode1 bin]# cp acfs* /sbin; chmod 755 /sbin/acfs*[root@racnode1 bin]# cp advmutil* /sbin; chmod 755 /sbin/advmutil*[root@racnode1 bin]# cp fsck.acfs* /sbin; chmod 755 /sbin/fsck.acfs*[root@racnode1 bin]# cp mkfs.acfs* /sbin; chmod 755 /sbin/mkfs.acfs*[root@racnode1 bin]# cp mount.acfs* /sbin; chmod 755 /sbin/mount.acfs*

[root@racnode1 ~]# cd /u01/app/11.2.0/grid/install/usm/cmds/bin


11/43

[root@racnode1 bin]# cp acfs* /sbin; chmod 755 /sbin/acfs*[root@racnode1 bin]# cp advmutil* /sbin; chmod 755 /sbin/advmutil*[root@racnode1 bin]# cp fsck.acfs* /sbin; chmod 755 /sbin/fsck.acfs*[root@racnode1 bin]# cp mkfs.acfs* /sbin; chmod 755 /sbin/mkfs.acfs*[root@racnode1 bin]# cp mount.acfs* /sbin; chmod 755 /sbin/mount.acfs*

------------------------------------------------------------------------------------

[root@racnode2 ~]# cd /u01/app/11.2.0/grid/install/usm/EL5/x86_64/2.6.18-8/2.6.18-8.[root@racnode2 bin]# cp acfs* /sbin; chmod 755 /sbin/acfs*[root@racnode2 bin]# cp advmutil* /sbin; chmod 755 /sbin/advmutil*[root@racnode2 bin]# cp fsck.acfs* /sbin; chmod 755 /sbin/fsck.acfs*[root@racnode2 bin]# cp mkfs.acfs* /sbin; chmod 755 /sbin/mkfs.acfs*[root@racnode2 bin]# cp mount.acfs* /sbin; chmod 755 /sbin/mount.acfs*

[root@racnode2 ~]# cd /u01/app/11.2.0/grid/install/usm/cmds/bin[root@racnode2 bin]# cp acfs* /sbin; chmod 755 /sbin/acfs*[root@racnode2 bin]# cp advmutil* /sbin; chmod 755 /sbin/advmutil*[root@racnode2 bin]# cp fsck.acfs* /sbin; chmod 755 /sbin/fsck.acfs*[root@racnode2 bin]# cp mkfs.acfs* /sbin; chmod 755 /sbin/mkfs.acfs*[root@racnode2 bin]# cp mount.acfs* /sbin; chmod 755 /sbin/mount.acfs*

Now, running acfsload start -s will complete without any further messages:



Check that the modules were successfully loaded on all Oracle RAC nodes:

[root@racnode1 ~]# lsmod | grep oracleoracleacfs 877320 4oracleadvm 221760 8oracleoks 276880 2 oracleacfs,oracleadvmoracleasm 84136 1

[root@racnode2 ~]# lsmod | grep oracleoracleacfs 877320 4oracleadvm 221760 8oracleoks 276880 2 oracleacfs,oracleadvmoracleasm 84136 1

Configure the Oracle ASM volume driver to load automatically on system startup on all Oracle RACnodes. You will need to create an initialization script (/etc/init.d/acfsload) that contains the

runlevel configuration and the acfsload command. Change the permissions on

the /etc/init.d/acfsload script to allow it to be executed by root and then create links in the rc2.d,

rc3.d, rc4.d, and rc5.d runlevel directories using 'chkconfig --add':


12/43

[root@racnode1 ~]# chkconfig --list | grep acfsload

[root@racnode2 ~]# chkconfig --list | grep acfsload

=======================================================

[root@racnode1 ~]# cat > /etc/init.d/acfsload


13/43

[root@racnode1 ~]# su - grid -c crs_stat | grep acfs

[root@racnode2 ~]# su - grid -c crs_stat | grep acfs

=======================================================

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/crsctl add type ora.registry.acfs.type \-basetype ora.local_resource.type \-file /u01/app/11.2.0/grid/crs/template/registry.acfs.type

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/crsctl add resource ora.registry.acfs \-attr ACL=\'owner:root:rwx,pgrp:oinstall:r-x,other::r--\' \-type ora.registry.acfs.type -f

=======================================================

[root@racnode1 ~]# su - grid -c crs_stat | grep acfsNAME=ora.registry.acfsTYPE=ora.registry.acfs.type

[root@racnode2 ~]# su - grid -c crs_stat | grep acfsNAME=ora.registry.acfsTYPE=ora.registry.acfs.type

As a final step, modify any of the Oracle ACFS shell scripts copied to the /sbin directory (above) to

include the ORACLE_HOME for Grid Infrastructure. The successful execution of these scripts requiresaccess to certain Oracle shared libraries that are found in the Grid Infrastructure Oracle home. Sincemany of the Oracle ACFS shell scripts will be executed as the root user account, the ORACLE_HOME

environment variable will typically not be set in the shell and will result in the executable to fail. Forexample:

[root@racnode1 ~]# /sbin/acfsutil registry/sbin/acfsutil.bin: error while loading shared libraries: libhasgen11.so:cannot open shared object file: No such file or directory

An easy workaround to get past this error is to set the ORACLE_HOME environment variable for theOracle Grid Infrastructure home in the Oracle ACFS shell scripts on all Oracle RAC nodes. TheORACLE_HOME should be set at the beginning of the file after the header comments as shown in thefollowing example:

#!/bin/sh## Copyright (c) 2001, 2009, Oracle and/or its affiliates. All rights reserved.#

ORACLE_HOME=/u01/app/11.2.0/grid

ORA_CRS_HOME=%ORA_CRS_HOME%if [ ! -d $ORA_CRS_HOME ]; thenORA_CRS_HOME=$ORACLE_HOME


14/43

fi...

Add the ORACLE_HOME environment variable for the Oracle Grid Infrastructure home as noted above tothe following Oracle ACFS shell scripts on all Oracle RAC nodes:

/sbin/acfsdbg

/sbin/acfsutil

/sbin/advmutil

/sbin/fsck.acfs

/sbin/mkfs.acfs

/sbin/mount.acfs

Verify ASM Disk Group Compatibility Level

The compatibility level for the Oracle ASM disk group must be at least 11.2 in order to create an OracleASM volume. From the Oracle ASM instance, perform the following checks:

SQL> SELECT compatibility, database_compatibility2 FROM v$asm_diskgroup3 WHERE name = 'DOCSDG1';

COMPATIBILITY DATABASE_COMPATIBILITY---------------- -----------------------10.1.0.0.0 10.1.0.0.0

If the results show something other than 11.2 or higher (as the above example shows), we need to set thecompatibility to at least 11.2 by issuing the following series of SQL statements from the Oracle ASM

instance:


SQL>ALTER DISKGROUP docsdg1 SET ATTRIBUTE 'compatible.asm' = '11.2';

Diskgroup altered.

SQL>ALTER DISKGROUP docsdg1 SET ATTRIBUTE 'compatible.rdbms' = '11.2';

Diskgroup altered.

SQL>ALTER DISKGROUP docsdg1 SET ATTRIBUTE 'compatible.advm' = '11.2';

Diskgroup altered.

If you receive an error while attempting to set the'compatible.advm' attribute, verify that theOracle
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driverhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driver


15/43

ASM volume driveris running:

SQL>ALTER DISKGROUP docsdg1 SETATTRIBUTE 'compatible.advm' = '11.2';ALTER DISKGROUP docsdg1 SET ATTRIBUTE'compatible.advm' = '11.2'

*ERROR at line 1:ORA-15032: not all alterationsperformedORA-15242: could not set attributecompatible.advmORA-15238: 11.2 is not a valid valuefor attribute compatible.advmORA-15477: cannot communicate with thevolume driver

Verify the changes to the compatibility level:

SQL> SELECT compatibility, database_compatibility2 FROM v$asm_diskgroup3 WHERE name = 'DOCSDG1';

COMPATIBILITY DATABASE_COMPATIBILITY---------------- -----------------------11.2.0.0.0 11.2.0.0.0


This section includes step-by-step instructions on how to create an Oracle ASM cluster file system using

the Oracle ASM Configuration Assistant (ASMCA). Note that at the time of this writing, ASMCA onlysupports the creation of volumes and file systems. Deleting an Oracle ASM volume or file system requiresthe command-line.

Create Mount Point

From each Oracle RAC node, create a directory that will be used to mount the new Oracle ACFS:

[root@racnode1 ~]#mkdir /documents1


Create ASM Cluster File System

As the Oracle Grid Infrastructure owner, run the ASM Configuration Assistant (asmca) from only one nodein the cluster (racnode1 for example):

[grid@racnode1 ~]$ asmca
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driverhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Verify%20Oracle%20ASM%20Volume%20Driver


16/43

Screen Name Response Screen Shot

Disk Groups When the Oracle ASM configuration assistant starts you are presented with the 'Disk Groups' tab.

Volumes Click on the 'Volumes' tab then click the [Create] button.

Create ASM VolumeCreate a new ASM volume by supplying a "Volume Name", "Disk Group Name", and "Size". For the purpose of this example, I will

be creating a 32GB volume named "docsvol1" on the "DOCSDG1" ASM disk group.

After verifying all values in this dialog are correct, click the [OK] button.

Volume CreatedAfter the volume is created, acknowledge the ' Volume: Creation' dialog.When returned to the "Volumes" tab, the "State" for the new ASM volume shou ld be ENABLED for all Oracle RAC nodes (i.e.

'ENABLED(2 of 2)').

ASM Cluster File

SystemsClick on the 'ASM Cluster File Systems' tab then click the [Create] button.

Create ASM Cluster File

System

Verify that the newly created volume (DOCSVOL1) is selected in the 'Volume' list.Select the 'General Purpose File System' option.

Enter the previously created mount point directory (/documents1) or leave the suggested mount point.

Select the 'Yes' option for 'Register MountPoint'.

After verifying all values in this dialog are correct, click the [OK] button.

ASM Cluster File System

CreatedAfter the ASM Cluster File System is created, acknowledge the 'ASM Cluster File System: Creation' dialog.

ASM Cluster File

Systems

The newly created Oracle ASM cluster file system is now listed under the 'ASM Cluster File Systems' tab.

Note that the new clustered file system is not mounted. That will need to be performed manually on all Oracle RAC nodes as a

privileged user (root) after exiting from the ASMCA.Exit the ASM Configuration Assistant by clicking the [Exit] button.

Mount the New Cluster File System

Now that the new Oracle ASM cluster file system has been created and registered in the Oracle ACFSmount registry, log in to all Oracle RAC nodes as root and run the following mount command:

[root@racnode1 ~]# /bin/mount -t acfs /dev/asm/docsvol1-300 /documents1/sbin/mount.acfs.bin: error while loading shared libraries: libhasgen11.so:cannot open shared object file: No such file or directory

[root@racnode2 ~]# /bin/mount -t acfs /dev/asm/docsvol1-300 /documents1

/sbin/mount.acfs.bin: error while loading shared libraries: libhasgen11.so:cannot open shared object file: No such file or directory

If you don't have the ORACLE_HOME environment variable set to the Oracle Grid Infrastructure home asexplained in the prerequisites section to this guide, the mount command will fail as shown above. In order

to mount the new cluster file system, the Oracle ASM ACFS binaries need access to certain sharedlibraries in the ORACLE_HOME for Grid Infrastructure. An easy workaround to get past this error is to setthe ORACLE_HOME environment variable for Grid Infrastructure in the file /sbin/mount.acfs on all

Oracle RAC nodes. The ORACLE_HOME should be set at the beginning of the file after the headercomments as follows:



ORA_CRS_HOME=%ORA_CRS_HOME%if [ ! -d $ORA_CRS_HOME ]; then
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Add%20Grid%20Infrastructure%20Oracle%20Home%20to%20ACFS%20Shell%20Scriptshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s08.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s07.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s06.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s05.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s04.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s03.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s02.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_asmca_s01.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Add%20Grid%20Infrastructure%20Oracle%20Home%20to%20ACFS%20Shell%20Scripts


17/43

ORA_CRS_HOME=$ORACLE_HOMEfi...

You should now be able to successfully mount the volume:



Verify Mounted Cluster File System

To verify that the new cluster file system mounted properly, run the following mount command from all

Oracle RAC nodes:

[root@racnode1 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:PUBLIC on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)

[root@racnode2 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:Public on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)

/dev/asm/docsvol1-300 on /documents1 type acfs (rw)

Set Permissions

With the new cluster file system now mounted on all Oracle RAC nodes, change the permissions to allowuser access. For the purpose of this example, I want to grant the oracle user account and dba group

read/write permissions. Run the following as root from only one node in the Oracle RAC:


18/43

[root@racnode1 ~]# chown oracle.dba /documents1[root@racnode1 ~]# chmod 775 /documents1

Test

Now let's perform a test to see if all of our hard work paid off.

Node 1

Log in to the first Oracle RAC node as the oracle user account and create a test file on the new cluster

file system:

[oracle@racnode1 ~]$ iduid=1101(oracle) gid=1000(oinstall)groups=1000(oinstall),1201(asmdba),1300(dba),1301(oper)

[oracle@racnode1 ~]$ echo "The Hunter Family: Jeff, Melody, and Alex" >/documents1/test.txt

[oracle@racnode1 ~]$ ls -l /documents1total 72drwxr-xr-x 5 root root 4096 Nov 23 21:17 .ACFS/drwx------ 2 root root 65536 Nov 23 21:17 lost+found/-rw-r--r-- 1 oracle oinstall 42 Nov 23 21:25 test.txt

Node 2

Log in to the second Oracle RAC node as the oracle user account and verify the presence and content

of the test file:



[oracle@racnode2 ~]$ cat /documents1/test.txtThe Hunter Family: Jeff, Melody, and Alex


This section presents a second method that can be used to create an Oracle ASM cluster file system;namely, Oracle Enterprise Manager (OEM). Similar to theASM Configuration Assistant (ASMCA), OEMprovides a convenient graphical user interface for creating and maintaining ASM cluster file systems.
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)


19/43

Create Mount Point

From each Oracle RAC node, create a directory that will be used to mount the new Oracle ACFS:



Create ASM Cluster File System

Log in to Oracle Enterprise Manager (OEM) as a privileged database user.

[oracle@racnode1 ~]$ emctl status dbconsoleOracle Enterprise Manager 11g Database Control Release 11.2.0.1.0Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.https://racnode1.idevelopment.info:1158/em/console/aboutApplicationOracle Enterprise Manager 11g is running.------------------------------------------------------------------Logs are generated in directory/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_racdb/sysman/log

[oracle@racnode2 ~]$ emctl status dbconsoleOracle Enterprise Manager 11g Database Control Release 11.2.0.1.0Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.https://racnode1.idevelopment.info:1158/em/console/aboutApplicationEM Daemon is running.------------------------------------------------------------------Logs are generated in directory

/u01/app/oracle/product/11.2.0/dbhome_1/racnode2_racdb/sysman/log


OEM HomeFrom the OEM home screen, click the 'Cluster' tab. Scroll to the bottom of the OEM home page, then click on one of the ASM

instances listed.

Automatic Storage

ManagementOn the resulting ASM home screen, click on the 'ASM Cluster File System' tab. Next, click the [Create] button.

Create ASM Cluster

File SystemClick the [Create ASM Volume] button.

Create ASM Volume

Create a new ASM volume by supplying a "Volume Name", "Disk Group Name", and "Size". For the purpose of this example, I will be

creating a 32GB volume named "docsvol2" on the "DOCSDG1" ASM disk group.

After verifying all values in this dial og are correct, click the [OK] button.

Create ASM Cluster

File System

The newly created ASM volume (/dev/asm/docsvol2-300) will now be entered in the 'Volume Device'field.

Enter a 'Volume Label'. For example, DOCSVOL2.

Select the option for 'Register ASM Cluster File System Mount Point '.

Enter the previously created mount point directory (/documents2).After verifying all values in this dial og are correct, click the [OK] button.

Automatic Storage

Management

OEM will create the new cluster file system, register it in the Oracle Grid Infrastructure registry, and leave it in the ' Dismounted'

state. Select the new ASM volume and click the [Mount] button.

Cluster Select Hosts Accept the default node selection (all Oracle RAC nodes) and click the [Continue] button.

Mount ASM Cluster

File SystemEnter the 'Mount Point' and click the [Generate Command] button.

Show CommandRun the suggested command as the root user on all Oracle RAC nodes:
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s09.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s08.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s07.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s06.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s05.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s04.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s03.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s02.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s01.gif


20/43


[root@racnode1 ~]# /bin/mount -t acfs/dev/asm/docsvol2-300 /documents2[root@racnode2 /]# /bin/mount -t acfs

/dev/asm/docsvol2-300 /documents2Note: If you receive an error loading Oracle shared libraries, refer to the followingworkaround.

After mounting the ASM volume, click the [Return] button on this and the previous screen.

Automatic Storage

ManagementThe new Oracle ASM Cluster File System is ready for use!



Oracle RAC nodes:

[root@racnode1 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:PUBLIC on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)/dev/asm/docsvol2-300 on /documents2 type acfs (rw)

[root@racnode2 ~]#mount

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:Public on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)/dev/asm/docsvol2-300 on /documents2 type acfs (rw)

Set Permissions


http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#First%20Time%20to%20Mount%20the%20New%20Cluster%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#First%20Time%20to%20Mount%20the%20New%20Cluster%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#First%20Time%20to%20Mount%20the%20New%20Cluster%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/images/ASMClusterFileSystemIntro/acfs_oem_s10.gifhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#First%20Time%20to%20Mount%20the%20New%20Cluster%20File%20System


21/43


Test


Node 1


file system:




Node 2


of the test file:



[oracle@racnode2 ~]$ cat /documents2/test.txt

The Hunter Family: Jeff, Melody, and Alex

Command Line / SQL

This section presents the third and final method described in this guide that can be used to create anOracle ASM cluster file system; namely, the command line interface and SQL. Unlike using theASMConfiguration Assistant (ASMCA) orOracle Enterprise Manager (OEM), using the command line tools
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)


22/43

does not require a graphic user interface and is the preferred method when working remotely over a slownetwork.

Create Oracle ASM Dynamic Volume

Log in as the Oracle Grid Infrastructure owner and switch to the Oracle ASM environment on node 1 of

the RAC:

[grid@racnode1 ~]$ iduid=1100(grid) gid=1000(oinstall)groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper)

[grid@racnode1 ~]$ . oraenvORACLE_SID = [+ASM1] ? +ASM1The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/grid

[grid@racnode1 ~]$ dbhome/u01/app/11.2.0/grid

[grid@racnode1 ~]$ echo $ORACLE_SID+ASM1

As the Oracle Grid Infrastructure owner, log in to the Oracle ASM instance using SQL*Plus and issue thefollowing SQL:


SQL>ALTER DISKGROUP docsdg1 ADD VOLUME docsvol3 SIZE 20G;

Diskgroup altered.

The same task can be accomplished usingthe ASMCMD command-line utility:[grid@racnode1 ~]$ asmcmd volcreate -Gdocsdg1 -s 20G --redundancy unprotecteddocsvol3

To verify that the new Oracle ASM dynamic volume was successfully created, query theview V$ASM_VOLUME (or since we are using Oracle RAC, GV$ASM_VOLUME). Make certain that

the STATE column for each Oracle RAC instance is ENABLED:

break on inst_id skip 1

column inst_id format 9999999 heading "Instance ID" justify leftcolumn volume_name format a13 heading "Volume Name" justify leftcolumn volume_device format a23 heading "Volume Device" justify leftcolumn size_mb format 999,999,999 heading "Size (MB)" justify rightcolumn usage format a5 heading "Usage" justify right


23/43

column state format a7 heading "State" justify right

============================================================================

SQL> select inst_id, volume_name, volume_device, size_mb, usage, state2 from gv$asm_volume

3 order by inst_id, volume_name;

Instance ID Volume Name Volume Device Size (MB) Usage State----------- ------------- ----------------------- ------------ ----- -------

1 DOCSVOL1 /dev/asm/docsvol1-300 32,768 ACFS ENABLEDDOCSVOL2 /dev/asm/docsvol2-300 32,768 ACFS ENABLED

DOCSVOL3 /dev/asm/docsvol3-300 20,480 ENABLED

2 DOCSVOL1 /dev/asm/docsvol1-300 32,768 ACFS ENABLEDDOCSVOL2 /dev/asm/docsvol2-300 32,768 ACFS ENABLED

DOCSVOL3 /dev/asm/docsvol3-300 20,480 ENABLED

The same task can be accomplished using

the ASMCMD command-line utility:

[grid@racnode1 ~]$ asmcmd volinfo -Gdocsdg1 -aDiskgroup Name: DOCSDG1

Volume Name: DOCSVOL1Volume Device:

/dev/asm/docsvol1-300State: ENABLEDSize (MB): 32768Resize Unit (MB): 256Redundancy: UNPROT

Stripe Columns: 4Stripe Width (K): 128Usage: ACFSMountpath: /documents1


/dev/asm/docsvol2-300State: ENABLEDSize (MB): 32768Resize Unit (MB): 256Redundancy: UNPROTStripe Columns: 4Stripe Width (K): 128

Usage: ACFSMountpath: /documents2


/dev/asm/docsvol3-300State: ENABLEDSize (MB): 20480Resize Unit (MB): 256


24/43

Redundancy: UNPROTStripe Columns: 4Stripe Width (K): 128Usage:Mountpath:

Additional checks can be performed on the new ASM volume using the /sbin/advmutil command:

[root@racnode1 ~]# /sbin/advmutil volinfo /dev/asm/docsvol3-300Interface Version: 1Size (MB): 20480Resize Increment (MB): 256Redundancy: unprotectedStripe Columns: 4Stripe Width (KB): 128Disk Group: DOCSDG1Volume: DOCSVOL3

As a final check, list the newly created device file on the file system:

[root@racnode1 ~]# ls -l /dev/asm/*brwxrwx--- 1 root asmadmin 252, 153601 Nov 26 16:55 /dev/asm/docsvol1-300brwxrwx--- 1 root asmadmin 252, 153602 Nov 26 16:55 /dev/asm/docsvol2-300brwxrwx--- 1 root asmadmin 252, 153603 Nov 26 17:20 /dev/asm/docsvol3-300

Create Oracle ASM Cluster File System

The next step is to create the Oracle ASM cluster file system on the new Oracle ASM volume created inthe previous section. This is performed using the mkfs OS command from only one of the Oracle RAC

nodes:

[grid@racnode1 ~]$ /sbin/mkfs -t acfs -b 4k /dev/asm/docsvol3-300 -n "DOCSVOL3"mkfs.acfs: version = 11.2.0.1.0.0mkfs.acfs: on-disk version = 39.0mkfs.acfs: volume = /dev/asm/docsvol3-300mkfs.acfs: volume size = 21474836480mkfs.acfs: Format complete.

In the above mkfs command, the -t flag indicates that the new file system should be of type ACFS. The

block size is set to 4K (-b). Finally, specify the Linux dynamic volume device (/dev/asm/docsvol3-

300) and the volume label (DOCSVOL3).

Mount the New Cluster File System

The mkfs command in the previous section only prepared the volume to be mounted as a file system, but

does not actually mount it. To mount the new cluster file system, first create a directory on each OracleRAC node as the root user account that will be used to mount the new Oracle ACFS:


25/43



Mount the cluster file system on each Oracle RAC node using the regular Linux mount command asfollows:



If you don't have the ORACLE_HOME environment variable set to the Oracle Grid Infrastructure home asexplained in the prerequisites section to this guide, the mount command will fail as shown above. In order

to mount the new cluster file system, the Oracle ASM ACFS binaries need access to certain sharedlibraries in the ORACLE_HOME for Grid Infrastructure. An easy workaround to get past this error is to setthe ORACLE_HOME environment variable for Grid Infrastructure in the file /sbin/mount.acfs on all

Oracle RAC nodes. The ORACLE_HOME should be set at the beginning of the file after the headercomments as follows:



ORA_CRS_HOME=%ORA_CRS_HOME%if [ ! -d $ORA_CRS_HOME ]; thenORA_CRS_HOME=$ORACLE_HOME

fi...

You should now be able to successfully mount the volume:





Oracle RAC nodes:
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Add%20Grid%20Infrastructure%20Oracle%20Home%20to%20ACFS%20Shell%20Scriptshttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Add%20Grid%20Infrastructure%20Oracle%20Home%20to%20ACFS%20Shell%20Scripts


26/43

[root@racnode1 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)

/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:PUBLIC on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)/dev/asm/docsvol2-300 on /documents2 type acfs (rw)/dev/asm/docsvol3-300 on /documents3 type acfs (rw)

[root@racnode2 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)domo:Public on /domo type nfs (rw,addr=192.168.1.121)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)/dev/asm/docsvol2-300 on /documents2 type acfs (rw)/dev/asm/docsvol3-300 on /documents3 type acfs (rw)

Set Permissions




Register New Volume

When creating the Oracle ACFS using theASM Configuration Assistant (ASMCA)andOracle EnterpriseManager (OEM), I glossed over this notion of registering the new volume in the Oracle ACFS mountregistry. When a node configured with Oracle ACFS reboots, the newly created file systems do notremount by default. The Oracle ACFS mount registry acts as a global file system reference, much likethe/etc/fstab file does in a UNIX/Linux environment. When mount points are registered in the Oracle

ACFS mount registry, Oracle Grid Infrastructure will mount and unmount volumes on startup andshutdown respectively.
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#ASM%20Configuration%20Assistant%20(ASMCA)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Oracle%20Enterprise%20Manager%20(OEM)


27/43

Use the /sbin/acfsutil utility on only one of Oracle RAC nodes to register the new mount point in the

Oracle ACFS mount registry:

[root@racnode1 ~]# /sbin/acfsutil registry -f -a /dev/asm/docsvol3-300 /documents3acfsutil registry: mount point /documents3 successfully added to Oracle Registry

Query the Oracle ACFS mount registry from all Oracle RAC nodes to verify the volume and mount pointwas successfully registered:

[root@racnode1 ~]# /sbin/acfsutil registryMount Object:Device: /dev/asm/docsvol1-300Mount Point: /documents1Disk Group: DOCSDG1Volume: DOCSVOL1Options: none

Nodes: allMount Object:Device: /dev/asm/docsvol2-300Mount Point: /documents2Disk Group: DOCSDG1Volume: DOCSVOL2Options: noneNodes: all

Mount Object: Device: /dev/asm/docsvol3-300Mount Point: /documents3Disk Group: DOCSDG1Volume: DOCSVOL3

Options: noneNodes: all

Test


Node 1


file system:



[oracle@racnode1 ~]$ ls -l /documents3


28/43

total 72drwxr-xr-x 5 root root 4096 Nov 24 18:44 .ACFS/drwx------ 2 root root 65536 Nov 24 18:44 lost+found/-rw-r--r-- 1 oracle oinstall 42 Nov 24 18:56 test.txt

Node 2


of the test file:


[oracle@racnode2 ~]$ ls -l /documents3total 72drwxr-xr-x 5 root root 4096 Nov 24 18:44 .ACFS/

drwx------ 2 root root 65536 Nov 24 18:44 lost+found/-rw-r--r-- 1 oracle oinstall 42 Nov 24 18:56 test.txt

[oracle@racnode2 ~]$ cat /documents3/test.txtThe Hunter Family: Jeff, Melody, and Alex

ACFS Snapshots

Oracle ASM Cluster File System includes a feature called snapshots. An Oracle ACFS snapshot is anonline, read-only, point in time copy of an Oracle ACFS file system. The snapshot process uses Copy-On-Write functionality which makes efficient use of disk space. Note that snapshots work at the blocklevel instead of the file level. Before an Oracle ACFS file extent is modified or deleted, its current value is

copied to the snapshot to maintain the point-in-time view of the file system.

When a file is modified, only the changed blocks arecopied to the snapshot location which helpsconserve disk space.

Once an Oracle ACFS snapshot is created, all snapshot files are immediately available for use.Snapshots are always available as long as the file system is mounted. This provides support for onlinerecovery of files inadvertently modified or deleted from a file system. You can have up to 63 snapshotviews supported for each file system. This provides for a flexible online file recovery solution which canspan multiple views. You can also use an Oracle ACFS snapshot as the source of a file system backup,as it can be created on demand to deliver a current, consistent, online view of an active file system. Oncethe Oracle ACFS snapshot is created, simply backup the snapshot to another disk or tape location tocreate a consistent backup set of the files.

Oracle ACFS snapshots can be created and deletedon demand without the need to take the file systemoffline. ACFS snapshots provide a point-in-timeconsistent view of the entire file system which can beused to restore deleted or modified files and toperform backups.


29/43

All storage for Oracle ACFS snapshots are maintained within the file system which eliminates the need forseparate storage pools for file systems and snapshots. As shown in the next section, Oracle ACFS filesystems can be dynamically re-sized to accommodate addition file and snapshot storage requirements.

Oracle ACFS snapshots are administered with the acfsutil snap command. This section will provide

an overview on how to create and retrieve Oracle ACFS snapshots.

Oracle ACFS Snapshot Location

Whenever you create an Oracle ACFS file system, a hidden directory is created as a sub-directory to theOracle ACFS file system named .ACFS. (Note that hidden files and directories in Linux start with leading

period.)

[oracle@racnode1 ~]$ ls -lFA /documents3total 2851148drwxr-xr-x 5 root root 4096 Nov 26 17:57 .ACFS/-rw-r--r-- 1 oracle oinstall 1239269270 Nov 27 16:02

linux.x64_11gR2_database_1of2.zip-rw-r--r-- 1 oracle oinstall 1111416131 Nov 27 16:03linux.x64_11gR2_database_2of2.zip-rw-r--r-- 1 oracle oinstall 555366950 Nov 27 16:03 linux.x64_11gR2_examples.zipdrwx------ 2 root root 65536 Nov 26 17:57 lost+found/

Found in the .ACFS are two directories named repl and snaps. All Oracle ACFS snapshots are stored

in the snaps directory.

[oracle@racnode1 ~]$ ls -lFA /documents3/.ACFStotal 12

drwx------ 2 root root 4096 Nov 26 17:57 .fileid/drwx------ 6 root root 4096 Nov 26 17:57 repl/drwxr-xr-x 2 root root 4096 Nov 27 15:53 snaps/

Since no Oracle ACFS snapshots exist, the snaps directory is empty.

[oracle@racnode1 ~]$ ls -lFA /documents3/.ACFS/snapstotal 0

Create Oracle ACFS Snapshot

Let's start by creating an Oracle ACFS snapshot named snap1 for the Oracle ACFS mountedon /documents3. This operation should be performed as root or the Oracle Grid Infrastructure owner:

[root@racnode1 ~]# /sbin/acfsutil snap create snap1 /documents3acfsutil snap create: Snapshot operation is complete.
http://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20Systemhttp://www.idevelopment.info/data/Oracle/DBA_tips/Automatic_Storage_Management/ASM_50.shtml#Resize%20File%20System


30/43

The data for the new snap1 snapshot will be stored in /documents3/.ACFS/snaps/snap1. Once the

snapshot is created, any existing files and/or directories in the file system are automatically accessiblefrom the snapshot directory. For example, when I created the snap1 snapshot, the three Oracle ZIP files

were made available from the snapshot /documents3/.ACFS/snaps/snap1 :

[oracle@racnode1 ~]$ ls -lFA /documents3/.ACFS/snaps/snap1total 2851084drwxr-xr-x 5 root root 4096 Nov 26 17:57 .ACFS/-rw-r--r-- 1 oracle oinstall 1239269270 Nov 27 16:02linux.x64_11gR2_database_1of2.zip-rw-r--r-- 1 oracle oinstall 1111416131 Nov 27 16:03linux.x64_11gR2_database_2of2.zip-rw-r--r-- 1 oracle oinstall 555366950 Nov 27 16:03 linux.x64_11gR2_examples.zip?--------- ? ? ? ? ? lost+found

It is important to note that when the snapshot gets created, nothing is actually stored in the snapshotdirectory, so there is no additional space consumption. The snapshot directory will only

contain modifiedfile blocks when a file is updated or deleted.

Restore Files From an Oracle ACFS Snapshot

When a file is deleted (or modified), this triggers an automatic backup of all modified file blocks to thesnapshot. For example, if I delete the file /documents3/linux.x64_11gR2_examples.zip , the

previous images of the file blocks are copied to the snap1 snapshot where it can be restored from at a

later time if necessary:

[oracle@racnode1 ~]$ rm /documents3/linux.x64_11gR2_examples.zip

If you were looking for functionality in Oracle ACFS to perform a rollbackof the current file system to asnapshot, then I have bad news; one doesn't exist. Hopefully this will be a feature introduced in futureversions!

In the case where you accidentally deleted a file from the current file system, it can be restored bycopying it from the snapshot, back to the the current file system:

[oracle@racnode1 ~]$ cp /documents3/.ACFS/snaps/snap1/linux.x64_11gR2_examples.zip/documents3

Display Oracle ACFS Snapshot Information

The '/sbin/acfsutil info fs' command can provide file system information as well as limited

information on any Oracle ACFS snapshots:

[oracle@racnode1 ~]$ /sbin/acfsutil info fs /documents3/documents3

ACFS Version: 11.2.0.1.0.0


31/43

flags: MountPoint,Availablemount time: Sat Nov 27 03:07:50 2010volumes: 1total size: 26843545600total free: 23191826432primary volume: /dev/asm/docsvol3-300

label: DOCSVOL3flags: Primary,Availableon-disk version: 39.0allocation unit: 4096major, minor: 252, 153603size: 26843545600free: 23191826432

number of snapshots: 1snapshot space usage: 560463872

From the example above, you can see that I have only one active snapshot that is consumingapproximately 560MB of disk space. This coincides with the size of the file I removed earlier(/documents3/linux.x64_11gR2_examples.zip ) which triggered a back up of all modified file

image blocks.

To query all snapshots, simply list the directories under '/.ACFS/snaps '. Each

directory under the snaps directory is an Oracle ACFS snapshot.

Another useful technique used to obtain information about Oracle ACFS snapshots is to query theview V$ASM_ACFSSNAPSHOTS from the Oracle ASM instance:

column snap_name format a15 heading "Snapshot Name"column fs_name format a15 heading "File System"

column vol_device format a25 heading "Volume Device"column create_time format a20 heading "Create Time"

======================================================================

SQL> select snap_name, fs_name, vol_device,2 to_char(create_time, 'DD-MON-YYYY HH24:MI:SS') as create_time3 from v$asm_acfssnapshots4 order by snap_name;

Snapshot Name File System Volume Device Create Time--------------- --------------- ------------------------- --------------------snap1 /documents3 /dev/asm/docsvol3-300 27-NOV-2010 16:11:29

Delete Oracle ACFS Snapshot

Use the 'acfsutil snap delete' command to delete an existing Oracle ACFS snapshot:

[root@racnode1 ~]# /sbin/acfsutil snap delete snap1 /documents3acfsutil snap delete: Snapshot operation is complete.


32/43

Managing ACFS

Oracle ACFS and Dismount or Shutdown Operations

If you take anything away from this article, know and understand the importance of dismounting anyactive file system configured with an Oracle ASM Dynamic Volume Manager (ADVM) volume device,

BEFORE shutting down an Oracle ASM instance or dismounting a disk group! Failure to do so will resultin I/O failures and very angry users!

After the file system(s) have been dismounted, all open references to Oracle ASM files are removed andassociated disk groups can then be dismounted or the Oracle ASM instance shut down.

If the Oracle ASM instance or disk group is forcibly shut down or fails while an associated Oracle ACFS isactive, the file system is placed into an offline error state. When the file system is placed in an offline errorstate, applications will start to encounter I/O failures and any Oracle ACFS user data and metadata beingwritten at the time of the termination may not be flushed to ASM storage before it is fenced. If aSHUTDOWN ABORT operation on the Oracle ASM instance is required and you are not able to dismountthe file system, issue two sync command to flush any cached file system data and metadata to persistent

storage:

[root@racnode1 ~]# sync[root@racnode1 ~]# sync

Using a two-node Oracle RAC, I forced an Oracle ASM instance shutdown on node 1 to simulate afailure:

This should go without saying, but I'll say it anyway.DO NOT attempt the following on a productionenvironment.

SQL> shutdown abortASM instance shutdown

Any subsequent attempt to access an offline file system on that node will result in an I/O error:

[oracle@racnode1 ~]$ ls -l /documents3ls: /documents3: Input/output error

[oracle@racnode1 ~]$ df -kFilesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00

145344992 22459396 115383364 17% //dev/sdb1 151351424 192072 143346948 1% /local/dev/sda1 101086 12632 83235 14% /boottmpfs 2019256 0 2019256 0% /dev/shmdf: `/documents1': Input/output errordf: `/documents2': Input/output errordf: `/documents3': Input/output errordomo:PUBLIC 4799457152 1901758592 2897698560 40% /domo


33/43

Recovering a file system from an offline error state requires dismounting and remounting the OracleACFS file system. Dismounting an active file system, even one that is offline, requires stopping allapplications using the file system, including any shell references. For example, I had a shell session thatpreviously changed directory (cd) into the /documents3 file system before the forced shutdown:

[root@racnode1 ~]# umount /documents1[root@racnode1 ~]# umount /documents2[root@racnode1 ~]# umount /documents3umount: /documents3: device is busyumount: /documents3: device is busy

Use the Linux fuser orlsof command to identify processes and kill if necessary:

[root@racnode1 ~]# fuser /documents3/documents3: 16263c

[root@racnode1 ~]# kill -9 16263

[root@racnode1 ~]# umount /documents3

Restart the Oracle ASM instance (or in my case, all Oracle Grid Infrastructure services were stopped as aresult of me terminating the Oracle ASM instance):

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop cluster

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/crsctl start cluster

All of my Oracle ACFS volumes were added to the Oracle ACFS mount registry and will thereforeautomatically mount when Oracle Grid Infrastructure starts. If you need to manually mount the file system,verify the volume is enabled before attempting to mount:

[root@racnode1 ~]#mount/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)proc on /proc type proc (rw)sysfs on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw,gid=5,mode=620)/dev/sdb1 on /local type ext3 (rw)/dev/sda1 on /boot type ext3 (rw)

tmpfs on /dev/shm type tmpfs (rw)none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)oracleasmfs on /dev/oracleasm type oracleasmfs (rw)domo:PUBLIC on /domo type nfs (rw,addr=192.168.1.121)/dev/asm/docsvol1-300 on /documents1 type acfs (rw)/dev/asm/docsvol2-300 on /documents2 type acfs (rw)/dev/asm/docsvol3-300 on /documents3 type acfs (rw)


34/43

Resize File System

With Oracle ACFS, as long as there exists free space within the ASM disk group, any of the ASMvolumes can be dynamically expanded which means the file system gets expanded as a result. Note thatif you are using another file system other than Oracle ACFS, as long as that file system can supportonline resizing, they too can be dynamically re-sized. The one exception to 3rd party file systems is online

shrinking. Ext3, for example, supports online resizing but does not support online shrinking.

Use the following syntax to add space to an Oracle ACFS on the fly without the need to take any type ofoutage.

First, verify there is enough space in the current Oracle ASM disk group to extend the volume:

SQL> select name, total_mb, free_mb, round((free_mb/total_mb)*100,2) pct_free2 from v$asm_diskgroup3 where total_mb != 04 order by name;

Disk Group Total (MB) Free (MB) % Free--------------- ------------ ------------ -------CRS 2,205 1,809 82.04DOCSDG1 98,303 12,187 12.40FRA 33,887 22,795 67.27RACDB_DATA 33,887 30,584 90.25

The same task can be accomplished usingthe ASMCMD command-line utility:[grid@racnode1 ~]$ asmcmd lsdg

From the 12GB of free space in the DOCSDG1 ASM disk group, let's extend the file system (volume) by

another 5GB. Note that this can be performed while the file system is online and accessible by clients no outage is required:

[root@racnode1 ~]# /sbin/acfsutil size +5G /documents3acfsutil size: new file system size: 26843545600 (25600MB)

Verify the new size of the file system from all Oracle RAC nodes:

[root@racnode1 ~]# df -k

Filesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00

145344992 21952712 115890048 16% //dev/sdb1 151351424 192072 143346948 1% /local/dev/sda1 101086 12632 83235 14% /boottmpfs 2019256 1135852 883404 57% /dev/shmdomo:PUBLIC 4799457152 1901103872 2898353280 40% /domo/dev/asm/docsvol1-300

33554432 197668 33356764 1% /documents1


35/43

/dev/asm/docsvol2-30033554432 197668 33356764 1% /documents2

/dev/asm/docsvol3-30026214400 183108 26031292 1% /documents3

[root@racnode2 ~]# df -kFilesystem 1K-blocks Used Available Use% Mounted on/dev/mapper/VolGroup00-LogVol00

145344992 13803084 124039676 11% //dev/sdb1 151351424 192072 143346948 1% /local/dev/sda1 101086 12632 83235 14% /boottmpfs 2019256 1135852 883404 57% /dev/shmdomo:Public 4799457152 1901103872 2898353280 40% /domo/dev/asm/docsvol1-300

33554432 197668 33356764 1% /documents1/dev/asm/docsvol2-300

33554432 197668 33356764 1% /documents2/dev/asm/docsvol3-300

26214400 183108 26031292 1% /documents3

Useful ACFS Commands

This section contains several useful commands that can be used to administer Oracle ACFS. Note thatmany of the commands described in this section have already been discussed throughout this guide.

ASM Volume Driver

Load the Oracle ASM volume driver:


Unload the Oracle ASM volume driver:

[root@racnode1 ~]# /u01/app/11.2.0/grid/bin/acfsload stop

Check if Oracle ASM volume driver is loaded:

[root@racnode1 ~]# lsmod | grep oracle

oracleacfs 877320 4oracleadvm 221760 8oracleoks 276880 2 oracleacfs,oracleadvmoracleasm 84136 1

ASM Volume Management

Create new Oracle ASM volume using ASMCMD:


36/43

[grid@racnode1 ~]$ asmcmd volcreate -G docsdg1 -s 20G --redundancy unprotecteddocsvol3

Resize Oracle ACFS file system (add 5GB):

[root@racnode1 ~]# /sbin/acfsutil size +5G /documents3acfsutil size: new file system size: 26843545600 (25600MB)

Delete Oracle ASM volume using ASMCMD:

[grid@racnode1 ~]$ asmcmd voldelete -G docsdg1 docsvol3

Disk Group / File System / Volume Information

Get detailed Oracle ASM disk group information:

[grid@racnode1 ~]$ asmcmd lsdg

Format an Oracle ASM cluster file system:

[grid@racnode1 ~]$ /sbin/mkfs -t acfs -b 4k /dev/asm/docsvol3-300 -n "DOCSVOL3"mkfs.acfs: version = 11.2.0.1.0.0

mkfs.acfs: on-disk version = 39.0mkfs.acfs: volume = /dev/asm/docsvol3-300mkfs.acfs: volume size = 21474836480mkfs.acfs: Format complete.

Get detailed file system information:

[root@racnode1 ~]# /sbin/acfsutil info fs/documents1

ACFS Version: 11.2.0.1.0.0flags: MountPoint,Available

mount time: Fri Nov 26 18:38:48 2010volumes: 1total size: 34359738368total free: 34157326336primary volume: /dev/asm/docsvol1-300

label:flags: Primary,Available,ADVMon-disk version: 39.0allocation unit: 4096major, minor: 252, 153601


37/43

size: 34359738368free: 34157326336ADVM diskgroup DOCSDG1ADVM resize increment: 268435456ADVM redundancy: unprotectedADVM stripe columns: 4

ADVM stripe width: 131072number of snapshots: 0snapshot space usage: 0

/documents2ACFS Version: 11.2.0.1.0.0flags: MountPoint,Availablemount time: Fri Nov 26 18:38:48 2010volumes: 1total size: 34359738368total free: 34157326336primary volume: /dev/asm/docsvol2-300

label:flags: Primary,Available,ADVMon-disk version: 39.0allocation unit: 4096major, minor: 252, 153602size: 34359738368free: 34157326336ADVM diskgroup DOCSDG1ADVM resize increment: 268435456ADVM redundancy: unprotectedADVM stripe columns: 4ADVM stripe width: 131072


/documents3ACFS Version: 11.2.0.1.0.0flags: MountPoint,Availablemount time: Fri Nov 26 18:38:48 2010volumes: 1total size: 26843545600total free: 26656043008primary volume: /dev/asm/docsvol3-300

label: DOCSVOL3flags: Primary,Available,ADVMon-disk version: 39.0allocation unit: 4096major, minor: 252, 153603size: 26843545600free: 26656043008ADVM diskgroup DOCSDG1ADVM resize increment: 268435456ADVM redundancy: unprotectedADVM stripe columns: 4ADVM stripe width: 131072



38/43

Get ASM volume information:

[grid@racnode1 ~]$ asmcmd volinfo -aDiskgroup Name: DOCSDG1

Volume Name: DOCSVOL1Volume Device: /dev/asm/docsvol1-300State: ENABLEDSize (MB): 32768Resize Unit (MB): 256Redundancy: UNPROTStripe Columns: 4Stripe Width (K): 128Usage: ACFSMountpath: /documents1

Volume Name: DOCSVOL2Volume Device: /dev/asm/docsvol2-300

State: ENABLEDSize (MB): 32768Resize Unit (MB): 256Redundancy: UNPROTStripe Columns: 4Stripe Width (K): 128Usage: ACFSMountpath: /documents2

Volume Name: DOCSVOL3Volume Device: /dev/asm/docsvol3-300State: ENABLEDSize (MB): 25600Resize Unit (MB): 256

Redundancy: UNPROTStripe Columns: 4Stripe Width (K): 128Usage: ACFSMountpath: /documents3

Get volume status using ASMCMD command:

[grid@r

using the oracle asm cluster file system (oracle acfs) on linux - (11gr2)

Documents