d67016gc20 exadata workshop part2

O ti i i D t b P f ithOptimizing Database Performance with Exadata

Copyright © 2010, Oracle and/or its affiliates. All rights reserved.

Objectives

After completing this lesson, you should be able to:

• Describe the recommendations for optimizing Exadata performance


Exadata and Database Machine Administration Workshop 7 - 2

Optimizing Performance

• Start with best practices for ASM and Oracle Database

• Areas for special consideration:– Flash memory usage

– Compression usage– Compression usage

– Index usage

– ASM allocation unit size

– Minimum extent size

Optimizing Performance

Optimizing the performance of your Exadata environment begins with following good practices for database design and application development From an administration


practices for database design and application development. From an administration viewpoint, you should continue to follow the best practices for ASM and Oracle Database in conjunction with advice and statistics provided by tools such as SQL monitor and SQL Tuning Advisor.

In addition, there are a number of areas for special consideration listed in the slide. These are covered in the remainder of this lesson.


Flash Memory Usage

• Each Exadata server contains 384 GB of high performance flash memory.

• Flash memory can be configured in two ways:– Exadata Smart Flash Cache:Exadata Smart Flash Cache:

— Speeds up access to frequently accessed data

— Uses all available flash memory by default

— Can be managed automatically for maximum efficiency— Users can only provide optional hints to influence caching priorities.

— Beneficial for OLTP and Data Warehouse workloads

– Flash-based permanent storage:— Uses flash memory as a premium storage device

— Must be configured by an administrator

— Requires deliberate planning to use efficiently

— Additional benefits for specific applications

Flash Memory Usage

Each Exadata server contains 384 GB of high performance flash memory.

B d f lt h E d t i fi t fi d thi fl h i fi d E d t


By default, when Exadata is first configured, this flash memory is configured as Exadata Smart Flash Cache. Exadata Smart Flash Cache provides a caching mechanism for frequently accessed data on each Exadata cell. It is a write-through cache which is useful for absorbing repeated random reads, and very beneficial to online transaction processing (OLTP). Exadata Smart Flash Cache also benefits data warehousing applications by caching the most frequently accessed data.

Exadata Smart Flash Cache automatically focuses on caching frequently accessed data and y g q yindex blocks, along with performance critical information such as control files and file headers. In addition, DBAs can influence caching priorities using the CELL_FLASH_CACHE storage attribute for specific database objects.

For example, to prioritize caching for the CALLDETAIL table use the following SQL command:

ALTER TABLE calldetail STORAGE (CELL_FLASH_CACHE KEEP)

Because of this setting, Exadata will cache data for the CALLDETAIL table more aggressively


than otherwise. Without this setting, Exadata caches individual table and index blocks. With this setting, Exadata attempts to cache scans on the CALLDETAIL table. This makes it more likely that subsequent scans on the CALLDETAIL table will also be satisfied using Exadata Smart Flash Cache.

Flash Memory Usage (continued)

This setting also causes Exadata to clear out other tables from the cache before clearing out cached data for the CALLDETAIL table. This means that CALLDETAIL records will remain in Exadata Smart Flash Cache longer than cached data for other tables. ada a S a as Cac e o ge a cac ed da a o o e ab es

If the CALLDETAIL table is spread across several Exadata servers, each one will cache its part of the table in its own Exadata Smart Flash Cache. If the caches are sufficient in size, the CALLDETAIL table is likely to be completely cached over time.

Alternatively, you can configure a smaller Exadata Smart Flash Cache and use the remaining flash memory as a flash disk drive. You can then create grid disks based on flash memory. These flash disks can be used to configure high performance ASM disk groups housing their own tablespaces. In this case, the flash memory is not used to cache data, rather it becomes a high performance permanent store for some of your data. You should consider configuring flash-based permanent storage areas in the following cases:

• Heavy write applications: For applications that perform many updates, or in cases of heavy data ingest, you may consider storing specific tables, partitions or indexes on flash disks. In these cases, the high volume of writes can consume a considerable portion of the physical disk bandwidth. Storing insert- and update-intensive tables, partitions, and p y g p , p ,indexes on flash-based storage means that there are no physical disk I/Os associated with those objects. The physical disk bandwidth saved using this configuration can be used by other operations increasing the overall throughput of the system. Flash disks can also be used effectively in conjunction with partitioning. In many time-based partitioning scenarios, the most current data can be quite volatile while older data is subject to few updates, if any. In these situations, you can host your most current partitions in a flash-based tablespace and routinely move these partitions to a disk-based tablespace as theirtablespace and routinely move these partitions to a disk based tablespace as their volatility diminishes due to age.

• Star-schemas: In a data warehouse based on a star schema, you may be able to store your entire fact table on flash disk. This effectively acts as a permanent cache for this key table. If your fact table is too large to store on flash disk, you can partition it and store the most referenced partition on flash disk.

The flash memory on each cell can be divided between Exadata Smart Flash Cache and permanent flash-based storage. Both areas can be any size as long as the total does not exceed 384 GB on each cell. In practice, the effectiveness of Exadata Smart Flash Cache drops if the cache size is very small so in most cases, you would allocate a substantial proportion of flash memory to Exadata Smart Flash Cache.

Note that Exadata Smart Flash Cache is largely automatic. After it is configured, you can only influence caching priorities for specific tables using the CELL_FLASH_CACHE storage attribute. In contrast, configuring permanent flash-based storage requires accurate space planning toIn contrast, configuring permanent flash based storage requires accurate space planning to make the most effective use of this premium resource.


Compression Usage

CompressionMethod

CREATE/ALTER TABLE Syntax

CompressionRatio

CPUOverhead

Typical Applications

Basic Compression

COMPRESS [BASIC]

High for direct path inserts

Conventional path inserts and updates are not compressed

Minimal: Oracle Database performs compression and decompression

DSS

are not compressed

OLTP Compression

COMPRESS FOR OLTP

High for all transaction types

Minimal: Oracle Database performs compression and decompression, compression for DML is performed in batches

OLTP and DSS

Warehouse Compression

COMPRESS FOR QUERY [LOW|HIGH]

Higher for direct path inserts

High for conventional

Higher: Decompression is performed by Exadata

DSS

path inserts and updates

Online Archival Compression

COMPRESS FOR ARCHIVE [LOW|HIGH]

Highest for direct path inserts

High for conventional path inserts and updates

Highest: Decompression is performed by Exadata

Archiving

Compression UsageMany people think of data compression as they approach the capacity limits of their storage. Commonly data compression is seen as a performance overhead which must be tolerated in


Commonly, data compression is seen as a performance overhead, which must be tolerated in order to deliver extra storage capacity. However, in cases where I/O bandwidth is limited, data compression can be an effective tool to increase performance by using available CPU capacity to effectively increase the I/O throughput of a storage system.

Oracle Database provides the following modes of data compression:

• Basic compression compresses data at the Oracle block level. It allows more data to be stored in each block by replacing duplicate values with a smaller symbolic

t ti F l if th i l t 50 ti irepresentation. For example, if the numeric value 99999 was present 50 times in a block of data it could be replaced by 50 occurrences of the # symbol along with an entry in a symbol table (also stored in the block) defining the use of the symbol. The degree of compression depends on the level of duplication in each block. Higher compression ratios can be achieved by using larger block sizes or by sorting data to increase the coincidence of duplicate values. Data remains in row-major format where the columns in each row are stored together. Compression occurs when data is loaded using a direct


path load operation, such as CREATE TABLE AS SELECT or direct path SQL*Loader. Tables that use basic compression support DML operations, however any data that is inserted or updated remains uncompressed. Basic compression is useful in decision support systems (DSS) where the compressed data is subject to minimal change.

Compression Usage (continued)• OLTP compression uses the same compression technique as basic compression, but

provides support for compressing transactional data from all DML statements, not just direct path loads. This makes it useful in DSS and online transaction processing (OLTP) p p g ( )environments. Instead of compressing changes as they are written, all the changes to a data block are compressed as a batch whenever a change causes the block to become fuller than an internally controlled threshold. In other words, only the DML statements that trigger compression experience the overhead of compression for the associated data blocks, however most DML statements experience no compression overhead at all. This means that compression can be implemented on OLTP systems without significantly impacting the overall transactional throughput of the system.impacting the overall transactional throughput of the system.

In addition to the basic and OLTP compression modes provided by Oracle Database, Exadata provides Exadata Hybrid Columnar Compression. Exadata Hybrid Columnar Compression technology uses a modified form of columnar storage instead of row-major storage. Sets of rows are stored in an internal structure called a compression unit. Within each compression unit, the values for each column are stored together along with metadata that maps the values to the rows. Compression is achieved by replacing repeating values with smaller symbolic references. Because a compression unit is much larger than an Oracle block and because columnBecause a compression unit is much larger than an Oracle block, and because column organization brings similar values together, Exadata Hybrid Columnar Compression can deliver much better compression ratios than both basic and OLTP compression. The best rates of compression are achieved using direct path loads.

Exadata Hybrid Columnar Compression provides a choice of compression modes to achieve the proper trade-off between disk usage and CPU overhead:

• Warehouse compression: This type of compression is optimized for query performance,Warehouse compression: This type of compression is optimized for query performance, and is intended for DSS and data warehouse applications.

• Online archival compression: This type of compression is optimized for maximum compression ratios, and is intended for historical data and data that does not change.

Exadata Hybrid Columnar Compression supports DML operations on compressed data. However, such operations attract a considerable CPU overhead and yield a lower compression ratio than for direct path loads. In addition, DML on tables using Exadata Hybrid Columnar Compression requires the entire compression unit to be locked, which may reduce concurrency. Finally, updates to rows using Exadata Hybrid Columnar Compression cause rowids to change. As a result, Exadata Hybrid Columnar Compression is recommended for situations where data changes are infrequent or where data sets are reloaded rather than substantially changed.

In conclusion, Exadata Hybrid Columnar Compression makes effective use of Exadata server hardware to deliver the highest levels of compression for data in an Oracle database. It is best suited to cases where the data is not subject to substantial change. For transactional data sets, su ed o cases e e e da a s o subjec o subs a a c a ge o a sac o a da a se s,you should consider OLTP compression instead of Exadata Hybrid Columnar Compression. In all cases, you should be aware of the relative merits and overheads associated with each compression type in order to choose the best approach for your situation.


Index Usage

• With Exadata, scans might perform better than index lookups for some queries.

• Consider removing indexes where scans deliver acceptable performance.p p

• Removing unnecessary indexes will:– Improve DML performance

– Save storage space

• Test the effect of removing indexes by making them invisible:invisible:

SQL> ALTER INDEX <index_name> INVISIBLE;

Index Usage

Some queries that require indexes when using conventional storage will perform acceptably without indexes using Exadata Review your queries that use indexes to determine if they


without indexes using Exadata. Review your queries that use indexes to determine if they would run acceptably with Exadata scans. To test if queries run acceptably without an index, you can make the index invisible to the optimizer. An invisible index still exists and is maintained by DML operations, but it is not used by the optimizer for queries. To make an index invisible, use the following command:

ALTER INDEX <index_name> INVISIBLE

Removing unnecessary indexes will aid the performance of DML operations and will save storage space.


ASM Allocation Unit Size

• By default, ASM uses an allocation unit (AU) size of 1 MB.

• For Exadata, the recommended AU size is 4 MB.– AU size must be set when a disk group is created.

– AU size cannot be altered after a disk group is created– AU size cannot be altered after a disk group is created.– AU size is set using the AU_SIZE disk group attribute:

SQL> CREATE DISKGROUP data NORMAL REDUNDANCY

DISK 'o/*/data_CD*'

ATTRIBUTE 'compatible.rdbms' = '11.2.0.0.0',

'compatible.asm' = '11.2.0.0.0',p ,

'cell.smart_scan_capable' = 'TRUE',

'au_size' = '4M';

ASM Allocation Unit Size

To achieve fast disk scan rates, it is important to lay out segments with at least 4 MB of contiguous space This allows disk scans to read 4 MB of data before performing another


contiguous space. This allows disk scans to read 4 MB of data before performing another seek at a different location on disk. To ensure segments are laid out with 4 MB of contiguous space, set the Oracle ASM allocation unit size to 4 MB. The allocation unit can be set with the AU_SIZE disk group attribute when creating the disk group.


Minimum Extent Size

• Large segments should have initial extents of 8 MB: – Stops needless proliferation of small extents in the database

– Minimizes sub-optimal reads for small extents

– Compliments the 4 MB ASM AU size recommendationCompliments the 4 MB ASM AU size recommendation

• Large initial extents can be specified using the:– INITIAL storage parameter for specific objects or for all

objects in a locally managed tablespace– CELL_PARTITION_LARGE_EXTENTS initialization

parameter for all new segments in a partitioned object:parameter for all new segments in a partitioned object:— TRUE: Sets an 8 MB minimum extent size, but will trim last

unloaded extents to 64 KB boundary for direct loads— ALWAYS: Sets an 8 MB minimum extent size and also minimizes

the creation of suboptimal extent sizes at the end of a load— DEFAULT: Uses traditional extent allocation

Minimum Extent SizeExtent size is managed automatically in locally managed tablespaces. This option automatically increases the size of the extent depending on segment size available free


automatically increases the size of the extent depending on segment size, available free space in the tablespace, and other factors. By default, the extent size starts at 64 KB and increases in increments of 1 MB, 8 MB, or 64 MB.

Generally speaking, it is recommended that large segments should be defined with larger than default initial extents to minimize the needless proliferation of small extents in the database. For Exadata, the recommendation is to have initial extents of 8 MB for large segments. This setting complements the recommendation to set the ASM allocation unit size to 4 MB The following options can be used to influence the database to allocate largeto 4 MB. The following options can be used to influence the database to allocate large extents:

• The INITIAL storage parameter will set the size of the first extent for a newly created object. It can be set specifically for individual objects or it can be set at the tablespace level for locally managed tablespaces. For objects that are large, set INITIAL to 8 MB.

• The CELL_PARTITION_LARGE_EXTENTS initialization parameter can be used to set the INITIAL storage parameter to 8 MB for all new segments in a partitioned object.


g p g p jThis parameter is relevant only to locally managed tablespaces created with the AUTOALLOCATE option, not the UNIFORM option. The parameter does not affect nonpartitioned objects. The parameter may be set to the following values: TRUE, ALWAYS, and DEFAULT. In general, Oracle recommends setting this parameter to TRUE.

Quiz

For which of the following scenarios would it make sense to consider altering the default Exadata Smart Flash Cache in order to configure a flash-based tablespace in your database.

1. When you can fit 10 or more of your tables into a flash-based t bltablespace.

2. When you can fit at least 80% of your most used tables into a flash-based tablespace.

3. When moving specific tables and indexes to a flash-based tablespace will soak up heavy write activity, freeing disk I/O bandwidth for other operationsbandwidth for other operations.

4. When you have key application objects (such as a star-schema fact table) that you wish to cache all the time, you understand their space requirements intimately, and you are prepared to accept the additional management effort of configuring a flash-based tablespace.

Answer: 3, 4



Quiz

Which of the following CREATE TABLE compression clauses are only available to use in conjunction with Exadata storage.1 COMPRESS BASIC1. COMPRESS BASIC

2. COMPRESS FOR OLTP

3. COMPRESS FOR QUERY

4. COMPRESS FOR ARCHIVE

Answer: 3, 4



Summary

In this lesson, you should have learned how to:

• Describe the recommendations for optimizing Exadata performance



Additional Resources

• Lesson Demonstrations (Viewlets)– Configuring Flash-Based Disk Groups

— http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/071ExadataFlashDiskGroups/071exadataflashdiskgroups_viewlet_swf.html

– Examining Exadata Hybrid Columnar Compression— http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/072ExadataCompressionAdmin/072exadatacompressionadmin_viewlet_swf.html

– Index Elimination with Exadata— http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/073ExadataIndexElimination/073exadataindexelimination_viewlet_swf.html



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/071ExadataFlashDiskGroups/071exadataflashdiskgroups_viewlet_swf.html

http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/072ExadataCompressionAdmin/072exadatacompressionadmin_viewlet_swf.html

http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/073ExadataIndexElimination/073exadataindexelimination_viewlet_swf.html

Practice 7 Overview: Optimizing Database Performance with ExadataOptimizing Database Performance with Exadata

In these practices, you will explore how to optimize database performance using:

• Flash-based disk groups

• Exadata Hybrid Columnar CompressionExadata Hybrid Columnar Compression

• Index elimination



D t b M hi O i d A hit tDatabase Machine Overview and Architecture


Objectives


• Describe the architecture and various configuration options for Database Machine



Introducing Database Machine

• Fully integrated platform for Oracle Database

• Based on Exadata storage

• High-performance and high-availabilityHigh performance and high availability for all Oracle Database workloads

• Balanced hardware configurations

• Well suited as a database consolidation platform

Si l d f t t i l t• Simple and fast to implement

Introducing Database MachineDatabase Machine is a fully-integrated Oracle Database platform based on Exadata storage technology. Database Machine provides a high-performance, highly-available solution for all


database workloads, ranging from scan-intensive data warehouse applications to highly concurrent OLTP applications.

Special attention has been paid to ensure that Database Machine is a well balanced platform. Throughout the hardware architecture of Database Machine, components and technologies have been specially matched to eliminate bottlenecks while maintaining good hardware utilization.

Using the unique clustering and workload management capabilities of Oracle DatabaseUsing the unique clustering and workload management capabilities of Oracle Database, Database Machine is well-suited for consolidating multiple databases onto a single Database Machine. Delivered as a complete package of software, servers, and storage, Database Machine is simple and fast to implement.

Note: While Database Machine is a fully-integrated platform solution comprised of specific hardware and software components, Oracle offers Database Machine as a series of separately purchasable components. Customers can choose from the different hardware configurations that are available Appropriate licensing of Oracle Database and Exadata software is also


that are available. Appropriate licensing of Oracle Database and Exadata software is also required. In addition, Database Machine is highly complementary with clustering and parallel operations, so Oracle Real Application Clusters and Oracle Partitioning are highly recommended software options for Database Machine.

Database Machine X2-2 Full Rack

8 Intel-based 2 CPUSun Fire database servers

14 Exadata Storage Servers (all high performance or all high capacity)

2 Power Distribution Units (PDUs) at rear

3 Sun Datacenter InfiniBand Switch 36(36-port managed QDR

i h 40Gb/ )

Keyboard, Video, Mouse (KVM) hardware

1 Cisco Ethernet switch(48-port)

switch - 40Gb/s)


Database Machine is offered in 4 configurations. The contents of a Database Machine X2-2 Full Rack configuration is illustrated in the slide.


g


X2-2 Database Server Hardware Details(Sun Fire X4170 M2)(Sun Fire X4170 M2)

Processors 2 Six-Core Intel® Xeon® X5670 processors (2.93 GHz)

Memory 96 GB (12 x 8 GB)

Local Disks Four 300 GB 10K RPM SAS disks

Disk Controller Disk controller HBA with 512 MB battery backed cache

Network Two InfiniBand 4X QDR (40 Gb/s) ports (1 dual-port PCIe 2.0 HCA)

Four 1GbE Ethernet ports

Two 10GbE Ethernet SFP+ ports (1 dual-port 10GbE PCIe 2.0 network card based on the Intel 82599 10GbE Controller technology)

Remote Management 1 Ethernet port (ILOM)

Power Supplies 2 redundant hot-swappable power supplies

X2-2 Database Server Hardware Details (Sun Fire X4170 M2)

Database Machine X2-2 uses standardized database server hardware based on 64 bit Intel-based Sun Fire servers The slide shows a description of the database server hardware


based Sun Fire servers. The slide shows a description of the database server hardware.


Start Small and Grow

X2-2 Full Rack

X2-2 Quarter Rack

X2-2 Half Rack

Start Small and Grow

In addition to a Database Machine X2-2 Full Rack, which contains eight database servers, 14 Exadata cells and 3 InfiniBand switches, Oracle offers two other Database Machine X2-2


configurations which allow customers to start small and grow:

• A Quarter Rack contains two database servers, three Exadata cells and 2 InfiniBand switches.

• A Half Rack contains four database servers, seven Exadata cells and 3 InfiniBand switches.

All of the Database Machine X2-2 configurations are packaged in a standard 19-inch 42U rack. All configurations are pre-cabled and supplied with a KVM and Ethernet switch to complete the configuration. In addition, Oracle offers hardware upgrade kits that allow customers to upgrade from a Quarter Rack to a Half Rack, or to upgrade from a Half Rack to a Full Rack.



2 Intel-based 8 CPUSun Fire database servers

14 Exadata Storage Servers (all high performance or all high capacity)

2 Power Distribution Units (PDUs) at rear

3 Sun Datacenter InfiniBand Switch 36(36-port managed QDR

i h 40Gb/ )

1 Cisco Ethernet switch(48-port)

switch - 40Gb/s)


In addition to the three Database Machine X2-2 configurations, Oracle now offers Database Machine X2-8 Database Machine X2-8 is only offered in a Full Rack There are no smaller


Machine X2 8. Database Machine X2 8 is only offered in a Full Rack. There are no smaller configurations.

In many ways Database Machine X2-8 is essentially the same as Database Machine X2-2 Full Rack. Both contain 14 Exadata cells, 3 InfiniBand switches, 2 power distribution units (PDUs) and an Ethernet switch.

The major difference is that the eight 2 CPU database servers in Database Machine X2-2 Full Rack are replaced by two 8 CPU database servers. Each of the larger database servers p y gcontains more RAM and is equipped with eight 10 Gb/sec Ethernet ports for high-speed client network access in addition to the 1 Gb/sec Ethernet ports that support administrative access.

Database Machine X2-8 does not come equipped with KVM hardware.

Database Machine X2-8 complements the X2-2 models. Like the X2-2 models, it is designed for database consolidation, large OLTP and data warehouse applications. The large memory configuration of Database Machine X2-8 makes it a particularly good platform for database

lid ti C t th t tl th i li ti l SMP t bconsolidation. Customers that currently run their applications on large SMP systems may be more comfortable choosing Database Machine X2-8.


X2-8 Database Server Hardware Details(Sun Fire X4800)(Sun Fire X4800)

Processors 8 x Eight-Core Intel® Xeon® X7560 Processors (2.26 GHz)

Memory 1 TB (128 x 8 GB)

Local Disks Eight 300 GB 10K RPM SAS disks

Disk Controller Disk controller HBA with 512 MB battery backed cache

Network Eight InfiniBand 4X QDR (40Gb/s) ports

(4 dual-port PCE 2.0 express modules)

Two Network Express Modules (NEM), providing a total of

Eight 1GbE Ethernet ports and

Eight 10 GbE Ethernet SFP+ ports (4 Fabric Express Modules (FEM) using Intel 82599 10GbE controller technology)

Remote Management 1 Ethernet port (ILOM)

Power supplies 4 redundant hot-swappable power supplies

X2-8 Database Server Hardware Details (Sun Fire X4800)

The slide shows a description of the database server hardware used in Database Machine X2-8


X2 8.


Database Machine Capacity

X2-2 Quarter Rack

X2-2 Half Rack

X2-2 Full Rack

X2-8 Full Rack

High Performance 21 TB 50 TB 100 TB 100 TB

Raw Disk1High Performance 21 TB 50 TB 100 TB 100 TB

High Capacity 72 TB 168 TB 336 TB 336 TB

Raw Flash1 1.1 TB 2.6 TB 5.3 TB 5.3 TB

User Data2High Performance 6 TB 14 TB 28 TB 28 TB

1 - Raw capacity calculated using 1 GB = 1000 x 1000 x 1000 bytes and 1 TB = 1000 x 1000 x 1000 x 1000 bytes.

2 - User Data: Actual space for uncompressed end-user data, computed after single mirroring (ASM normal redundancy) and after allowing space for database structures such as temporary space, logs, undo space, and indexes. Actual user data capacity varies by application. User Data capacity calculated using 1 TB = 1024 * 1024 * 1024 * 1024 bytes.

User DataHigh Capacity 21 TB 50 TB 100 TB 100 TB

Database Machine Capacity

The table in the slide gives you some key capacity metrics for the available Database Machine configurations.


g


Database Machine Performance

X2-2 Quarter Rack

X2-2 Half Rack

X2-2 Full Rack

X2-8 Full Rack

Raw Disk High Performance 5.4 GB/s 12.5 GB/s 25 GB/s 25 GB/sRaw Disk Throughput1

g

High Capacity 3 GB/s 7 GB/s 14 GB/s 14 GB/s

Raw Flash Throughput1 11 GB/s 25 GB/s 50 GB/s 50 GB/s

Raw Disk IOPS2

High Performance 10,800 25,000 50,000 50,000

High Capacity 5 400 12 500 25 000 25 000

1 – Bandwidth is peak physical disk scan bandwidth, assuming no compression.

2 – IOs per second (IOPS) – Based on IO requests of size 8K

IOPS High Capacity 5,400 12,500 25,000 25,000

Flash IOPS2 225,000 500,000 1,000,000 1,000,000

Database Machine Performance

The above table gives you some key performance metrics for the different Database Machine configurations.


g

Note: These metrics do not take into account compression. As a result, they are quite conservative. With compressed data, you can achieve much higher throughput rates. In all cases, actual performance will vary by application.


Database Machine X2-2 Architecture

Database Machine X2-2 Architecture

The diagram in the slide shows how the major components of Database Machine X2-2 are connected to each other The diagram shows the KVM hardware Ethernet switch and


connected to each other. The diagram shows the KVM hardware, Ethernet switch and InfiniBand switches. For the sake of clarity, only one database server and one Exadata server are shown. Database Machine contains three major networks:

• Management/ILOM Network. This is a standard Ethernet/IP network which is used to manage Database Machine. Using this network, administrators can access the database servers and the Exadata servers using the KVM hardware or by using remote login facilities such as Secure Shell (SSH). The database servers and Exadata servers

l id Eth t i t f f I t t d Li ht O t M t (ILOM) ILOMalso provide an Ethernet interface for Integrated Lights-Out Management (ILOM). ILOM provides a powerful set of remote administration facilities. Using ILOM, administrators can remotely monitor and control the state of the server hardware. The InfiniBand switches also provide an Ethernet port for remote configuration and management purposes.

• Client Access Network. There is also a standard Ethernet network which is primarily used to provide database connectivity via Oracle Net software. The client access network can be separate from the management network (as illustrated in the diagram) or both functions can be provided by the same network. Single-network configurations are quite typical.


Database Machine X2-2 Architecture (continued)When Database Machine is initially configured, customers can choose to configure the database servers with a dedicated client network interface (NET1) or they can choose to configure a bonded network interface (using NET1 and NET2). A bonded g ( g )client access network interface can provide protection if a network interface fails however using bonded interfaces may require additional configuration in the customer’s network.

• InfiniBand Network. The InfiniBand network is to provides a reliable high-speed interconnect for Database Machine. The InfiniBand network in described in greater detail later in this lesson.

I dditi th NET2 d NET3 i t f th d t b b ti llIn addition, the NET2 and NET3 interfaces on the database servers can be optionally configured for additional networks.

Note: Database Machine X2-8 is essentially the same except that each database server has more physical Ethernet ports and each database server also contains four bonded InfiniBand network interfaces connecting to the storage network.


InfiniBand Network Architecture

• Sun Datacenter 36-port managed QDR (40 Gb/s) switches– 2 leaf switches used to connect server InfiniBand ports

– 1 spine switch in Full Rack and Half Rack configurations only

• Exadata Servers and X2-2 Database Servers– Each server has one dual-port QDR (40 Gb/s) InfiniBand HCA

– Each HCA port is connected to a different leaf switch for high availability

– Active-passive bonding with single IP address— Performance is limited by PCIe bus, so active-active not needed

• X2 8 Database Servers• X2-8 Database Servers– Each server has four dual-port QDR (40 Gb/s) InfiniBand HCAs

– Each pair of HCA ports are connected to different leaf switches for high availability

– Active-passive bonding for each bonded pair

• Connections are cabled in the factory

InfiniBand Network Architecture

Within Database Machine, the primary role of the InfiniBand network is to provide a reliable high-speed interconnect to support database server to storage server communications It is


y

high speed interconnect to support database server to storage server communications. It is also used to facilitate communication and coordination between database servers using Oracle Clusterware and Oracle Real Application Clusters (RAC) software. It is important to note that the InfiniBand network is completely internal to Database Machine and there is no requirement for additional InfiniBand infrastructure within the customer’s network.

Each Database Machine configuration uses at least two Sun Datacenter 36-port managed QDR (40 Gb/s) InfiniBand switches. Each database server and each Exadata server is

t d t b th it h i b d d t k i t f T it hconnected to both switches using one or more bonded network interfaces. Two switches are used to provide high availability. If a switch fails, then the InfiniBand network can be maintained by the remaining switch. In such a case there would be no performance loss because each switch is capable of supporting the full system bandwidth. The switches that connect the database servers and Exadata servers within a rack are known as leaf switches.

Full Rack and Half Rack Database Machine configurations contain a third InfiniBand switch, known as the spine switch. The spine switch connects the 2 leaf switches and is also used to p pconnect multiple Database Machines. Quarter Rack configurations do not include a spine switch.


X2-2 Leaf Switch Topology

X2-2 Leaf Switch Topology

The diagram in the slide illustrates how the two leaf switches in each Database Machine X2-2 rack are connected to the database servers Exadata servers and to each other Each


rack are connected to the database servers, Exadata servers and to each other. Each database server and each Exadata server is configured with a bonded network interface (BOND0) which uses the two InfiniBand HCA ports on the server. Active/passive bonding is used. On each server, each InfiniBand port is connected to a different leaf switch. The active and passive connections are spread across the leaf switches to balance the workload. In addition, the leaf switches are connected to each other via seven inter-switch links. All the connections are pre-cabled in the factory. The topology described on this page is common to all Database Machine X2-2 configurations contained in a single rack; Quarter Rack Half Rackall Database Machine X2-2 configurations contained in a single rack; Quarter Rack, Half Rack and Full Rack. Multiple rack configurations are discussed later in the lesson.

Note: The switch port mappings shown in the diagram are illustrative only.

Note: Database Machine X2-8 has a similar topology except that each database server has four bonded network interfaces with connections spread between the leaf switches.


Full Rack Spine and Leaf Topology

Full Rack Spine and Leaf Topology

The diagram in the slide shows the added spine switch which exists in Full Rack and Half Rack Database Machine configurations only As shown in the diagram each leaf switch is


Rack Database Machine configurations only. As shown in the diagram, each leaf switch is connected to the spine switch using a single network link. The purpose of the spine switch is to connect multiple Database Machines together, creating a single large-scale database grid.


Scale Performance and Capacity

Scalable• Scale to eight racks by adding

Redundant and Fault Tolerant• Failure of any component g y g

cables• Scale beyond eight racks by

adding InfiniBand switches• Scale to hundreds of storage

servers to support multi-petabyte databases

y pis tolerated.

• Data is mirrored across storage servers.

Scale Performance and Capacity

If you need to scale beyond the performance or capacity of a single Database Machine configuration, you can scale up to eight Full Racks by simply adding some cabling between


g y g y y g gthem. There are enough free ports in the InfiniBand switches and enough bandwidth for eight Full Rack Database Machines running at full performance and with full redundancy.

If you want to extend beyond eight racks, you need to add InfiniBand switches.

In addition, Oracle now enables customers to interconnect a Half Rack Database Machine with other Half Rack or Full Rack configurations. Interconnecting Quarter Rack Database Machines is not supported.


Scaling Out to Multiple Full Racks

• Single InfiniBand network based on a Fat Tree topology– Up to 8 racks supported with existing switches

– Database and storage server cabling unchanged

• Two rack example:Two rack example:

Scaling Out to Multiple Full RacksWhen multiple Full Rack Database Machines are connected to form a large-scale database grid, the InfiniBand network is modified to resemble a Fat Tree topology. The database server


g , p gyand storage server connections to the leaf switches remain unchanged. In a standalone Full Rack Database Machine, each leaf switch has 8 links that connect it to the other leaf switch in the rack (7 links) and to the spine switch (1 link). When multiple full racks are interconnected, these 8 links are redistributed to the spine switches. The diagram in the slide illustrates this using a two rack example. As shown in the diagram, the seven inter-switch links that originally connected the leaf switches in each rack have been redistributed so that each leaf switch has 4 links to each spine switch. There are no direct links between spine

it h d th di t li k b t l f it hswitches and there are no direct links between leaf switches.To connect eight Full Rack Database Machines, the eight leaf-to-spine links on each leaf switch are configured to provide exactly one link to each spine switch. For configurations between 2 and 8 racks, the eight leaf-to-spine links on each leaf switch are distributed as evenly as possible amongst the total number of spine switches.Oracle supports interconnecting up to eight Full Rack Database Machines using the existing switches in each rack. Scaling beyond eight racks requires larger external switches.Note: Up to three Full Rack Database Machines can be interconnected using the cabling supplied from the factory. To interconnect more than three Database Machines, additional cabling must be purchased.


Quiz

Bonded interfaces are used for the InfiniBand storage network on Database Machine servers to provide:

1. Fault Tolerance and High Availability

2 Increased Bandwidth and Performance2. Increased Bandwidth and Performance

3. Both of the above

Answer: 1

Each database server and Exadata server is configured with bonded network interfaces which use the InfiniBand HCA ports on the server. Active/passive bonding is used so the aggregate


g gg gbandwidth of the links is not used simultaneously. For each bonded network interface, each InfiniBand port is connected to a different leaf switch. This configuration provides fault tolerance and high availability by removing the InfiniBand switch as a potential single point of failure.


Quiz

Which of the following scalability options are supported?

1. Upgrading a Quarter Rack Database Machine to a Half Rack Database Machine

2 Upgrading a Half Rack Database Machine to a Full Rack2. Upgrading a Half Rack Database Machine to a Full Rack Database Machine

3. Interconnecting two Quarter Rack Database Machines

4. Interconnecting two or more Full Rack Database Machines

Answer: 1, 2, 4

Interconnecting Database Machines is only supported for Half Rack and Full Rack configurations


configurations.


Summary


• Describe the architecture and various configuration options for Database Machine



D t b M hi C fi tiDatabase Machine Configuration


Objectives


• Describe the installation and configuration process for Database Machine.

• Describe the default configuration for Database MachineDescribe the default configuration for Database Machine.

• Describe supported and unsupported customizations for Database Machine.



Database Machine Implementation Overview

Four phases:

1. Pre-installation– Various planning and scheduling activities including:

— Site planning: space power cooling logistics— Site planning: space, power, cooling, logistics…

— Configuration planning: hostnames, IP addresses, databases…

— Network preparation: DNS, NTP, cabling…

– Oracle and customer engineers work together

2. Installation and configuration– Hardware and software installation and configurationHardware and software installation and configuration

– Result is a working system based on a default configuration

– Conducted by Oracle engineers


The process of successfully implementing Database Machine requires cooperation and coordination between various Oracle and customer representatives and engineers At a high


coordination between various Oracle and customer representatives and engineers. At a high level the process involves four main phases:

1. Pre-installation involves various planning and scheduling activities including those listed in the slide. Typically, Oracle will guide the customer through this phase by collaborating with them in order to complete a key document known as the Configuration Worksheet. The Configuration Worksheet drives the installation and configuration phase (Phase 2). Once it is completed there are additional preparation tasks that the customer will t i ll l t Thi h i d ib d i t d t il l t i th ltypically complete. This phase is described in greater detail later in the lesson.

2. Installation and configuration is the process of taking a virgin Database Machine and installing it into the customer environment. It involves numerous steps conducted by hardware and software engineers. It is highly recommended that Oracle hardware and software engineers are used to conduct this phase because of the specialized nature of the tasks involved and to ensure that the Database Machine is configured in a standard and supportable way. The result of this phase is a working system which is configured according to the Configuration Worksheet. An example is shown later in the lesson.



3. Additional configuration– Additional activities for production readiness including:

— Reconfigure storage using non-default settings

— Create additional databases

— Configure Enterprise Manager

— Configure backup and recovery

– Conducted by customer or Oracle services engagement

4. Post-installation– Ongoing monitoring and maintenanceg g g

– Conducted by customer or Oracle services engagement

Database Machine Implementation Overview (Continued)3. After installation and configuration, the Database Machine is configured on the customer

network Depending on the contents of the Configuration Worksheet the Database


network. Depending on the contents of the Configuration Worksheet, the Database Machine may include one or more running databases. At this point customers can commence the process of loading or migrating data and prepare Database Machine for production use. However, customers may wish to vary the configuration of Database Machine in order to create customized databases or to organize the available storage in a non-default way. Such additional configuration tasks are typically conducted by the customer or Oracle services are engaged to conduct the work. Other examples of additional configuration tasks include configuring Enterprise Manager to assist withadditional configuration tasks include configuring Enterprise Manager to assist with ongoing monitoring and maintenance, or the configuration of a backup and recovery environment. Common supported and unsupported additional configuration activities are discussed later in the lesson.

4. During post-installation the focus of activities shifts to on-going monitoring and maintenance of Database Machine. This is usually conducted by the customer.

The remainder of this lesson focuses on topics that will assist you to understand some of the key activities in the implementation process. The aim is not to replace the available documentation, rather the material is presented to assist you to better understand the available configuration options and to better appreciate the results of making different selections.


Configuration Worksheet Overview

The Configuration Worksheet:

• Is a Microsoft Excel spreadsheet with macros

• Contains site-specific configuration settings including:Host and domain names– Host and domain names

– IP addresses

– Region and time zone information

– Name servers and NTP time servers

– Database server OS user and group settings

E d t ll tifi ti tti– Exadata cell notification settings

– ASM diskgroup definitions

– Oracle database definitions

• Generates a series of configuration files that drive the installation and configuration programs

Configuration Worksheet Overview

One of the most important Database Machine pre-installation tasks is to complete the Configuration Worksheet Completing the Configuration Worksheet is often a collaborative


Configuration Worksheet. Completing the Configuration Worksheet is often a collaborative effort involving database, network and system administrators. Typically, Oracle engineers will participate to provide assistance. The information in the configuration worksheet is used to configure Database Machine. It also defines the network names and addresses that need to be configured within your network before installing Database Machine. The following section examines the Configuration Worksheet in greater detail.


Getting Started

Getting Started

The slide shows a brand new Configuration Worksheet. The values in the worksheet are all pre-defined defaults The worksheet version is displayed at the very top of the worksheet


pre defined defaults. The worksheet version is displayed at the very top of the worksheet. Please check with your Oracle representative to ensure that you are using the latest available version.

Notice that the Configuration Worksheet implies a set of conventions for naming the various components of Database Machine. For example, the Compute Node Hostname is derived by concatenating the DB Machine Prefix along with the Compute Node Base Name and then adding a two digit numeric identifier at the end.

Notice also that the private InfiniBand network is configured by default to use addresses in the 192.168.10.X range. Because this network is self-contained within Database Machine, in most cases there is no requirement to change this setting.

While the Configuration Worksheet is quite flexible and allows many opportunities for customization, it is highly recommended that customers should stick to the default conventions wherever possible. This not only reduces the possibility for errors or misconfigurations but it also helps to ensure that the resulting configuration is easier tomisconfigurations, but it also helps to ensure that the resulting configuration is easier to support.


Configuration Worksheet Example


The slide shows an example of the first part of a completed Configuration Worksheet. The following slide shows the rest of the example The example illustrates a typical configuration


following slide shows the rest of the example. The example illustrates a typical configuration using as many default values as possible. In this part of the Configuration Worksheet the following fields are set to non-default values:

• Customer Name: Setting the Customer Name to Example Industries has no effect on the configuration of Database Machine. The field simply helps to identify this instance of the Configuration Worksheet.

• DB Machine Prefix: DB Machine Prefix is an identifier used to generate host names for network interfaces for all the systems. Indirectly, this value is also used in other identifiers such as cell disk and grid disk names. Oracle recommends that this value should be 4 characters or less to avoid possible issues with names that are too long. In this example, eidm is an abbreviation of Example Industries Database Machine. Customers should choose a setting that reflects their own naming conventions.

• dbMachine Model: Select a value from the drop-down list to reflect the environment being configured. Setting this field modifies the rest of the Configuration Worksheet so be g co gu ed Sett g t s e d od es t e est o t e Co gu at o o s eet sothat the correct number of database servers and Exadata cells are configured.


Configuration Worksheet Example (Continued)• Domain Name: This field is set to the domain name of the management network where the

Database Machine resides.

• Region: This field is set to the geographic region where the database machine resides• Region: This field is set to the geographic region where the database machine resides.

• Time Zone: This field is set to the time zone where the database machine resides.

• NameServer: Use these fields to identify up to four DNS servers in the network where the Database Machine resides. While it is possible to implement Database Machine without DNS, Oracle recommends that DNS should be used in conjunction with Database Machine.

• NTP Time Server: Use these fields to identify up to four NTP time servers in the network NTP Time Server: Use these fields to identify up to four NTP time servers in the network where the Database Machine resides. NTP services are a mandatory requirement for Database Machine. NTP provides coordinated timing which synchronizes services across Database Machine. Without NTP, a lack of coordination can lead to database nodes being evicted from the cluster or Exadata cells being excluded from the storage pool. NTP services also ensure that the timestamps written to the various log files are coordinated across Database Machine. Note that the Cluster Time Synchronization Service (CTSS) introduced in Oracle Clusterware11g Release 2 cannot be used to provide time services tointroduced in Oracle Clusterware11g Release 2 cannot be used to provide time services to Exadata cells, hence it is not used for Database Machine.

• First Management/ILOM IP address [eth0]: Set this field to the first IP address in the range of IP addresses allocated for the management network. When this field is set, the Configuration Worksheet sequentially allocates IP addresses for each network interface as shown in the Specific Network Information section of the example worksheet. You can change the default addressing policy by changing values in the Specific Network Information section of the worksheet or you can change individual addresses by modifyingInformation section of the worksheet or you can change individual addresses by modifying the addresses generated after you click the Generate button (discussed later in the lesson). Beware that if you modify the default addressing policy you must ensure that the resulting addresses are valid and there are no duplicates.

• First Client Access IP address [eth1]: Set this field to the first IP address in the range of IP addresses allocated for the client access network. When this field is set, the Configuration Worksheet sequentially allocates IP addresses for each network interface as shown in the Specific Network Information section of the example worksheet. You can change the default addressing policy by changing values in the Specific Network Information section of the worksheet or you can change individual addresses by modifying the addresses generated after you click the Generate button (discussed later). Beware that if you modify the default addressing policy you must ensure that the resulting addresses are valid and there are no duplicates.

In this example all the other fields contain default values. Other fields that might typically be setIn this example all the other fields contain default values. Other fields that might typically be set to non-default values include:

• Cell Disk Size: Set this field to 600G for systems with high performance disks or 2TB for systems with high capacity disks.

• Backup requirements: Select from one of the available options (Tape backup, Disk backup only, Both Disk and Tape backup). This setting influences the default size of the ASM disk groups that are configured later in the worksheet.

• Subnet Mask: If the default values are not suitable, the various subnet mask fields should be adjusted to reflect the network where the Database Machine resides.

• Gateway: If the default values are not suitable, the various gateway fields should be adjusted to reflect the network where the Database Machine resides.



Configuration Worksheet Example (Continued)

Following on from the previous slide, this slide shows an example of the second portion of a completed Configuration Worksheet In this part of the Configuration Worksheet the following


completed Configuration Worksheet. In this part of the Configuration Worksheet the following fields are set to non-default values:

• Email To Notification Address, Email From Notification Address, Email From Name, Email Server, Notify via Email: This group of fields are set to facilitate email notification of Exadata cell alerts. Depending on the email server setup you may also need to set other fields in the SMTP and SNMP Details section of the worksheet. Exadata cell alerts can also be delivered via SNMP however this option is not configured in the example.

• Number of Diskgroups: Set this field to a non-zero value to configure ASM disk groups. By default, a setting of 2 results in the creation of disk groups named DATA and RECO as shown in the slide. This section of the Configuration Worksheet is described in greater detail on the next slide.

• Number of Databases: Set this field to a non-zero value to create databases on the Database Machine. The example in the slide shows the default settings for one database. database


Configuration Worksheet Example (Continued)Note the limited set of database configuration attributes that are available in the Configuration Worksheet. Using the Configuration Worksheet you cannot set many common database attributes, such as database character set for example. The defaultcommon database attributes, such as database character set for example. The default database created during installation uses Unicode database character set AL32UTF8. Oracle recommends Unicode for all new deployments. To use a different database character set after installation, or to create databases with other non-default attributes, run Database Configuration Assistant (DBCA) to delete the default database, and to create a new database using the desired settings.

Note also that if you do not specify any databases in the Configuration Worksheet, the configuration programs will not install the Oracle Database software on the Databaseconfiguration programs will not install the Oracle Database software on the Database Machine. Therefore it is highly recommended that you specify at least one database in the Configuration Worksheet, which can be deleted later if it is not required.


Configuring ASM Disk Groupswith Configuration Worksheetwith Configuration Worksheet

DATA(220G)

or SYSTEMDG (Disks 2-11)(29G)

System Area (Disks 0-1)

Exadata Cell nExadata Cell 2

Exadata Cell 1

RECO(ALL ~ 309G)

Configuring ASM Disk Groups with Configuration Worksheet

The slide illustrates how ASM disk groups are configured using the Configuration Worksheet.

B d f lt ASM di k fi d i th C fi ti W k h t t i d


By default, ASM disk groups configured using the Configuration Worksheet are striped across every disk in every Exadata cell in the Database Machine. The worksheet allows you to specify a range of cells to limit the striping to all the disks those cells. However, you cannot specify a subset of disks within Exadata cells using the Configuration Worksheet. The policy of striping disk groups across all the available disks is recommended by default. Deviating from this policy is recommended only where data isolation is required; such as in circumstances where security policies govern that data in different databases must be stored

h i ll t di kon physically separate disks.

Regardless of how many disk groups are specified in the Configuration Worksheet, a disk group named SYSTEMDG is always created. The primary purpose of the SYSTEMDG disk group is to store the shared clusterware files; Oracle Cluster Registry (OCR) and voting disks. The SYSTEMDG disk group occupies a grid disk that consumes approximately 29 Gigabytes on every disk except the first two disks in each cell. On the first two disks (numbered 0 and 1) in each cell, approximately 29 Gigabytes is used to store the System Area, which contains the , pp y g y y ,OS image, swap space, Exadata software binaries, metric and alert repository and various other configuration and metadata files.


Configuring ASM Disk Groups with Configuration Worksheet (Continued)Because of the layout of the System Areas and the SYSTEMDG disk group, the remaining space on every disk is equal.

As shown in the slide specifying two disk groups in the Configuration Worksheet results in aAs shown in the slide, specifying two disk groups in the Configuration Worksheet results in a configuration with disk groups named DATA and RECO. The disk group names can be modified. The amount of space allocated to each disk group on each disk can also be set. The default size values are provided as a guide only and are influenced by Backup Requirements setting which is near the top of the worksheet. In this example the default sizes are used, so the DATAdisk group occupies a grid disk that is allocated 220 Gigabytes on each cell disk, and the RECOdisk group occupies another grid disk that is allocated the remaining space (approximately 309 Gi b t i hi h f di k ) h di kGigabytes assuming high performance disks) on each disk.

Specifying only 1 disk group results in a disk group named DATA which contains all of the non-system disk space on each disk (approximately 529 Gigabytes assuming high performance disks).

If you wish to specify 3 or more disk groups, the first two disk groups will be named DATA and RECO by default and you will need to supply names for the additional disk groups. You will also need to specify the per-disk space allocation for each disk groupneed to specify the per disk space allocation for each disk group.

Note that the grid disks that are created to allocate disk group space on each cell disk are created in the order in which the associated disk groups appear in the Configuration Worksheet. Because of this ordering, it is recommended that you list the disk groups with the highest performance requirements first in the Configuration Worksheet.

Note also that the values generated by the worksheet under the heading Estimated Size of DG represent the approximate space available for user data factoring in the default 2-way mirroring policy (ASM NORMAL redundancy) that is applied during disk group creation. The Estimated Size of DG values also factor in the loss of one Exadata cell. For example, in the example worksheet the Estimated Size of DG value for the DATA disk group is approximately 2.6 TB, however the actual disk space allocated to the disk group is approximately 7.9 TB (220 GB x 12 disks per cell x 3 cells) and the space available for user data is approximately 3.9 TB after 2-way mirroring is factored in.


Generating the Configuration Files

1

2

Generating the Configuration FilesAfter the colored fields in the Configuration Worksheet are set, click the Generate button to generate individual host names and IP addresses in the area below the colored fields. An


gexample is shown in the slide. You can customize the entries in this area of the worksheet to set specific host names and IP addresses, however it is recommended that the default settings should be used wherever possible to minimize the risk of errors or misconfigurations. If you change one of the colored fields in the Configuration Worksheet you must click the Generate button again for the change to be reflected in the generated section.

After you are satisfied with the generated settings, click the Create Config Files button to generate the configuration files that are used to drive the Database Machine installation and configuration programs A series of files is written to a directory named according to theconfiguration programs. A series of files is written to a directory named according to the following convention:

c:\dbmachine_<Customer Name>\<DB Machine Prefix>\

While you can examine the entries in the generated files it is not recommended that the files are modified directly because doing so may result in errors during the execution of the installation and configuration programs. If you make a change to the Configuration Worksheet after creating the configuration files, you must re-create the configuration files for the change to be incorporated in themto be incorporated in them.


Other Pre-Installation Tasks

• Network Preparation– Configure Domain Name Service (DNS) server

– Configure Network Time Protocol (NTP) server

• Physical PreparationPhysical Preparation– Provide adequate cooling capacity

– Provide proper ventilation and airflow

– Provide adequate floor space with adequate floor loading

– Provide required power outlets

Provide required network drops– Provide required network drops

Other Pre-Installation Tasks

Once the Configuration Worksheet is complete there are a series of important tasks that must be completed before the installation of Database Machine These can be logically grouped


be completed before the installation of Database Machine. These can be logically grouped into two main areas; network preparation and physical preparation.

To prepare your network for Database Machine configure DNS to include the host names and IP addresses allocated in the Configuration Worksheet. While the use of DNS is not mandatory, it is highly recommended. If one is not already available, you must configure a NTP service for use in conjunction with Database Machine. NTP configuration is mandatory because it is used to synchronize the InfiniBand switches, not just the Oracle Clusterware

ftsoftware.

Physical preparation involves provisioning your data center to accept Database Machine. The main provisioning tasks are listed in the slide. Your Oracle representative will provide specific requirements based on your proposed Database Machine configuration and location.


The Result After Installation and Configuration

DB Server 2DB Server 1

DB Instance: dbm1

ASM Instance: +ASM1

DB Instance: dbm2

ASM Instance: +ASM2

eidm01 172.16.1.101

eidm01-vip 172.16.1.103

eidm02 172.16.1.102

eidm02-vip 172.16.1.104

eidmdb0110.7.7.101

eidmdb0210.7.7.102

eidmdb01-ilom10.7.7.106

eidmdb02-ilom10.7.7.107

eidm-scan 172.16.1.105172.16.1.106172.16.1.107

eidmdb01-priv192.168.10.1

eidmcel03-priv192.168.10.5


eidmdb02-priv192.168.10.2


ASM Instance: +ASM1

Cluster: eidm

ASM Instance: +ASM2

Cluster: eidm

SYSTEMDG SYSTEMDG SYSTEMDG

InfiniBand Storage Network

873GB total

Exadata Server 1

SYSTEMDG

DATA

RECO

Exadata Server 2

SYSTEMDG

DATA

RECO

Exadata Server 3

SYSTEMDG

DATA

RECO

eidmcel0110.7.7.103

eidmcel01-ilom10.7.7.108

eidmcel0210.7.7.104


eidmcel0310.7.7.105


873GB total

7,920GB total

11,112GB total

The Result After Installation and Configuration

The diagram in the slide illustrates the result after installation and configuration based on the Configuration Worksheet example shown earlier in this lesson


Configuration Worksheet example shown earlier in this lesson.

The host names and IP addresses are based on the Configuration Worksheet settings. Note that each host name starts with the DB Machine Prefix (eidm). Note also the default IP address allocation scheme.

By default, the storage is configured symmetrically. That is, the storage in each Exadata cell is allocated in the same way. On each storage server:

• A cell disk is defined for each physical disk. The cell disks use the default namingA cell disk is defined for each physical disk. The cell disks use the default naming convention of CD_<nn>_<cell host name> where <nn>is a two digit identifier in the range 00-11 and <cell host name> is the canonical host name for the storage server. For example, CD_01_eidmcel01.

• On every cell disk, apart from the first two (<nn>=00 or 01), a grid disk with the prefix SYSTEMDG is created. These grid disks are each approximately 29 GB in size. These grid disks are allocated to the SYSTEMDG disk group. The 10 grid disks on each Exadata server total approximately 291 GB adding up to 873 GB across all three storage serversserver total approximately 291 GB, adding up to 873 GB across all three storage servers in the example Quarter Rack configuration.


The Result After Installation and Configuration (continued)• On every cell disk, a grid disk with the prefix DATA is created and another grid disk with the

prefix RECO is created. These grid disks are allocated to the DATA and RECO disk groups respectively. Each DATA grid disk is 220 GB which adds up to 2640 GB per cell (12 x 220 p y g p p (GB) or 7920 GB across the Quarter Rack Database Machine. Likewise, each RECO grid disk is approximately 309 GB, totalling 11112 GB across the Quarter Rack Database Machine.

• All the available flash storage is configured as Exadata Smart Flash Cache.

The Database Servers are also configured symmetrically by default. Each database server is installed with the same operating system configuration, including the same Oracle user account and group definitions. The database servers are configured as a cluster under the control of Oracle Clusterware software. An ASM cluster is also configured. Finally, Oracle Database software is installed on each database server and a Real Application Clusters (RAC) database is established across all the cluster nodes.

Note the following particulars regarding the installation and configuration of Oracle Grid Infrastructure and Oracle Database on Database Machine.

• Grid Naming Service (GNS) is not configured DNS is the preferred naming service for• Grid Naming Service (GNS) is not configured. DNS is the preferred naming service for Database Machine.

• RAC databases created as part of the installation and configuration process are administrator-managed by default. To create a policy-managed RAC database on Database Machine you must manually create the database using the Database Configuration Assistant (DBCA).

• By default, Database Control is not configured for databases created by the installation and configuration process.


Supported Additional Configuration Activities

• Earthquake protection using a 3rd party Seismic Isolation Platform

• Replace the Ethernet switch

• Connect a tape library for backup

• Create and configure additional databases

• Configure database features– Oracle Data Guard

– Database File System (DBFS)

• Configure Enterprise Manager

• Customize the storage configuration; ASM and Exadata

Supported Additional Configuration Activities

The slide lists some typical configuration activities which customers can undertake using their own labor and at their own expense The following points expand on some of these:


own labor and at their own expense. The following points expand on some of these:

• Customers are permitted to implement earthquake protection with a 3rd party Seismic Isolation Platform as long as the Database Machine is not physical altered or re-racked.

• Customers are permitted to replace the Cisco Ethernet switch with an equivalent switch of their choice. Alternatively, customers can implement 3rd party gateway switches to isolate Database Machine from other components on the network.

• Customers are permitted to connect additional servers or devices to Database Machine pvia Ethernet or InfiniBand. Typically, customers require such connections to connect their backup and recovery infrastructure or to facilitate data transfer from other systems.


Unsupported Configuration Activities

• Hardware re-racking

• Adding components to servers

• Adding servers to Quarter Rack or Half Rack configurationsconfigurations

• Interconnecting Quarter Rack configurations

• Swapping Linux distributions

• Configuring ACFS

Unsupported Configuration Activities

The following configuration activities are not supported with Database Machine:

Hardware re racking: Customers sometimes wish to re rack a Database Machine to


• Hardware re-racking: Customers sometimes wish to re-rack a Database Machine to comply with a data center policy, to achieve earthquake protection or to overcome a physical limitation of some sort. Apart from being inherently error-prone, re-racking can cause component damage, thermal management issues, cable management issues and other issues. As a result, hardware re-racking of Database Machine is not supported.

• Adding components to servers: Customers sometimes wish to add components to Database Machine. A typical example is the desire to add a Host Bus Adapter (HBA) to the database servers so that they can be attached to existing SAN storage. Adding components to servers in Database Machine is not supported because of the potential for driver and firmware incompatibilities that could undermine the system.

• Adding servers to Quarter Rack or Half Rack configurations: Outside of a supported upgrade, physically adding servers to a Quarter Rack or Half Rack is not supported because it changes the environmental and power characteristics of the system. It also impacts on the future ability to conduct a supported upgrade. Note that customers can pacts o t e utu e ab ty to co duct a suppo ted upg ade ote t at custo e s caadd Exadata cells to any Database Machine configuration by placing the additional cells in a separate rack and by using the spare InfiniBand ports to connect to them.


Unsupported Configuration Activities (continued)• Interconnecting Quarter Rack configurations: Interconnecting Quarter Rack configurations

has not been tested and certified by Oracle. In addition, such configurations cannot be replicated by Oracle Support in order to analyze and remedy problems.replicated by Oracle Support in order to analyze and remedy problems.

• Swapping Linux distributions: Oracle Linux is provided as the operating system underpinning the database servers and Exadata servers in Database Machine. Swapping Linux distributions is not supported.

• Configuring ACFS: ASM Cluster File System (ACFS) is currently unavailable on Database Machine.

Note: Consult your Oracle representative to determine whether other configuration activities are y p gsupported or not.


Quiz

Using the Configuration Worksheet you can set specific IP addresses for each database server and Exadata server in Database Machine:

1. True

2. False

Answer: 1

You can set individual IP addresses by modifying the addresses generated after you click the Generate button in the Configuration Worksheet. Beware that if you modify the default


g y yaddressing policy you must ensure that the resulting addresses are valid and there are no duplicates. Note that while the Configuration Worksheet is quite flexible and allows customers many opportunities for customization, it is highly recommended that customers should stick to the default conventions wherever possible. This not only reduces the possibility for errors or misconfigurations, but it also helps to ensure that the resulting configuration is easier to support.


Quiz

Which of the following options for connecting to existing SAN storage is supported in conjunction with Database Machine?

1. Install a fiber channel HBA into each database server

2 Use a server connected to the existing SAN as a storage2. Use a server connected to the existing SAN as a storage gateway and connect it to Database Machine using NFS over Ethernet

3. Use a server connected to the existing SAN as a storage gateway and connect it to Database Machine using NFS over InfiniBandover InfiniBand

Answer: 2, 3

Connecting Database Machine to other servers using Ethernet or InfiniBand is supported. Adding components to Database Machine servers is not supported.


g


Summary


• Describe the installation and configuration process for Database Machine.

• Describe the default configuration for Database MachineDescribe the default configuration for Database Machine.

• Describe supported and unsupported customizations for Database Machine.




• Lesson Demonstrations (Viewlets)– Database Machine Configuration Example using

Configuration Worksheet— http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/091DBMConfigWorksheet/091dbmconfigworksheet_viewlet_swf.html



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/091DBMConfigWorksheet/091dbmconfigworksheet_viewlet_swf.html

Mi ti D t b t D t b M hiMigrating Databases to Database Machine


Objectives


• Describe the general steps to migrate your database to Database Machine

• List the main approaches for migrating your database toList the main approaches for migrating your database to Database Machine

• Identify the most appropriate approach for different circumstances



Migration Best Practices Overview

Legacy storage to Database Machine

1 2 3 4

Capacityplanning

Choose migrationapproach

Migration Post-migrationtasks

1 2 3 4

Migration Best Practices Overview

The slide depicts the four important phases to migrate your existing database to Database Machine. Each phase is discussed in the remainder of this lesson.



Performing Capacity Planning

Sourcedatabase

…ExadataStorageServer

High High

CALIBRATE

AWRReport

High Performance Disk

High Capacity Disk

Exadata Smart Flash Cache 384 GB 384 GB

Raw Disk Capacity 7.2 TB 24 TB

Uncompressed Data Capacity Up to 2 TB Up to 7 TB

Raw Disk Throughput (MBPS) Up to 1 800 Up to 1 000

Size IOPS MBPS

Raw Disk Throughput (MBPS) Up to 1,800 Up to 1,000

Effective Throughput with Flash (MBPS) Up to 3,600 Up to 3,600

Effective Throughput with Flash and Compression (MBPS)

Up to 36,000 Up to 36,000

Disk I/Os per Second (IOPS) Up to 3,600 Up to 1,440

Flash I/Os per Second (IOPS) Up to 75,000 Up to 75,000

Performing Capacity Planning

Before migrating to Database Machine, you must understand the I/O characteristics of your current environment. Collect the size and throughput of your current system. The key measures


g y y yof I/O throughput are I/Os per second (IOPS) and megabytes per second (MBPS). Use the system statistic ‘physical I/O disk bytes’ to derive the current MBPS for the system. Use the system statistics ‘physical reads’ and ‘physical writes’ to determine the current IOPS for the system. These statistics are available in an Automatic Workload Repository (AWR) report.

After you understand the capacity of your current system, you can determine the required number of Exadata cells, and from there determine the appropriate Database Machine configuration It is important to size both for performance and capacity One Exadata serverconfiguration. It is important to size both for performance and capacity. One Exadata server configured with 12 high performance drives provides approximately 2 TB of mirrored uncompressed usable storage, 1,800 MBPS throughput for large I/Os, and 3,600 IOPS for small I/Os, however much greater effective throughput can be achieved by making efficient use of Exadata Smart Flash Cache and Exadata Hybrid Columnar Compression. Verify that the cell provides the expected throughput by running the CALIBRATE command.

It is also important to consider failures when planning capacity. Exadata cell and disk failures l l d i A i S M (ASM) d d H


are transparently tolerated using Automatic Storage Management (ASM) redundancy. However, it is a best practice to ensure that post-failure I/O capacity is sufficient to meet the redundancy requirements and performance service levels.

Database Machine Migration Considerations

• Platform is Oracle Linux Release 5 x86_64– Byte order is little endian

• Exadata, ASM and database software must run release 11.2.0.1 or greater.11.2.0.1 or greater.

• ASM disk group attributes:– COMPATIBLE.ASM=11.2.0.0.0

– COMPATIBLE.RDBMS=11.2.0.0.0

– CELL.SMART_SCAN_CAPABLE=TRUE

AU SIZE 4M– AU_SIZE=4M

• Database initialization parameters:– COMPATIBLE=11.2.0.0.0

– DB_BLOCK_CHECKSUM=TYPICAL|FULL

• Database extent sizes should be a multiple of 4 MB.

Database Machine Migration ConsiderationsFollowing are the configuration requirements and recommendations for Database Machine that you should bear in mind when planning your database migration:


• Database Machine is based on Oracle Linux Release 5 x86_64. The byte order for Oracle Linux x86_64 is little endian.

• The Exadata, ASM and database software on Database Machine must be release 11.2.0.1 or later.

• For ASM disk groups containing Exadata grid disks, specify the attribute settings shown in the slide. Although these are not mandatory, they are highly recommended in order to enable all of the Exadata features and to optimize performanceenable all of the Exadata features and to optimize performance.

• To enable all the Exadata features in your database, your Oracle Database software must be release 11.2.0.1 or later, and you must set the COMPATIBLE initialization parameter as shown in the slide. Because Exadata storage is Hardware Assisted Resilient Data (HARD) compliant, you should set DB_BLOCK_CHECKSUM to TYPICAL or FULL to ensure that database checksums are computed and stored in the blocks. Note that this initialization parameter is set to TYPICAL by default.


• Exadata performs best when scanning contiguous chunks at least 4 MB in size. To ensure this occurs, the ASM disk group allocation unit size (AU_SIZE) should be set to 4 MB and the database extent sizes should be a multiple of 4 MB.

Note: The parameters in red in the slide are required to enable smart storage capabilities.

Choosing the Right Migration Path

• Determine what to migrate– Avoid methods that migrate what you will discard

• Consider the configuration of the source system– Source Oracle Database version and platform matters– Source Oracle Database version and platform matters

– Target system is fixed: 11.2, ASM, Linux x86_64

• Weigh up the costs and favor methods that facilitate best practices– Implementing best practices is important in the long term

because your future performance depends on itbecause your future performance depends on it

– Remember:— ASM 4MB AU size can only be set at disk group creation

— Large extents (8MB) are set at extent allocation

Choosing the Right Migration PathMigrating an existing database to Database Machine can be achieved using a number of different approaches. The recommended approaches are discussed later in this lesson. Before

id i th it i th hil t t l t f t th t h l i h i th


considering them, it is worthwhile to contemplate some factors that can help in choosing the right migration path.Before choosing a migration approach, you should clearly define what you want to migrate. This will help you to avoid wasted effort such as migrating data which is not required. Clearly defining the scope of the migration also helps you to clearly identify the source systems. You need to understand the source systems because their composition may limit the available migration options. For example, Database Machine is a little endian platform, so if you are migrating from a big endian platform there are a number of physical migration approaches which becomea big endian platform there are a number of physical migration approaches which become unavailable.When choosing the right migration approach, you should weigh up the short term requirement to perform the migration with the long term impact of using the selected migration approach. Specifically, you should favor methods that facilitate the implementation of best practices and avoid methods that hinder your ability the implement best practices. For example, Exadata performs best with an ASM AU size of 4 MB and database extents that are a multiple of 4 MB. If the source database extent sizes are not a multiple of 4 MB and it is


are a multiple of 4 MB. If the source database extent sizes are not a multiple of 4 MB and it is impractical to reorganize the database before migration, then you should favor a migration approach that allows you to reorganize the database during the migration. If you choose an approach that does not allow the extents to be reorganized, you may be able to deliver a quicker and easier migration however you may also end up paying an ongoing performance penalty.

Logical Migration Approaches

Migration approach Source database must be release

11.2

Can adjustASM

AU sizeto 4MB

Source platform support

Time and effort

requirement

Migration outage

time

L i l t db d t b N Y Li it d Hi h LLogical standby database Minimal down time

No Yes Limited High Low

StreamsMinimal down time

No Yes Broad High Low

Data PumpFor migration during a planned maintenance period

No Yes Broad Low High

Logical Migration Approaches

Migration to Database Machine can be done logically or physically.

Using a logical approach you can change the database extent size and alter other physical


Using a logical approach, you can change the database extent size and alter other physical characteristics of your database, which is not possible using a physical migration approach.

The table in the slide summarizes some logical migration approaches:

• Logical standby database: If your application service-level agreements permit little or no down time, you can use an Oracle Data Guard logical standby database to replicate the database on Database Machine, and track and merge the changes while the source database continues to run. After the configuration is established, the logical standby database (on Database Machine) can be switched to assume the role of primary database and the original source database can be decommissioned. The procedure documented in My Oracle Support bulletin 737460.1 can be used to change physical storage attributes of the database, such as segment extent sizes. Refer to My Oracle Support bulletin 1085687.1 for information on support for heterogeneous platform support in the same Data Guard configuration.


Logical Migration Approaches (continued)• Streams: If your application service-level agreements permit little or no down time,

you can also use Oracle Streams to propagate the data, and to track and merge the changes while the source database continues to run. For more information about this gapproach, see Appendix D in Oracle Streams Concepts and Administration 11g Release 2 (11.2). Streams provides broader source database platform support than the logical standby database approach.

• Data Pump: If a suitable maintenance window can be accommodated, use Data Pump to move the data in bulk from the legacy system to Database Machine. Data Pump is easy to use and provides broad support across different platforms and database versionsdatabase versions.

Note: Logical Standby Database and Streams both have additional considerations which may contradict their use. Firstly, both approaches are unable to natively handle all Oracle Database data types. While there are methods to overcome this limitation, the extra effort required to implement and maintain these methods should not be overlooked. Also, both approaches will fail to duplicate NOLOGGING operations that are conducted on the primary database.


Physical Migration Approaches

Migration approach Source database must be

11.2

Can adjustASM

AU sizeto 4MB

Source platform support

Time and effort

requirement

Migration outage

time

ASM Online MigrationBest when already using ASM

Yes No Linux x86_64 only

Medium LowBest when already using ASM and 4 MB AU's

y

Recovery ManagerTablespace level migration

Yes Yes Linux x86_64 only

Medium Low

Create new tablespacesIn conjunction with rolling partitions

Yes Yes Linux x86_64 only

Medium Low

Physical standby database No Yes Limited Low LowMigration of DB server and storage

Transportable DatabaseMigration to a different platform with the same endian format

Yes Yes Little endian Low Medium

Transportable TablespaceMigration to a different platform with different endian format

No Yes Broad Medium Medium

Physical Migration Approaches

The table in the slide summarizes some physical migration approaches:

• ASM Online Migration: This method is only applicable if your database is already using


ASM Online Migration: This method is only applicable if your database is already using ASM and you do not need to adjust the ASM allocation unit (AU) size. To use this method you must also be able to connect your current database storage to Database Machine and migrate the database instances to Database Machine. After migrating the database instances, the data migration is very simple, just add new Exadata-based grid disks to your ASM disk groups and drop existing disks from your ASM disk groups to migrate the data.

• Recovery Manager: This method is useful if your source database does not use ASM and• Recovery Manager: This method is useful if your source database does not use ASM and you are already using the Linux x86_64 platform for your current database server. To use this method you must be able to connect your current database storage to Database Machine and migrate the database instances to Database Machine. Then you essentially migrate the database into ASM by creating RMAN backups of the source database files and using the RMAN SWITCH...TO COPY command to perform the migration. See Chapter 8 of the Oracle Database Storage Administrator's Guide 11g Release 2 (11.2) for f rther information abo t sing RMAN for ASM migrations


further information about using RMAN for ASM migrations.

Physical Migration Approaches (continued)• Create new tablespaces: This method is useful if you are already using the Linux

x86_64 platform for your current database server and you use partition roll-in and roll-out to naturally age data in and out of the database. To use this method you must be y g yable to connect your current database storage to Database Machine and migrate the database instances to Database Machine. After migrating the database instances, you create new Exadata-based tablespaces and create your new data partitions in the new tablespaces. Over time more of your database is housed on Database Machine. To migrate the remainder of your database to Database Machine you need to employ other methods such as ASM Online Migration or Recovery Manager.

• Physical standby database: Create a physical standby database on Database• Physical standby database: Create a physical standby database on Database Machine and perform a Data Guard switchover to migrate. See My Oracle Support bulletin 413484.1 for information about heterogeneous platform support. If the source database version is earlier than 11.2 you will need to use the transient logical rolling database upgrade feature. My Oracle Support bulletin 413484.1 also contains additional information on this feature.

• Transportable Database: Use the Transportable Database feature to migrate the ti d t b T thi th d i j ti ith D t b M hi thentire database. To use this method in conjunction with Database Machine, the

source database must be on a little endian platform. See the Platform Migration Using Transportable Database white paper at http://www.otn.oracle.com/goto/maa for more information about Transportable Database.

• Transportable Tablespaces: Use the Transportable Tablespace feature to migrate tablespaces from your current system to a new database hosted on Database Machine. This is the only physical database migration method that provides broad y y gplatform support and supports migration from earlier Oracle Database versions.

Note: Database segments created using Oracle Database versions prior to 11g may not align properly on ASM extent boundaries. For data files copied from pre-11g databases, this misalignment can cause a single I/O request issued by the database to result in two physical I/Os split across an ASM extent boundary, and possibly across two physical disks. Because this behavior is most evident when large contiguous I/Os are issued, the impact is likely to be low for OLTP workloads However for workloads that contain many full tablelikely to be low for OLTP workloads. However, for workloads that contain many full table scans, the impact can be significant. To create the most efficient extent alignment, data must be loaded into new segments. This can be achieved using a logical migration approach, or using a CREATE TABLE ... AS SELECT command to re-create the segments into a new tablespace after a physical migration.


Other Approaches

• Data Integration Tools– Oracle GoldenGate

– Oracle Data Integrator

• Custom CodeCustom Code– Query over database link

– PL/SQL routines

• Hybrid Approaches– For example, Transportable Tablespaces from current

production database to staging database outside Databaseproduction database to staging database outside Database Machine, then Data Pump to unload from staging database and load into Database Machine

Other ApproachesThe previous slides introduced a number of migration approaches based on tools and technologies that are part of Oracle Database. In addition, there are many other migration


g p , y gtechnologies and approaches including:• Data integration technologies such as Oracle GoldenGate or Oracle Data Integrator can

be used to migrate, and if necessary transform data. Typically, such environments provide a high degree of flexibility along with a productive way of defining and implementing the required data integrations and transformations. Also, these technologies are often used to facilitate data migration while the source system continues to run.

• In some cases, customers have specific circumstances that result in the use of custom code for data migration. This approach typically uses queries over database links as the main data transportation mechanism. These distributed queries are sometimes encapsulated in PL/SQL routines. Custom code is often used in situations where existing routines can be easily repurposed to perform a migration to Database Machine.

• There are also many potential hybrid approaches that combine two or more of the methods introduced in this lesson. For example, a customer may already use Transportable Tablespaces in their environment so using them as a data source for aTransportable Tablespaces in their environment so using them as a data source for a migration would be easy and convenient. However, the physical organization of the data in the Transportable Tablespaces may not be ideal for Database Machine so they may choose to re-load the data in fresh segments using Data Pump.


Post-Migration Best Practices

• Check ASM disk groups are balanced:– Manual rebalance may be needed.

– Script available at My Oracle Support bulletin 367445.1

– Enterprise Manager alert for disk group imbalanceEnterprise Manager alert for disk group imbalance

• Assess index requirements:– With Exadata, full scans might deliver acceptable

performance

– Determine indexes that are not required and remove them

• Configure I/O Resource Management• Configure I/O Resource Management

Post-Migration Best Practices

Perform the following tasks after completing the migration:

• One of ASM’s functions is to ensure that data is evenly distributed across all disks in a


One of ASM s functions is to ensure that data is evenly distributed across all disks in a disk group. This happens automatically, but occasionally a disk group may become imbalanced due to uncommon errors, such as a failed rebalance. It is, therefore, an operational best practice to check disk groups on a regular basis and run a manual rebalance if necessary. A script is available to check disk group balance in My Oracle Support bulletin 367445.1. Also, Enterprise Manager Grid Control displays an alert if a disk group becomes unbalanced beyond a customizable threshold.

• The superior scan rates available from Database Machine make it possible that indexes• The superior scan rates available from Database Machine make it possible that indexes, previously required for good performance, are no longer required. You should assess execution plans that use indexes to see if they would run acceptably with Smart Scans. To determine if queries would perform acceptably without an index, you can make the index invisible to the optimizer. An invisible index is maintained by DML operations, but it is not to be used by the optimizer for queries. To make an index invisible, use the following command:


ALTER INDEX <index_name> INVISIBLE;

• After you perform the above tasks, you can configure I/O Resource Management (IORM).

Quiz

What are two important best practices when migrating to Exadata storage?

1. Double the size of your SGA.

2 Configure ASM allocation units to 4 MB2. Configure ASM allocation units to 4 MB.

3. Double the size of your disk space.

4. Configure database extents to multiples of 4 MB.

Answer: 2, 4



Quiz

Which of the following physical migration approaches is most universally applicable?

1. ASM Online Migration

2 Physical Standby Database2. Physical Standby Database

3. Transportable Tablespaces

Answer: 3

Using transportable tablespaces you can migrate to Database Machine from a much broader range of platforms than either of the other approaches.


g


Summary


• Describe the general steps to migrate your database to Database Machine

• List the main approaches for migrating your database toList the main approaches for migrating your database to Database Machine

• Identify the most appropriate approach for different circumstances




• Lesson Demonstrations (Viewlets)– Migrating to Database Machine Using Transportable

Tablespaces— http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/101DBMMigrationTTS/101dbmmigrationtts viewlet swf htmlTTS/101dbmmigrationtts_viewlet_swf.html

• My Oracle Support Notes– Changing Storage Definition in a Logical Standby Database

— https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=737460.1

– Data Guard Support for Heterogeneous Primary and Standby Systems in Same Data Guard Configuration

htt // t l /CSP/ i / ti l ? d h &t NOT— https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=413484.1

– Script to Report the Percentage of Imbalance in all Mounted Diskgroups

— https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=367445.1



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/101DBMMigrationTTS/101dbmmigrationtts_viewlet_swf.html

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=737460.1




• Documentation and White Papers– Oracle Streams Concepts and Administration

11g Release 2 (11.2)— http://www.oracle.com/pls/db112/to_toc?pathname=server.112/

e10704/toc.htm

– Oracle Database Storage Administrator's Guide 11g Release 2 (11.2)

— http://www.oracle.com/pls/db112/to_toc?pathname=server.112/e10500/toc.htm

Platform Migration Using Transportable Database– Platform Migration Using Transportable Database— http://www.oracle.com/technology/deploy/availability/pdf/MAA_

WP_10gR2_PlatformMigrationTDB.pdf



http://www.oracle.com/pls/db112/to_toc?pathname=server.112/e10704/toc.htm

http://www.oracle.com/pls/db112/to_toc?pathname=server.112/e10500/toc.htm

http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_PlatformMigrationTDB.pdf

Practice 10 Overview: Migrating to Databases Machine usingMigrating to Databases Machine using

Transportable Tablespaces

In this practice you will use Oracle Recovery Manager (RMAN), in conjunction with the transportable tablespace feature of Oracle Database, to migrate data from a big endian platform to , g g pDatabase Machine; a little endian platform.



B lk D t L di ith D t b M hiBulk Data Loading with Database Machine


Objectives


• Describe the best practices for bulk data loading into Database Machine

• Describe the process to configure the Database FileDescribe the process to configure the Database File System (DBFS) feature for staging input data files

• Use external tables based on input data files stored in DBFS to perform high-performance data loads



Bulk Data Loading Overview

Database MachineSource System

Staging Database

Target Database

DBFS

Network FileTransfer

CREATE ... AS SELECT ... INSERT ... SELECT ...

Data File

Transfer

(FTP, NFS, etc.) Data file staged inDatabase File System

(DBFS)

Target tables

External tablebased on

DBFS stageddata

Bulk Data Loading Overview

This lesson provides information on how to load your Database Machine databases with data from other systems using data files as the intermediate data store The lesson focuses on the


from other systems using data files as the intermediate data store. The lesson focuses on the recommended approach for loading file-based data into Oracle on Database Machine. This approach is illustrated on the slide. The remainder of this lesson describes this approach in greater detail.


Preparing the Data Files

• Data files can be in any format supported by the external table feature

• To facilitate high-performance parallel access:– Oracle automatically divides the files into 10MB granulesOracle automatically divides the files into 10MB granules

— Exceptions: compressed files, data read from a pipe or a tape

– If granules cannot be created then:— Each file is treated as a granule

— The number of files determines the maximum degree of parallelism

— You must use multiple files to manually enable parallelism

– General rules of thumb— If using multiple files, then try to keep them similar in size

— If the file sizes vary significantly, then list them in order from largest to smallest in the external table definition

Preparing the Data Files

The recommended approach for bulk data loading into Database Machine relies on the external table feature of Oracle Database The data files used for bulk data loading can be in


external table feature of Oracle Database. The data files used for bulk data loading can be in any format supported by external tables. The process for creating the data files is outside the scope of this lesson and mostly depends on the facilities available in the source system.

However the data files are created, the following should be taken into account in order to facilitate high-performance parallel access to the data files while they are being loaded:

• When accessing large data files through external tables, where possible Oracle automatically divides the files into 10 MB granules. These granules can be processed in separate parallel processing threads. Oracle is unable to use this approach with compressed files or data read from a pipe or a tape device.

• If granules cannot be used then each separate data file can be treated as a granule and the number of files determines the maximum degree of parallelism that is available. You can manually divide a large file into separate smaller files and use them to manually enable parallelism.

• If you are using multiple input data files in conjunction with a single external table then• If you are using multiple input data files in conjunction with a single external table, then you should try to keep the data files similar in size. If the file sizes do vary significantly, then list them in order from largest to smallest in the external table definition.


Staging the Data Files

• Data files should always be staged in DBFS– Not database server internal drives

• DBFS enables the database to be used as a file system– Shared storage for staging or storing data files scripts– Shared storage for staging or storing data files, scripts,

reports and other application files

– Files are stored as SecureFiles LOBs inside database tables that are stored in Exadata

– Files are exposed to the database servers using a POSIX-compatible file system interface

– Files are protected like any Oracle data – ASM mirroring, Data Guard, Flashback, and so on

– 5 to 7 GB/sec file system I/O throughput possible on a Full Rack Database Machine

Staging the Data Files

It is recommended that you stage your data files inside Database Machine using Database File System (DBFS). DBFS is an Oracle Database feature that enables the database to be used as


y ( )a high-performance POSIX-compatible file system on Linux. Using the available space on internal database server disk drives for staging data files is highly discouraged.

Inside DBFS files are stored as SecureFiles LOBs. A set of PL/SQL procedures implement the file system access primitives, such as open, close, create, and so on. The dbfs_client utility enables the mounting of a DBFS file system as a mount point on Linux. It also provides the mapping from file system operations to database operations. The dbfs_client utility runs completely in user space and interacts with the kernel through the FUSE library infrastructurecompletely in user space and interacts with the kernel through the FUSE library infrastructure.

Note: ASM Cluster File System (ACFS) is not supported in conjunction with Exadata.


Configuring the Staging Area

DBFS should be configured in a separate staging database

• Use DBCA to create a database based on the OLTP template– Redo logs at least 8 GBRedo logs at least 8 GB

– 4 GB buffer cache

– 1 GB shared pool

– 8 KB or 16 KB block size

• Create a bigfile tablespace for DBFS storage

C t DBFS t• Create a DBFS user accountSQL> create user dbfs identified by dbfs quota unlimited on <DBFS Tablespace>;

SQL> grant create session, create table, create procedure, dbfs_role to dbfs;


While DBFS is fully functional if it is co-located with your target database, it is recommended to configure DBFS in a separate staging database Use the Database Configuration Assistant


to configure DBFS in a separate staging database. Use the Database Configuration Assistant (DBCA) to create a database based on the provided OLTP template. If you wish to access DBFS across multiple Database Machine database servers create the staging database as a RAC database across the desired servers. Default settings are mostly suitable when configuring the staging database however bear in mind the following:

• Redo logs should be at least 8 GB in size

• A 4 GB buffer cache is generally adequate

• A 1 GB shared pool is generally adequate

• An 8 KB or 16 KB block size is suitable

After the staging database is created, create a bigfile tablespace to use as the DBFS store. Also, create a DBFS user account as suggested on the slide.



• Additional database server operating system configuration– Add the Oracle software owner, or user that will mount the

DBFS file system, to the fuse group# usermod –a –G fuse oracle

– As root, create /etc/fuse.conf containing the entry: user_allow_other

# echo "user_allow_other" > /etc/fuse.conf# chmod 644 /etc/fuse.conf

– Create a mount point for DBFS with ownership and group i i t t th O l ft th tpermissions set to the Oracle software owner, or user that

will mount the DBFS file system# mkdir /data# chown oracle:dba /data

Configuring the Staging Area (continued)

To configure DBFS you must also perform the operating system configuration tasks listed on the slide They should be performed on each database server where DBFS will be mounted


the slide. They should be performed on each database server where DBFS will be mounted. The examples show a typical configuration using the Oracle software owner as the DBFS mount point owner. You can create and use an alternative operating system user if you wish to separate DBFS access away from the Oracle software owner and it’s database administration privileges. You can also use the dcli utility to replicate these configuration steps to multiple database servers.



• Creating the DBFS store$ cd $ORACLE_HOME/rdbms/admin$ sqlplus dbfs/dbfsSQL> @dbfs_create_filesystem_advanced.sql <TS Name> <FS Name> nocompress nodeduplicate noencrypt<FS Name> nocompress nodeduplicate noencrypt non-partition

• Mounting DBFS$ nohup $ORACLE_HOME/bin/dbfs_client dbfs@<StagingDB> -o allow_other,direct_io /data < passwd.txt &

• Using DBFS• Using DBFS– Access DBFS through <DBFS Mount Point>/<FS Name>

— For example: /data/mydbfs

– Copy data files to DBFS using network file transfer methods such as FTP and NFS

Configuring the Staging Area (continued)

After the staging database is created and the required operating system configuration is completed you can create the DBFS store Use the script located at


completed, you can create the DBFS store. Use the script located at $ORACLE_HOME/rdbms/admin/dbfs_create_filesystem_advanced.sql. The script must be run by the DBFS database user (created earlier in the configuration process). The script accepts numerous parameters. In the example on the slide, <TS Name> represents the name of the tablespace you created to house the DBFS store, and <FS Name> represents the name of the DBFS store, such as mydbfs for example. This name is used later after DBFS is mounted to name the directory that appears under the DBFS mount directorydirectory.

To mount DBFS you can use dbfs_client as suggested in the example on the slide. Note that the example references /data as the DBFS mount directory. Also, note that the DBFS user password is contained in a file called passwd.txt. The password file is only required when mounting DBFS and is not required while DBFS is being used. Alternatively you can configure an Oracle Wallet so that the DBFS client can mount a DBFS store without a password. You can also configure DBFS to automatically mount using fstab.


Configuring the Staging Area (continued)After DBFS is mounted you can access it through the mount directory (/data in the slide example). In the mount directory you will find a subdirectory named after the DBFS store (<FS Name> in the slide); the contents of this subdirectory is the contents of the DBFS store.Name> in the slide); the contents of this subdirectory is the contents of the DBFS store.

Note that it is possible to have multiple copies of dbfs_client accessing the same shared file system. The sharing and caching semantics are similar to NFS. Like NFS, the default mode caches writes on the client and flushes them after a timeout or when the file being modified is closed. Also like NFS, writes to a file are only visible to clients that open the file after the writer closed the file. This behavior is commonly referred to as close-to-open cache consistency.

For more information regarding DBFS on Database Machine refer to My Oracle Support bulletin 1054431.1. See also the chapters on DBFS located in the Oracle Database SecureFiles and Large Objects Developer's Guide 11g Release 2 (11.2).


Configuring the Target Database

• Prerequisites for data file access using external tables– Create an Oracle directory object that references the DBFS

staging area directory

– Grant the required permissions on the Oracle directory objectq p y j

– Create the required external tables

• Ensure efficient space management– Use bigfile tablespaces

– Use 8MB initial extents for large segments— Set the INITIAL storage parameter to 8MBSet the INITIAL storage parameter to 8MB

— Set the CELL_PARTITION_LARGE_EXTENTS initialization parameter to TRUE or ALWAYS

• Use unlimited quotas to bypass quota management

• Use the parallel clause to set the default degree of parallelism for the target tables

Configuring the Target Database

After the DBFS-based staging area is configured it can be populated with the required data files To access the data files using external tables you must:


parallelism for the target tables

files. To access the data files using external tables you must:

• Create an Oracle directory object that references the DBFS staging area directory.

• Grant the required permissions on the Oracle directory object so that the data files can be referenced by the required target database users.

• Create the required external tables referencing the data files stored in DBFS.

You should also prepare your target database more generally to optimize the load process. You should ensure efficient space management by following the general space managementYou should ensure efficient space management by following the general space management recommendations for use in conjunction with Exadata that are listed on the slide. In addition, you can optimize the performance of your loads if you bypass tablespace quota management and you set an appropriate default degree of parallelism for your target tables.


Loading the Target Database

• Recommended approach uses external tables– Parallel direct path loading for high-performance

– In-flight processing using SQL— Transformations using SQL functionsg Q

— Sort data while loading to optimize Exadata storage indexes

– No need to re-stage the data

– Other advanced features— For example: Input file preprocessing

• SQL*Loader can also be usedSQL Loader can also be used– Parallel direct path loading can also be achieved

– Less processing flexibility compared with external tables

– Existing SQL*Loader scripts can be easily reused


The recommended method for loading the data files uses external tables. Loading data using external tables provides the following key benefits:


external tables provides the following key benefits:

• You can use direct path loading and parallel processing to achieve high-performance.

• You can use the flexibility and power of SQL to efficiently process the data while it is being loaded. For example, you can use standard or user-defined SQL functions to transform the data during loading and you can sort the data while you load it to optimize the storage indexes that are automatically maintained by Exadata.

• Because you can process the data while it is being loaded, it is unlikely that you will y p g , y yneed to re-stage the data inside the target database.

• External tables provide some useful advanced features. A primary example is the ability to preprocess a data file using a user-defined program which provides additional flexibility and power to process the data using routines outside the database.

SQL*Loader can also be used to load data files. It can deliver comparable performance but does not provide the same level of processing flexibility when compared with external tables. SQL*L d i d h i h t l d h SQL*L d b d i tSQL*Loader remains a good choice when customers already have SQL*Loader based scripts that they wish to reuse in a Database Machine environment.



• Parallel loading using external tables– A CREATE TABLE ... AS SELECT statement

automatically uses the defined degree of parallelism– An INSERT ... SELECT statement needs parallel DML to

be enabledSQL> alter session enable parallel dml;

• Direct path loading using external tables– A CREATE TABLE ... AS SELECT statement

automatically uses direct path loadingautomatically uses direct path loading– An INSERT ... SELECT statement needs an APPEND

hint to enable direct path loadingSQL> insert /*+ APPEND */ into my_table select * from my_external_table;

Loading the Target Database (continued)Bulk data loading using external tables is achieved using a CREATE TABLE ... AS SELECT statement or an INSERT SELECT statement


SELECT statement or an INSERT ... SELECT statement.

To utilize parallel processing, it is recommended that you set the default degree of parallelism for the external tables used in the load and the target table being created or inserted into. Alternatively, you can set parallelism within the statement using a PARALLEL hint.

After parallelism is configured, a CREATE TABLE ... AS SELECT statement will automatically use the defined degree of parallelism. An INSERT ... SELECT statement requires parallel DML to be enabled before it can execute in parallel.q p p

To utilize direct path lading from external tables, you must use an APPEND hint for an INSERT ... SELECT statement. CREATE TABLE ... AS SELECT statements automatically use direct path loading.

Remember that the bulk data loading approach outlined in this lesson can be used in conjunction with other techniques such as partition exchange loading for example.


Quiz

To facilitate parallel loading from typical flat files your external table definitions must reference numerous smaller data files rather than one file containing all the data:

1. True

2. False

Answer: 2

Oracle automatically divides typical flat files into 10MB granules to facilitate parallel processing Oracle is unable to use this approach with compressed files or data read from a


processing. Oracle is unable to use this approach with compressed files or data read from a pipe or a tape device.


Quiz

While DBFS is fully functional if it is co-located with your target database, it is recommended to you should configure DBFS in a separate staging database:

1. True

2. False

Answer: 1



Summary


• Describe the best practices for bulk-data loading into Database Machine

• Describe the process to configure the Database FileDescribe the process to configure the Database File System (DBFS) feature for staging flat files

• Use external tables based on flat files stored in DBFS to perform high-performance data loads




• Lesson Demonstrations (Viewlets)– Bulk Data Loading with Database Machine

— http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/111DBMBulkDataLoad/111dbmbulkdataload_viewlet_swf.html



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/111DBMBulkDataLoad/111dbmbulkdataload_viewlet_swf.html

Practice 11 Overview: Bulk Data Loading with Database MachineBulk Data Loading with Database Machine

In this practice you will perform a bulk data load on Database Machine. You will configure a database file system (DBFS) and use it to stage a CSV formatted file. You will then use the external table feature of Oracle Database to reference the CSV file. Finally, you will use a CREATE TABLE AS SELECT statement to copy the CSV file data into a table in your database.



B k d R ith D t b M hiBackup and Recovery with Database Machine


Objectives


• Describe how RMAN backups are optimized using Exadata

• Describe the recommended approaches for disk-basedDescribe the recommended approaches for disk based and tape-based backups of databases on Database Machine.

• Describe the recommended best practices for backup and recovery on Database Machine.



Backup and Recovery Overview

• Backup and recovery of databases on Database Machine– Use RMAN

– Typical strategies:— Disk-based backupsp

— Tape-based backups

— Hybrid strategy

• Backup and recovery of Database Machine software– Database server software

– Exadata storage server softwareExadata storage server software

Backup and Recovery Overview

This lesson focuses on the best practice recommendations for backup and recovery in conjunction with Database Machine The slide lists the topics that are considered throughout


conjunction with Database Machine. The slide lists the topics that are considered throughout the lesson.


Using RMAN with Database Machine

• Using RMAN on Database Machine is essentially the same as using RMAN elsewhere– Same concepts

– Same commands

• Incremental backup performance is improved– Block filtering is offloaded to Exadata

– Fewer blocks need to be processed by RMAN

– Offload processing is automatic and transparent

Using RMAN with Database Machine

Oracle recommends the use of RMAN for database backup and recovery in conjunction with Database Machine In essence using RMAN with Database Machine is the same as using


Database Machine. In essence, using RMAN with Database Machine is the same as using RMAN without Database Machine.

To optimize the performance of incremental backups, the database can offload block filtering to Exadata. This optimization is only possible when taking backups using RMAN. The offload processing is done transparently without user intervention. During offload processing, Exadata filters out the blocks that are not required for the incremental backup in progress. Therefore, only the blocks that are required for the backup are sent to the database. This can i ifi tl i b k f i ll if th ti f h d bl k isignificantly improve backup performance, especially if the proportion of changed blocks is

low.


General Recommendations for RMAN

Use RMAN to backup and recover databases on Database Machine

• Use RMAN incremental backups and block change trackingg

• Use an external RMAN recovery catalog repository• Set DB_RECOVERY_FILE_DEST_SIZE to bound space

used in the Fast Recovery Area

General Recommendations for RMAN

The following recommendations apply when using RMAN in conjunction with Database Machine:


Machine:

• Use RMAN incremental backups and block change tracking. For fast incremental backups, enable block change tracking. Block change tracking allows RMAN to avoid scanning blocks that have not changed, when creating incremental backups. Also, when performing incremental backups of databases on Exadata storage, additional block inspection is offloaded from the database servers. Block change tracking provides the greatest benefit for databases where fewer than 20% of the blocks are changed b t i t l b k Y till b fit b i bl k h t ki ithbetween incremental backups. You may still benefit by using block change tracking with change rates greater than 20%, but testing is recommended to ensure that backup times are reduced.

• Use an external RMAN recovery catalog repository. The RMAN recovery catalog should be hosted on a server outside Database Machine. In practice, it is common to have a server that hosts the RMAN catalog along with other management repositories such as the Oracle Enterprise Manager repository and the Oracle Secure Backup catalog.


General Recommendations for RMAN (continued)• Set DB_RECOVERY_FILE_DEST_SIZE to bound space used in the Fast Recovery

Area. The database writes archived redo log files and any additional recovery files to the Fast Recovery Area. These include any disk backup files such as level 0 image copiesFast Recovery Area. These include any disk backup files such as level 0 image copies and level 1 backup sets as well as Flashback log files (if enabled). It is important that you set the value of this parameter to less than the total free space in the disk group, taking into account at least one disk failure, and preferably one Exadata cell failure. Additionally, if multiple databases are sharing the Fast Recovery Area, ensure that the sum of the space allocated to the different databases is less than the free space in the disk group.


Disk Based Backup Strategy

For disk-based database backups, Oracle recommends:

• Use a Fast Recovery Area

• Perform an initial level 0 (full) backup

• Perform periodic incremental level 1 backups• Perform periodic incremental level 1 backups

• Update your level 0 backup by applying the second to last level 1 backup

Disk Based Backup Strategy

For disk-based database backups, Oracle recommends incremental backups to a Fast Recovery Area The recommended strategy is outlined in the slide Using this approach you


Recovery Area. The recommended strategy is outlined in the slide. Using this approach, you can achieve a good compromise between the time and effort required during backup and recovery operations. The approach also efficiently manages the amount of storage required for backups.


Disk Based Backup Configuration

• Fast Recovery Area (FRA) configuration:– Default: FRA disk group striped across all available Exadata

storage along with data disk groups

– Alternative: FRA disk group and data disk groups on g p g pseparate Exadata storage servers

• Additional RMAN recommendations:– Instances and channels:

— Initially, run RMAN using one database instance and two RMAN channels

— Add another instance with two channels if required

— Add an additional two channels per instance if required

– Configure an Oracle Service to use as the RMAN target

Disk Based Backup Configuration

By default, the installation and configuration procedure for Database Machine yields a disk group for the Fast Recovery Area (FRA) which is spread across all of the disks in all of the


group for the Fast Recovery Area (FRA) which is spread across all of the disks in all of the Exadata storage servers. In this configuration, the FRA shares each disk with disk groups containing database data files. By default, all the disk groups are created with NORMAL ASM redundancy (2-way mirroring). The result is that Database Machine can tolerate the loss of an entire Exadata storage server and still maintain full data integrity.

Where customers desire a separation between database files and the FRA, the recommended alternative is to configure the FRA disk group on a dedicated set of Exadata t d t fi th i d d t di k t tstorage servers and to configure the required data disk groups on separate storage servers.

This configuration means that the I/O for a single workload can no longer benefit from being striped across all the storage servers, however it also provides additional protection if multiple simultaneous failures affect either the database or FRA. Multiple simultaneous failures to both the database and FRA can still result in data loss.

Another strategy is to purchase additional high capacity Exadata servers specifically to store the FRA. This allows your databases to leverage the full capacity of the Database Machine y g p ystorage servers. To implement this strategy, the FRA storage servers need to be hosted in a separate rack and they need to be connected to the Database Machine storage network using the spare ports on the Database Machine InfiniBand switches.


Disk Based Backup Configuration (continued)

In addition to the general recommendations for RMAN in conjunction with Database Machine, Oracle recommends the following configuration for disk based backups:

• Testing shows that optimal backup rates are achieved with 2 database instances and 2 to• Testing shows that optimal backup rates are achieved with 2 database instances and 2 to 4 RMAN channels per instance. For the 2 database instances designated as backup servers, less than 10% CPU and less than 40% IO bandwidth are utilized. During backup operations, I/O intensive parallel queries should avoid these designated backup servers. Start by running RMAN on one database instance using two RMAN channels. If greater backup throughput is required, use a second instance with two more channels. Finally, if required add two more channels per instance.

• Configure an Oracle Service to run against specific database instances in the cluster. The service is used by RMAN to automatically spread the backup load evenly among the target instances offering the service.


Tape Based Backup Strategy

For tape based database backups, Oracle recommends:

• Use media management software that is integrated with RMAN, like Oracle Secure Backup

• Perform periodic level 0 (full) database backupsPerform periodic level 0 (full) database backups

• Perform more frequent cumulative level 1 backups and also backup the Oracle Secure Backup catalog

Tape Based Backup Strategy

For tape based database backups, Oracle recommends the use of Oracle Secure Backup or other media management software that is integrated with RMAN The recommended backup


other media management software that is integrated with RMAN. The recommended backup strategy is to perform periodic (weekly) level 0 (full) database backups. In addition, more frequent (daily) cumulative level 1 backups should be taken along with a backup of the Oracle Secure Backup catalog if it is the media management software which is used.


Tape Based Backup Configuration

Database Machine Sun StorageTekSL500

Oracle Secure BackupAdministration Server

(Sun Fire X4170)

Oracle Secure BackupMedia Servers

(Sun Fire X4275)

Fiber Channel SANInfiniBandNetworkNetwork


The slide illustrates the recommended reference architecture for a tape based backup configuration using Oracle Secure Backup The key recommendations associated with the


configuration using Oracle Secure Backup. The key recommendations associated with the configuration follow:

• Connect the media servers to Database Machine using the high-performance InfiniBand network. The InfiniBand connection to each media server should be bonded for high availability. Alternatively, you can use a Gigabit Ethernet network between the media servers and Database Machine, however if you use this configuration be aware that the network will likely be the constraining factor for backup and recovery performance.

• Typically, backup performance is limited by tape drive throughput. Backup performance scales when you add more tape drives and RMAN channels. Allocate a sufficient number of tape drives so the media servers can achieve their maximum backup and restore rates. Add tape drives until the bandwidth of the media servers is saturated.

• Start with at least two media servers. Add media servers if you have enough tape drives to keep them busy without saturating Database Machine resources.



• Media server InfiniBand configuration recommendations:– Configure bonding of the media server InfiniBand interfaces

– Update OpenFabrics Enterprise Distribution on the media server

– Configure IP over InfiniBand connected mode for best performance

– Set the message transfer unit (MTU) size to 65520 for the InfiniBand interface

– Configure the media management software to use the InfiniBand network

• Media server SAN configuration recommendation:– Configure persistent bindings for tape devices

Tape Based Backup Configuration (continued)

Media servers can be directly connected to the Database Machine InfiniBand fabric by adding an InfiniBand Quad Data Rate (QDR) host channel adapter (HCA) to the media server For


an InfiniBand Quad Data Rate (QDR) host channel adapter (HCA) to the media server. For high availability, connect the HCA to two different Database Machine InfiniBand leaf switches to eliminate the switch as a single point of failure. This provides transparent failover if connectivity is lost to one of the ports. Follow these recommendations:

• Configure bonding of the media server InfiniBand interfaces. Active-passive bonding is recommended, which is consistent with the database servers and Exadata servers in Database Machine.

• You must use an OpenFabrics Enterprise Distribution (OFED) version on the media server that is compatible with the version found in Database Machine. For details, refer to My Oracle Support bulletin 888828.1.

• Configure IP over InfiniBand connected mode for best performance. On Linux, edit the /etc/ofed/openib.conf file so that it contains the entry SET_IPOIB_CM=yes. Reboot the server to enable the setting. To verify the setting, check the contents of /sys/class/net/ib0/mode and /sys/class/net/ib1/mode. Both files should / y / / / / / y / / / /contain the entry connected.


Tape Based Backup Configuration (continued)• Set the message transfer unit (MTU) size to 65520 for the InfiniBand interface. On Linux,

edit the /etc/sysconfig/network-scripts/ifcfg-ib* and /etc/sysconfig/network-scripts/ifcfg-bond0 files so that they contain the / / y g/ p / g yentry MTU=65520. Verify the MTU setting by examining the output of the ifconfigcommand for the InfiniBand interfaces.

• To direct the backup and restore traffic over the InfiniBand network, configure the media management software to favor InfiniBand. Note that each media management software type has its own method of enabling this configuration. For example, Oracle Secure Backup has the concept of a preferred network interface, which can be set on the media server for a specific list of clients Other media management software may require thisserver for a specific list of clients. Other media management software may require this configuration to be defined when the software is installed. See the media management software for information about how to direct traffic over a particular network.

For SAN attached tape devices you must configure persistent bindings so the device address does not change. If the device address changes, the media servers cannot access the device unless you update the device configuration within the media server software. Therefore, it is very important that your environment maintains consistent device addresses.

Persistent bindings are a part of your SAN infrastructure setup. Typically you will configure persistent bindings through the HBA or the operating system. The configuration steps may vary by platform and vendor.



• Run RMAN across all the available database instances– Create a Database Service that runs across the cluster

$ srvctl add service –d <dbname> -s <service name> -r <instance1>, ... ,<instancen>

– Use the service name and SCAN address to connect RMAN$ rman target sys/<passwd>@<scan_address>/<service_name> catalog …

• Allocate one RMAN channel per tape drive

• Configure IORM and DBRM to control resource allocationConfigure IORM and DBRM to control resource allocation between backups and application workloads

Tape Based Backup Configuration (continued)

Further recommendations:

For your RAC databases configure RMAN to run across all the database instances It is


• For your RAC databases, configure RMAN to run across all the database instances. It is recommended that you create a Database Service to run across the RAC cluster. Then when running RMAN, use the service name and SCAN address in the connect string for the RMAN target as shown in the slide.

• Allocate one RMAN channel per tape drive. A single RMAN channel in Database Machine can stream data at a rate of 749 MB/sec from Database Machine to the media server. Typical tape drive backup rates are between 100 MB/sec and 240 MB/sec, depending on the drive type and compression options. Note that tape drive compression becomes less effective when backing up tables that are compressed at the database level.

• If Database Machine resources must be prioritized between application workloads and backups, then configure I/O Resource Manager (IORM) and Database Resource Manager (DBRM). This is more likely to be required in cases where time-consuming backups to large databases must run at the same time as production application bac ups to a ge databases ust u at t e sa e t e as p oduct o app cat oworkloads.


Hybrid Backup Strategy

• A hybrid backup strategy combines the disk-based and tape-based backup approaches:– Level 0 (full) database backups are stored on tape

— Relatively cheap backup media that can be stored off sitey p p

– Level 1 incremental backups are stored on disk in the FRA— Readily available with high performance access

• Follow recommendations for both approaches

Hybrid Backup Strategy

A hybrid backup strategy combines the disk-based and tape-based backup approaches. For example a reasonable hybrid approach would result in level 0 (full) database backups being


example, a reasonable hybrid approach would result in level 0 (full) database backups being stored on tape, while the level 1 incremental backups would be stored on disk in the FRA. The hybrid approach also combines the benefits of each storage type as indicated on the slide. If you choose to implement a hybrid backup approach you should follow the recommendations for disk-based backups and tape-based backups.


Restore and Recovery Recommendations

• Restore into existing data files if possible– Restore performance is better

– Restore using all database instances

• If no existing data files are present restore using up to 2If no existing data files are present, restore using up to 2 database instances

• Recommended number of RMAN channels:– For disk-based restoration, use 2 to 4 RMAN channels per

database instance

– For tape-based restoration set the number of RMAN– For tape-based restoration, set the number of RMAN channels based on the total number of tape drives

Restore and Recovery Recommendations

Higher restore rates are achieved when avoiding the overhead of initial data file allocation. If existing files are present prior to database restoration do not delete the data files and perform


existing files are present prior to database restoration, do not delete the data files and perform database restoration into the existing files to take advantage of this optimization.

With pre-existing files, create a restore service across all database instances. If no existing data files are present prior to the restore operation, create a restore service with only two database instances. Use 2 to 4 RMAN channels per database instance for disk-based restoration. For tape-based restoration, the number of channels should be set to the number of tape drives.


Backup and Recovery of Database Machine SoftwareSoftware

• Database Server software– Perform file system level backup and recovery

— Use your chosen file system backup management software or Oracle Secure Backup can be used

— Copies of the Oracle Cluster Registry are automatically maintained on the Database Server file system and should be included in Database Server file system backups

• Exadata software– File system level backups are not recommended

System areas are mirrored— System areas are mirrored— Use CellCLI commands to recover if one system disk fails

— Use the Exadata Software Rescue Procedure if both system disks fail simultaneously

— The rescue procedure uses a built-in USB flash drive

Backup and Recovery of Database Machine Software

It is recommended that you use file system level backup and recovery techniques for the database servers Use your chosen file system backup management software and


database servers. Use your chosen file system backup management software and infrastructure or use Oracle Secure Backup if desired. Note that a backup of the Oracle Cluster Registry, which also contains Voting Disk information, is automatically maintained on the file system of the first database server and should be included in your database server file system backups. The default location for the Oracle Cluster Registry backup is <Grid_Home>/cdata/<Cluster_Name>, where:

• <Grid_Home> is the location of the Grid Infrastructure software as specified during the initial configuration of Database Machine The default G id H location isinitial configuration of Database Machine. The default <Grid_Home> location is /u01/app/11.2.0/grid.

• <Cluster_Name> is the name of your cluster. This is the same as the DB Machine Prefix specified during the initial configuration of Database Machine.

File system level backups are not recommended for Exadata software. Exadata maintains mirrored system areas on separate physical disks. In the rare event that both system disks fail simultaneously, you must use the rescue functionality provided on the CELLBOOT USB flashsimultaneously, you must use the rescue functionality provided on the CELLBOOT USB flash drive that is built into every Exadata server.


Quiz

To facilitate disk-based backups you can install additional high capacity Exadata servers into the available space in a Half Rack Database Machine based on high performance disks:

1. True

2. False

Answer: 2

Adding Exadata servers into the same rack as a Half Rack Database Machine is only supported using the Half Rack to Full Rack upgrade package and the resulting Full Rack


supported using the Half Rack to Full Rack upgrade package, and the resulting Full Rack Database Machine can only utilize one disk type; high performance or high capacity. The only supported way to add high capacity cells to a Half Rack Database Machine using high performance cells is to install the high capacity cells into a separate rack and connect them to the Database Machine storage network using the spare ports on the Database Machine InfiniBand switches.


Quiz

How many RMAN channels should you use for tape-based backups?

1. 2 per database instance

2 4 per database instance2. 4 per database instance

3. 1 per tape drive

Answer: 3



Summary


• Describe how RMAN backups are optimized using Exadata

• Describe the recommended approaches for disk-basedDescribe the recommended approaches for disk based and tape-based backups of databases on Database Machine.

• Describe the recommended best practices for backup and recovery on Database Machine.




• Lesson Demonstrations (Viewlets)– Backup Optimization Using RMAN and Exadata

— http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/121DBMBackupRMAN/121dbmbackuprman_viewlet_swf.html

– Recovery Optimization Using RMAN and Exadata— http://st-

curriculum.oracle.com/demos/db/11g/r2/dbmach/122DBMRecoverRMAN/122dbmrecoverrman_viewlet_swf.html



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/121DBMBackupRMAN/121dbmbackuprman_viewlet_swf.html

http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/122DBMRecoverRMAN/122dbmrecoverrman_viewlet_swf.html

Practice 12 Overview: Using RMAN Optimizations for Database MachineUsing RMAN Optimizations for Database Machine

In these practices, you will examine the backup and recovery optimizations that are enabled when Oracle Recovery Manager (RMAN) is used in conjunction with Exadata storage.



M it i d M i t i i D t b M hiMonitoring and Maintaining Database Machine


Objectives


• Describe the monitoring tools provided with Database Machine

• Describe the support update and patching considerationsDescribe the support, update and patching considerations that apply to Database Machine



Monitoring Tools Overview

• Most Database Machine monitoring tools are not specific to Database Machine; for example:– Enterprise Manager

– Operating system tools and utilities– Dynamic performance views (V$ views) and wait events

– Alerts logs and trace files

• Exadata-specific monitoring capabilities– Exadata metrics, alerts and active requests– Exadata-specific V$ view entries and wait events

Enterprise Manager Plug In for Exadata– Enterprise Manager Plug-In for Exadata

• Additional monitoring capabilities for Database Machine– Integrated Lights Out Manager (ILOM)– The dcli utility

– InfiniBand diagnostic utilities

Monitoring Tools Overview

Monitoring Database Machine is largely the same as monitoring Oracle RAC on Linux. The biggest difference is the use of Exadata storage and the use of InfiniBand for the storage


biggest difference is the use of Exadata storage and the use of InfiniBand for the storage network and cluster interconnect. Most of the tools and utilities available for Database Machine are the same tools and utilities used by Oracle administrators and system administrators apart from Database Machine.

As a Database Machine administrator, Oracle recommends that you use Enterprise Manager in conjunction with other lower level monitoring capabilities such as the dynamic performance views, wait events, alert logs and trace files available in both ASM and Oracle Database. In

dditi ti t t l d tiliti h iaddition, you can use operating system tools and utilities such as vmstat, iostat, top, syslog, and so on.

The slide also lists a set of Exadata-specific monitoring capabilities that have already been discussed in the lesson Exadata Performance Monitoring and Maintenance.

Finally, the slide lists three additional monitoring capabilities which are likely to be unfamiliar to Oracle administrators outside of Database Machine. The following slides introduce the Integrated Lights Out Manager (ILOM) the dcli utility and the InfiniBand diagnostic utilitiesIntegrated Lights Out Manager (ILOM), the dcli utility, and the InfiniBand diagnostic utilities.


ILOM Overview

ILOM Capability What You Can Do

Dedicated service processor and resources

• Manage the server without consuming system resources • Continue to manage the server using standby power even when the server is powered-off

• Monitor system status and event logs • Monitor power supplies fans disks CPUs memory and motherboard

Remote hardware monitoringMonitor power supplies, fans, disks, CPUs, memory, and motherboard

• Monitor component temperatures • Monitor sensors, including voltage and power, and indicators (LEDs)

Hardware inventory and presence

• Identify part numbers, versions, and product serial numbers • Identify NIC card MAC addresses

Remote Access• Redirect the system serial console• Access keyboard, video, and mouse (KVM) on remote systems

System power control and • Power the system on or off, either locally or remotely monitoring • Force power-off for emergency shutdown or perform a graceful shutdown

Error and fault management• Monitor system BIOS, POST, and sensor messages • Monitor hardware and system-related errors, as well as ECC memory errors, reported into SP logs, syslog, and remote log-host

System alerts, including SNMP traps, IPMI PETs, remote syslog, and email alerts

• Monitor components using industry-standard SNMP commands and the IPMItool utility

ILOM Overview

Integrated Lights Out Manager (ILOM) provides advanced service processor hardware and software that you can use to manage and monitor Database Machine servers ILOM’s


software that you can use to manage and monitor Database Machine servers. ILOM s dedicated hardware and software is preinstalled on the database servers and Exadata servers inside Database Machine.

ILOM enables you to actively manage and monitor the server independently of the operating system state. With ILOM, you can proactively:

• Learn about hardware errors and faults as they occur

• Remotely control the power state of your servere ote y co t o t e po e state o you se e

• View the graphical and non-graphical consoles for the host

• View the current status of sensors and indicators on the system

• Determine the hardware configuration of your system

• Receive generated alerts about system events in advance through IPMI PETs, SNMP traps, or email alerts.

The slide shows a broader list of ILOM capabilitiesThe slide shows a broader list of ILOM capabilities.


ILOM Overview (continued)

The ILOM service processor (SP) runs its own embedded operating system and has a dedicated Ethernet port, which together provide an out-of-band management capability. In addition you can access ILOM from the server’s host operating system Using ILOM you canaddition, you can access ILOM from the server s host operating system. Using ILOM, you can remotely manage your server as if you were using a locally attached keyboard, monitor, and mouse.

ILOM automatically initializes as soon as power is applied to the server. It provides a full-featured, browser-based web interface and has an equivalent command-line interface (CLI). There is also an industry-standard SNMP interface and IPMI interface.

You can easily integrate these management interfaces with other management tools and processes that you might have working already with your servers, such as Oracle Enterprise Manager Ops Center. In addition, you can integrate ILOM with many third-party management tools, such as BMC Patrol, CA Unicenter, and HP OpenView for example.


ILOM Example

$ ssh root@eidmdb01-ilom

Password: <Password>

Sun(TM) Integrated Lights Out Manager

Version 3.0.6.10.b r52264

...

-> show /SYS/T_AMB

/SYS/T_AMB

Targets:

Properties:

type = Temperature

ipmi_name = T_AMB

class = Threshold Sensor

value = 23.000 degree C

upper_nonrecov_threshold = 50.000 degree C

upper_critical_threshold = 45.000 degree C

...

-> reset /SYS

Are you sure you want to reset /SYS (y/n)? y

Performing hard reset on /SYS

ILOM Example

The slide shows a simple example using the command line interface to ILOM. In the example, a remote connection is established to ILOM using the dedicated network address associated


a remote connection is established to ILOM using the dedicated network address associated with ILOM on the specified server. After the connection is established you can execute ILOM commands to perform various functions. In the example, the first ILOM command shows information relating to the server’s temperature sensors, while the second command reboots the server. Note that the ILOM session continues beyond the reboot and other ILOM commands can be executed even while the server reboots.

For more information, refer to the ILOM documentation located at htt //d / /d / ll/il 3 0http://docs.sun.com/app/docs/coll/ilom3.0


DCLI Overview

• The dcli utility allows you to simultaneously execute a command on multiple Database Machine servers

• Command types:– Operating system commandsOperating system commands

– CellCLI commands

– Operating system scripts

– CellCLI scripts

• Commands are executed in separate parallel threads

I t ti i t t d• Interactive sessions are not supported

• Python 2.3 and SSH user-equivalence are required

• Command output is collected and displayed in the terminal executing the dcli utility

DCLI OverviewThe dcli utility facilitates centralized management across Database Machine by automating the execution of a command on a set of servers and returning the output to the centralized


the execution of a command on a set of servers and returning the output to the centralized management location where dcli was run. The types of commands supported by dcliincludes operating system commands, CellCLI commands, operating system scripts, and CellCLI scripts.

The dcli utility runs commands on multiple servers in parallel threads. However, it does not support an interactive session with a remote application on a server. By default, the dcliutility is located at /opt/oracle/cell/cellsrv/bin/dcli on each Exadata server and

t / /l l/bi /d li h d t b Y l th d li tilit tat /usr/local/bin/dcli on each database server. You can also copy the dcli utility to a server outside of Database Machine and centrally execute commands from that server.

The dcli utility requires Python version 2.3 or later on the server running dcli. You can determine the version of Python by running the python -V command. In addition, dclirequires prior setup of SSH user-equivalence between all the servers. You can use the dcliutility initially with the -k option to set up SSH user-equivalence between a group of servers.

Command output (to stdout and stderr) is collected and displayed after command execution isCommand output (to stdout and stderr) is collected and displayed after command execution is finished on the specified servers. The dcli options allow command output to be abbreviated to filter output, such as removing messages showing normal status.


DCLI Examples

$ dcli –g mycells date

eidmcel01: Sun May 16 20:48:09 CDT 2010

eidmcel02: Sun May 16 20:48:09 CDT 2010

$ dcli –c eidmcel01,eidmcel02 cellcli –e list cell

eidmcel01: eidmcel01 online

eidmcel02: eidmcel02 online

$ dcli –g mycells –x cellclicommands.scl

$ dcli –g mydbservers –l root –x dbwork.sh

DCLI ExamplesThe slide shows a series of examples using the dcli utiltiy.

The first example shows d li being used to execute the operating system d t command


The first example shows dcli being used to execute the operating system date command. The –g option specifies a file (mycells) which identifies the list of target servers to which the command (date) is sent. The servers can be identified by host names or IP addresses. The servers can be database servers or Exadata servers.

The second example uses the –c option to specify the target servers (eidmcel01,eidmcel02) on the command line. It invokes CellCLI to report the cell status.

The third example uses the –x option to specify a command file The command file must existThe third example uses the x option to specify a command file. The command file must exist on the server executing the dcli utility. The command file is copied to the target servers and executed. A file with the .scl extension is run by the CellCLI utility on the target server. A file with a different extension is run by the operating system shell on the target server. The file is copied to the default home directory of the user on the target server. Files specified using the –x option must have execute privileges or else dcli will report an error.

The final example adds the use of the –l option to specify the user to connect as on the t B d f lt th d li tilit t th ll d iremote servers. By default, the dcli utility connects as the celladmin user.

For more information, refer to the chapter entitled Using the dcli Utility in the Oracle Exadata Storage Server Software User's Guide 11g Release 2 (11.2).


InfiniBand Diagnostic Utilities

• verify-topology

– Verifies the presence of the expected Database Machine storage network links

– Example:p

# cd /opt/oracle.SupportTools/ibdiagtools# ./verify-topology

[ DB Machine Infiniband Cabling Topology Verification Tool ]Is every external switch connected to every internal switch..........[SUCCESS]Are any external switches connected to each other....................[SUCCESS]Are any hosts connected to spine switch..............................[SUCCESS]Check if all hosts have 2 CAs to different switches..................[SUCCESS]Leaf switch check: cardinality and even distribution [SUCCESS]Leaf switch check: cardinality and even distribution.................[SUCCESS]Check if each rack has an valid internal ring........................[SUCCESS]

• infinicheck

– Verifies connectivity across the Database Machine storage network links and checks bandwidth across the links

InfiniBand Diagnostic Utilities

On each Database Machine server there are two useful InfiniBand diagnostic utilities, verify-topology and infinicheck which are located at


verify topology and infinicheck, which are located at /opt/oracle.SupportTools/ibdiagtools. You must be the root user or a member of the root operating system group in order to run the utilities.

verify-topology conducts a series of topology-level tests to verify that the expected storage network links are cabled correctly. An example of the expected output for verify-topology is shown in the slide. You can modify the behavior and output of verify-topology through the use of optional arguments, however in most cases you will find this

E t if l h f li t f th ti l tunnecessary. Execute verify-topology –h for a list of the optional arguments.

infinicheck conducts a series of tests that verify connectivity across each Database Machine storage network link. It also stress-tests each link to check that the available bandwidth for each link is as expected. infinicheck reports output to the terminal and logs the terminal output along with additional diagnostic information to a file named diagnostics.output. Execute infinicheck –h to view a list of the optional arguments for infinicheck. The following page shows and example of the terminal output from g p g p pinfinicheck.


InfiniBand Diagnostic Utilities (continued)Here is an example of the terminal output from infinicheck.

# ./infinicheck

Verifying User Equivalance of user=root to all hosts.(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

Verifying User Equivalance of user=root to all cells.(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)keys to all the nodes)

#### CONNECTIVITY TESTS ####[COMPUTE NODES -> STORAGE CELLS]

(30 seconds approx.)[SUCCESS]..............Connectivity verified

[ ] ll h lk ll ll[SUCCESS]....... All hosts can talk to all storage cells

Verifying Subnet Masks on Hosts and Cells[SUCCESS] ......... Subnet Masks is same across the network

Checking for bad links in the fabric[SUCCESS].......... No bad fabric links found

[COMPUTE NODES -> COMPUTE NODES](30 seconds approx.)

[SUCCESS]..............Connectivity verified

[SUCCESS]....... All hosts can talk to all other nodes

#### PERFORMANCE TESTS ######## PERFORMANCE TESTS ####

[(1) Every COMPUTE NODE to its STORAGE CELL](30 seconds approx.)

[SUCCESS]........ Network Bandwidth looks OK........... To view only performance results run ./infinicheck -d -p

[(2) Every COMPUTE NODE to another COMPUTE NODE](30 seconds approx.)

[SUCCESS]........ Network Bandwidth looks OK....... To view only performance results run ./infinicheck -d -p

[(3) Every COMPUTE NODE to ALL STORAGE CELLS](90 seconds approx.) (looking for SymbolErrors)

[SUCCESS]....... No port errors found[SUCCESS]....... No port errors found#


Database Machine Support Overview

Support Offering Key Features

Oracle Hardware Warranty

• Included with Database Machine, 1 year term• Any-time Web access, Local business hours phone access• 4 hour response during normal business hours • On-site response and parts exchange within 2 business days subject to availability and locationsubject to availability and location

Oracle Premier Support

• Support for Oracle Database and Exadata software• Any-time Web or phone access• Software enhancements, fixes and upgrades• Proactive tools, including alerts and configuration guidance

Oracle Premier Support for Systems

• Support for server and storage hardware and firmware; includes Oracle Linux• Any-time Web or phone access

– See also http://www.oracle.com/support/policies.html

for Systemsy p

• On-site hardware response with 2 hours if Database Machine is within a 2 hour service coverage area

Oracle Customer Dataand Device Retention

• Provides replacements for failed disk drives• Customer retains the failed disk drives• Provides additional security for sensitive data

Database Machine Support Overview

Oracle provides a complete and integrated set of support offerings for Database Machine. Customers can use a single contact point to access all of the support services outlined on the


Customers can use a single contact point to access all of the support services outlined on the slide. There is also a single point of accountability, meaning that issues will never remain unresolved while separate support organizations wait for each other to progress.

The support services outlined in the slide are modular so that customers can choose the right level of support for their needs. Oracle Hardware Warranty is included with every Database Machine and is the minimum level of support. Typically, Database Machine is deployed for enterprise-scale business-critical applications, and in these cases Oracle recommends that

t h O l P i S t d O l P i S t f S t O lcustomers purchase Oracle Premier Support and Oracle Premier Support for Systems. Oracle Customer Data and Device Retention is recommended where security or privacy requirements compel customers to ensure that sensitive data never leaves their enterprise.

In addition to the support offerings outlined here, Oracle provides installation and configuration services for Database Machine. These services are highly recommended to ensure an efficient and trouble-free start with Database Machine. Additional services exist to help customers with upgrades from a Quarter Rack to a Half Rack, or to upgrade from a Half p pg , pgRack to a Full Rack. A specialized service is also available for customers who wish to interconnect multiple Half Rack and Full Rack Database Machines.


Patching and Updating Overview

• There are three categories of software that must be maintained in Database Machine:– Software and firmware in the Exadata servers

– Software and firmware in the database servers

– Software and firmware for other components, like the InfiniBand switches for example

• Compatibility between these different pieces of software needs to be maintained

• Patches and updates are “rolling” in nature whereverPatches and updates are rolling in nature wherever possible

• Key information is maintained in My Oracle Support bulletin 888828.1

Patching and Updating Overview

There are three broad categories of software that must be maintained in Database Machine. That is software and firmware in the Exadata servers database servers and other


That is, software and firmware in the Exadata servers, database servers and other components like the InfiniBand switches. Compatibility between these different pieces of software is vital for Database Machine to function.

Wherever possible, Database Machine patches and updates are “rolling” in nature, meaning that they are applied in a manner that facilitates ongoing system availability by “rolling” the patch across the environment one server at a time rather affecting all the servers at once.

Important information relating to Database Machine patches and updates is maintained in My p g p p yOracle Support bulletin 888828.1. This bulletin is constantly updated with new information. Oracle recommends that all Database Machine customers should sign up for an automated alert when this bulletin changes.

The remainder of this lesson outlines some additional recommendations regarding Database Machine patching and updating.


Maintaining Exadata Software

• Patches for Exadata are provided by Oracle as a patch bundle

• The patch bundle maintains consistency across all the Exadata software componentsp– Oracle Linux

– Exadata Storage Server software

– InfiniBand software

– Firmware

• Patches can generally be applied online while the• Patches can generally be applied online, while the database is up and running

• Exadata automatically maintains firmware levels

• No additional software, RPMs or otherwise, should be installed

Maintaining Exadata Software

Patches for Exadata are provided by Oracle as a patch bundle. The patch bundle maintains consistency across all the Exadata software components Do not manually update Exadata


consistency across all the Exadata software components. Do not manually update Exadata firmware or software unless directed by Oracle Support.

Exadata patches can generally be applied in a “rolling” manner, while the database continues to run. Note that the ability to perform rolling Exadata patches is not available if you use EXTERNAL redundancy disk groups which are not recommended on Database Machine. Patches may also include instructions for parallel installation on multiple Exadata servers. This approach generally requires down time but may be preferred if a scheduled maintenance

i d i t f th twindow exists for the system.

As described in the lesson Exadata Performance Monitoring and Maintenance, Exadata automatically maintains firmware within the server. Firmware levels are periodically checked while Exadata is running and the correct firmware is automatically applied to components when the server reboots. Disk firmware is also automatically updated when a disk is replaced.

Oracle recommends that no additional software, whether RPMs or otherwise, should be installed on Exadatainstalled on Exadata.


Maintaining Database Server Software

• You can patch and update the database server software as you would for an Oracle Database server outside of Database Machine– Oracle Database patches applied by OPatchp pp y

— Double-check compatibility of patches with Database Machine

— Oracle supplies Bundle Patches for Database Machine— Periodic bundle of database patches recommended for

Database Machine

– Operating system and firmware updated by Unbreakable Linux Network (ULN)Linux Network (ULN)

— Must maintain consistency with InfiniBand (OFED) software

• Check Exadata patches for database server firmware and operating system updates

Maintaining Database Server Software

In essence, you can patch and update the database server software using the same techniques and approaches as you would for an Oracle Database server outside of Database


techniques and approaches as you would for an Oracle Database server outside of Database Machine.

OPatch is used to apply and manage Oracle Database patches. It is recommended that you raise a Service Request with Oracle Support to verify the compatibility of patches with Database Machine. In addition to regular database patches and updates, Oracle supplies periodic Bundle Patches for Database Machine. These bundle a series of database patches that are recommended for use in conjunction with Database Machine. Oracle recommends th t h ld l B dl P t h t f t hi ithat you should apply Bundle Patches as part of your patching regime.

Operating system and firmware patches should be managed using standard Linux patching practices, such as using the Unbreakable Linux Network (ULN). My Oracle Support bulletin 888828.1 lists current constrains and requirements for operating system and firmware patches. The main consideration is to ensure that any patches or updates maintain consistency with the OFED software version which underpins the InfiniBand network.

Some Exadata patches also include database server firmware and operating system updatesSome Exadata patches also include database server firmware and operating system updates. This may occur, for example, when the firmware for the InfiniBand HCA is updated on Exadata and the update must also be applied to the database servers.


Maintaining Other Software

• Other components in Database Machine that have software or firmware:– InfiniBand switches: Software and firmware must be

maintained in accordance with Oracle guidelines published in My Oracle Support bulletin 888828.1

– KVM module: May be updated according to customer requirements

– Cisco Ethernet switch: May be updated according to customer requirements

Maintaining Other Software

Other Database Machine components that have software or firmware include:

InfiniBand switches: The software and firmware on the InfiniBand switches must be


• InfiniBand switches: The software and firmware on the InfiniBand switches must be maintained in accordance with Oracle guidelines published in My Oracle Support bulletin 888828.1.

• Keyboard, Video and Mouse (KVM) module: The KVM can be updated according to customer requirements. There are no specific requirements or constraints from Oracle.

• Cisco Ethernet switch: The Cisco Ethernet switch can be updated according to customer requirements. There are no specific requirements or constraints from Oracle.q p q


Quiz

ILOM enables you to actively manage and monitor an individual server independently of the operating system state. With ILOM, you can proactively:

1. Learn about hardware errors and faults as they occury

2. Learn about Exadata software faults as they occur

3. Remotely control the power state of your server

4. View the graphical and non-graphical consoles for the host

5. View the contents of the database alert log

6. View the current status of sensors and indicators on the server

7. View information about the hardware configuration of the server

Answers: 1, 3, 4, 6, 7

ILOM does not generate Exadata software alerts, however a hardware fault may generate an ILOM alert and an Exadata alert


ILOM alert and an Exadata alert.

ILOM does not natively provide a view of the database alert log, however you can use ILOM to remotely access the console for a database server and using that console access you can log into the sever as a DBA user and view the database alert log.


Quiz

Which of the following statements is a broad overview of the recommended approach for patching and updating Database Machine:

1. All updates for Database Machine are specific to Database p pMachine, and these are the only updates that should be applied to Database Machine

2. Use Unbreakable Linux Network (ULN) to update Linux on database servers and Exadata servers, and apply specific database and Exadata updates for the remaining softwarep g

3. Use Exadata-specific update bundles for the Exadata servers, and use normal database updating practices for the database servers

Answer: 3



Summary


• Describe the monitoring tools provided with Database Machine

• Describe the support update and patching considerationsDescribe the support, update and patching considerations that apply to Database Machine




• Lesson Demonstrations (Viewlets)– Using the distributed command line utility (dcli)

— http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/131DBMDCLI/131dbmdcli_viewlet_swf.html

• Documentation and Support Bulletins– ILOM Documentation

— http://docs.sun.com/app/docs/coll/ilom3.0

– Database Machine patches and updates— My Oracle Support bulletin 888828.1

– Oracle Database Machine monitoring— My Oracle Support bulletin 1110675.1



http://st-curriculum.oracle.com/demos/db/11g/r2/dbmach/131DBMDCLI/131dbmdcli_viewlet_swf.html

Practice 13 Overview: Using the distributed command line utility (dcli)Using the distributed command line utility (dcli)

In this practice you will use the distributed command line utility (dcli) provided with Database Machine to simultaneously execute monitoring and administration commands across multiple servers.



N F t i U d t R l 11 2 1 3 1New Features in Update Release 11.2.1.3.1


Objectives


• Describe the new features of Exadata and Database Machine introduced in update release 11.2.1.3.1


Exadata and Database Machine Administration Workshop A - 2

New Features Overview

Release 11.2.1.3.1 includes:

• Auto Service Request (ASR)– An automatic service requests is opened with Oracle Support

when a hardware failure condition is detectede a a d a e a u e co d o s de ec ed

• Oracle Linux 5.5– Includes security fixes not present in Oracle Linux 5.3

• Enhanced operating system security– Unnecessary packages and services are removed

Fi ll f ti lit i i l t d– Firewall functionality is implemented

• Pro-active disk quarantine – Reduces the potential for data loss due to multiple failures

• Alerts for disk failures on database servers

• SNMP alerts from InfiniBand switches

New Features Overview

The slide lists the main new features associated with Exadata software release 11.2.1.3.1. Each feature is further introduced in the remainder of this lesson


Each feature is further introduced in the remainder of this lesson.


Auto Service Request (ASR)

• Automatically and securely opens service requests with Oracle for common hardware faults– Minimal data is collected and transmitted

– No IP addresses are communicated

– One-way (customer to Oracle) SSL encrypted communications are used

• Enables fast and accurate resolution of the hardware faults– Improved availability, less downtime

• Can be integrated with existing monitoring tools• Can be integrated with existing monitoring tools– ASR manager can send SR notifications via SNMP traps to

existing monitoring tools

• Included with hardware warranty and Oracle Premier Support for Systems

Auto Service Request (ASR)

Oracle Auto Service Request for Sun Systems is a secure, scalable, customer-installable software solution available as a feature of your Oracle or Sun hardware warranty and Oracle


software solution available as a feature of your Oracle or Sun hardware warranty, and Oracle Premier Support for Systems or valid Sun support plan. The ASR software helps to resolve problems faster by using auto-case generation for Oracle's Sun server and storage systems when specific hardware faults occur.

With the release of the Exadata software version 11.2.1.3.1, ASR functionality is extended to Exadata and Database Machine. ASR automatically opens service requests (SRs) with Oracle Support when specific hardware faults occur either in the Exadata servers or the d t b N t th t ASR i li bl l f h d f lt d t t d thdatabase servers. Note that ASR is applicable only for hardware faults detected on the following server components: CPUs, disk controllers, disks, flash cards, flash modules, InfiniBand cards, memory modules, system boards, power supplied, and fans.


The ASR process

Customer Datacenter

Oracle Field

FRU replaced byField Engineer

Oracle Support Services

Oracle Support Engineer

Field Replaceable Unit (FRU)

SR creationemail notification

to customer

Customer

Engineer

Fault occurs

Unit (FRU) dispatched by

Support EngineerService Request routed to Support Engineer

Oracle Case Management

Fault telemetry securely

transmitted to Oracle

Management System

Service Request (SR) created

ASR ServiceSNMP trap sent to

ASR Manager

ASRManager

The ASR process

The diagram in the slide illustrates the ASR process. It shows how a Service Requests (SR) is automatically opened by the ASR Manager after it receives an SNMP trap that is triggered


is automatically opened by the ASR Manager after it receives an SNMP trap that is triggered by a hardware fault. Within Oracle Support Services the automatic service request is acknowledged via email and serviced according to normal procedures.

Customers should note that there are occasions when a SR may not be automatically filed. This can happen due to loss of connectivity to the ASR manager, for example. Oracle recommends that customers continue to monitor their systems for faults and engage Oracle Support if they notice a fault but do not receive notice that a service request has been

t ti ll fil dautomatically filed.


ASR requirements

• ASR Manager requirements:– Qualified Sun server running Oracle Linux 5.3 (or higher) or

Solaris 10 update 6 (or higher)

– ASR Manager version 2.7 or higherg g

– Connectivity to the management network of Database Machine

– https connectivity to Oracle Support

– Can upgrade an existing ASR Manager to monitor Database Machine

– Can use a Database Machine database server as the ASR Manager, though not recommended

• Technical system administrator contact for the Database Machine registered with Oracle Support

ASR requirements

The slide lists the requirements for using ASR in conjunction with Database Machine. See http://www oracle com/asr for more information including a list of qualified Sun servers which


http://www.oracle.com/asr for more information including a list of qualified Sun servers which can be used to run the ASR Manager.


Oracle Linux 5.5

• Upgrading Exadata to version 11.2.1.3.1 causes Oracle Linux to be upgraded from 5.3 to 5.5 as well

• For Database Machines shipping from the factory with the Exadata software version 11.2.1.3.1 the database servers are imaged with Oracle Linux 5.5

• For existing Database Machines the database servers can be upgraded manually using standard Oracle Linux upgrade procedures.

Oracle Linux 5.5

With Exadata software version 11.2.1.3.1, the operating system in the Exadata servers is upgraded to Oracle Linux 5 5 This version of Oracle Linux includes security fixes not included


upgraded to Oracle Linux 5.5. This version of Oracle Linux includes security fixes not included in the previous version used in Exadata and Database Machine, Oracle Linux 5.3.

Similarly, for Database Machines shipping from the factory with the Exadata software version 11.2.1.3.1, the database servers are also imaged with Oracle Linux 5.5. However, note that for existing deployments, when applying the Exadata 11.2.1.3.1 patch the operating system on the database servers is not be automatically upgraded to Oracle Linux 5.5. The database servers continue to use Oracle Linux 5.3 until the customer chooses to upgrade them

ll i t d d O l Li d dmanually, using standard Oracle Linux upgrade procedures.


Enhanced operating system security

• Unnecessary Linux packages are not installed

• Unnecessary Linux services are disabled

• Firewall functionality (using iptables) is availableImplemented with pre configured rules on Exadata servers– Implemented with pre-configured rules on Exadata servers

– Available but not implemented on database servers

Enhanced operating system security

In addition to upgrading the operating system on the Exadata servers to Oracle Linux 5.5, security has been enhanced by eliminating the installation of unnecessary Linux packages


security has been enhanced by eliminating the installation of unnecessary Linux packages. Also, unnecessary services like nfs, ftp and telnet are disabled by default to minimize security vulnerabilities. In addition, firewall functionality (using iptables) with pre‐configured rules is implemented on the Exadata servers.

For Database Machines shipping with version 11.2.1.3.1, or for freshly re‐imaged database servers, the number of Linux packages that are pre‐installed is also minimized and unnecessary services are disabled by default. Customers also have the ability to configure

d bl fi ll f ti lit i i t bl th d t band enable firewall functionality using iptables on the database servers.


Pro-active disk quarantine

• If a disk or flash module fails, or is about to fail:– Associated grid disks are immediately dropped from

corresponding ASM disk groups— and from Exadata Smart Flash Cache for flash modules

– ASM re-creates lost data using mirrored extents sooner

– Greater protection from potential data loss associated with multiple overlapping disk failures

Pro-active disk quarantine

With Exadata 11.2.1.3.1, if the cell server detects a failed disk or a disk predicted to fail soon, it will pro‐actively drop the associated grid disks from the ASM disk groups they belong to


it will pro actively drop the associated grid disks from the ASM disk groups they belong to rather than waiting for the time specified in the DISK_REPAIR_TIME disk group attribute (3.6 hours by default). This ensures that ASM can immediately start to re‐create the data stored on the failed disk using the mirrored extents on other disks, thus preserving the data redundancy in the ASM disk group. This behavior improves the overall availability of the system because previously the loss of another disk during the DISK_REPAIR_TIME window could have resulted in the loss of data.

Si il l if th ll d t t fl h d l th t h f il d i b t t f il it d thSimilarly, if the cell server detects a flash module that has failed, or is about to fail, it drops the flash module from the Exadata Smart Flash Cache and drops any associated flash-based grid disks from the corresponding ASM disk groups.


Other new features

• Alerts for disk failures on database servers– SNMP traps can be configured to alert when a disk fails, or is

about fail, on a Database Machine database server

• SNMP alerts from InfiniBand switches– Upgraded InfiniBand switch software facilitates SNMP traps

to enable improved monitoring of the switch

Other new features

In conjunction with the Exadata 11.2.1.3.1 update, it is now possible to configure and send SNMP traps when a disk fails or is about to fail in the Database Machine database servers


SNMP traps when a disk fails, or is about to fail, in the Database Machine database servers.

Also, in conjunction with this update the InfiniBand switch software in upgraded to version 1.1.3.2 which includes software ILOM functionality. This new functionality sends SNMP traps, about the overall health of the InfiniBand switch, which can be used to better monitor the InfiniBand switch.


Summary


• Describe the new features of Exadata and Database Machine introduced in update release 11.2.1.3.1



d67016gc20 exadata workshop part2

Documents

exadata smart flash

available flash memory

exadata cell

exadata environment

exadata performancecopyright

data warehouse workloads

accessed data

cache keepbecause