exachk for exadata presentation
TRANSCRIPT
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Health Checks amp Best Practices for Exadata
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The following is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
2
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
3
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
4
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Oracle Engineered Systems
Configuration specific Health Checks
Saves Time Saves Money
Why EXAchk
5
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The following is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
2
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
3
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
4
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Oracle Engineered Systems
Configuration specific Health Checks
Saves Time Saves Money
Why EXAchk
5
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
3
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
4
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Oracle Engineered Systems
Configuration specific Health Checks
Saves Time Saves Money
Why EXAchk
5
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
4
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Oracle Engineered Systems
Configuration specific Health Checks
Saves Time Saves Money
Why EXAchk
5
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Oracle Engineered Systems
Configuration specific Health Checks
Saves Time Saves Money
Why EXAchk
5
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
ndashAutomated risk identification and proactive notification before business is impacted
ndashHealth Checks based on most impactful reoccurring problems across Oracle customer base
ndashRuns in your environment ndash no need to send anything to Oracle
ndashScheduled email Health Check reports
ndashFindings can be integrated into other tools of choice
Why EXAchk
Engineered Systems
Non Engineered Systems
EXAchk
ORAchk
Common Framework
Lightweight amp non intrusive Health Check framework for the Oracle stack
6
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Engineered to work together with Engineered Systems
bull Exadata DBM is engineered for best fastest and stable database performance and availability
bull Hundreds Thousands of QA Test amp MAA tests through Exadata DBM Lifecycles
bull Feedback from internal tests customers and development teams including Maximum Availability Architecture (MAA) is integrated into the product andor EXAchk
7
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Configuration specific Health Checks
bull EXAchk provides configuration specific up to date health check across the entire DBM stack
o Continuously evolving configuration checks for the entire Oracle Exadata Database Machine and the software that runs upon it
o Exadata DB GI ASM critical issues list specific for the environment
o Exadata full stack software planner
o MAA scorecard that highlights MAA configuration gaps and provide guidance to MAA and consolidation best practices
o Automated scheduling ability and automatic difference identification between runs with email notification
8
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Saves Time
bull No time required on your part to develop checks (unless you want tohellipcovered later)
o Hundreds of checks for best practices are provided
bull Configuration is minimal
bull Provides an automation framework which can be run fully automatically at scheduled times or invoked adhoc on demand
bull Executes quickly
o Parallelized execution keeps typical wall clock time low
bull Provides prioritized categorized report of findings focused on high priority issues first
bull Provides self service guidance for each check
bull Results can be integrated into your existing tools such as Oracle Enterprise Manager 3rd party tools such as Kibana or your own custom applications
9
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Saves Money
bull EXAchk is provided as a ldquovalue addrdquo as part of your existing Support Contract
oNo extra cost for EXAchk included within your support fee
oAvailable to download from My Oracle Support (10709541)
bull No development costs required by your organization
oHealth Checks developed and provided for free by Oracle
oWith User Defined Checks you can use EXAchk as an automation platform to run your own checks
10
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
11
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk is a lightweight full-service best practice and critical issues evaluator for the Exadata DBM
bull Release cycles are targeted to 90 days with Beta made available approximately two weeks before production
o Interim releases may be released sooner if significant Critical Issues are discovered
o Also timed to coincide with major software releases and or hardware changes
bull Maintains backward compatibility
Development Methodology
12
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Continuous improvement from
o Enhancement requests feedback from customers
o Exadata MAA Best Practices team
o Critical Issues discovery
o Internal testing Proofs of Concepts amp Support Partners
Development Methodology
13
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Development Methodology
bull EXAchk does not duplicate the function of other established tools
o Enterprise Manager for continuous monitoring
o AWR reports
o SQL tuning tools
o Advanced diagnostic gatherers
o Automated Service Request
o Storage server alert mechanisms
bull EXAchk validates a given Exadata DBM configuration against a critical subset of the Oracle MAA Exadata Best Practices (published in MOS doc 7575521)
14
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
15
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Recommended Usage ndash Automated Risk Notification
1) Schedule EXAchk to run in daemon mode weekly or daily and email report
2) Identify actions easily by viewing automated comparison of previous runs
3) Act on recommendations
16
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Schedule EXAchk
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
example exachk -set AUTORUN_SCHEDULE=820 25 will schedule runs on Tuesday and Friday at 0800 amp 2000
AUTORUN_SCHEDULE =
17 17
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull First email will contain the HTML report
Email Notification
18
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Health score
bull Summary of EXAchk run
bull Table of content
bull Controls for report features
bull Findings
bull Recommendations
19
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Report
bull Table of content
bull Controls for report features bull Checks which passed are not shown by default
bull Quickly show or hide checks based on their status
bull Show or hide major sections of the report
bull Collapse or expand check findings
bull Show check Ids
bull Remove the findings from the report ndash This doesnrsquot change the report but optionally hides findings on a check by check basis
bull Display a printable view
20
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View EXAchk Findings
bull Check status
bull Type of Check
bull Check Message
bull Where the check was run
bull Link to expand details
21
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull What to do to solve the problem
bull Links to relevant Knowledge docs
bull Where recommendation applies
bull Where problem doesnrsquot apply
bull Example of data the recommendation is based on
View Recommendations
22
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Review MAA Score Card
bull Critical Issues in MAA Scorecard
o All issues reported in ldquoSOFTWARE MAINTENANCE BEST PRACTICESrdquo
bull Software version mapping table
bull Installed software versions checked for noncurrent or incompatible feature usage
(Many other areas of report shown in demo sample later)
23
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Subsequent emails compare results to previous run
bull Easily see if something has changed
bull Email attachment has
o Latest report
o Previous report
o Diff Report
Email Notification
24
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Diff overview
bull Summary of this run vs previous
Diff Output
25
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Differences between each run
Diff Output
26
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Upgrade or Patching
o Detects all databases registered in the clusterware automatically
o Presents list of databases to check
o Pre Upgrade Run during upgrade planning phase
Asks which version you plan to upgrade to
o Post Upgrade Run after upgrade
bull Other typical times to run EXAchk
o Machine moves
o Hardware failure repair
o Problem troubleshooting
o In addition to go-live testing
bull Other Key My Oracle Support notes
o MOS 7575521 publishes the MAA Exadata Best Practices
o MOS 8888281 is the master Exadata version reference
o Exadata Critical Issues (Doc ID 12700941)
Other Recommended Usage
exachk -u ndasho pre
exachk -u ndasho post
27
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
28
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Report Demo
29
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
30
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Supported Environments
bull All supported hardware types operating systems firmware versions and Oracle versions for the following Oracle Engineered Systems
o Big Data Appliance
o Exadata (V2 and above HP hardware based V1 systems are not covered)
o Exalogic
o Exalytics
o SuperCluster
o Zero Data Loss Recovery Appliance
31
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Included in base image and latest OEDA
2 Download latest version from My Oracle Support (install lt 1 min)
3 Auto update when later version available
Roll Out amp Maintain EXAchk
32
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
1 Download the exachk bundle from MOS note 10709541 to your local machine
2 Unzip it and transfer exachkzip to a directory on the target Engineered System
3 Unzip exackzip
o As owner of oracle database or grid home
Installation
33
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Run as root (recommended)
o EXAchk will su to lower privileged owners of RDBMS or grid homes
o To specify a user other that root for these situations
bull Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level checks ndash several options
1 Provide the root userid password at prompts
or
2 Set up sudo
or
3 Pre-configure passwordless SSH connectivity
Which User to Run EXAchk as
Connect via SSH amp
Run Checks on Default User
Change User By exporting user id in this Environment Variable
Exadata Storage Server
root RAT_CELL_SSH_USER
InfiniBand switches
root (when run as root)
RAT_IBSWITCH_USER nm2user (when run as other user)
34
Note You may only choose from the provided lower privileged accounts
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Option 1 With internet connection
o When EXAchk is older than 120 days it will prompt you to let it automatically download newer version from My Oracle Support
o Can also be specifically triggered with
Option 2 No internet connection
a) Download the latest exachkzip to a shared network staging location
b) Set environment variable RAT_UPGRADE_LOC
c) Next time exachk script is started it will prompt to allow it to upgrade itself
Maintaining EXAchk
exachk -download
export RAT_UPGRADE_LOC=ltstaging dirgt
35
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Automated Usage with the Daemon
36
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options When What amp Who to Tell
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
AUTORUN_SCHEDULE
bull Schedule when exachk will be run
bull Hour day of month month of year amp day of week
bull Comma separate multiple values for same timeframe
bull Wildcard
Hour (0 ndash 23)
Day of month (1 ndash 31)
Month (1 ndash 12)
Day of week (0 ndash 6) (0 to 6 are Sunday to Saturday)
exachk ndashset ldquoAUTORUN_SCHEDULE=2 135rdquo
AUTORUN_FLAGS
bull Command line options to be passed through to exachk run
exachk ndashset ldquoAUTORUN_FLAGS=-profile dba ndashtag dbardquo
NOTIFICATION_EMAIL
bull Comma separated list of emails to send daemon notifications to
exachk ndashset ldquoNOTIFICATION_EMAIL=somepersonacompanycomanotherpersonacompanycomrdquo
37
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Maintenance
exachk ndashset ldquoltoption_1gt=ltoption_1_valuegtltoption_2gt=ltoption_2_valuegtltoption_ngt=ltoption_n_valuegtrdquo
COLLECTION_RETENTION
bull Number of days to keep files created by scheduled run files older than this will be deleted
exachk ndashset ldquoCOLLECTION_RETENTION=30rdquo
PASSWORD_CHECK_INTERVAL
bull Frequency in hours of password validation
bull When found invalid daemon stops amp notifies via log amp email
exachk ndashset ldquoPASSWORD_CHECK_INTERVAL=48rdquo
38
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Set Daemon Options Multiple IDs
bull Only one Daemon process should be used across a database cluster
o Multiple daemon profiles can be configured using IDs
o Allows for multiple different types of exachk runs
exachk ndashd ltIDgt ndashset ldquoltoptiongt=ltvaluegtrdquo
39
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get Daemon Option
bull Find a specific daemon option which has been set
bull Use with or without ID
exachk ndashd ltIDgt ndashget ltoptiongt
Get value of particular option for specific id
exachk ndashget ltoptiongt
Get value of particular option for all IDs
40
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Get All Daemon Options
bull Find any Daemon options which have been set
bull Used with or without ID
exachk ndashd ltIDgt ndashget all
Get value of all options for specific id
exachk ndashget all
Get value of all options for all IDs
41
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Start Daemon
exachk ndashd start
1 Start the daemon
2 Follow the prompts
42
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Information
exachk ndashd status
Check if the daemon is running
exachk ndashd info
Get more detailed information about the daemon
43
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Next Autorun
exachk ndashd nextautorun exachk ndashid ltidgt ndashd nextautorun
Find when the next auto run will happen Find when the next auto run will happen for a specific ID
44
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Daemon Stopping
exachk ndashd stop_client exachk ndashd stop
Stop an exachk run mid-flow Stop the daemon
45
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Auto Restart Daemon
bull Restart daemon if machine restarts o Uses passwordless ssh
user equivalence to root
exachk ndashinitsetup
46
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Restart status of daemon can be queried If you want to remove restart settings
Auto Restart Daemon Check Status amp Remove Auto Restart
exachk ndashinitcheck exachk ndashinitrmsetup
47
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
With or Without the Daemon
bull When daemon is running if exachk is run in adhoc mode it will connect to the daemon by default and make use of pre-provided prompt values
o To avoid connecting to the daemon use ndashnodaemon
o To only run exachk when the daemon is running use ndashdaemon
exachk ndashnodaemon
exachk ndashdaemon
48
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Email Notifications bull Run in ad hoc mode and receive html report via email
bull Verify email configuration function
bull Use ndashtestemail all to use email addresses stored in daemon configuration
exachk -sendemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail NOTIFICATION_EMAIL=abccompanycomxyzcompanycom
exachk -testemail all
49
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Additional EXAchk Capabilities amp Features
50
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Health Check Catalog bull Bundled within zip download
bull Audit_Checks_Report_Exachkhtml
bull Contains all published checks
bull Filterable amp searchable bull Engineered System
bull Profiles
bull Alert Level
bull Release Check Authored
bull Platforms
bull Look up check id without running report
51
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Database Checks
bull EXAchk runs checks against all database nodes in the cluster by default
o To specify only a subset of nodes use
o Only local node
bull Automatically discovers all databases and prompts for which should be checked
o Do not prompt but run all checks on all discovered database
o Do not prompt and skip all database related checks
o Only run checks against a subset of databases
exachk ndashclusternodes ltnode_1gtltnode_2gt
exachk ndashlocalonly
exachk ndashdball
exachk ndashdbnone
exachk ndashdbnames ltdb_1gtltdb_2gt
52
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Checks for Cells amp Switches
bull Run on a subset of cells using ndashcells
bull Run on a subset of switches using -ibswitches
exachk ndashcells ltcell_1gtltcell_2gt
exachk ndashibswitches ltswitch_1gtltswitch_2gt
53
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Virtualization
1 Run from dom0 on first compute to cover all dom0s cells and switches
o Install EXAchk into the management domain (DOM0) amp run as root
o Will discover all compute nodes storage servers and InfiniBand switches in the entire InfiniBand fabric and run EXAchk on all
o If a single machine has been separated into multiple management domains
bull Use -clusternodes -cells -ibswitches to detail the components of the separate management domains
2 Run once on each virtual cluster
54
exachk ndashclusternodes ltnode_1gt ltnode_2gt ndashcells ltcell_1gtltcell_2gt ndashibswitches ltswitch_1gtltswitch_2gt
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
EXAchk Output
bull EXAchk will output the collection results to the directory it is run from
o Output can be directed to a different directory with ndashoutput
bull Output will be directory and a zip of the same name
exachk ndashoutput ltOUTPUT_DIRgt
o log various log files
o outfiles collection results checks are based on
o Reports subreports used to build the main report
o Scripts scripts used during collection
o Upload files for upload of collection into database or integration into other tools
o exachk_html Main HTML report output
55
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Temporary Working Directory
bull EXAchk will create temporary files during execution
bull Default location is $HOME of the user EXAchk is running as
bull Location can be changed by setting RAT_TMPDIR
bull If using sudo access to root from a lower privileged user id temporary directory must be reflected in etcsudoers file
export RAT_TMPDIR=ltTEMP_DIRgt
exachk
ltusergt ALL=(root) NOPASSWDltTEMPDIRgtroot_exachksh
56
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Parallel Execution
bull Database collections are executed in parallel
bull The default number of slave processes used is calculated automatically
bull Default can be changed with ndashdbparallel lt slave processesgt or -dbparallelmax
bull Parallel execution can be disabled altogether if required with -dbserial
exachk ndashdbparallel lt slave processesgt exachk ndashdbparallelmax
exachk ndashdbserial
57
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Tagging Merging amp Comparing Reports
bull Collections are typically of the format exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgthtml
bull Tag collections so output contains another word to help differentiate it
exachk_ltdbservergt_ltdatabasegt_ltdategt_lttimestampgt_lttag_namegthtml
bull Merge multiple reports into one with ndashmerge and list of collection directories or zip files
bull Force the merge of collections from dom0 and domu or global and local zones by adding ndashforce
bull Compare collections with ndashdiff
exachk ndashmerge ltcollection_1gtltcollection_2gt
exachk ndashmerge ltcollection_1gtltcollection_2gt -force
exachk ndashdiff ltcollection_1gtltcollection_2gt
exachk ndashtag lttag_namegt
58
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Profiles provide logical grouping of checks which are about similar topics
bull Run only checks in a specific profile
bull Run everything except checks in a specific profile
Profiles
exachk ndashprofile ltprofilegt
exachk ndashexcludeprofile ltprofilegt
Profile Description asm ASM Checks avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks control_VM Checks only for Control VM(ec1-vm ovmm db pc1 pc2)
No cross node checks corroborate Exadata checks needs further review by user to determine
pass or fail dba DBA Checks ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks ovn Oracle Virtual Networking
platinum Platinum certification checks preinstall Pre-installation checks prepatch Checks to execute before patching security Security checks
solaris_cluster Solaris Cluster Checks storage Oracle Storage Server Checks switch Infiniband switch checks
sysadmin Sysadmin checks user_defined_checks Run user defined checks from user_defined_checksxml
59
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Run or exclude individual checks
bull Granular control to execute or exclude a single check
bull Ideal for testing new checks or troubleshooting
bull Run only specific check(s)
bull Exclude a specific check
bull Find check id either from report or Health Check Catalog
exachk -check ltcheck_id_1gtltcheck_id_2gt
exachk ndashexcludecheck ltcheck_id_1gtltcheck_id_2gt
60
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
61
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Dashboard
62
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Dashboard Filters
Filter by Interval
Filter by configurable
business units Filter by systems
Click on color coded area to drill down
63
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures amp Warnings
Click to see the recommendation details
64
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Failures
Click to drill into failures
65
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Most Warnings
Click to drill into warnings
66
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
No difference OR No regression failed in current collection
At least one regression from Non-WARNING to WARNING OR Found WARNING regression in current collection
At least one regression from Non-FAIL to FAIL OR Found FAIL regression in current collection
Non clickable green flag - Preceding collection not found
Recent Collections
Health Score Warning count
Fail count Info count Pass count
Ignore count
67
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Collection Link
68
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
View Collection
Recommendation
69
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks bull Use as a Health Checking Platform
bull You write your own business specific User Defined Checks
bull Collection Manager authoring UI very similar to Oraclersquos internal authoring tool
bull OS or SQL logic
bull Generates user_defined_checksxml sample in install directory
bull Utilizes framework features such as result output email notification CM storage etc
70
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
User Defined Checks
bull Have their own profile user_defined_checks
bull Can be excluded exachk -excludeprofile user_defined_checks
bull Have their own section of the report
exachk -profile user_defined_checks
bull Can be run on their own
71
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Manager Setup
1 Ensure APEX 42 is installed amp configured
2 Follow Collection Manager installation in Collection Manager User Guide
3 Login to Collection Manager Application via a URL like the following
o Format will depend on choices during installation
httphostnameportapexfp=ApplicationID
httphostnameportplsapexfp=ApplicationID
72
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Collection Storage Table
bull Collection zip files are stored in the RCA13_DOCS table
o Already created during collection manager installation
bull Provide EXAchk where to upload collection results by providing connection string amp password via ndashsetdbupload and completing prompts
bull Get current values with bull Unset values with
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk ndashunsetdbupload RAT_UPLOAD_PASSWORD
exachk -getdbupload
73
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Store DB Upload Variables in Wallet bull Set all with
bull Set specific variables by specifying comma separated list
bull Unset all with
bull Check if variables are set correctly
exachk -setdbupload all
exachk -setdbupload RAT_UPLOAD_CONNECT_STRINGRAT_UPLOAD_PASSWORD
exachk -unsetdbupload all
exachk -checkdbupload
74
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
75
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Oracle Enterprise Manager Cloud Control 131
76
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Enterprise Manager Integration
bullCheck results integrated into EM compliance framework via plugin
bullView results in native EM compliance dashboards
bullRelated checks grouped into compliance standards
bullView targets checked violations amp average score
bullDrill down into compliance standard to see individual check results
bullView break down by target
77
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Integration is via the Enterprise Manager ORAchk Healthchecks plugin with the following Support
bull The following prerequisites must be met before you can deploy the plug-in o Verify that your Engineered Systems hardware
and software are at the supported level as described in Supported Hardware and Software Versions
o All Engineered System plug-ins should be deployed
o InfiniBand switches and storage cells should be an Enterprise Manager-managed target for the respective engineered system
o Expect package should be installed on the hosts
Enterprise Manager Plugin Perquisites
Hardware Types Supported By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No Recovery appliance Yes Exalogic (physical configuration) Yes Exalogic (virtualized configuration) Yes Oracle SuperCluster No Oracle Private Cloud Machine No
78
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Setting Up Enterprise Manager EXAchk Plugin
bull The plugin is already installed by default with Enterprise Manager 131
1 Deploy the plugin using the Enterprise Manager Plugin Deployment feature
2 Provision the plugin to setup the EXAchk daemon
79
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Use Enterprise Manager provisioning feature and select ORAchkEXAchk
bull After selected this will launch the provisioning wizard choose the system type
Provision EXAchk
80
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Provide new or select existing credentials
bull Specify install location
bull Select when daemon should be run
81
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Provision EXAchk
bull Verify CRS Home
bull Finally choose Cell amp InfiniBand configuration
82
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Drill into applicable standard and view individaul checks amp target status
View Results by Compliance Standard
Filter by Exachkrdquo
Click individual checks for recommendation details
83
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With 3rd Party Tools
84
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
JSON Output to Integrate with Kibana Elastic Search etc
85
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull EXAchk results are also output in JSON format in the upload directory of the collection
bull Writing JSON Results With syslog
1 JSON output results can be sent to the syslogd Daemon with ndashsyslog option eg
2 EXAchk uses the message levels of ldquocritrdquo ldquoerrrdquo ldquowarnrdquo and ldquoinfordquo
3 You can verify syslog configuration by running the following commands
4 Then verify in your configured message location (eg varadmmessages) that each test message was written
JSON Result Output
exachk ndashset ldquoAUTORUN_FLAGS=-syslogrdquo
86
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Integrating With Your Own Application
87
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Custom App Integration
bull If you donrsquot use Collection Manager and have your own application which consumes the EXAchk results
1 Create the tables auditcheck_result auditcheck_patch_result amp RCA13_DOCS tables
2 Tell EXAchk about these tables
exachk -setdbupload all
bull EXAchk results will be uploaded into the tables after execution which can be consumed by your own App
88
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Agenda
1
2
Why EXAchk
Development Methodology
Recommended Usage Overview
Report Structure
Detailed Usage Options
Collection Manager
Integration With Other Tools
Troubleshooting
3
4
5
6
7
8
89
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull View Component Elapsed Times
o Identify if one particular component is slow
bull View top 10 Time Consuming Checks
o If some in particular are slow show check_id and run again with
Troubleshooting Performance
exachk ndashexcludecheck ltcheck_idgt
bull Try increasing the number of parallel slave processes (Note will increase resource usage)
exachk ndashdbparallelmax
90
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull If checks are being killed this will be due to timeout
bull Timeouts can be caused by slow checks or interactive prompts
bull Try increasing default times with applicable environment variables
bull EBS checks query application data which can vary hugely in size depending on the size of your data
o If you suspect EBS checks try excluding EBS profile
oUse RAT_TIMEOUT to increase to a value where they are no longer killed
Troubleshooting Timeouts
exachk ndashexcludeprofile ebs
91
Environment Variable Timeout Controlled Default Value
(seconds)
RAT_TIMEOUT non-root individual
commands 90
RAT_ROOT_TIMEOUT root userid command
sets 300
RAT_PASSWORDCHECK_TIMEOUT ssh login DNS
handshake 1
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
bull Debugging via Daemon bull Debugging adhoc EXAchk runs
Collecting Debug
exachk ndashd start_debug exachk ndashdebug
bull Debug output
92
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved |
Safe Harbor Statement
The preceding is intended to outline our general product direction It is intended for information purposes only and may not be incorporated into any contract It is not a commitment to deliver any material code or functionality and should not be relied upon in making purchasing decisions The development release and timing of any features or functionality described for Oraclersquos products remains at the sole discretion of Oracle
93
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94
Copyright copy 2016 Oracle andor its affiliates All rights reserved | 94