Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizing the Enterprise Manager 12c Environment Enterprise Manager 12c, Release 4
Kellyn Pot’Vin-Gorman Consulting Member of Enterprise Manager Technical Team Strategic Customer Program May, 2015
Oracle Confidential
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential 4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
Oracle Management Service, (OMS)
Oracle Management Repository, (OMR)
Metric Collections and Jobs
Weblogic and Agent Tuning
Summary
Oracle Confidential 5
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The Challenge •Slow Login Page Response
•Slow EM Page Response
•No Data Results to Screen
•Slow Refresh of Data Upon Request
•Job Delays
•Upload or Collection Delays/Not Uploading
Oracle Confidential 6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 7
Architecture
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda
1
2
3
4
5
Oracle Cloud Control User Interface, (Console)
Oracle Confidential 8
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Health Overview
Oracle Confidential 9
Setup Manage Cloud Control Health Overview
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Console Performance Page
Setup Manage Cloud Control Health Overview
OMS and Repository Monitoring Page Performance
Oracle Confidential 10
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Page Performance
Oracle Confidential 11
OMS and Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Page Correlation vs. Network/OMR/OMS
Oracle Confidential 12
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Which Pages Are Highest Impacting
Oracle Confidential 13
OMS Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Java or Agent?
Oracle Confidential 14
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
SQL and Plan
SQL from Page
Plans
Compare to awrsqrpt.sql
(aka sql_id specific awr)
Oracle Confidential 15
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 16
Job Run
Not to be confused with user sessions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Load Balancers
Oracle Confidential 17
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Why a Load Balancer?
Load Balancer setup and configuration
– Users require session affinity to a particular OMS
This allows for new connections to perform Round-Robin vs. Most Requested which is less efficient.
Allows for simpler scaling for environments to support future growth.
Oracle Confidential 18
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 19
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Distributing Work Load to Multiple Oracle Mgmt Servers
Oracle Confidential 20
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Using the Access Log
$OMS_HOME/EMGC_OMS1/access.log
•Tracks each login, along with app/module logged in from:
Use NSLookup to locate workstation name.
Oracle Confidential 21
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Using EM CLI to Track UI Page
Oracle Confidential 22
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Controlling Login and Login Time
•Automatically controls logout of inactive sessions
•Single configuration setting for ALL OMS in environment-
•default is 45 minutes, but can be changed with EMCTL command:
$emctl set property -name
oracle.sysman.eml.maxInactiveTime -value 30
Oracle Confidential 23
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Have You Succeeded? Bottom of Page Performance Page:
Oracle Confidential 24
OMS and Repository Monitoring Page Performance
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
2
3
4
5
Oracle Management Service, (OMS)
Oracle Confidential 25
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
What We Think Of When We Think of EM12c
Oracle Confidential 26
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Enterprise Level- Enterprise Manager Environment
Oracle Confidential 27
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
EM12c Project Consideration
•Although it requires licensing, consider RAC-One Node for any production repository to allow for easy scalability in future.
•8Gb for a small OMR, 12Gb+ for medium and 16Gb+ for large.
•Consider ASM to assist with tablespace maintenance and enhanced management. Remember, everything is in the MGMT_TABLESPACE.
•Consider installing the OMS and OMR on separate hardware. Each requires different resources, but can compete with each other if constrained.
Oracle Confidential 28
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Initialization Parameters for the Repository Database
Note the Compliance Checks
Oracle Confidential 29
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Why ODL Format Log Files Are Important
Oracle Diagnostic Log Files provide a lot of data in an aggregated format that allows you to locate what you need if you know what you are looking for!
Located in following directories for EM12c: $OMS_HOME/EMGC_OMS1 $OMS_HOME/EMCG_ADMINSERVER $OMS_HOME/emoms $OMS_HOME/nodemanager $OMS_HOME/opmn_logs, etc., etc. *Agent has similar directory structures
Oracle Confidential 30
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
EMGC AdminServer Diagnostics Log File
Oracle Confidential 31
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Format Translations
Oracle Confidential 32
Timestamp Source and Type
Weblogic Server Mgmt Product
Component ID
Error Message
Execution Context ID
Transaction ID
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Google is our Friend
Search using the error message first, then add in filters with the context or type information to assist.
Oracle Confidential 33
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Tuning Java Heap Size
JAVA heap size for the OMS is a crucial setting. This value is used to accommodate memory for each user logged in.
$OMS_HOME/user_projects/domains/GCDomain/bin/startEMServer.sh
– Use OMS_HEAP_MAX, OMS_HEAP_MIN, OMS_PERMGEN_MAX, OMS_PERMGEN_MIN parameters to change the heap settings
$emctl set property -name OMS_HEAP_MAX -value 2560
– Stay between JAVA Heap size between 2Gb and 4Gb
Oracle Confidential 34
Default values: OMS_HEAP_MAX: 1740M OMS_HEAP_MIN: 56M OMS_PERMGEN_MAX: 768M OMS_PERMGEN_MIN: 128M
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Using EM Diagnostics
Repvfy(Repository Verify)
•Run level 9 –details
•Review Jmap output file.
Oracle Confidential 35
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
EMGC_AdminServer.out
Includes Java Heap Settings in first five lines.
$OMS_HOME/EMGC_ADMINSERVER/EMGC_AdminServer.out
Oracle Confidential 36
JAVA Memory arguments: -Xms256m -Xmx512m -XX:MaxPermSize=512m
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
3
2
4
5
Oracle Management Repository, (OMR)
Oracle Confidential 37
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Repository Support
Oracle Confidential 38
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The OMR Schema
Oracle Confidential 39
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 40
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Repository Space Usage in Table View
Oracle Confidential 41
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Partition Retention
Oracle Confidential 42
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Retention Killing Your Space?
Change at Table/Partition Level
> execute gc_interval_partition_mgr.set_retention('SYSMAN', <table name>, <number
of partitions to retain>);
Oracle Confidential 43
Modify retention or view impact of changing.
> repvfy send set_purge_retention -name {ALERTS|ERROR} -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Management Server Incidents
Oracle Confidential 44
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
OMR High Redo
Very common, consider at least 500M-1G Redo logs with at least 5 groups for high demand EM environments and consider retention data
>repvfy send set_purge_retention -name CONFIGURATION -id <num days>
Oracle Confidential 45
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The “Weight” of Different Tasks
Oracle Confidential 46
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Most Important Graphs for OMR
Setup Manage Cloud Control Repository Repository Tab
Oracle Confidential 47
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Upload Performance
Oracle Confidential 48
•Uploads are collections of data from agents to the OMS. •If backlogs occur, experiences of-
•Delayed notifications •Unresponsive agents •Inaccurate information to console •Delayed response to incidents from jobs
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
SQL Baselines
For the SYSMAN schema, no SQL plan baselines should get captures, and no baselines should be used. The use of SQL plan baselines can severely impact the performance of the OMS and the repository (in a very negative way).
SQL> ALTER SYSTEM SET
optimizer_capture_sql_plan_baselines=false
SCOPE=both;
SQL> ALTER SYSTEM SET
optimizer_use_sql_plan_baselines=false SCOPE=both;
Oracle Confidential 49
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Repository Performance
•Average Collection Duration in Seconds
•Number of Collections in Backlog**
•Throughput per Second
•Metric Data Rollup Performance, (metric extensions included)
Oracle Confidential 50
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Task Worker Threads
These workers process tasks and messages from the internal subsystems. These internal housekeeping jobs are executed by 2 main DBMS_SCHEDULER jobs in the repository database:
One to execute small jobs (SHORT worker pool), and one to execute larger, more time consuming jobs (LONG worker pool). The number of 'workers' for each of these pools is configurable, and can be changed. To set the minimum of 2 workers for both the short and long running pool, use this command:
>repvfy send set_workers
Oracle Confidential 51
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
5
2
3
4
Metric Collections and Jobs
Oracle Confidential 52
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
All Target Metrics Are Not Created Equal- Uploads by Day
Exadata Storage Server Target: 27K rows, 9.5M
Cluster Target: 23K rows, 29M
Infiniband Switch: 27K rows, 11M
Database Instance: 122K rows, 7M
Oracle Home: 8K rows, 3M
Oracle Confidential 53
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
How Much Metric Data Am I Receiving?
Data can be gathered by running an EMDiag Report:
>repvfy dump metric_stats
Or….
Oracle Confidential 54
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Top Data Loading Metrics
Oracle Confidential 55
Setup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 56
Setup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Demands and Consequences
Oracle Confidential 57
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Cluster Detail Bubble Map
Oracle Confidential 58
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metric Collection Errors
Oracle Confidential 59
Setup Manage Cloud Control Repository Metrics Tab
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Collection Performance Reconfigure Update from Short Running to Long Running Graphed
Oracle Confidential 60
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Metric Raw Data
Metric data represents all the metric collections that are enabled against a target (3 main settings).
Raw Data, 7 days by default. Largest data, take care in changing retention.
Once per day, raw data is aggregated by day, month and 366 days. To change this retention:
>repvfy send set_data_retention -name {RAW|HOUR|DAY} -id <num days>
Oracle Confidential 61
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Advanced Troubleshooting
•Highlight a metric alert type
•Hover cursor to view options to:
Oracle Confidential 62
•Problem Analysis •Metric Details •Target Home
OMS and Repository Monitoring Metrics Collections
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Problem Analysis
Oracle Confidential 63
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Files Processed Vs. Backoff
Oracle Confidential 64
Outage
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Collection Performance- Improvement?
Oracle Confidential 65
Setup Manage Cloud Control Repository
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Repository Job Activity
Oracle Confidential 66
>repvfy send set_job_retention -name <job module> -id <num days>
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizing OMS Jobs
•Backload of Jobs, consider updating the parameter job_queue_processes up to 10.
•Edit the schedule for the daily jobs to run in off-peak time or more often to lessen impact.
Oracle Confidential 67
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Job Pool Tuning
In any environment relying on EM to execute jobs and deployment procedures, the size of the job connection pool from the OMS to the repository will have to get tuned. To change the site of the job pool connection pool, run this command on each OMS in the environment:
>emctl set property -name
oracle.sysman.core.conn.maxConnForJobWorkers
-value <recommended> -module emoms
Oracle Confidential 68
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
5
2
3
4
Weblogic and Agent Tuning
Oracle Confidential 69
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Weblogic Server- Stuck Threads
Stuck Thread Timeout (BEA-337 incidents in the WebLogic log/trc files)
From OMS in Admin Console:
Stuck Thread Timeout = 1800
Stuck Thread Timer Interval = 120rom OMS in
Admin Console:
Oracle Confidential 70
Admin Console → Environment → Servers → EMGC_OMSx→ Tuning
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Weblogic: LDAP Configuration Flag
For LDAP and SSO setups, always make sure the local LDAP can be used.
Without this set, users are unable to login, (even as SYSMAN) remotely. The console will report remote authorization and authentication as down or unavailable. To correct, set in the Admin Console:
LDAP Provider Control Flag = Sufficient
Oracle Confidential 71
Admin Console → Security Realm -> myrealm -> Providers → LDAP Provider
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agents
•Separate Management Area from all OMS/OMR Performance Screens.
•How many targets are monitored by the agent.
•Incidents involving the agent or dependent targets.
•Last successful upload to the OMS.
Oracle Confidential 72
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agents
Oracle Confidential 73
Setup Manage Cloud Control Agents
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agent Down Detection
To eliminate staleness of agents after downtime and allow OMS's to catch up, force the agents to send heartbeats with a more effective grace period by defining in the system. This grace period serves as a time period right after an outage during which the OMS’s will NOT report any Agent as 'down'.
SQL> exec gc_diag2_ext.SetPingGracePeriod;
To set the grace period to a specific value, use this command:
SQL> exec gc_diag2_ext.SetPingGracePeriod(90);
The value for the grace period should be between 60 and 300 seconds. Higher values should only be used if specifically instructed by Oracle.
Oracle Confidential 74
Setup Manage Cloud Control Agents
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
View by Name Default
Oracle Confidential 75
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agent Heartbeat
Incoming heartbeats from agents are “batched” to allow for bulk updates in the repository. For optimal performance, the size of such a bulk operation performed by a single thread in the OMS should never exceed 1000 Agents.
The default number of threads configured per OMS is two, which should be sufficient to handle up to 2000 Agents per OMS. If the number of Agents is above this, the number of threads to do the Agent heartbeat processing should be increased:
>emctl set property -name
oracle.sysman.core.omsAgentComm.ping.heartbeatPingRe
corderThreads –value "3"
Oracle Confidential 76
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Auto Tuning Set in Agents
To make sure the auto tuning is enabled check the value of the 'enableAutoTuning' parameter:
>emctl getproperty agent -name enableAutoTuning
If the value is not set to true, change it, and restart the Agent.
Oracle Confidential 77
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agents with Variety of Target Types
If the Agent is monitoring a various number of different target types, the overall timeout value can be changed, to avoid having to specify the timeout for each individual target type one by one:
>emctl setproperty agent -allow_new -name dynamicPropsComputeTimeout -value 180
Oracle Confidential 78
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agent Tuning on Incoming to OMS
Similarly, if there are a lot of targets (+50) of a given type discovered on the Agent, and those targets are administered in bulk (by applying templates, submitting jobs, or performing repetitive tasks), the number of incoming connections from the OMS to the Agent can exceed the default of 25. For those cases, the default value can be increased. Values above 50 should be extreme cases, and need to be validated and verified by Oracle.
>emctl setproperty agent -allow_new -name MaxInComingConnections –value 40
Oracle Confidential 79
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Agent Patching Tips
Oracle Confidential 80
Pre-stage patches
To apply a set of patches automatically to an Agent while deploying it via the console, stage the patches
in the >$OMS_HOME/install/oneoffs/<version>/Generic/ (generic platform
independent patches)
and >$OMS_HOME/install/oneoffs/<version>/<platform>/ (platform specific
patches)
Every Agent path to apply will have to get copied on each of the OMS’s, to make sure that regardless
which OMS does the Agent install, the pre-staged patches will get applied.
Download Agent Software and Opatch
The Agent software for the managed platforms can be downloaded using the Self Update pages in the
console.
Before installing additional agents for other platforms, ensure that the OPatch download job has
completed. This job runs on a 24hr cycle, and will get the latest OPatch from My Oracle Support.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Program Agenda with Highlight
1
5
2
3
4
Summary
Oracle Confidential 81
6
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Summary
•Best Performance requires different skills for optimizing Enterprise Manager Cloud Control than for a database or engineered system.
•There are impressive enhancements and bug fixes in the PSU patches and yes, they exist for EM12c- the OMS, OMR and Agent. Address this FIRST!
•Don’t get overwhelmed- use the tools provided in the Manage Cloud Control and rep/agt/oms vfy to inspect the environment on a quarterly basis.
•Don’t just settle with the default values for the environment- many times these are set by development and you most often use your EM12c differently than they imagined or plugins, etc. can change usage.
Oracle Confidential 82
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Staying on Top of PSU Patching
96% of issues experienced are addressed by applying OMS and Agent patches. Please remember to provide tender loving care to your Enterprise Manager…
Oracle Confidential 83
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Tip Working With MOS and Service Requests
omsvfy “Diagnostic dump” contains a file called “env”.
File contains all lib files, all environment information, ports, etc. that will be required for an service request. Upload this one file and save a LOT of time..
Oracle Confidential 84
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Want to Learn More?
DBAKevlar Blog Posts: http://dbakevlar.com/2014/10/em12c-release-4-health-overview/ http://dbakevlar.com/2014/11/em12c-rel-4-oms-and-omr-health-part-ii/ http://dbakevlar.com/2014/12/oms-and-omr-performance-part-iii-metrics-page/ http://dbakevlar.com/2014/11/agents-management-via-em12c-release-4-console/ Change Retention: http://docs.oracle.com/cd/E24628_01/doc.121/e24473/repository.htm#EMADM12785
Oracle Confidential 85
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 86
Connect with me via Social Media: