Download - ERP Support Tips
-
8/8/2019 ERP Support Tips
1/61
Oracle Apps: Art of Supporting Oracle Applications ERP
Art of Supporting Oracle Applications ERP
1. Problem Resolution
CLARIFYING THE PROBLEM
The following questions can assist in identifying and resolving problems in a timely
manner.
Has this function worked previously?
If the answer is yes, then you will need to explore areas such as patch application.
Has this problem occurred before, during the testing phase of your implementation or
an upgrade?
You could review your records at this stage to see if this has occurred previously and
how it was resolved.
Has it stopped working as a result of a stand-alone Patch or Patchset being applied?
At this point it is also a good idea to obtain the current Patchset level you have for
the particular module.
Have there been any changes to the configuration of the system technical or
functional?
Technical eg upgrade of the Database, Application, Forms or Reports, Functional eg
Values in the System Options form have been modified within a particular module.
Is the problem user specific?
You will need to identify if other users are experiencing the problem reported or if itrelates to one user only. If it is a single user, things such as personal profile options
can be reviewed. Or, is it a group of users accessing the same responsibility? Review
the differences in the responsibilities to try to pin point the issue.
What environment is it happening in? Is the problem reproducible?
-
8/8/2019 ERP Support Tips
2/61
Is this a Production, Test, User Acceptance Testing (UAT) or Development system? Is it
occurring in all, some or only one of the above environments? Try to examine the
differences between the environments if the problem exists in one but not the other.
It is also important to try to reproduce the problem in another environment if
possible. This will aid the problem solving process if the issue needs to be logged withOracle Support Services.
Is this standard functionality or a customisation of Oracle applications?
If the problem is standard functionality and the initial research has been done, such
as referring to MetaLink and User Guides, Technical References and so on, and the
problem remains unsolved, then a call can be logged with Oracle Support for
assistance. If it is a customisation made to the application, then it will not be
supported by Oracle Support Services, and you may need to seek the services of a
Consulting organisation, if you cannot resolve the problem.
Reviewing your concurrent log files.
I cant stress enough the importance of reviewing the log file from your Concurrent
processes. Even if your concurrent request completed without an error status, it is
always good practice to just quickly review the log file. Processes such as the Oracle
Assets Periodic Mass Copy and the General Ledger Journal Import may complete
successfully, but that does not mean that there were no exceptions encountered.With processes such as these, which transfer information, or with interfaces, always
review the log file and associated report output for any records that were not
successful in the process, as they will need to be addressed.
Always take note of the errors that appear in your log file, and save the file to a
secure directory, to avoid losing it should the Concurrent Manager requests get
purged.
METALINK
Before logging a call with Oracle Support Services, the above questions should be
-
8/8/2019 ERP Support Tips
3/61
answered as well as performing research on MetaLink. MetaLink is a Web support
service available from Oracle Support Services. It was developed to provide Oracle
customers with a 24x7 resource to access information from repositories regarding
support information, including solutions. The following address allows you to register
on and access MetaLink - www.oracle.com/support/
There are various facets to MetaLink, however the focus of this paper will be on the
search function for previous reported incidents and the logging, updating and
reviewing of Technical Assistance Requests (TARS), and the reviewing of Patches, Bugs
and briefly on the Forums available.
The Search Function
The Basic Search
Enter the search criteria in the search field (To the left of the Search Button). Thenclick on the search button and your results will be displayed.
The Advanced Search
The Advanced Search page allows you to refine and further customise your querywhen searching for information on the site. You can narrow your search by selecting
one or more options. These options include selecting the type of search, the order in
which information is shown, and the source or area of the site you wish to search.
To ensure that your search is effective, take some time to review the actual search
methods available. There are five methods and generally their use will vary on the
-
8/8/2019 ERP Support Tips
4/61
type of problem/error that you encounter.
Below is the screen dump of the Advanced search screen.
This is the information the above search retrieved.
Match All (AND++) Search
This is used for general keyword searches and is the default for both basic andadvanced searches.
Using AND++ without a comma:
EXAMPLE: Using the keywords,"APP-43446 TAXCODE will return all documents in
which both "APP-43446" and "TAXCODE" appear. The precise phrase "APP-43446
TAXCODE" does not have to appear in any of these documents. The more frequently
the keywords exist in the document the higher the score on the hit list returned.
Using AND++ with a comma:All words BEFORE the COMMA must exist, while each word AFTER the comma scores
higher if present.
EXAMPLE: Using the keywords, "APP-43446 unable to find, taxcode for this
transaction" will return all documents in which "APP-43466 unable to find" appear but
"taxcode for this transaction" are optional. All documents with the words "APP-43446
unable to find" will appear in the hit list but those documents with "taxcode for this
-
8/8/2019 ERP Support Tips
5/61
transaction" also appearing in the document will score higher than those without
these optional words.
ConText Syntax Search
Use the ConText Syntax option if you want to conduct a detailed search that involves
more than keywords. This option searches the site using the ConText Query language.The ConText Query language uses special search operators that define terms within a
query expression.
The following table provides examples for some commonly used search operators:
Symbol-Operator
Input Result Description
% Wildcard Character 10565%
te%st 10565.6
test, teeniest, tenderest The percent symbol matches any substring.
( _ ) Wildcard Character 80_86
_est 80186,80286, 80z86
best, jest, pest, rest, test The underscore matches any single character.
( ) Group Character (peter, or paul and mary) peter and/or paul and mary The open
parenthesis indicates the beginning of a group. The first close parenthesis
encountered is the end of the group. Any open parentheses encountered before the
close parenthesis indicate nested groups.
{ } Group Character {end-of-file}{ORA-600} end of file
ORA-600 Performs the same function as regular brackets except it prevents the
penetration of other expansion operators.
$ Stem Expansion $scream scream screaming screamed Expands a query to include all
terms having the same stem or root word as the specified term.
? Fuzzy Expansion ?feline feline defined filtering Expands a query to include all terms
with similar spellings as the specified term (English-language text only).
Match any (OR) Search
Use the Match any (OR) option to find information on multiple topics or for synonyms.
The Match any (OR) option is a Boolean expression that allows you to find documents
in which either of two or more search words appear, again without needing to know
the ConText search search syntax.
-
8/8/2019 ERP Support Tips
6/61
EXAMPLE: Using the keywords, APP-43446 Taxcode will find all documents in which
APP-43446 appears and all documents in which Taxcode appears.
Doc ID Search
Querying by document ID number allows you to retrieve a specific document fromOracle Support Services' databases. A document ID number is a unique identification
number assigned to every file stored in Oracle's Support Knowledge Base. Document ID
numbers can be obtained from an Oracle analyst or by looking at the document ID in
hitlists from other types of MetaLink searches.
Weighted (FUZZY) Search
Use the Weighted (FUZZY) search option to find documents that contain any of the
keywords (like a Match any (OR) search) but by using a comma you can rank the
importance of the keywords.
EXAMPLE: Using the keywords, "ORA-942, AR_STATEMENT_HISTORY" would find
documents that contain either of those keywords, however it would rank "ORA-942" as
more important than AR_STATEMENT_HISTORY. Scores take into account word
frequency so it is possible that an article containing 10 occurrences of
AR_STATEMENT_HISTORY would score more than an article containing both terms.
Technical Assistance Request (TAR) - Create, Update and Modify
With the correct privileges, you have the ability to create, update and modify the
TARS in Metalink. When you create an internet TAR (iTAR), it is sent to a queue in the
Metalink System. All TARS are transferred into the internal tracking system every 15
minutes and an analyst will then be assigned to work on it.
Below is a screen dump of a Global TAR search by a single Customer Support Identifier
(CSI) Or Multiple CSIs within the same country.
-
8/8/2019 ERP Support Tips
7/61
Below is a screen dump representing a search for a specific TAR number.
Patches
A patch is a collection of fixed bugs assembled and tested by support analysts and
developers within Oracle. If you find a problem that has been reported before and
there is a patch available you can download the patch directly from the web site. You
can also view a list of available patches for a product and platform.
Type in the desired patch that you wish to download, the release, product, platform,language and type. If you know the file that the problem occurred in, then type this
in the Includes File box. Ie Statement Generation program file arxsgp.lpc. Then it will
list all the patches that contain this file. You can choose to order it by Patch Number
or Release Date. Then click submit. It is very important to read the README.txt as
there could be pre-requisite or other important instructions you need to follow.
Bugs
These are issues that are logged into the bug database and this is the primary tool
used by Development to determine the problem and devise a solution. The Bug
Database Search page allows you to specify your search either by a bug number or by
other search criteria such as product version, platform, or status. You can also
-
8/8/2019 ERP Support Tips
8/61
determine the order of your query results by sorting how you would like the
information displayed.
If you can not retrieve a bug then it may be due to the following reasons;
1) The bug may not be published
2) Bugs have to be newer than June 1998
3) You can only view bugs that match the products and platforms that are licensed
according to the CSIs you have inserted in the Show Licence section of Metalink.
This will retrieve the history of the bug and provide a resolution.
Forums
A forum is an interactive area for discussions and commentaries dedicated to a
certain topic. Forums allow you to post questions and comments and receive
responses within 2 business days from a Support Analyst who has expertise in that
product area. If this is not obtainable within 2 days then advice from Oracle will be
given. It also gives you the opportunity to reply to questions posted by other people.
The forums are intended to address problems with a severity level 2,3 or 4. If the
problem is a severity 1 then you must make a call to support. Forums are threaded, so
a reply to a particular posting becomes part of the thread.
Select the relevant Forum Product Groups from the drop-down list.
Click on the type of Thread: All, New, Changed.
All Threads posted since the date indicated in the posting period.
New- Only those threads posted since your last visit.Changed- Only those threads with changes ie new replies
From the time drop down list, select the number of days for which you would like to
view messages.
Option to click on Only Threads in which I participate, that is the forum threads
which you have posted a message or replied to a message.
-
8/8/2019 ERP Support Tips
9/61
Click on the Open Forum Button
For Further information about Metalink please review- Metalink Quick Reference
Sheet. Document id 115465.1
THE EXAMINE/DEBUG TOOL
The Examine utility can be very useful to view values of hidden fields in a form that
relate directly to columns in the Oracle tables in your database. It is often used to
find the value of an Id column, which is usually the primary key used to identify
records in a table.
The Navigation path for this is as follows -
Release 10.7 Character
\Other Debug
Release 10.7 GUI 11
Help > Tools > Examine
-
8/8/2019 ERP Support Tips
10/61
Release 11iHelp > Diagnostics > Examine
Utilities:Diagnostics
This profile determines if the diagnostic utility Examine requires an Oracle database
password. It also controls access to most of the other functions on the Tools menu.
1) The setting of Yes means that you can automatically use the Examine tool when
the choice is on the menu.
2) The setting of No mans that you must enter a password to use the Examine tool
when it appears on the menu.
Your System Administrator controls the setting of this profile option.
You may have noticed on your Tools menu in Rel 10.7SC, that there is also a debug
option available. This will effectively run your form in debug mode, displaying debug
messages as each step executes. Please note that this option does not work with
version 4.5 of Forms, but is fixed in version 6.
HOW DO I IDENTIFY THE VERSION OF A FORM, REPORT, and PROGRAM?
It is important to identify the current version of the program that is experiencing the
problem, as a bug in the version of the program you are running may be the cause of
your problem.
Form
A Form version is easily obtained via the front-end application. The navigation path is
as follows
Release 10.7 Character
-
8/8/2019 ERP Support Tips
11/61
\Help Version
Release 10.7 GUI 11i
Help > About Oracle Applications
This will provide you with several pieces of useful information, including the Database
version, the Applications version, the Form Name and Form Version.
Report/Executable
To obtain the version of a report or concurrent executable, you need to firstly log on
to the operating system, and go to the appropriate directory -
For Reports:
Release 10.7
Cd $/srw
Release 11
Cd $/reports
For a Spawned Concurrent program:
Cd $/bin
Issue the following command to retrieve the version number of the report program or
the libraries used by a concurrent executable
Unix:
Strings -a |grep Header
Or
Ident |more
-
8/8/2019 ERP Support Tips
12/61
NT:
Find /i Header
FORM ERRORS
If the problem is occurring in a standard Oracle Form, obtain the following
information by navigating to Help-> About Oracle Applications in the Forms menu -
Oracle Forms Version (example 4.5.10.10.2), this will help determine the form
patchset level.
Form Name: (example FNDSCSGN), will assist in finding previous form issues.
Individual Form Version (example 11.0.30), this is important if reporting a bug to
development is necessary.
You can try to regenerate the Oracle Standard Forms (All platforms) and relink form
executables (UNIX Only)
Regenerate the forms that are causing the problems. Forms can be regenerated by
either using the adadmin installation utility, or manually at the command line. If you
are on a UNIX platform, you will also need to relink the form executables.
Refer to the Installations Manuals that are appropriate for your Release and Platform.
HOW DO I RUN A PROGRAM AND REPORT FROM THE COMMAND LINE?
Sometimes there is insufficient information in the Concurrent log file to determine
the reason why a particular concurrent process is failing or completing with errors. Inthis situation, if a TAR is logged with Oracle Support Services, they will quite often
request you to run the process from the command line. This will also rule out if there
is in fact a concurrent manager problem. To do this you need to firstly take note of
the exact parameters as listed in the parameters field when you view your concurrent
request. Then, go to the command line in your operating system and change directory
to where the program resides: Cd $/sub-directory.
-
8/8/2019 ERP Support Tips
13/61
Executable Program
Enter the following command, substituting the parameters from the Concurrent
Manager, and the userid and password:
0 Y Y
is the parameters you see from that concurrent request
Note: when you run a concurrent program executable from the command line, you
should use a space instead of a comma to separate between each parameter.
/ is the username/password of the apps account for your database.
Eg For the General Ledger Posting program, if the parameters you see in the
Concurrent request are 1, 101, 2546, you would enter the following command
GLPPOS 0 Y 1 101 2546 Y
This will execute the process and produce a debug log file in the format Lxxxxx.log.
This log file will reside under the directory. The information in this log file can give
Oracle Support Services a better idea of what may be causing the problem.
Reports
Report programs reside in:
Product_TOP/srw (Apps 10.7)
Product_TOP/reports (Apps 11)
To run a report from the command line, issue the following command -
Unix
r25run userid= report= .rdf
destype= desname=desformat=batch=yes
-
8/8/2019 ERP Support Tips
14/61
NT
In NT, use the r25run32.exe executable to run a report:
r25run32 userid=apps/apps@vd11 report=.rdf
destype= desname=batch=yes
TRACE TOOLS
SQL*Trace
SQL*Trace is a facility which assists you in identifying the SQL statements that are
executed by a process. Additionally, it provides you with statistical information for
further analysis, which can be useful when there are performance issues or problems
with database objects.
There are four levels in your Oracle Applications at which SQL*Trace can be enabled
Database
Form
Report
Profile (applicable to Release 10.7 only)
Database Trace
A Database level trace requires a shutdown and startup of both the Concurrent
Managers and the Database. If there are a number of users on the system or the
system is unavailable due to 24 x 7 requirements, a database level trace may not
always be feasible.
The Database trace process is as follows
1) Perform a shut down of the concurrent managers, then the database.
2) Modify the start up parameter file (init.ora) to ensure the following initialisation
parameters are set
SQL_TRACE=TRUE
USER_DUMP_DEST (destination directory for trace output)
-
8/8/2019 ERP Support Tips
15/61
TIMED_STATISTICS=TRUE
3) Restart the database, then the concurrent managers.
4) Run the process for which the problem is occurring.
5) Retrieve the trace file(s) from the USER_DUMP_DEST directory.6) Perform a shut down of the concurrent managers, then the database.
7) Restore the original init.ora file.
8) Restart the database, then the concurrent managers.
Steps 1-3 and 6-8 above should always be performed by your Database Administrator.
The value of the initialisation parameter SQL_TRACE dictates whether the SQL trace
facility is enabled or disabled when you begin a session in ORACLE. Its presence in the
init.ora file sets the initial value for this trace. This will apply to all sessions on an
ORACLE instance. You can additionally enable or disable the SQL trace facility just for
a specific session with the SQL_TRACE option of the ALTER SESSION command. This
allows you to obtain smaller and more readable trace files and is particularly useful
for developers. The SQL statement to achieve this is
ALTER SESSION
SET SQL_TRACE = TRUE
Form Trace
A Form level trace requires the Trace option to be available as a menu item. The
trace process is as follows
- Navigate to the Form for which you wish to set the Trace.- Turn the Trace feature on when you are at the point in the form that you want to
turn trace on
GUI versions
Help > Tools > Trace
-
8/8/2019 ERP Support Tips
16/61
Rel 10.7 Character
\ Other Trace On
Rel 11i
Help > Diagnostics > Trace
- Note the time trace is started.
- Execute the actions you wish to trace.
- Turn the Trace feature off. Make sure this step is done, or you will severely degrade
your system performance.
GUI versions
Help > Tools > Trace
Rel 10.7 Character
\Other Trace Off
Rel 11i
Help > Diagnostics > Trace
- Locate the trace file(s) with the time stamp when you ran your trace. The trace file
will be in the USER_DUMP_DEST directory.
Note: if a trace file is being generated to track performance, the parameter
TIMED_STATISTICS in the init.ora file must be set to TRUE.
Report TraceA Report level trace would generally require a developer or technical resource, and
involves the following steps
1) Logon as applmgr and change directory to where your .rdf file resides.
2) Create a backup copy of your current .rdf and .rex files in the appropriate srw
directory. Create the .rex text of the report if necessary.
-
8/8/2019 ERP Support Tips
17/61
3) If the .rex does not exist, you can easily regenerate the report executable by doing
the following:
UNIX
R25convm userid=/ source=stype=rdffile dtype=rexfile dest=
overwrite=yes batch=yes
NT
R25conv32 syntax:
R25convm userid=/ source=
stype=rdffile dtype=rexfile dest=
overwrite=yes batch=yes
4) Modify the .rex file for the report.
Include an additional line after the statement FND SRWINIT in the Before Report
Trigger
srw.do_sql (alter session set sql_trace=TRUE)
This will activate the SQL Trace in the report
Include an additional line after the statement FND SRWEXIT in the After Report
Trigger
srw.do_sql (alter session set sql_trace=FALSE)
This will deactivate the SQL Trace in the report after the report completes.
5) Convert the modified .rex file into an .rdf file
UNIX
R25convm userid=/ source=
stype=rexfile dtype=rdffile dest=
overwrite=yes batch=yes
-
8/8/2019 ERP Support Tips
18/61
NT
R25conv32 syntax:
R25convm userid=/ source=
stype=rexfile dtype=rdffile dest=overwrite=yes batch=yes
6) Run the report.
7) Locate the trace file(s) from the USER_DUMP_DEST directory.
There is another method that can be used to trace an individual report within Oracle
Applications for Release 10.7:
In Character Mode:
1) Log on as System Administrator.
2) \Navigate Concurrent Program Define.
3) Enter Query mode.
4) Enter your report-name in the short name field.
5) Execute Query.
6) Navigate to the next block.
7) Cursor is now in the concurrent program details.
8) Select Parameters.9) Arrow down through the parameters, looking for the name = 'trace'.
If not found, then create a new parameter:
Add sequence number, such as 999
Name = Trace
Description = Report Trace
Enabled = YES
Value Set = Yes_No
Required = NoEnable Security = No
Default Type = Profile
Default Value= NO
10) Save the record
11) Run the report, making sure that profile option trace = YES.
-
8/8/2019 ERP Support Tips
19/61
In GUI Mode:
1) Log on as System Administrator.
2) Navigate to Concurrent>Program>Define.
3) Enter Query mode.
4) Enter your report_name in the short name field5) Execute Query.
6) Click on the [Parameters] button.
7) Search through parameters, looking for parameter name = trace.
If not found, then create a new parameter:
Enter Seq = 999
Parameter = trace
Description = report trace
Click on Enabled
Value Set = Y or N
Default Type = profile
Default Value= No
Required = no
Enable Security = no
8) Save the record
9) Run the report, making sure that profile option trace = Y.
Utilities:SQL TraceSQL trace files can also be generated for concurrent programs through the use of this
profile option. The trace can be enabled at all levels, i.e. Site, Application,
Responsibility and User, by setting the profile Utilities:SQL Trace to Yes. This
option is controlled by your System Administrator - users can see the profile option
but cannot update it.
This profile option is only applicable to Release 10.7.
In Release 11, the equivalent trace utility for concurrent programs is set within each
individual Concurrent Program definition. It is only the System Administrator or
Application Developer responsibility that has access to define concurrent programs.
The information produced by a SQL*Trace is stored in output files with a .trc
-
8/8/2019 ERP Support Tips
20/61
extension. As the format of the trace output is difficult to read, there is a utility
available called TKPROF which should be applied to the trace file(s) to reformat the
output in a more readable format.
The basic command syntax of the TKPROF utility is as follows
$ tkprof
If you do not know the destination directory for the creation of your trace files, this
value can be found easily through the following SQL query using SQL*Plus
SELECT value
FROM v$parameter
WHERE name = 'user_dump_dest';
DEBUG/TRACE PROFILE OPTIONS
There are several profile options across the Oracle Applications that can assist with
problem resolution. These profile options serve to provide more information in the
Concurrent log file in the form of debug tracing messages
Product Updateable Profile Option CommentsOracle Cost Management At all levels for the System Administrator
CST:Cost update debug level
(Release 10.7 11i) For the cost update program, available values are as follows
None Do not print debug messages.
Regular Print a debug message for each subroutine executed.Extended Print a debug message for each SQL statement executed.
Full Print a debug message for each SQL statement executed and keep any temporary
data in the database
Oracle Master Scheduling/MRP At all levels MRP:Trace Mode
(Release 10.7 11i) Indicates whether to enable the trace option. Available values are
-
8/8/2019 ERP Support Tips
21/61
Yes or No
At all levels MRP:Debug Mode
(Release 10.7 11i) Indicates whether to enable debug messages within MRP and
Oracle Work in Process. Available values are Yes or NoOracle Assets Application
Responsibility
User FA: Print Debug
(Release 10.7 - 11i) This profile option is used by Oracle Support Services as a tool to
identify a problem with the code. Available values are Yes or No
Application
Responsibility
User FA: Print Timing Diagnostics
(Release 10.7 - 11i) This profile option is used by Oracle Support Services as a tool to
identify a problem with the code. Available values are Yes or No
Oracle Order Entry Application
Responsibility
User OE:Debug
(Release 10.7 11i) Indicates whether debug messages are displayed on forms.Available values are Yes or No
Application
Responsibility
User OE:Debug Trace
(Release 10.7 11i) This profile option is used by Oracle Support Services as a tool to
identify a problem with the code. Indicates whether a trace file is created during a
Concurrent Program execution. Available values are Yes or NoOracle Work in Process At all levels TP:WIP Concurrent Message Level
(Release 10.7 - 11i) Determines the level of detail reported in the move transaction
concurrent log file during move and resource transaction validation/processing.
Available values are
-
8/8/2019 ERP Support Tips
22/61
Message level 0 Reports only errors.
Message level 1 Reports processing activities and errors.
Message level 2 Reports and time stamps processing activities and errors.
Oracle General Ledger At all levels FSG: Message Detail
(Release 10.7 11i) Determines the level of detail in your error message log file for
FSG reports. Error messages are divided into three categories
Catalog I contains detailed statistics useful for program debugging
Catalog II contains process messages useful for finding out where a process failed
Catalog III contains only error messages
Available values for this profile option are
None No messages.
Minimal Catalog III messages.
Normal Catalog II and III messages.
Full Catalog I, II and III messages.
Must be set by System Administrator. GL:Debug Directory
(Release 11-11i) Specifies the directory where the Workflow debug files are created.The database must have write permissions to the specified directory. Available values
are
Yes The database may create debug files in the nominated directory.
No The database cannot create debug files.
Oracle Projects Application
Responsibility
User PA:Debug Mode
(Release 11 -11i) Determines if PA processes and reports are run in debug mode.
Available values are Yes or No.
Oracle Purchasing At all levels PO:Set Debug Concurrent ON
(Release 11 11i) This profile option is used for finding problems with Requisition
-
8/8/2019 ERP Support Tips
23/61
Import. For performance reasons, the value should ALWAYS be set to No, unless
otherwise instructed by Oracle Support Services for debugging purposes.
At all levels PO:Set Debug Workflow ON
(Release 11 11i) This profile option is used for finding problems with PurchasingWorkflow processes. For performance reasons, the value should ALWAYS be set to No,
unless otherwise instructed by Oracle Support Services for debugging purposes.
Account Generator At all levels Account Generator:Run in Debug Mode
(Release 11 11i) Determines if Account Generator processes are run in debug mode.
Available values are Yes or No.
COMMON UNIX O/S SIGNAL ERRORS
Signal errors are how Unix processes communicate with each other and the operating
system. This is not a problem with the operating system, but is part of the operating
systems mechanism of protecting itself. When programs terminate abnormally, this is
communicated to other processes and the kernel by means of a Signal code. The
actual code associated with the signal indicates the nature of the termination. The
meaning of the signal message is contained in a file called signal.h, which is located
usually in the /usr/include/sys sub-directory.
Some of the more common Signal errors are as follows
SIGNAL 4
A Signal 4 error usually indicates an illegal/unknown instruction was encountered.
This is typically caused by bugs in the code. Explore Metalink for the error to find out
if your particular problem is related to a known bug for which there may be a patch.
Some encounters with a Signal 4 error, for which there are patches available, include
Process Possible Cause Possible Resolution
AR to GL Transfer In Rel 10.7 Product 16.1, you may only have 5 accounting periods
open at one time. Fixed in Patch 932703 or in 11.02 Patch 713261. Fix is also included
in 11.0.AR.D for Rel 11.
-
8/8/2019 ERP Support Tips
24/61
FA Mass Revaluation Preview Report Possible cause is not having set up the
Revaluation Reserve and Amortization Accounts for all Categories/Books. Perform
relevant FA setup
Open and Close GL periods Retained Earnings accounts contain disabled or expired
segments Apply database Server side patch 642941 for Rel 10.7. This will providemore meaningful error messages on the setup issues/steps not completed prior to
running the Open Next Period process.
SIGNAL 10
A Signal 10 usually suggests a memory or addressing error, but it can also be caused
by a bug in the code, or if an .rdf file has been incorrectly ftpd from the client in
ASCII mode. There are several patches available on Metalink for Signal 10 errors
caused by bugs in the various Oracle products. If it is a memory-related problem,
sometimes shutting down the Concurrent Managers and the database, and booting the
machine will eliminate any run away processes and reclaim resources.
Examples of Signal 10 errors include -
Process Possible Cause Possible Resolution
AP Mass Additions Create Lack of space in the database. Ask your DBA to check the
available space in the FA data and index tablespaces, and allocated an additional
datafile if necessary.Oracle Alert process If email (Unix Sendmail) is part of the action items defined in the
Alert definition, and Sendmail is not configured correctly for the system, is non-
functional or is missing. Ask your Unix Administrator to verify the Sendmail
configuration, as it is specific to certain variations of Unix.
THE DREADED SIGNAL 11!!
A Signal 11 error indicates that a memory fault has occurred, where an Oracle
executable has attempted to access memory incorrectly, e.g. program
pointers/addresses for memory allocations are incorrect or corrupted, or an attempthas been made to use a segment of memory for which it does not have permission.
The most common causes of a Signal 11 are
1) Data Corruption
2) File Size insufficient disk space in $APPLCSF/out directory where concurrent
manager output is created
-
8/8/2019 ERP Support Tips
25/61
3) Kernel parameter problems
4) Lack of storage space in Tablespaces, Rollback Segments etc
5) Reports ftpd using incorrect mode ASCII instead of BINARY
Examples of Signal 11 errors include -
Process Possible Cause Possible Resolution
FA Periodic Mass Copy Header file corruption Fixed with Patches 924617 and 958462.
FA Mass Change Bug in Mass Change program
For Rel 10.7 -Fixed with Patch 534568 (or Patchset C or higher)
For Rel 11 Fixed with 709527 (or Patchset 11.0.FA.C part of the 11.0.3
Maintenance Pack)
The problem with Signal errors is that whilst we know the meaning of the Signal code
that was encountered, how do we pinpoint which of the possible causes actually
caused this violation ?
What to look for with Signal errors
Trap the Core Dump file !!
When a Signal error is encountered by the system, a core dump is usually produced.
This core dump is created at the instant the process was aborted, and is produced in
the form of a file called core. The information contained in the core dump file is
most valuable in helping to resolve a Signal error. It is important to trap the file as
soon as it is created, as these types of files are overwritten the next time a core
dump occurs. It is always a good idea to rename the core dump file produced and
move it to a different directory.
The core dump file is usually contained in the directory where the command was
issued, or in $ORACLE_HOME/dbs.
If a core dump is not generated, this is usually due to the dump destination directory
being full, or the process not having write access to the dump directory.
-
8/8/2019 ERP Support Tips
26/61
Produce a debugged version of the Core Dump file !!
You can relink the executable in debug mode by using the Adrelink utility. This will
allow you to rerun the process causing the Signal error, producing a debugged version
of the core dump. This provides more comprehensive statistics than the standard core
dump file produced without debug switched on.
An example of using the debug feature is outlined below, for the Oracle Assets
Periodic Mass Copy process (executable FAMCP)
1) Log in as APPLMGR to the operating system and environment affected.
2) Change directory to the directory in which the executable resides e.g.
$FA_TOP/bin
3) Make a backup copy of the current executable.
4) Issue the following command -
Adrelink force=y ranlib=y link_debug=y fa FAMCP
This will produce a log file called adrelink.log, which can be located in
$APPL_TOP/install/log. This log file will be appended to every time a relink command
is initiated.
5) Re-run the process that produced the Signal error, e.g. Periodic Mass Copy.
6) Turn debug off by typing in the following command
Adrelink force=y ranlib=y link_debug=n fa FAMCP
The problem with a core dump file is that it is not easily decipherable. You will need
to enlist the help of your DBA or a technical resource with expertise in this area, as
you will need to determine which debugger exists on your system, and use it to
produce a stack trace from the core dump. The Unix command script should be
used to capture the output of the debugger. This can then be sent to Oracle Support
Services to help them determine why the core dump occurred.
Checklist for Signal errors
The following general checklist, utilising the techniques described earlier in this
White paper, can be used as a guide to trying to determine the cause of a Signal error
-
8/8/2019 ERP Support Tips
27/61
Action
1. Review Concurrent Request log for any useful information on the error encountered
2. Locate core dump file rename and move to a separate directory
3. Identify full version numbers of the product, RDBMS and the program thatencountered the error
4. Answer the basic questions
Has the process worked before ? If so, when, and has anything changed since then,
such as application of patches, upgrades or configuration ?
What is the current Patch Set level for the product?
Is the error reproducible in the same environment ?
Is the error reproducible in another environment ?
Is the problem related to a specific User or Responsibility?
If using Release 10.7SC, is the error reproducible in Character mode ?
Is it standard or customised functionality ? If its a custom report, and is based on a
standard report, run the standard report to see if it too errors
5. Check Metalink for known bugs relating to this Signal code and any patches
available
6. Check storage space in tablespaces, rollback segments, etc
7. Bounce the Concurrent Manager
8. Run the process from the operating system command line
9. If Signal error occurred when running a report :1) Check disk space in $APPLCSF/out directory
2) Check report was ftpd in BINARY mode
3) Check SRW.USER_EXIT (FND SRWINIT) in Before Report Trigger
4) Check SRW.USER_EXIT (FND SRWEXIT) in After Report Trigger
5) Check for P_CONC_REQUEST_ID parameter in user parameters
10. Rerun the process in debug mode
1) Application debug mode e.g. FA:Print Debug (profile option)
2) ADRELINK utility11. Check for data corruption:
1) Check parameters entered when submitting the Concurrent Request. If it is a
report process, try narrowing down the parameters e.g. smaller date range
2) If submitted by batch feed, check input file for errors in field mapping or the
presence of invalid characters
3) Process of elimination. Identify the records being processed. Run the SQL
-
8/8/2019 ERP Support Tips
28/61
statements being executed by the program, and check the status of each record for
data corruption.
12. Log TAR with Oracle Support Services
13. Send debugged core dump file and concurrent log file to1) Oracle Support Services
2) DBA
Include all relevant version numbers
TIPS AND HINTS FOR COMMON APPLICATION ERRORS
Asset Trace
This SQL script is designed to provide detailed information on a single asset. It is
extremely useful for troubleshooting problems with your assets. Whenever you log a
TAR with Oracle Support Services for problems with particular assets, you will ALWAYS
be asked to run this script for the offending asset(s). The script captures the data
contained in all the Oracle Assets tables for an individual asset, and outputs the
details to a spool file. You should find this file in the admin/sql directory under
$FA_TOP. To run the script, log into SQL*Plus as the APPS user and type -
@trace.sql or start trace.sql
You will be prompted to enter the ASSET_ID and BOOK_TYPE_CODE [uppercase].
To get the ASSET_ID run the following statement in SQL*Plus -
select asset_id from fa_additions where asset_number = '[the asset number]';
If you cannot find the script on your system, you can locate it on Metalink, or
alternatively contact Oracle Support Services and get them to e-mail the script to
you. The script is called trace10.sql (Rel 10) or trace11.sql (Rel 11), and produces anoutput file called t107.lis and t11.lis respectively. There is also a tracehtml.sql that
produces the output in html format (fa_trc.html), which can be viewed using IE
Explorer 4.0+ or Netscape 4.0+. This version of the trace script works with both
Release 10 and Release 11 of Oracle Assets.
To effectively analyse the asset data in the trace output, you really need to have a
-
8/8/2019 ERP Support Tips
29/61
good understanding of the Oracle Asset tables in order to determine if there is data
corruption. Alternatively, log a TAR with Oracle Support Services and send the trace
results to them for further investigation.
Profile Option FA:Deprn SingleThis profile option should be used specifically when a depreciation run fails in Oracle
Assets. It controls the caching buffer used when depreciation is run. You can set the
buffer to either No or Yes. If you set the value to No, then the cache is reset
after every 20 assets. If the value is set to Yes, then the cache is reset after EVERY
asset.
When depreciation encounters an error in Rel 10.7 and 11, the program will fail at
that point and rollback any depreciation that was committed. In this situation, change
the value of this option to Yes under the Fixed Assets Manager responsibility, and
the value of the FA: Print Debug profile option to Yes, and rerun depreciation - the
process will fail again, but you will find that the log file now produces detailed
information on the actual asset causing the failure. Once your resolve this, you will be
able to resubmit the depreciation process. If there are no other errors, the process
will complete successfully.
In Release 11i, the run depreciation and close periods have been broken out to
separate processes. If depreciation fails, and you set the FA:Deprn Single value toYes, it will process all the assets in your Depreciation Book and capture ALL errored
assets in the log file. This way you can fix all of your errors in one pass, resubmit
depreciation, and this time only the corrected assets will process.
Please note you should only ever change the value of this profile option to Yes when
depreciation fails.
ap375496.sql AP Trial Balance Rebuild scriptThis script can be useful if your AP Trial Balance report is not reporting correctly on
outstanding liability. The script is a data fix script that can be run to recreate the
trial balance records in the AP_TRIAL_BALANCE table for all records (payments and
invoice distributions) that have been transferred to GL within a specified range of
accounting dates. This script is contained in patch 375496 and can be downloaded
from Metalink.
-
8/8/2019 ERP Support Tips
30/61
Note the script is not needed for cash basis installations.
Account Generator FA Test Script
This script assists in diagnosing problems in Oracle Assets concurrent processes thatfail due to account generation. It replaces the Flexbuilder Test screen used in Release
10. The output from the script tells the User the following
1) The code combination id that Account Generator attempted to build.
2) The reason why Account Generator could not build this code combination.
3) If Dynamic insertion is set to Yes.
When you run the script, it will prompt you for a number of parameters. All the
information that feeds this script is contained in the log file from the failed run.
Release 11 of Oracle Assets is delivered with this SQL*Plus test script called
faxagtst.sql. You should find this in the $FA_TOP/admin/sql directory. The latest
script comes with 11.03 Minipack. Contact Oracle Support Services for the latest
version.
Account Generator Purchasing Workflow Test Script
In order to find out what is wrong with the Account Generator workflow, a script
called wfstatus.sql can be run. The parameter WF_ITEM_KEY is required to run thisscript. The wfstatus.sql gives important debug messages, including error messages for
the processes that are erroring. This provides the same information that the Workflow
Monitor does, if the customer has Workflow Monitor set up. Running the script
involves the following steps
1) Check the profile option Account Generator: Run in Debug mode. This should be set
to Yes. (No or Null value means that the account generator process is being run in
'Synch' mode, in which case no information is logged in the workflow tables, andhence wfstatus.sql would not retrieve any information)
2) After setting the profile option to Yes, simulate the error, by navigating to the
Distribution zone of Enter Purchase Order or Enter Requisition, and trying to build the
account again.
3) After you receive the error, go to Menu Help -> Tools -> Examine, and look for the
value for field CHARGE_ACC_WF_ITEMKEY (Click on the arrow next to the Field Zone
-
8/8/2019 ERP Support Tips
31/61
and then Query up the CHARGE_ACC_WF_ITEMKEY). Note down the itemkey value
required to run wfstatus.sql. (NOTE: If the value for the field is _SYNCH#_, the profile
option Account Generator: Run in Debug mode has not been set to Yes).
4) After getting the itemkey value, discard changes in the distribution zone, navigate
to lines/header zone and commit (Save). This is important for the wfstatus.sql toretrieve any data.
How to create an AR Debug log file
This function is normally used in the Transactions, Customers and Receipts screens.
The output gives details of the packages that are being used on the process when an
error occurs. The Debug steps are as follows
1) Go through the steps that you would perform to encounter the error/problem but
stop just before you would get the actual error. i.e. Don't hit the key that will invoke
the error.
2) From the Help menu at the top of the form, select: Help, Tools, Examine. Enter
the password if prompted. This will give you a box with 3 fields. Enter the Following:
In Block: Parameter
In Field: AR_DEBUG_FLAG
In Value: FS Where F = File S = Screen.
relates to the UTL_FILE_DIR directory where the database writes files. There are afew ways to determine what the UTL_FILE_DIR setting is
3) Run the following in SQL*Plus for the related instance:
Select value from v$parameter where upper(name) = 'UTL_FILE_DIR';
4) The UTL_FILE_DIR setting should be held in the init.ora file under the
$ORACLE_HOME/dbs directory. There is no default setting for this parameter if it is
not set. If the setting needs to be added to the init.ora file, then please note thatyour DBA will need to bring down the database instance and bring it back up for the
changes to be effective.
is whatever you wish to call the output created. i.e. Customers.log
5) Hit the key that will invoke the error/problem. A log file should then be created in
the output directory with the filename you have specified.
-
8/8/2019 ERP Support Tips
32/61
ORA-01000 Maximum Number Of Cursors Exceeded
This is one of the most common errors that can occur. The reason for this error is that
Oracle has reached the set limit of open cursors allowed for that executable or User
session. It is the initialisation parameter OPEN_CURSORS that determines themaximum number of cursors. If it is a custom program that fails with this error, then
a developer should review the code to see if the program can be modified to use
fewer cursors. If the error occurs often within your Oracle Applications, your DBA
should shut down Oracle, modify the OPEN_CURSORS parameter to increase its value,
then restart the database.
ORA-01562:failed to extend rollback segment number (n)
ORA-01628:max # extents (n) reached for rollback segment
Errors with rollbacks segments are common in the Oracle database when storage
becomes a problem. If you encounter errors with rollback segments, or a max #
extents error referring to a database object such as a table or index, contact your
DBA, as the size of the rollback segment will need to be increased, or the value of the
maximum extents parameter increased to allow for further growth.
REP-1222 Invalid body size
REP-1212 Object is not fully enclosed by its enclosing object
REP-1211 Object is larger than enclosing bodyThese are common errors that can occur with the running of reports. The cause of
these errors is usually that the Rows/Columns defined in the Printer Style you are
trying to use do not cover the whole of the report. If it is a Standard applications
report, then the solution is to increase the Col/Row for the Printer style. If it is a
customised report you need to establish the size of the report and set-up a Printer
Style as appropriate.
FIXES FOR DATA CORRUPTION
If your problem is in fact caused by data corruption, a data fix will be required to
update the records in error. If you need to run a data fix script, it is essential that you
perform a backup of the Oracle Application tables that will be affected by the fix.
Additionally, to form an effective audit trail, output should always be spooled before
and after the data fix to verify its results. Always review any data fix scripts sent to
-
8/8/2019 ERP Support Tips
33/61
you by Oracle Support Services before you run them. Back up the tables the script is
either inserting, updating or deleting records from, and if the script does not contain
a SPOOL command, then update the script to insert the SPOOL command before
execution of the script, and insert a SPOOL OFF command after the last SQL
statement. Example as follows
SPOOL UPD_PAY_DIST
Update ap_payment_distributions_all
set amount = base_amount
where invoice_payment_id = 13548
and payment_line_number = 33;
SPOOL OFF
Last, but not least, always run the data fix script in a Test environment before
running it against Production. It is always good practice to perform regular refreshes
of your Test environment to keep your Test environment up to date with your
Production environment.
-
8/8/2019 ERP Support Tips
34/61
2. THE ORACLE APPLICATIONS ARCHITECTURE
Depending on the Oracle Applications version you are running, the architecture of the
Oracle Applications environment will vary. The fundamental differences in the
architecture are as follows -
Release 10
The Release 10 architecture supports both the Graphical User Interface (GUI) Client
and Character Mode forms. Release 10 SmartClient (SC) works with an Oracle
Applications server running Release 10.7. Through the character mode forms tool, you
can access data through workstation clients. There are generally two tiers in this
architecture
Database tier (contains the database, application logic and the concurrent mangers)
Client tier (contains logic required to display the GUI front-end forms and some of the
business logic and error handling)
(Source: Oracle Applications for MS Windows Clients Installation Manual. Applications
Server for Window NT Edition. Release 10SC Production 16.1 pg 2-2)
-
8/8/2019 ERP Support Tips
35/61
Release 10.7 NCA Release 11
A third tier is introduced in the later releases of Oracle Applications. The client
interface is provided through a Java Applet Viewer or Web browser that launches the
applications forms server from the middle tier. The three-tier structure is as follows Database tier contains the data manipulation logic and manages the database.
Application tier contains the applications specific logic and manages the Oracle
Applications and other development tools (the forms and procedures that define the
application), including the Web Application Server software.
Desktop client tier provides the User interface display.
The introduction of the middle Application tier eliminates the need to install and
maintain application software on each desktop client, and helps to reduce network
traffic. The fundamental difference with the Internet Computing architecture is that
only the presentation layer of Oracle Applications is on the desktop tier in the form of
a plug-in to a standard Internet Browser.
(Source: Oracle Applications Release 11 for Unix Concepts pg 1-2)
Release 11i (11.5)
The Internet Computing architecture of Release 11i provides a framework for multi-
tiered, distributed computing that supports Oracle Applications products. It is the
100% Internet Oracle Applications suite that allows companies to run their worldwide
operations from a single, centrally managed site. The Database tier manages an
Oracle8i (8.1.6) Relational Database Management System (RDBMS) which offers
additional features only used by this Release.
(Source: White Paper - Upgrading to and administering Release 11i: A Technical
Perspective)
-
8/8/2019 ERP Support Tips
36/61
2. Basic Structure of the $APPL_TOP
It is important to have a general understanding of Oracles APPL_TOP and the
directory structure of the Oracle Applications product files. It helps to know where
the applications code resides for the various products, as some of the tools andtechniques for investigating support issues require you to directly access these
specific program files.
In the basic directory structure, there is a top Applications directory that holds one
set of product files. You create this directory and set the environment variable
$APPL_TOP to point to it. The Autoinstall process sets up a directory tree for each
fully installed or dependent product within this directory.
A product directory tree starts with a directory that uses the products abbreviation,
eg ap for Oracle Payables, gl for Oracle General Ledger. Below that is a sub-directory
that uses the product version. This is known as the Product Top directory eg
$GL_TOP, $FA_TOP, $AP_TOP. Beneath each Product top directory are various sub-
directories to hold the different types of product files. Forms programs will reside in
the forms directory, report programs will reside in the reports directory, sql scripts in
the sql directory and so on.
Custom programs should reside in a custom directory structure under the customapplication top directory.
A Look at $APPL_TOP
The Oracle Application software is installed in one common area in a subdirectory
tree fashion. The top directory for this repository is defined as $APPL_TOP. All other
subdirectories will key off of $APPL_TOP. For each product module that you haveinstalled, there will be another $_TOP variable defined. For example, if you have
Accounts Payable, General Ledger, and Purchasing, then you will have a $AP_TOP,
$GL_TOP, and $PO_TOP. Underneath these directories, you will notice a common
directory tree including /bin, /forms, /lib, /sql, /install, /srw, etc. Each directory
has its own unique use, and is similar for every product top. You should become
familiar with these standards and the contents of these directories.
-
8/8/2019 ERP Support Tips
37/61
Note that there exists other "tops" not associated with the product modules that your
site may use. These are either "shared" products or supporting products that Oracle
installs for you. We will explain a few of them in more detail. Note the file called
APPLSYS.env under $APPL_TOP. This is one of the environment files that are sourcedin to define your environment variables and directories necessary for your
applications. It will define $APPL_TOP, your product tops, and other variables. We
will discuss some of the "output" directory variables in a moment.
A couple of other directories worth noting under $APPL_TOP include
$APPL_TOP/install and $APPL_TOP/patch. The /install subdirectory holds many of the
utilities used for the initial installation. It also has a /log subdirectory where
installation, patches, and other adadmin utility log files will go. You will visit this
directory often. Typically, you will have $APPL_TOP/patch for holding patches that
Oracle Support may send you (will send you). You can put your patches in another
directory, but this is the default. You will become very familiar with patches, too.
A Look at $FND_TOP
Another top that you will find is $FND_TOP. When Oracle started writing the financial
applications software, they developed a core set of utilities and objects called
Application Object Library (AOL). From these objects, they wrote the "foundation" forthe Oracle Applications, referred to as Application Foundation. This foundation code
is stored under $FND_TOP. As product modules developed, they hooked these into the
Application Foundation infrastructure utilizing the AOL objects. Examples of these
AOL objects and FND products include the concurrent managers, quick picks, zooms,
etc.
Notice that $FND_TOP has a very similar directory tree to the other product modules.
You may have noticed that when you fire up the applications, that you call the scriptfile "found" (short for Application Foundation) which executes the $FND_TOP/bin/aiap
executable and passes the username/password stored in the variable $GWYUID
(typically applsyspub/pub) to get you to your initial login screen. See, its not magic --
just code. Many of the topics that we will cover, especially the Concurrent Managers,
are found in $FND_TOP.
-
8/8/2019 ERP Support Tips
38/61
A Look at $AD_TOP
Most of the other utilities used by the Oracle Fin-Apps DBA and which we will discuss
in detail are found in $AD_TOP. Of particular interest are the /bin and /sql
subdirectories. You will find the following executables in $AD_TOP/bin: adaimgr(autoinstall for the installation or upgrade of the software), adpatch for administering
patches, and adadmin which is a menu driven utility for maintaining both the Oracle
Applications database and software. Many of these utilities in turn call other $AD_TOP
utilities.
Output Directories of the Oracle Applications
There are several directories where output is written. These directories require
routine cleanup and maintenance. The jobs that are run from the concurrent
managers create both log and output files (reports). The location of these files
depends upon the variables $APPLCSF, $APPLLOG, $APPLOUT, and $*TMP. The
$APPLLOG and $APPLOUT variables are typically set to "/log" and "/out", respectively,
but they can be set to other values. The location of these subdirectories depends
upon the value of $APPLCSF.
If $APPLCSF is set to a directory, then all of the product modules (AR, AP, PO, etc.)
output will go to a common "log" or "out" directory. The typical setting, though, is tonot have $APPLCSF set to any value. When this is true, then the output for the
product modules defaults to the specified "log" and "out" directories under the
corresponding product module top. For example, concurrent manager jobs run from
an Account Receivables responsibility would find the logs and data output in
$AR_TOP/log and $AR_TOP/out.
I would advise you to not set $APPLCSF. This way, you can more easily find and
categorize your output. There is generally a lot of output, anyway, and you can stressthe inodes by having too many files. Be sure that your temporary directories, such as
$APPLTMP or $REPTMP get cleaned up and don't fill up a file system. Note that any
SYSADMIN responsibility output will go in $FND_TOP/log or $FND_TOP/out.
-
8/8/2019 ERP Support Tips
39/61
3. The Concurrent Managers
One of the most attractive features of the Oracle Application Software is the
concurrent manager. Basically, this is a batch job scheduling system for running
reports and background tasks. From the concurrent managers you can manage queues,
-
8/8/2019 ERP Support Tips
40/61
workshifts, access and security, job priorities, job log and output, job automation,
and assorted job compatibility (or incompatibility) rules.
This feature is one of the key areas that can consume much of the Oracle Fin-Apps
DBA/SYSADMIN time. To find more complete instructions on how to setup and use theconcurrent managers and the jobs that they run, refer to the AOL Reference Manual.
For the purposes of this presentation, we will discuss major concepts in setting up the
managers, performance issues, and other general tips and suggestions.
Basic Tuning of the Concurrent Manager
We go back to the age old concepts of computer tuning and balance loading for OLTP
versus Batch Processing. OLTP (on-line transaction processing, or "real-time"
computing) is where you have end-users doing their work on the screen needing quick,
real-time results -- especially if they are servicing clients in person or on the phone.
These requests need to be completed as soon as possible as to not disrupt the
business and revenue flow! An example of these transactions may be your Order Entry
people in customer services.
Note: Just because an on-line transaction submits a job to the concurrent manager (or
the "background") that this does not necessarily qualify it as a "batch-processing" job.
On the other hand, batch-type jobs can afford to be completed at a later time than
when initially entered. They usually can be grouped together (batched) and processed
outside of the normal business hours. Examples of these type of reports could be
financial reports, summary reports, end-of-day processing, etc. Some jobs are
required to assist the on-line transaction processing but can be batched (like a sales
forecast or open ticket report) but needs to be completed prior to the days activities,
rather than after.
You may be in a 7x24 shop where OLTP is always a priority. Balancing your OLTP
versus batch jobs may be a little more complicated. Still, your objective is to reduce
the impact of the non-critical resource hungry jobs against the OLTP transactions. The
batch jobs will just have to work when OLTP demands drop. You do this by managing
queues, workshifts, priorities, incompatibility rules, and . . . end-user training or
awareness.
-
8/8/2019 ERP Support Tips
41/61
-
8/8/2019 ERP Support Tips
42/61
Example script for starting the managers:
#strmgr.sh
dateecho "Executing strmgr.sh script ..."
echo "Starting Concurrent Managers ..."
startmgr sysmgr="apps/fnd" mgrname=prd sleep=20
#exit
Actually, I would advise you to use symbolic parameters for the APPS password
instead of hard coding it. The "sleep" parameter tells the internal manager to search
fnd_requests every 20 seconds for new requests, rather than the 60 second default.
The internal log file will be called prd.mgr (typically found in $FND_TOP/log). There
are other parameters available, too, such as the debug option. Consult your manual
for more details.
Example script for stopping the managers:
#stopmgr.sh
date
echo 'Stopping Concurrent Managers ...'#The following is one command line
$FND_TOP/bin/CONCSUB apps/fnd SYSADMIN 'System Administrator' SYSADMIN WAIT=Y
CONCURRENT FND DEACTIVATE
#End of command line
ps -ef | grep LIBR
date
echo 'Concurrent Managers Stopped'
exit 0
Notice that stopmgr.sh does not run a command line executable to directly stop the
managers. Instead, it submits a concurrent job via the concsub utility. The WAIT
parameter tells the job not to process any further until all the managers have
shutdown before proceeding and eventually exiting the script.
-
8/8/2019 ERP Support Tips
43/61
Debugging Concurrent Manager Errors
Look for errors in the logs. The internal manager's log file will usually be in
$FND_TOP/log (see previous discussion on defining log and out directories) defaulting
to std.mgr or named as you specified in the command line parameter, mgrname=. Theinternal manager monitors the other queue managers. You will see the startup,
shutdown, print requests, and other information in this log. You may also find errors
as to why the internal or subsequent slave managers could not start.
All of the other managers have dedicated logs, too. They are preceded with a "w" or
"t" followed by an identity number, such as w139763.mgr. Each queue will have one of
these log files. You can see individual jobs and associated request ids in each of these
files. You can review error messages, too. Occasionally, a job will fail and take the
manager down with it. The internal manager will sense that the queue is down and
restart it on the next pmon cycle.
Suggestion: We will discuss purging of the fnd_concurrent_request table and
associated log and output files, later, but I would make this suggestion. Purge these
manager files frequently (daily) so that you can easily perform a search on "error"
when trying to debug concurrent manager errors.
Kick Starting Dead Managers
Sometimes you may encounter difficulty in starting either the internal concurrent
manager or the other slave queues. Consult the log files for error messages and take
appropriate action to resolve the problem. If you are unsuccessful, then enter the
"verify" command in the concurrent manager screen to force the internal manager to
read and initiate the target number of queues specified. If that doesn't work, try to
deactivate or terminate the managers, then restart them. If you have trouble bringing
them down, you may have to perform a "kill" on the background process. You canidentify the managers with "ps -ef|grep LIBR" command. If you still encounter
problems, make sure that there aren't any processes still tied to the managers. If you
find any, kill them.
If you still encounter problems, then the statuses are probably improperly set in the
tables. For example: You may see the error in the internal std.mgr log file stating
-
8/8/2019 ERP Support Tips
44/61
that it was unable to start because it has already started! You have verified that
there are no "FNDLIBR" background processes. The problem is that the tables have
improper statuses. You will have to clean up these tables. Here are some queries. I
put them into scripts and keep them handy for when the time arises because the
statuses are not that easy to remember.
Reset the concurrent queues:
UPDATE fnd_concurrent_queues
SET running_processes=0, max_processes=0;
Remove any completed jobs: (optional)
DELETE FROM fnd_concurrent_requests
WHERE conc_process_status_code='C';
Set jobs with a status of Terminated to Completed with Error: (optional)
UPDATE fnd_concurrent_requests
SET status_code='E',phase_code='C'
WHERE status_code='T';
Delete any current processes:
DELETE FROM fnd_concurrent_processes;
I have listed these in descending order of frequency that I have had to use them.
There is a paper available from Oracle Support which describes these and more.
Purging Concurrent Manager Logs and Output
The concurrent managers create several table entries and file output in the /log and
/out directories. You should purge these frequently to reduce excessive table growth
and fragmentation, and avoid performance degradation of the concurrent manager
processes. You should also decrease the used space on your disks from old log and
report files. This will also relieve stress on the inodes from a large number of files.
Under SYSADMIN, setup a reoccurring report called "Purge Concurrent Request and/orManager Data". There are several parameters, but I typically try to setup two jobs. 1)
One job for "Manager" data -- that's the concurrent manager log files typically found in
$FND_TOP/log. I set the frequency to daily, and have it purge down to one day. 2)
Another job for the "Request" data -- this is for all other modules outside of the
SysAdmin responsibility, such as AR, PO, GL, etc. I typically try to keep only one
week's worth of data out there on the system. Your needs and capacity may vary, so
-
8/8/2019 ERP Support Tips
45/61
set accordingly.
This purge process does two things: 1) Deletes rows from the fnd_concurrent_requests
tables, and 2) Deletes both the log and output files from the associated $XX_TOP/log
or /out directories. If for any reason the file delete did not complete, but the table
data was purged, then you will need to manually purge the output files from the /logand /out directories. This can happen if the privileges were incorrectly set, or you
replicated a copy of the production database to your development environment, or
the file system was not mounted, etc.
Purge Signon Audit Data
This is another purge report, like above. Only this purges the signon audit data which
records every login to the Oracle Applications. Set the frequency and retention equal
to that of your request data purge.
Performance Tuning of Concurrent Manager Jobs
What has been described thus far is balancing job throughput. Yet, the jobs
themselves may be in need of sql tuning or resolving problems in the database. We
won't go into detail of sql tuning -- that is a typical skill set that should be handled by
the IT staff. What I want to discuss here are ways of identifying and classifying
problems within the Oracle Applications.
FND Tables Can Speak Volumes
The concurrent manager is just a scheduling system that keeps track of jobs,
parameters, scheduling information, and completion statuses of every job submitted.
By querying these tables, you can learn much about the patterns of your site,
including performance trends.
I strongly suggest that you become familiar with these tables and develop reports
against these tables. Some of the most useful tables are the fnd_concurrent_% tables.
Things to look for are which jobs are run, how many times executed, completion
status (especially "errors"), and run times for these jobs.
Where Can I Get Help?
-
8/8/2019 ERP Support Tips
46/61
When it comes to looking for established help on tuning your concurrent manager
jobs, there is an excellent reference that can never be exploited enough... the white
paper on Managing the Concurrent Managers or ("How to Herd Cats") by Barbara
Matthews. See proceeding papers from the OAUG Fall 1997 convention. Thispresentation has been very useful to me. I have modified several of these scripts to
my clients' needs.
My favorites are daily errors, daily and weekly hogs, the min/max reports, and the job
schedule report (note that these are not the exact names that you'll find). Here are
some ideas on how to use these reports.
The daily errors report shows me every job that completed with an error status. I
review these from time to time to look for trends. The error could be caused by a bug
(so then you open a tar and look for an existing patch). The problem is usually
attributed to user error, such as bad parameter input. But don't let the error go on --
it could be an indication that the user needs some training or other help (you'll know
the user name because it provides the request id number that allows you to view all
the details and log of the job -- if you haven't purged it, yet).
The hog reports flag every job that exceeds some set time threshold (such as 20
minutes). It also sets a submission time range, such as weekdays 6:00 AM to 6:00 PM.The idea here is that we are looking for jobs with very lengthy completion times
running during standard operating business hours (the prime OLTP window). If a job
exceeds this limit, then it is taking resources away from your OLTP users and should
either be 1) tuned to reduce execution time, or 2) moved to the "batch" processing
window or queue during the off-hours.
Before you tune a "hog", I would suggest that you see if a performance patch has been
issued on this program. Many times there is, and this can save you the trouble oftuning it -- and crossing that dilemma of introducing a customized piece of code into
your environment.
The min/max reports can be modified to sort the jobs in ascending or descending
order based upon the execution time or number of times executed. This report takes
some interpretative skills. For example, lets say that you identify the job that has the
-
8/8/2019 ERP Support Tips
47/61
longest execution time... say 4 hours! At first glance, this looks like a sql tuning
candidate. A closer look, though, reveals that the minimum time it took to run the
job was only 2 minutes -- and that the average time for 300 submissions in one day
was only 5 minutes! Now, what you have is some sort of exception. You should cross-
reference this job to the "hogs" report -- it should be there. Or, see if it was in theerrors. By finding the request id of this aberrant job you can review the details. You
may find that the parameters specified a much larger data set, or was incorrect, or
many other things.
If you finally determine that the job was correctly submitted and that the rest of the
evidence points to an optimized sql code set, then you have probably encountered a
"non compatible" job! In other words, the job is fine by itself, but may suffer
drastically due to contention with other jobs run at the same time. With more
detective work, you should strive to find which jobs it is incompatible with and
rearrange queues, priorities, or compatibility rules to ensure that they will not run
simultaneously.
The job schedule report shows all the scheduled jobs that keep submitting
themselves, automatically. There are a few things I look for, here. One is the sheer
volume of jobs that may be scheduled -- are they really needed? Often these jobs get
scheduled, then forgotten, and are no longer useful. Or is it a batch oriented job that
runs during peak time that should be rescheduled to a more practical time slot? Or isthe owner of the job still an employee? I have seen many "ghost" jobs that were once
submitted by users who have left the company -- but their reports still run,
regardless!
One last item about scheduled jobs. See if the jobs are overlapping themselves. When
specifying the resubmission parameters, you can have a job start at a fixed time, or
reschedule at a time interval calculated when the jobs starts, or reschedule at a time
interval after the job completes. I often find jobs scheduled to resubmit some timeafter the first job starts, like every 15 minutes. Maybe the job used to complete in 5
minutes. Yet, as the database grows, the job may now be taking more than 15
minutes to complete. Hence, it submits the same job, again, when the first one hasn't
even completed, yet! Then this can cause contention degrading the performance time
of both jobs and the cycle repeats itself and degrades further and further. I would
suggest that you schedule jobs to resubmit themselves on a time delay after the
-
8/8/2019 ERP Support Tips
48/61
previous job completes!
I Didn't Know Those Scripts Were There!
There are some other existing scripts which may be of benefit to you, but I must firstput in a very strong disclaimer: CAUTION: Do not blindly run these scripts without
analyzing their purpose, impact, and possibly consulting with Oracle Support! Test
them in your development environment, first.
I must confess that I do not fully understand why all these files are here. I suspect
that many are used in the installation/upgrade and use of the applications. I have not
found deliberate documentation of these scripts, other than what I can see in some of
the script text. Yet, I have used some of these scripts to great satisfaction -- or at
least to learn about the usage of certain tables. These scripts are in $FND_TOP/sql.
The ones of interest for the concurrent managers are afcm*.sql and afrq*.sql. These
range from reports on the concurrent managers, locks, gridlock, etc. You can also find
useful scripts in $AD_TOP/sql, too. Again, BE CAREFUL!
Things to Avoid Regarding the Concurrent Managers
These following tips seem to be common sense, but I am still amazed at how often I
see these abuses and misunderstandings, so I will mention them...
Use of the Reprint Option: Do not allow your users to run jobs multiple times in order
to recreate the same output. They can view it offline or do a reprint on a previously
run job. There are other third party tools, too, that give more flexibility in viewing
and formatting the outputs, too.
Use Query Enter to Find Your Jobs: If a user cannot see their job on the immediate
screen, then scroll down or enter a query to further define the job that they arelooking for. I have seen sites where the user couldn't find the job they submitted on
the first screen, so they would submit it again!
Whoa! on the Refresh Screen: It is very, very common to have your whole company
just hitting that refresh key on the concurrent request screen in an effort to see their
job go into the queue or its completion status -- especially when performance is
already suffering! But this only contributes to the problem! This is one of the most
-
8/8/2019 ERP Support Tips
49/61
common queries possible. For one, the internal manager scans this table at whatever
the pmon interval (the concurrent manager pmon, not to be confused with the Oracle
background pmon process) where it scans the fnd_requests table for the next set of
pending jobs to be processed.
Discourage Multiple User Logins: Multiple logins by the same user to get more work
done is often contributing trouble to an over researched system. Sometimes this is
unavoidable because the user wears different "functional" hats and must view
different screens/data within multiple responsibilities. Some also find it annoying to
login and navigate to particular work screens, and then keep idle sessions active until
they need them. Try to educate your users that they consume resources (memory,
CPU, etc.) every time that they do this. In the newer NCA versions, navigating to
different screens and responsibility areas will be made easier via shortcuts and should
help to eliminate this abuse.
Eliminate Redundancy of Similar Jobs: Users often submit the same job multiple times
in the same time frame, distinguished only with minor changes to the parameters.
These jobs hit the same tables over and over again and can even create locks and
resource conflicts among themselves. Many times they would find the overall
throughput to be better if they single threaded the jobs one after the other. This can
be managed by user education or by the SYSADMIN single threading the queue or
placing incompatibility rules that limit the same program to run with itself.
Another variation of this problem is having different users running the same or similar
jobs at the same time. It may be better for the SYSADMIN to schedule these jobs to
resubmit themselves in the concurrent manager at predetermined intervals and take
away the ability for the end-users to submit the jobs, themselves. This should reduce
the frequency and burden on the system, yet allow the users to still have the jobs and
processes run in a timely manner for their use.
-
8/8/2019 ERP Support Tips
50/61
4. Utilities for Maintaining the Applications
There are many tools and utilities available to you for maintaining and upgrading your
applications. Some are well documented, others are more mysterious. I'll describe
some of the major utilities. Note that most of these utilities are in $AD_TOP/bin or
$FND_TOP/bin.
Patching with the Adpatch Utility
This utility is used for applying patches that you receive from Oracle Support. When
you uncompress the patch from Oracle, you will get at least one driver file
(patch.drv), a readme.txt, and new code to patch your applications. The patch.drv
file is read by the adpatch utility and performs a multitude of tasks. It basically
checks the versions of your code to make sure that the patch code is more recent,
-
8/8/2019 ERP Support Tips
51/61
moves the new code to the proper directories while making a copy of the original
suffixed with an "O", updates the library file, links object code to make new
executables, compiles or generates code, and logs all of its activities.
All of these tasks are performed by other utilities in the $AD_TOP/bin directory,including adlib*, admvcode, adrepgen, adrelink, adfrmgen, etc. Look at the log file
for your adpatch task and you will see the utilities that were called. These utilities
will match up to the operative key words in the patch.drv file.
You should ALWAYS review the readme.txt file prior to applying a patch. You need to
verify that the patch is going to do what you intended, and see if there are any other
manual tasks to perform either before or after applying the patch. If sql scripts are to
be performed, the patch.drv usually moves the sql script to the directory but does not
execute it. The readme.txt file will direct you to run adpatch again, and direct you to
specify the db*.drv file as the patch input. This will execute the sql scripts.
Patching Suggestions and Tips
Always make a backup of the directories that will be affected prior to applying a
patch -- a patch can be a very nasty thing to rollback! Even though admvcode will
make a backup copy of most files suffixed with a capital "O", it is not very reliable in
rolling out a patch. This is because sometimes patches are "bundled" up with otherpatches and the affected files may be patched multiple times. Hence, the backup file
"O" may actually be a backup of the backup! Without your own backup, you cannot
rollback to the original.
Regardless of what the readme.txt file says, to really be certain which files and
activities can occur, look at the patch.drv and db*.drv files. If it isn't in the patch
driver file, then it isn't going to happen.
When prompted for the patch log file, do not take the default name "adpatch.log". I
recommend that you use the same patch/bug number, such as .log. This enables you
to quickly review the results of your patch without stumbling through reams of
previous patches.
A running log of patches applied resides in $APPL_TOP/applptch.txt. DO NOT DELETE
-
8/8/2019 ERP Support Tips
52/61
THIS FILE! It is invaluable when determining which patches have been applied, when,
and what actually happened in the patch. I am finding this file even more critical in
considering the NCA upgrades and possible Y2K upgrades -- operations where you may
have to lay down a new baseline of the applications and reapply your patches to
recreate your current configuration!
To learn more about the patching process and several other utilities, investigate the
log and patch.drv files. Many of these utilities can be run by themselves. You may find
use (at least understanding) of these utilities.
Adadmin Utilities
This is an interactive menu available to you to maintain several aspects of your Oracle
Applications environment. The menu divides into two categories: Database and File
maintenance.
The database screen gives you options for creating or maintaining database
structures, data, or privileges. Most of these activities are encountered during
installation or upgrades. You