auto memory defined
TRANSCRIPT
Auto Memory DefinedDOCUMENT AUTHOR:
Shannon Heustess
DOCUMENT OWNER:
Shannon Heustess
DATE CREATED: Tuesday, March 23, 2010LAST UPDATED:
PROJECT: PowerCenter UpgradeCOMPANY:
Best Practices Documentation
Best Practices and Standards........................................................................................................................................... 1Memory Tuning Process....................................................................................................................................................3
Default Memory Allocation for Session Configuration 3Default Memory Allocation Background 3Default Memory Allocation Process Defined 3
How to Calculate Memory Cache for Objects................................................................................................................... 5How do I use the Informatica Sizing Calculator 5How do I get the values to plug into the calculator 9
Lookups 9Aggregator 9
Sorter 10Ranker 10
2 Informatica Velocity – Best Practices Documentation
Best Practices DocumentationMemory Tuning Process
Default Memory Allocation for Session ConfigurationDefault Memory Allocation BackgroundPowerCenter 811 brought forth the concept of Auto Memory when dealing with any object that interacts with memory.
Examples include: DTM Lookup Cache Joiner Cache Aggregator Cache Sorter Cache XML Target
This concept was hailed as a step in the right direction. Developer productivity would be lowered because it allowed developers allocate a memory pool to those objects and not have the developer spend time calculating optimized memory allocations per object.
Default Memory Allocation Process DefinedAt run time a complex set of steps will occur to allocate memory at the object level. The following will explain the process:
Determine Memory Pool SizeThis is determined by the following session’s properties:
Maximum Memory Allowed For Auto Memory Attributes Maximum Percentage of Total Memory Allowed For Auto Memory Attributes
Maximum Memory Allowed For Auto Memory Attributes – can be either a numeric or alpha numeric value that allocates a dedicated amount of memory for all objects set to Auto. You can Append KB, MB, or GB to the value to specify other units.
Maximum Percentage of Total Memory Allowed For Auto Memory Attributes – is a numeric value that relates to the maximum percentage of memory allocated for automatic cache.
The auto memory pool size will be the lesser of these two values.
Informatica Velocity 3
Best Practices DocumentationDetermine Memory Pool Math32 bit Operating SystemMaximum amount of free Memory 3600 MBMaximum Memory Allowed For Auto Memory Attributes Value: 512 MBMaximum Percentage of Total Memory Allowed For Auto Memory Attributes: 5% - 180 MB
Even though we have specified 512 in the Maximum Memory Allowed For Auto Memory Attributes Value attribute the Maximum Percentage of Total Memory Allowed For Auto Memory Attributes resolves to 180 MB and is the lower number. This will determine our total memory pool.Total Unit AllocationNow that our memory pool size has been defined we need to determine our memory unit allocation. This is conducted by looking at the mapping and then determining the number of objects that use Auto Memory, applying a unit cost to that object.
1. LKP, AGG, RANK, JNR index cache - 1 unit 2. LKP, AGG, RANK, JNR data cache - 2 units 3. SORTER - 8 units 4. XML Target - 10 units
The DTM will sum the total unit costs and then divide the sum into the Auto Memory pool to determine the size of each file.
Total Unit Allocation MathMapping contains the following objects1 Lookup – Unit Cost of 31 Aggregator – Unit Cost of 31 Sorter – Unit cost of 8
Total Unit Cost = 14Auto Memory Pool = 180 MBAmount of Memory per unit = 12.85 MB
Total Maximum Memory allowed per objectLookup index = 12.85Lookup data = 25.7Aggregator index = 12.85Aggregator data = 25.7Sorter File = 102.8
If the memory goes beyond this these sizes for each individual object then we will spool the file to disk. It is still considered best practice to analyze each object and calculate the object’s data and index size independently. This will prevent memory cache starvation due to the object mix of a mapping.
4 Informatica Velocity – Best Practices Documentation
Best Practices DocumentationHow to Calculate Memory Cache for ObjectsInside of Workflow manager Informatica has provided tool to calculate memory per object. This eliminates the need to remember per object the complex rule to determine the optimized value for memory.
How do I use the Informatica Sizing Calculator
1. Open workflow manager
2. Connect to the desired repository housing the session needing tuning.
3. Supply your user name and password
4. Click the connect button
Informatica Velocity 5
Best Practices Documentation
5. Right Click on the folder containing the session
6. Select the Open option
7. Right click on the workflow containing the object
8. Select Open
9. Right click on the desired sessions
10. Select Edit
6 Informatica Velocity – Best Practices Documentation
Best Practices Documentation
11. Select the mapping tab
12. Navigate to the transformation section
13. Select an object that would use memory for caching
14. Navigate down to the Objects Data Cache Size and Objects Index Cache Size Properties
15. Click down arrow in the value field to bring up the tool.
Informatica Velocity 7
Best Practices Documentation
16. Enable the calculate button and then specify the number of unique rows in the case of a lookup.
17. Enable the calculate button and then specify the number of groups in the case of a aggregator.
18. Enable the calculate button and then specify the number of rows to be sorted in the case of a sorter
8 Informatica Velocity – Best Practices Documentation
Best Practices Documentation
19. Enable the calculate button and then specify the number of groups, the number of ranks is specified inside of the mapping in the case of a rank.
How do I get the values to plug into the calculatorLookupsYou can determine the number of unique rows in a lookup using the following methods:
SQL query – Have either the lookup generate the query it will use at run time or use the sql override and query against the production DB and get the counts.
Look inside of the session log (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Run Performance Indicators when testing out a mapping (Note this is will only tell you the number at run time and limited to the set of test data you are running)
AggregatorYou can determine the number of groups in an aggregator using the following methods:
SQL query – If the aggregator is far enough up stream in the mapping and is not affected by any active transformations then you may be able to generate a SQL statement that mimics the group by functionality to determine the number of groups.
Look inside of the session log (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Run Performance Indicators when testing out a mapping (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Informatica Velocity 9
Best Practices Documentation
SorterYou can determine the number of rows in a sorter using the following methods:
SQL query – If the sorter is far enough up stream in the mapping and is not affected by any active transformations then you may be able to generate a SQL statement that will provide you with a count of the number of rows entering into the sorter.
Look inside of the session log (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Run Performance Indicators when testing out a mapping (Note this is will only tell you the number at run time and limited to the set of test data you are running)
RankerYou can determine the number of rows in a rank using the following methods:
SQL query – If the rank is far enough up stream in the mapping and is not affected by any active transformations then you may be able to generate a SQL statement that SQL statement that mimics the group by functionality to determine the number of groups.
Look inside of the session log (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Run Performance Indicators when testing out a mapping (Note this is will only tell you the number at run time and limited to the set of test data you are running)
Once you have this number you can then apply a growth factor based on needs. 0 – 40% growth factor should allow for proper sizing without having to constantly having to monitor for size adjustments. You will then plug this number into the respective field and specify you want to calculate the value for the data, index or both by selecting the check boxes. Once the values are calculated you would then review the settings and click the OK button.
10 Informatica Velocity – Best Practices Documentation