oaug whitepaper april 2014 alternative to engineered systems final

An Alternative to Exadata for Large Scale Enterprise Applications

Cliff Burgess, Director of Information TechnologyGentex Corporation

Musa Ramadhani, Oracle Applications DBAGentex Corporation

http://www.gentex.com/

Disclaimer

The following is intended to outline the general process followed by Gentex Corporation in deploying infrastructure related to the Company’s Oracle enterprise applications. It is not intended to provide instructions, guarantees or sizing calculations for the deployment of such applications. Gentex Corporation is not endorsing any particular vendor, technology or deployment strategy. The information discussed is for illustrative purposes only.

Collaborate 14 Copyright © 2014 by Cliff Burgess P a g e 2

Table of ContentsTable of Figures............................................................................................................................................................3

Overview......................................................................................................................................................................4

Background..................................................................................................................................................................4

Decision Making Process..............................................................................................................................................5

Journey to “End-state” Architecture............................................................................................................................5

Improve Performance & Stability of Existing Infrastructure.................................................................................6

Hardware & Storage Upgrade: Prototype............................................................................................................7

Hardware & Storage Upgrade: “End-state”..........................................................................................................8

Ongoing Efforts to Optimize Performance & Stability..........................................................................................9

Performance Efforts for Advanced Planning Server...................................................................................................10

Conclusion................................................................................................................................................................. 11

Table of FiguresFigure 1: Original Infrastructure...................................................................................................................................6Figure 2: AWR Overview of Beginning Infrastructure..................................................................................................6Figure 3: AWR Stale Data Base Stats............................................................................................................................7Figure 4: Prototype Configuration................................................................................................................................7Figure 5: "End-state" Architecture...............................................................................................................................8Figure 6: Disaster Recovery Architecture.....................................................................................................................9Figure 7: Initial Decentralized APS Configuration.......................................................................................................10Figure 8: Virtualized APS Configuration.....................................................................................................................10


Overview After nearly four years of operating on the Company’s original hardware, Gentex Corporation determined to upgrade the Company’s infrastructure. With Oracle’s recent acquisition of Sun, more emphasis was being placed on Engineered Systems by Oracle. In-memory Applications have also been recently introduced but require deployment on Engineered Systems or Oracle’s proprietary hardware. Gentex Corporation had traditionally deployed on commodity hardware. The Company faced a decision to either change its current deployment strategy or to update its current deployment with more modern infrastructure based on known technologies. Engineered Systems, such as Exadata, promise improved performance and manageability but are new to the ERP market and have a significant cost of entry.

During the analysis to determine the appropriate direction, Gentex Corporation uncovered a number of issues that resulted in performance opportunities, indicating the need for an upgrade. While many of these issues were resolved during the infrastructure upgrade, there was also a need to focus on business process and system maintenance to achieve optimal performance at the lowest total cost of ownership. Gentex Corporation did an exhaustive study comparing engineered systems to commodity hardware. This included a detailed proof of concept (POC) on both platforms to determine the appropriate infrastructure. The Company also leveraged outside consultants to improve business processes related to database management and ongoing efforts to maintain a focus on performance.

Ultimately, an investment in new infrastructure was deemed necessary but new infrastructure without an ongoing effort focused on performance and stability was determined to be insufficient. Gentex Corporation undertook a broader project to look at system performance as an ongoing effort to deliver improved results to the business while minimizing total investment in infrastructure and software. The Company continues these efforts today as part of a continuous improvement project focused on delivering highly-preformat, reliable infrastructure to the enterprise.

BackgroundGentex Corporation (GNTX) is a Zeeland, Michigan-based manufacturer of automotive automatic-dimming rearview mirrors, automotive electronics, dimmable aircraft windows, and fire protection products. The Company’s manufacturing operations are US based and require business systems to support operations. Distribution facilities support operations in North America, Europe and Asia for the Company’s global customer base. The Company implemented Oracle’s E-Business Suite going live with 12.0.6 on July of 2009. Gentex uses a large footprint of Oracle applications including ERP components for Finance, Procurement, Manufacturing and Warehouse Management. Gentex has also deployed Oracle’s value-chain planning products in a decentralized manner leveraging Oracle’s Advanced Planning Server (APS).

Due to the proprietary nature of the Company’s manufacturing processes, Gentex Corporation leverages a Manufacturing Execution System (MES) for shop-floor management. Manufacturing transactions are collected in the MES applications and directly interfaced into Oracle Manufacturing and Inventory Applications. Being in the Automotive Industry, the Company leverages Oracle’s Release Management and processes EDI transactions on a daily basis for inbound customer orders. A majority of Material Handling transactions are processed via Radio Frequency (RF) devices. The Company has a Production environment of more than 4 terabytes with an annual growth rate of nearly 1.5 terabytes. Some of the largest transactional tables grow at more than a million records per week.


Gentex Corporation opted to decentralize Oracle’s Planning Server and segregate the highly-intensive Memory Based Planner (MBP) engine with Oracle’s Value Chain Planning suite. These products sit outside ERP on their own infrastructure. The Planning server supports the Company’s deployment of Advanced Supply Chain Planning (ASCP) with Constraint Based Optimization (CBO), Advanced Planning Command Center (APCC), Production Scheduling (PS) and Inventory Optimization (IO).

Decision Making ProcessFor the investment in hardware, storage and the associated software licensing for Oracle Database and Middleware products; Gentex Corporation elected to work with multiple vendors in a benchmarking effort. The most processor, memory and storage intensive process consuming the most resources was the focus for the benchmarking exercise. Oracle, with the Exadata platform, took a data dump of the planning data and ran the plan in one of Oracle’s Exadata competency centers. To perform a similar benchmarking exercise, Gentex Corporation partnered with its existing hardware and storage vendors to build a prototype environment representing a commodity hardware environment to conduct similar tests. The cost of infrastructure was an important factor in the decision making process but software licensing was also a consideration. Oracle claimed that commodity hardware would require as much as 40% more CPUs than engineered systems to deliver similar performance. If this were proven to be true, software licensing would be a significant component in the cost of any new infrastructure investment.

Upon completion of the benchmarking exercise, it was determined that both solutions could deliver equal performance at similar configurations. Given the narrow focus of the benchmarking exercise, it was not possible to determine the ultimate configuration required to run the entire enterprise at an acceptable level of performance. Given the Company’s existing investment in commodity hardware and the lack of any experience with engineered systems, the decision was made to maintain the existing architectural approach and upgrade the components. However, Gentex Corporation needed to determine appropriate sizing before committing to a license agreement with Oracle. To minimize total cost of ownership, an extensive project was needed to determine the appropriate sizing and configuration of the “End-state” architecture.

Journey to “End-state” ArchitectureFollowing the Company’s decision to build its “End-state” architecture on commodity hardware and storage, a road-map was developed. The roadmap consisted of the following steps to reach the desired “End-state” architecture. First, ongoing efforts were made to improve the performance and stability of the existing infrastructure. Second, given the capacity of the existing infrastructure; Gentex chose to move to the POC environment to determine appropriate sizing. Once on the new infrastructure, the Company began efforts to implement the “End-state” architecture. Finally, Gentex continues its efforts to optimize performance and stability of its infrastructure to provide optimal performance at the lowest total cost of ownership.


Improve Performance & Stability of Existing InfrastructureThe original infrastructure configuration details are noted below.

CPUo 8 Per Boxo Model AMD Opteron 2222o Clock speed 3.0 Ghzo Total Cluster CPU 16

RAMo Maximum Capacity 32 GBo Installed 32 GBo Total Cluster RAM 64

Databaseo RAC 11gR1o DB Version 11.1.0.7

Applicationso EBS Release 12.0.6

Figure 1: Original Infrastructure

This configuration resulted in a number of performance and stability areas for focus. High wait events resulted from both concurrency and I/O. Concurrency issues indicated the need for improvements in RAM and processors. Production and non-production environments shared the same storage resulting in high I/O. Dynamic Memory Management was enabled resulting in an inefficient deployment of RAM and high overhead due to the constant reallocation of memory.

Figure 2: AWR Overview of Beginning Infrastructure


A number of routine maintenance items were not being addressed, with stats jobs taking too long to run. In addition, the lack of a number of routine data purges (for example AQTBL) resulted in performance issues. Insufficient time was allocated to performance management. The DBA Team began reviewing AWRs on a frequent basis to identify poor performing SQL and taking action to resolve issues. Below is an example of an Oracle Enterprise Manger chart during an event that resulted from outdated statistics on a single table.

Figure 3: AWR Stale Data Base Stats

As an intermediate step, Gentex added a third database node to the existing 2-node RAC cluster to provide additional capacity while preparing for the move to the commodity based infrastructure leveraged in the POC. In December of 2012, Gentex Corporation took the first step toward fully implementing its “End-state” architecture.

Hardware & Storage Upgrade: PrototypeAs an intermediate step toward its desired “End-state”, Gentex Corporation leveraged the hardware and storage used in the POC. The Company built a prototype that it could leverage to determine adequate sizing and performance metrics. The legacy platform did not provide an adequate bench-marking for sizing. Given Oracle’s estimates regarding processors required for commodity platforms, as compared to engineered systems; it was necessary to determine an accurate baseline for sizing.

DB CPUo 16 Per Boxo Model Intel E5-2643o Clock speed 3.3 Ghzo Total Cluster CPU 32

DB RAMo Maximum Capacity 256 GBo Installed 160 GBo Total Cluster RAM 320



Figure 4: Prototype Configuration


The guiding principles for the Company’s “End-state” architecture were reliability, scalability and performance at the lowest total cost. Toward that goal, the Applications Tier was virtualized. This allowed for a more robust deployment of concurrent managers to better segment traffic. CPU clock speeds were maximized and RAM was increased significantly. Since Oracle database licenses are processor based, clock speeds and RAM could be increased with no increase in license cost. Note that the number of processors was doubled to provide adequate headroom during the sizing. The prototype environment provided an adequate baseline for performance and sizing. During this time, the Company updated the Oracle database to 11gR2 in March, 2013.

Hardware & Storage Upgrade: “End-state” After completing the sizing and performance benchmarking and upgrading the Oracle database, Gentex Corporation deployed its desired “End-state” Architecture in July, 2013. The Applications tier remained virtual and the Database Tier remained physical. A 3-node RAC cluster was preferred over a 2-node cluster for increased reliability and performance. Total CPU count was increased to 24 from the original 16; however, this was a reduction from the prototype configuration and far less that the number of CPUs originally estimated by Oracle during the POC.

DB CPUo 8 Per Boxo Model Intel E5-2643o Clock speed 3.3 Ghzo Total Cluster CPU 24

DB RAMo Maximum Capacity Per Box 256 GBo Installed 160 GB Per Boxo Total Cluster RAM 768GB



Figure 5: "End-state" Architecture


Building upon the above infrastructure, Gentex Corporation extended its deployment to include a Disaster Recovery (DR) site. Gentex Corporation viewed this as another opportunity to improve performance without substantially increasing cost.

The Standby (DR) environment was leveraged to off-load key processes to reduce the workload of the Production Environment.

Backups were moved to the Standby instance to reduce overall load on the Production system. In addition Extract, Transform and Load (ETL) processes for the Company’s data warehouse were also moved to the Standby instance.

Moving these processes provide more flexibility with the timing and frequency of these processes without impacting performance of the Production system.

Figure 6: Disaster Recovery Architecture

Ongoing Efforts to Optimize Performance & StabilityWith new infrastructure in place, routine maintenance jobs have been scheduled in Oracle’s Enterprise Manager. Gentex Corporation implemented a Code Review Process for all new development. The DBA Team reviews all code prior to testing to ensure it adheres to code guidelines and any performance issues are resolved prior to deployment. AWRs are reviewed on a routine basis and are used to set the priorities for the DBA Team. Finally, the updated infrastructure has provided the platform to further expand efforts to provide improved performance to critical business processes. The first priority of Gentex Corporation was to reduce the overall plan time of its nightly ASCP run.


Performance Efforts for Advanced Planning ServerAs previously noted, Gentex Corporation decentralized the Advanced Planning Server (APS) prior to undertaking its infrastructure project. Decentralization improved performance and stability as the Material Based Planner (MBP) process is very memory intensive. Having that process conflict with ERP resources created stability and performance issues in a centralized deployment. Upon decentralization, the Company had a very basic infrastructure for its APS platform.

APS was first decentralized on a shared install platform. The database, applications and the concurrent managers all ran on a single physical server. This architecture quickly proved to be insufficient as countless failures were encountered when the MBP Concurrent Program would "run-away" and consume all the Memory available on the server. This would cause the server to be non-responsive and render the entire instance unusable. A hard server reboot was the only remedy to bring the server back into service.

The MBP process is very dynamic and dependent on the run-time variables submitted by the business user during the plan set-up and execution. While Gentex Corporation has a Development and Test platform for APS and makes every attempt to test these plans prior to submitting them in Production, MBP plans could still have unintended consequences due to a number of factors including the parameters of the Concurrent Process, the data available in the Production environment, profile options, etc. As a result, Gentex Corporation chose to modify the initial configuration of the APS infrastructure.

Figure 7: Initial Decentralized APS Configuration


During the hardware upgrade in March of 2012, Gentex Corporation virtualized APS providing the capability to separate the database tier from the application tier. The Company then went a step further and separated the applications tier into two parts; Applications OAF/Forms and Concurrent Managers. This architecture enhanced instance stability by segregating load and protecting the database from runaway MBP processes. After the segregation, the Company was able to leverage virtualization and reallocate server resources to optimize stability and performance. Since this architectural update, the Company has not encountered a runaway MBP processes.

Virtualization provides the ability to over-provision the server’s resources. Initially, APS was deployed with other applications on the physical host. Over-provisioning led to degradation in performance, resulting in increased plan runtimes. The architecture was modified to continue leveraging virtualization but only the APS server resided on the physical host.

Eliminating the over-provisioning of resources and dedicating 100% of the physical resources of the server to APS resulted in a 20% reduction in plan run-time.

Figure 8: Virtualized APS Configuration

Now that the infrastructure for APS has been proven, Gentex Corporation continues to look for opportunities to continue to reduce ASCP plan run-times. Shortening these run-times has significant business benefit as it enables manufacturing to be more responsive to the daily changes in demand often encountered in the Value Chain of a Tier1 Automotive Supplier. Following a 20% reduction in plan run-times after segregating the APS virtual server, Gentex focused on I/O.

MBP generates a significant number of both read and write operations during the planning process. To minimize the wait time, Gentex Corporation is currently investigating solid-state storage presented at the server level. A 700 gigabyte device is cost effective and can provide significant performance improvement. Gentex has taken that 700Gb device and allocated 600Gb to the concurrent Manager and 100Gb to the database.

Phase I testing focused on the temp files and log files generated by MBP during the plan run. Writing these files to local (solid-state) disk provided and additional 10% reduction in the MBP plan run-time. Adding re-do logs provided an additional 10% reduction while adding Undo Table Space actually increased plan run-time by nearly 10%. While adding re-do logs again reduced wait events, the Undo Table Space increased wait events. Gentex continues to test various scenarios to determine the optimum configuration to maximize performance.

In addition to read/write events, network throughput is also a variable to consider. Network packet sizes between the Concurrent Manager and the APS Database are being expanded. Another opportunity is to expand the packet sizes between the ERP Applications Database and the APS Database. These opportunities are currently being explored. As a result, performance metrics were not available at this time.


ConclusionPerformance continues to be an exercise in chasing bottlenecks among processors, memory and storage. Options such as Virtualization, Solid-state Disk, In-memory Applications and Engineered Systems provide new tools to address performance issues; but add complexity to the evaluation process. Gentex Corporation chose a path based on technologies that the Company understood and is focusing energy and resources in optimizing the Company’s resources and business processes to maximize utilization of those technologies.

Poor business processes related to database maintenance, data management, and change management will eventually overwhelm any infrastructure. A balanced approach focused on business process and maximizing performance was found to deliver the best results at the lowest total cost of ownership. Engineered Systems, In-memory Applications and other advanced options may present opportunities for further improvements in performance but Gentex Corporation made a decision to optimize existing technologies before abandoning platforms for unknown options. Today, the Company operates a broad portfolio of Oracle Applications while providing its users with acceptable reliability and performance. Efforts continue to deliver improvements while maximizing the utilization of the Company’s current investment.


oaug whitepaper april 2014 alternative to engineered systems final

Documents