![Page 1: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/1.jpg)
Eric Westfall – Indiana University
James Bennett – Indiana University
ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE
![Page 2: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/2.jpg)
KUALI RICE @ INDIANA UNIVERSITY
• Integrated with 26 enterprise applications
• Hosting 50 eDocLite applications
• Over 400 different KEW Document Types
• 24 x 7 x 365 availability
• Approximately 99.8% uptime in the last year (including maintenance)
![Page 3: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/3.jpg)
EXPANDING IMPLEMENTATION
2003 2004 2005 2006 2007 2008 2009 20100
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
9000000
10000000
11000000
12000000
13000000
Documents RoutedRequests GeneratedActions Taken
![Page 4: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/4.jpg)
USAGE
2003 2004 2005 2006 2007 2008 2009 20100
10000
20000
30000
40000
50000
60000
70000
80000
Number of Unique Users
# of Unique Users
![Page 5: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/5.jpg)
ARCHITECTURE AND INFRASTRUCTURE
• Red Hat Enterprise Linux on VMware
• Load balancer with four clustered Kuali Rice standalone server instances
• Java 6
• Tomcat servlet container
• Shared file system mount for attachments
• Oracle 11g database
![Page 7: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/7.jpg)
MAINTAINED BY
• A team of 3 FTE, who do…
• Production Support
• System Maintenance
• Upgrades
• Integration Projects
• Application and eDocLite Development
![Page 8: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/8.jpg)
PRODUCTION SUPPORT
• Areas of concern:
• Database resources
• Service bus
• Ingestion
• Clustering/Load Balancing
• Tools we use
![Page 9: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/9.jpg)
DATABASE RESOURCES
• Recently updated to Oracle 11g
• Clients primarily use embedded
• Concerns:
• Connection pool limit
• Database blocking
• Client proxy accounts
• Export/import job for refreshing test environments
![Page 10: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/10.jpg)
MANAGING THE SERVICE BUS
• Bad service endpoints
• Clean shutdown of client applications
• Monitoring/maintaining the message queue
![Page 11: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/11.jpg)
CLUSTERING/LOAD BALANCING
• Horizontally scalable
• Session affinity
• Session replication required for full failover
• Unique endpoints per machine
![Page 12: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/12.jpg)
TOOLS - DYNATRACE
• Performance monitoring
• Error reporting
• Drill down into requests
• Take memory/thread dumps
![Page 13: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/13.jpg)
DYNATRACE EXAMPLE
![Page 14: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/14.jpg)
TOOLS - DATABASE
• Reporting:
• Stuck documents
• Database blocking
• Cleanup
• User options table
• Message queue
![Page 15: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/15.jpg)
SYSTEM MAINTENANCE
• Rice customization/patches
• Environment configuration
• Load Testing
• Functional testing
• Data migration
![Page 16: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/16.jpg)
RICE CUSTOMIZATION/PATCHES• Two options:
• Override Spring bean (ex: IU Group Service)
• Override a particular file
• Process
• Copy source
• Make necessary changes
• Build the standard Rice source
• Overlay customized files
• Deploy or distribute changes
![Page 17: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/17.jpg)
ENVIRONMENT CONFIGURATION
• Java environment managed by a J2EE group
• Build process deploys:
• Application code
• Settings/security files
• Most Java applications share the same build process
• http.url configuration parameter
![Page 18: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/18.jpg)
LOAD TESTING
• Using Apache JMeter
• Perform standalone and coordinated load tests
• Load testing transactions include
• Document search
• Action list
• Taking mass actions
• Viewing route logs
• Creation and routing of documents
![Page 20: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/20.jpg)
FUNCTIONAL TESTING
• Uses Selenium
• Runs nightly through Bamboo
• Functional test suite includes:
• Most eDocLite applications
• Maintenance documents (group, rule, etc)
• Document search
• Action List/Preferences
![Page 21: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/21.jpg)
DATA MIGRATION
• Database job to move production data into testing environments
• Use the “export” feature
Dev Test Pre-production Production
![Page 22: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/22.jpg)
UPGRADING RICEProcess:
1. Bring in new Kuali Rice foundation jars
2. Make changes to the IU version of Rice which includes:
• Updating customizations
• Merging "patched" code
• Produce IU Rice client jars
3. Perform DB updates to test environment
4. Bring up standalone testing environments
5. Work with clients to bring in IU Rice client jars and update code
6. Testing before making the change to production
![Page 23: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/23.jpg)
INTEGRATION
• Client application configuration
• Keystore Administration
![Page 24: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/24.jpg)
CONFIGURING CLIENT APPLICATIONS
• Spring files (ex KIM overrides)
• http.url configuration parameter
• Document handler URL
• Namespace (if appropriate)
• Roles/permissions
• Security file deployment
![Page 25: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/25.jpg)
KEYSTORE ADMINISTRATION
• Keystore needed for KSB security
• Keytool commands
• genkey
• selfcert
• export
• import
• list
![Page 26: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/26.jpg)
KEYSTORE ADMINISTRATION
Items to consider in a production setting:• Keep certificate expiration dates in mind
• Testing and prod keystores should not be the same
• New applications must:
• Create a certificate
• Include a copy of the KR certificate in their keystore
• Give their certificate to be included in the KR keystore
![Page 27: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/27.jpg)
EDOCLITE DEVELOPMENT
Process:
1. Request comes in for a new eDocLite
2. Meet to review requirements
3. Develop initial eDocLite
4. Test and make changes until ready for prod
5. Ingest eDocLite into production
![Page 28: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/28.jpg)
IU EDOCLITE EXAMPLES
![Page 29: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/29.jpg)
IU EDOCLITE EXAMPLES
![Page 30: Eric Westfall – Indiana University James Bennett – Indiana University ADMINISTERING A PRODUCTION KUALI RICE INFRASTRUCTURE](https://reader036.vdocuments.us/reader036/viewer/2022062516/56649e5e5503460f94b58595/html5/thumbnails/30.jpg)
EDOCLITE MANAGEMENT• Currently have 50 eDocLite apps in production
• Reasons for changes to existing eDocLites
• Annual changes due to nature of form
• Addition of a new field
• Change to drop down contents
• Change to field validation