- e-vitals... · web viewindependent verification and validation. final report – including...

EVITALS PROJECT

Independent Verification and Validation

Final Report – Including Activity for August – October, 2007

Submitted November 15, 2007

Submitted by

10555 Montgomery Northeast Building 2 Suite 110

Albuquerque, NM 87111

“Quality is the Difference”

New Mexico Department of HealthEVITALS ProjectFinal IV&V Report – August/November 2007

Executive Summary:

This Independent Verification and Validation (IV&V) report for the EVitals project provides a final update on the status of this very special and important project. It also provides final observations of the project in its entirety.

The project team has been successful in implementing the birth and death registration and issuance system. The project team also implemented an interface to the Social Security Administration, and added an unscheduled task to the plan for the interface for Office of the Medical Investigator (OMI). The birth record conversion was officially removed from the EVitals project via a Change Order, but the Project Manager, Bette Perry was retained to continue to coordinate the death record data conversion that was completed November 11, 2007. This completed every scheduled task for the EVitals project.

This report will provide a summation of what should be considered a “Model Project.” Key factors for its success are simple:

There was a strong, interested and active Steering Committee who took their role seriously and stepped in to expedite authorizations and provide resolution when requested. Additionally, they had knowledge of processes in order to provide guidance to the team on how to proceed in various situations.

There was an experienced, excellent Project Manager who came with a comprehensive set of project management ‘tools’ in order to effectively guide, develop, monitor and report on the project.

There was an extremely supportive Bureau Management team who was directly involved in every aspect of implementing the system. Their willingness to actively participate in such a positive manner served as an exemplary model for their project staff.

There were team leaders and members who were the most consistently dedicated people ever observed, and who gave 150% effort to their every task.

There was a tremendous amount of support from ITSD who stepped in to assist with problems at critical times and who also provided excellent support for testing activity, Help Desk activity, network, hardware setup and conversion development, just to name a few.

The selected vendor had a good product in its ‘youth’ and was willing to dedicate as many staff as it took to assure the software was customized, installed and functional in a timely manner. The exception was the DTS load functionality which was several weeks overdue near the end of the project. This in particular caused the death record data conversion to run slightly over schedule. Earlier versions were more ‘buggy’ than was preferred, but there finally came a point where there were only minor issues to fix in version patches. The Birth Module software had significantly fewer issues than the Death Module. Remarkably,

Bency & Associates, LLC 2


considering the complexity of edits and rules the system contains, the vendor delivered a good, basic product that was made better through the testing and QA/QC process, issue tracking and additional enhancements provided at no cost to the State.

Last, the working relationship and cooperation between all parties was admirable, professional and sprinkled with humor. The team dynamics were the most positive ever observed.

The challenges on the EVitals project were external to the project team itself. The primary reason the project went to the final moments was the arduous procurement process for every item purchased for the project, e.g., hardware, software and contract staffing. This was further exacerbated by the funding issues that were at the root of many challenges. The difference between the estimated cost of the project and allowable funding was one million dollars. This caused delays that impacted deadlines and resulted in the loss of grant funds earmarked for items on the project. Team members had to serve dual roles by integrating the project demands into their full-time responsibilities resulting in a staff that spent two years working extra hours.

In addition, the SHARE implementation schedule was an unknown when the Evitals Project Plan was developed. This major change impacted some acquisitions as did other procurement-related issues that existed prior to SHARE and are expected to continue. A case in point is the fact that a software invoice approved and submitted in July was not paid to the vendor until October. The vendor met their obligations and should have been compensated in a timely manner. This is a systemic issue for the entire State and these observations are not meant to pass judgment on any particular area. Any process requiring more than four signatures will take additional time with a greater risk of documentation getting lost between stops (this has been observed as quite common across all departments, not just DOH.)

Observations on weaknesses include the following, but are not limited to this list:

There was a lack of general communication and coordination of processes.

There was a lack of procurement procedures and input on proper contract vehicle/templates for the procurement activity. It was also observed that a lack of communication regarding procedure or template changes resulted in re-initiating procurement processing. Since this process crosses departmental lines (individual departments, DFA, DoIT) and is complicated at best, the current solution for project planners is to build in extremely long periods of time for procurement activities. With the current process it is recommended that one should begin the procurement four to six months in advance of when the item/service is needed or up for renewal.

System performance issues were experienced due to network and infrastructure issues. The DOH network is saturated. The ITSD staff has done an admirable job with ‘band-aids and baling wire’, but improvements will be ongoing and expensive for some time to come. Keep in mind that there have been many installations of sophisticated systems within DOH that rely heavily on real-time



data transfers with increased communication demands across all points in the state and beyond. While not knowing the ultimate, long-term goals of DOH or the State, cost allocation for ITSD is a positive move and will hopefully result in additional, adequate funding to upgrade and expand their communications network and hardware.

A detailed discussion by the project team regarding lessons learned is provided in Attachment D. This document is very important in that it is an honest and complete, retrospective review of the accomplishments, strengths and challenges the project faced and states it better than could have been done by this reviewer. Please refer to that document for a more detailed analysis.

Despite all obstacles the project team persevered and delivered a high quality product of great benefit to the citizens of New Mexico. One can now obtain birth and death certificates at sixteen offices with an additional seven offices issuing the certificates for funeral homes. This is added with the other interfaces that were developed, and the major death record conversion was recently completed.

Lastly, Bency & Associates is pleased to have been a part of this most important and successful system implementation. Without question this project has been one of the best managed and documented projects this IV&V Specialist has ever had the privilege to review, which helped guide it through all the challenges and roadblocks that came along the project path. It is with great pleasure (and a twinge of sadness) that I provide this final report on a model project, thus fulfilling the IV&V agreement.

Final Summarized Analysis of Project Status

In August, the focus was on completing the OMI interface that was implemented without any major problems, and the death record conversion that spanned all weekends for processing through mid-November. The good news is that final reports on the conversion are very positive, and all records are now contained in the DAVE system.

For this final review, there is not much improvement to report in the remediation of system performance issues in the production environment, though purchase of parts and needed items is underway. The Bureau continues to host the production application, and it is not possible at this time to turn on replication. The good news is ITSD has set up to issue self-signed certification until the SSL feature could be purchased. The final word is that the SSL is in and is scheduled for installation within the next week.

For the final status update, the discussion of details for accomplishments and technical issues will be provided in the IV&V Analysis of Project Status section below. This information provides who, how, and where project information resides and responsibilities for each as the project closeout commences.

Risks Noted:

The final risk items for the EVitals project now dwindle to maintenance support funding.



Activities Completed this Period

Attended the EVITALS Project Steering Committee meeting for August, 2007. Reviewed minutes for six project status meetings. Attended the final Steering Committee meeting held on November 9, 2007.

(Many thanks to Dr. Armitage for her special apple cider and fancy plastic glasses for the ‘final toast’.)

Reviewed the Steering Committee meeting minutes. Reviewed five ISD project status reports provided by the PM. Reviewed the project budget and dashboard submitted to the OCIO for August

and September. Reviewed the ITSD Application Support plan. Reviewed the Lessons Learned document provided by the project team. Reviewed the Conversion meeting notes and Conversion test plan, issue logs

and status minutes from August through mid-November. Prepared the final monthly report. Participated in multiple update sessions on project status with the Project

Manager throughout this final reporting period.

Significant Findings and Pertinent Recommendations:

New Findings and Recommendations

There were no new findings or issues during this reporting period.

Analysis of Budget Status

Table 1 : Budget Status as of September 10, 2007 taken from project lifetime for the final budget review.

Total Budget

Project To-Date

Estimate to Complete

$1,800,000 $1,536,636 $263,364

Funds have been encumbered from last fiscal year in order to pay license/maintenance costs. Additionally, Bette Perry will be updating the expenditures with all final invoices in the closeout report for the project.

Statistics on Prior Findings and Recommendations

Table 2 provides a statistical summary of previous findings/issues (excluding current period findings). Findings/issues and recommendations from prior reporting periods are contained in Attachment D: IV&V Issues Log.



Table 2: Statistical Summary of Prior Findings/Issues

Potential Impact # Open # Closed# Severe 1 7

# Moderate/Mild 1 10

The open issues are few and should be monitored post-project until resolved. DOH Network/EVitals server configuration (system redundancy) DOH toolset for stress testing systems prior to implementation Adequate funding for maintenance.

END of EXECUTIVE SUMMARY



IV&V Periodic Review:

Analysis of Project Status

The project team made great strides during this final reporting period to meet all project objectives. It was a challenge as issues arose in the conversion process and contributed to the conversion going to the deadline. Some issues were systems related such as data base limits set too low or not properly adjusting automatically for the volume of records being loaded. Other issues were procedural such as the invoking of applications in the conversion process (‘triggers’ that cause the next application in the processing flow to begin or calls to external modules to begin). However, all of these issues were resolved, and it was simply a matter of continuing the load process first to the conversion data base while the new version is installed instead of directly to the production database, which is the ultimate goal. All was completed smoothly, and the final issues from an IV&V perspective have been resolved with the exception of system fail-over/redundancy and stress test mechanisms. Following is the final update/disposition of all project related tasks as provided by Bette Perry for the most complete status possible while still meeting the report deliverable deadline.

Status of Project Deliverables

The EVitals Project has successfully implemented the birth and death registration and issuance system, an interface for Social Security Administration, and an additional project deliverable, an interface between the DAVE™ application and the Office of the Medical Investigator’s X-Files system.

All major tasks in the scope of work have been completed. The conversion of legacy birth data was formally removed from the scope of work several months ago via a Change Request.

Data Conversion As of this writing, approximately 100% of the legacy death data has been converted in the NM Production environment.

There are two primary reasons that the death data conversion was not completed until mid-November: 1) Late delivery by the vendor of the DTS load functionality, which was promised before the end of July, was received approximately the third week of August. The vendor agreed to develop this back-end load functionality after it was determined that the DAVE application job scheduler could not handle the large file sizes.

2) Issues (database not expanding automatically, and some application trigger issues) arose after the conversion of several loads into the production database. These issues were resolved.



Due to critical bug fixes in DAVE™ release 7.1.4 and new birth module functionality in 7.2, these releases had to be in the production environment by the end of September. Since the window to complete the data conversion by the end of September was missed, the plan is to temporarily suspend conversion loads into the production environment until 7.1.4 and 7.2 are deployed into production. These releases were deployed to production at the end of September.

Contingencies to Complete Data Conversion

Kemtah Contract 4445 for project management services for the EVitals Project could not be renewed as Kemtah was not awarded a new State Pricing Agreement (SPA) for project management. Two contingency plans were developed to continue the conversion of legacy death data and close out the project:

1) A Purchase Order for staff augmentation for Bette Perry was routed through the system. This PO would be through another vendor who does have an SPA for project management.

2) If the staff augmentation purchase order was not approved, Debbie Dickerson was prepared to place another DOH project management resource with the EVitals Project to complete the data conversion.

Contingency #1 was preferred and utilized with Bette remaining until the end of the project.

In order to maintain momentum on the data conversion, the EVitals Project continued to do conversion loading in the conversion database against release 7.1.4/7.2 until the project determined that the system was ready to resume loads to the production environment.

Status of Payment of Vendor Invoices

A purchase requisition for VitalChek Network, Inc. in the amount $190,743.00 was submitted on July 20, 2007. This purchase order is being funded from carryover funds from the computer enhancement budget. As of this September, a purchase order had not been issued. This purchase requisition is for payment of the remaining FY08 deliverables for VitalChek. The vendor cannot submit invoices until the purchase order is completed. (Reference Contract 05/665.0009.0030 A1, Vendor #42160). This issue was finally resolved in October.

Kemtah submitted a final invoice for project management services for the month of September. The purchase order for these services is in place. Reference Contract #4445, Vendor #49201

Bency will submit a final invoice for IV&V services. The purchase order for these services is in place. Reference Contract #2041, Vendor #55484.

Status of Technical Issues



SSL - Permanent SSL solution – A temporary solution (self-signed certificates) was implemented prior to the rollout of the birth module in July. Permanent SSL certificates have been received from the vendor and are awaiting installation by DOH ITSD in November.

Replication between fail-over and production – Replication between the fail-over and production environments was turned off early in the EVitals implementation when the users began experiencing severe system performance issues. Due to the fact that there was not a timely resolution to the performance issues in the Runnels Building, a temporary production environment was established in the Vital Records Building. The former main production environment in the Runnels Building was taken off-line pending resolution of the performance issues. System performance in the Vital Records Building is far superior to that experienced when the production environment was in the Runnels Building. A third environment, testing/training/conversion was brought up in the Runnels Building. (These environments, testing/training/ conversion, experience the same slow response time as the former main production environment experienced in the Runnels Building.)

An update from Rod Skiver was received on September 25, 2007. The parts required for the infrastructure work in the Runnels Building have been received from the vendor.

Ultimately, at some point in time the main production environment will be moved to the Simms Building. Several months ago when the project began experiencing severe system performance issues, ITSD was going to move the servers to the Simms building. Unfortunately, the power was not adequate for the Dell 6850 database servers. Per information from Rod Skiver on September 25, 2007, a power conversion is scheduled for the Simms Building in late October. No additional servers will be moved to the Simms building until that work is done. There was no final status as to whether this was accomplished.

Once the performance issues for all of the environments are resolved, replication between the production and fail-over environments will need to be turned back on.

Upgrading of Internet Connections to Local Public Health Offices is being conducted.

Transition to Support Documents

The EVitals Support Plan was written as a collaborative effort between the EVitals Project and ITSD. The most recent copy of this plan is provided in Attachment F. As new information becomes available, the document should be updated. The EVitals Project Manager added additional VitalChek support information to the document on September 25, 2007 and has placed additional documents regarding deployment of DAVE™ releases on SharePoint in the EVitals section.

A printed copy will be given to Donna Dossey as part of the project close-out documents. An electronic copy was provided to Bency & Associates, and an electronic copy will be provided to Deborah Dickerson, Chief, Implementation Services Division.



Project Library

The EVitals Project has, from the beginning, maintained a Project Library:

\\10.136.5.84\E-Vitals

Documents from the Project Library are being archived on CD’s as of September 27, 2007 and will be given to Donna Dossey, EVitals Project Director. The documents in the Project Library will not be printed due to the volume of documentation. The documents being archived will be in excess of 4 GB. A CD containing all files received for IV&V and all reports submitted for this contract will be provided to either Donna Dossey or Deborah Dickerson for final disposition.

Lessons Learned

The EVitals Project Lessons Learned document has been written and is stored in the project folder entitled Project Closeout Documents. It is also provided here in Attachment D for closer study. This is an important document for preparation of future projects. A printed copy will be given to Donna Dossey, EVitals Project Director, and electronic copy to Deborah Dickerson, Chief, Implementation Services Bureau. An electronic copy was provided to Bency & Associates.

Project Closeout Report

The Project Closeout Report will be filled out by the EVitals Project Manager as far as it can be based upon the status of payment of invoices as outlined above. The document will be saved in the EVitals Project Library, Project Closeout Documents folder. A printed copy will be given to Donna Dossey, EVitals Project Director and electronic copy to Deborah Dickerson, Chief, Implementation Services Bureau.

Final IV&V Report

This document serves as the final IV&V report from Bency & Associates.

Formal Project Closeout

Formal project closeout will be performed when vendor invoices have been paid and the EVitals Project receives permission to officially close the project. It is unknown when this will occur.

Analysis of Steering Committee Meeting Minutes

A final Steering Committee meeting was held November 9, 2007. Following are the highlights of that session.

As of the meeting, 77% of the conversion records were loaded to production.



Everything was completed over the weekend of October 10, 2007, and the team worked Monday, Veteran’s Day, to finalize the process. The conversion is officially finished. A document containing all of the record statistics is provided in Attachment F, EVitals Conversion Data Load Statistics. Approximately 300,000 records were converted with only one that was not loaded in a batch for the year 2003. It was noted by Bette Perry that ITSD put forth a tremendous effort to make this task successful and was given the credit for its scheduled completion. Much appreciation for the conversion team was discussed in detail.

Preliminary meetings are occurring for the Birth Conversion. A load tool will be in place by the end of the year. It was noted that the same conversion team will be in place for the Birth conversion.

The OMI Interface was in full production in October. Several hundred records have already been processed, with no issues. It is working more smoothly than originally thought and the OMI loved the application! It has tremendously improved the working relationship with OMI and the Bureau.

It was unanimous that the entire committee and team appreciated the BT grant that made this new system possible.

The Outreach effort has been much more successful than originally anticipated with numerous physicians coming on board to utilize the system. Still, the bureau is planning to target the ‘big players’ (those physicians who issue 90% of the death certificates) and continue to market the new system.

ITSD officially ‘owns’ any further technical issues with one final meeting and updates to the technical systems manuals.

The last task is to get the vendor payments closed out. The meeting ended with a toast to a job well done with the help of Dr. Karen

Armitage bringing apple cider to officially end a most remarkable project.

The August Project Steering Committee meeting was held on Friday, August 17, 2007. The meeting for September was cancelled and not rescheduled in time for this final report. Following are the topics discussed during the August session:

Birth Module

The statewide birth module rollout to approximately 80 new users at 31 birth hospitals occurred on July 30, 2007. With the exception of a couple of technical issues, the rollout was very smooth. In fact, Bette reported that this was one of the smoothest “go-live” events she has seen. This is in great part due to excellent training team members: Dorothy Holland, Gaspar Chacon, Mary Anne Escudero, Renee Valencia, Melinda Roybal, Donna Dossey, Pete Varela, Karen Gary, Edward Beshears, and Marcus McBride. Training was provided in Santa Fe, Albuquerque, Roswell, and Las Cruces during the week immediately prior to the “go-live”.

The team expressed appreciation for the tremendous support from the DOH ITSD and DOH Help Desk in resolving technical issues and providing first level support for new clients. The first day, the Help Desk fielded 35 calls. The number of calls fell off later in the week as expected. From reviewing the HEAT tickets, it appears there are fewer calls coming in on the birth module than the death module.

Death Module



Outreach to Physicians

The Outreach team has been continuing the outreach to physicians. A training session for the hospital physicians at St. Vincent Hospital is scheduled. Dr. Karen Armitage met with George and Dorothy on August 2nd to talk about the challenges of getting the physicians on board. The Outreach team has also talked with the Outreach Director from Oregon. He recommended they use the “it’s the right thing to do” strategy which Karen agrees with. Karen shared the message she gives resident physicians about the importance of prompt and accurate completion of the death certificate. Compelling reasons include thwarting terrorists and identity thieves by ensuring quick updates to national and international data banks and by avoiding additional pain and suffering of families at the time of a loved one’s death. Karen also said that she supports the approach of targeting the “heavy users” first and make a personal visit to the physician’s office. She also suggested considering recruiting doctors who are “natural leaders” in communities first. Karen shared a physician contact in each region who could help the Outreach team identify these physicians and offer help if they need a speaker to draw a picture of how disastrous it can be if death certificates are late, inaccurate, or both.

Funeral Homes

Formal regional training sessions for the funeral homes were completed in April. There were a few funeral homes that came on to the system later. All funeral homes throughout the State are now using the DAVE™ application.

Data Conversion

Work on the data conversion of legacy death data continues. The DTS package for back-end load was received from VitalChek this week. The data conversion is several weeks behind the projected completion date due to having to wait for the DTS functionality. It was not feasible to attempt the data conversion through the DAVE™ application job schedule due to the small file size limitations. With the DTS load tool, it is expected to be able to move forward quickly with large conversion load testing. All are very grateful for the support and direction that Lori Hewig’s staff (Gary Hernes, Kim Ingraham, and Mike Baldonado) have provided to the EVitals Project. Mike has done an incredible job with the required programming, recoding and assisting in programmatically cleaning up data.

OMI Interface

Development of the OMI interface is complete. It is in QA at VitalChek. OMI is providing sample test files to VitalChek. The OMI interface will be loaded into our test environment on August 22nd or 23rd. Paul Bachand of VitalChek will conduct on-site training and testing of the OMI interface at OMI on August 24, 2007.

Project Transition Documentation

The EVitals Project met with DOH ITSD this week to review the first draft of the transition support document and review outstanding technical issues and remaining project tasks.



The final version of the document is due next week.

Risks/Issues

Working under a tight project schedule (legacy death data conversion, OMI interface).

Late delivery by VitalChek of the back-end load (DTS) tool for conversion of legacy death data.

Funding for production support (3-year hardware maintenance contracts were purchased and software maintenance contracts through 2009 are in the VitalChek contract). The EVitals Project has requested carryover of part of the FY07 funds to FY08. An Expansion Request was submitted for FY09 software maintenance costs.

Ensure continued availability of IT resources to assist with QA of new releases and ongoing technical support. Dorothy Holland will continue working with the Bureau after Bette Perry leaves at the end of September.

Technical

Prior to the rollout of the birth module, the following technical tasks were accomplished:

Load balancing of web servers and applications servers in temporary production environment. New birth users are using a new URL, https://evtest2.health.state.nm.us/EVITALS_PROD_TMP/.

Funeral homes and internal Vital Records staff are continuing to use https://evtest.health.state.nm.us/EVITALS_PROD_TMP/ .

This type of load balancing is working quite well so far. It has assisted in troubleshooting and can redistribute the load if necessary by having users log in to a particular URL.

Temporary SSL (self signed certificates) solution was implemented prior to the birth module rollout. Vendor is waiting on a copy of the Purchase Document. As soon as the PD is received by the vendor, the permanent certificates can be delivered.

Remaining Technical Tasks

Permanent SSL solution – The update from ITSD is that as soon as the vendor gets a copy of the approved purchase document, DOH will be able to get the permanent certificates.

Permanent resolution for performance problems in the main production environment – Leif has been assigned to head up this effort. (The EVitals Project was not able to move servers to GSD because the power required for the Dell 6850 database server was not available at GSD.)

Turn replication back on between production environment and fail-over – The original configuration for the fail-over was one web server, one application server,


https://evtest.health.state.nm.us/EVITALS_PROD_TMP/

https://evtest2.health.state.nm.us/EVITALS_PROD_TMP/


one database server and an integration server. In order to truly replicate the production environment, the Bureau should have two web servers, two application servers, and a database server. The integration server originally planned for BizTalk is no longer needed, so it can be utilized. This leaves the need to purchase just one additional server to complete the environment.

The original plan was to use BizTalk for reporting and for implementing interfaces. The EVitals Project purchased one BizTalk license and was going to share resources within another PHIN project (E-Reporting) for the other BizTalk environments. The projects did not sync up schedule-wise and VitalChek implemented a reporting database for use with the DAVE™ application. The OMI Interface was implemented as a web service interface.

Project Closeout – November

Bette Perry’s contract will be extended through a new vendor through the end of at least November, 2007 in order to complete the close-out for the EVitals Project.

END OF IV&V REPORT -


New Mexico Department of HealthEVITALS ProjectFinal IV&V Report November 2007

Attachment A - EVITALS August Project Steering Committee Meeting Notes

The Steering Committee convened on August 17, 2007. Minutes from this meeting are provided here. The final Steering Committee meeting for September was cancelled and not rescheduled.

Attachment B – August and September 2007 EVITALS Project Status Minutes & ISD Reports

Attachment C – EVitals Project Dashboard

Following is the final, available budget information for the project, for August and September, 2007.

15


Attachment D - LESSONS LEARNED, EVITALS PROJECT

(Reviewer’s Note: The following findings by the project management and team have full agreement by this reviewer. It is a constructive and informational discussion that can be used to improve process, policy and procedure for future projects.)

Project Mission:

The mission for the EVitals System Project is to acquire and implement an integrated web-enabled electronic birth and death certificate registration and issuance system. The new system will replace the current non-integrated birth and death systems with a web application that can be accessed through a standard web browser, automating many of the functions that are currently handled manually. The web-enabled solution will follow the IT standards proposed by National Center for Health Statistics (NCHS), National Association for Public Health Statistics and Information Systems (NAPHSIS), Centers for Disease Control (CDC), Public Health Consortium, and Social Security Administration (SSA).

Accomplishments:

1. The EVitals Project has successfully implemented a birth and death registration and issuance system throughout the State of New Mexico. All funeral homes (approximately 70) in the State of New Mexico and 33 birth hospitals are entering data into the DAVE™ application.

Both the birth and death module implementations included a “Pilot”. The death module was rolled out to funeral homes by regions. The birth module was implemented with all birth hospitals coming on to the DAVE™ application at the same time.

2. Citizens of New Mexico can now obtain birth and death certificates at 16 local registrar offices. An additional 7 local registrar offices are issuing death certificates for funeral homes.

3. The birth and death registration and issuance systems are now one integrated system, the VitalChek DAVE™ application.

4. The EVitals Project has successfully implemented an interface between the DAVE™ application and the Social Security Administration (OVS).

5. The EVitals Project has successfully implemented an interface between the DAVE™ application and the Office of the Medical Investigator (OMI) X-Files System. The OMI interface was added to the original scope of work.

6. The EVitals Project is successfully converting legacy death data from the legacy death application into the DAVE™ application.

16


Project Strengths:

Active project participation and support by Steering Committee. Monthly Steering Committee Meetings. Very supportive Executive Sponsors. Highly committed and actively engaged Project Director. Highly committed project team. Every project team member with the exception of

the Project Manager, Training Coordinator, and Outreach Coordinator had regular full-time responsibilities in their respective roles in the New Mexico Bureau of Vital Records and Health Statistics.

Excellent rapport between Contract Project Manager and EVitals Project Director, Sponsor, Steering Committee, and project team

Excellent communication between the Project Director, Project Manager, and project team.

Weekly (or more) project team meetings. Program area “ownership” of the project. Every project team member’s voice is heard at Project Team Meetings. Every

opinion has value. Strong training coordinator and training team. Hands-on training sessions were

provided by a variety of methods (class-room training, one-on-one training, Live Meeting training sessions. Training sessions for funeral homes were presented in every region. Training sessions for birth hospitals were presented in Santa Fe, Albuquerque, Las Cruces, and Roswell.

Outreach Coordinator has worked closely with the Training Coordinator to recruit funeral homes, birth hospitals, and physicians. The Outreach Coordinator has assisted in coordinating and scheduling training for our external stakeholders.

Contingency Planning Excellent relationship with the vendor, VitalChek.

o Successful negotiations for needed application functionalityo VitalChek sent two of their top IT staff to DOH at no charge to assist DOH

in troubleshooting performance problemso VitalChek helped us keep the project moving by providing access to test

sites when the purchase of our servers was delayed. Excellent working relationship with DOH ITSD technical staff (network, servers,

applications group, DBA group). ITSD does an incredible job supporting the applications with so few staff available.

The Applications Group has worked hand-in-hand with the EVitals Project as members of our Conversion Team.

Project Challenges: Fiscal Challenges

o Projects crossing fiscal years.o Getting budgets approved and carryover funds available for use by the

project.o Contracting process.

Delays in getting contracts through the system cause schedule delay issues and cost the projects money.

17


Example: The contract for VitalChek was not finalized until May 2005, several months after the Project Manager was brought on board.

Example: Project Manager contract renewal delays. Project Manager would have to go off site until contracts renewals were in place. This occurred three times during the life of the project.

o Procurement process. The procurement process was so slow in getting the purchase

documents through the system that the project lost part of its BT grant funding.

The project was not allowed to purchase the server brand recommended by the Vendor (Hewlett Packard) because that brand was not on the Save Smart list. We had to purchase Dell servers instead. The vendor did not have experience working with Dell servers.

The Project could not start development of the birth module until budgets were in place, encumbrance documents/ purchase orders were approved even though the vendor has a multi-year contract.

o Implementation of a new accounting system (SHARE) in the middle of the

project.

o Under-funding of the project. The original estimate for this project was $2.8 million. The project was funded at $1.8 million. Requests for additional funding were not approved.

Availability of IT resources when neededo EVitals Project Manager also served as QA Lead for this project because

QA resources in DOH ITSD are so limited. Also, due to the under-funding of the project, it was not feasible to fund a QA position.

o Competing demands for IT resources by many projects

System performance issues with the DAVE™ application when located in the Runnels Building.

o Infrastructure issues within DOH

o Inadequate power in the Simms Building to host the Dell 6850 database servers.

The lack of load/stress testing tools made it difficult to predict how the DAVE™ application would perform under different load scenarios. The lack of these tools as predictors slowed down the rollout of the death module.

18


Other Contributions to Lessons Learned from Project Team Members:

The time commitment for the program area that is necessary to do a project of this magnitude.

Frequency of application releases from the vendor has presented challenges. The need for workflow and process changes. The need to develop workarounds for application issues/limitations. Projects of this type need an assigned DBA. The challenges of being the first state to implement the DAVE™ birth module

and the first state to do a death data conversion from a legacy application to the DAVE™ application.

Learning to find ways to provide quality training with limited resources (due to under-funding of the project).

Resistance of some external users (primarily physicians) to coming on to the DAVE™ application because the “old way” was working for them.

19


Attachment E - IV&V Issues Log

Below are historical findings observed/recommended over the course of the IV&V. Findings are coded by red, yellow and green. Two findings remain open at the end of the project.

No. Issue Action(s) Taken Status1 The hardware procurement

needs assistance to move forward quickly, in order to meet the mid-July 2007 Pilot deadline.

Recommend the Steering Committee, using their experience with the procurement process, try to identify the best way to ‘hand carry’ it through to prevent further issues. This issue is on the critical path for commencement of the Pilot.

Closed

2 Current documentation is not available due to the absence of the Project Manager.

Missing documentation has been provided this month.

Closed

3 The system test plan was not completed on schedule and since the PM was assigned responsibility to develop this plan, it is not clear when the plan will be completed.

The test cases have now been modified for the new software release. They will be sent out prior to the August testing for review.

Closed

4 This is not a new finding, but particular emphasis to aggressively ‘push’ the hardware acquisition along as quickly as possible. Any executive intervention that could speed up the process and keep it a highly critical priority is necessary.

It is recommended that this documentation be monitored daily with status calls to the appropriate party in that stage of the process. All measures should be used to get funds paid and equipment delivered as soon as possible.

Closed

5 Testing is scheduled for the week of Feb. 13. The schedule is very aggressive. It is possible that the testing cannot be accomplished in that timeframe. Also, an error threshold for testing is not evident.

The majority of testing was accomplished, though additional testing and prioritization of ‘bug’s are still taking place.

Closed

6 The systems implementation work plan is out of date and does not accurately reflect the schedule for tasks and resources.

The Project Plan was submitted on August 7, 2007.

Closed

20


No. Issue Action(s) Taken Status7 The Vital Statistics Bureau

staff/project team is strained in terms of resources and the vacancy of the outreach coordinator is an issue. This contract position is due to expire in August and most likely will not be filled.

An outreach coordinator was hired for a one year term.

Closed

8 Server locations are an issue for various reasons such as inadequate power, environment or data communications speed. This issue is unresolved too close to the installation schedule date.

This has been resolved with an acceptable temporary and permanent solution.

Closed

9 FY07 Budget is being developed for operations based on the anticipated revenue for FY07, in order to prevent project interruption at the end of FY06 and beginning of FY07. This includes requesting the extension of the SSA funding to 12/31/06.

July Update – Additional funding was approved.

Closed

10 System Performance – The VitalChek system experiences problems with transaction processing response time.

The tactical team came together in July and resolved the system performance issues.

Closed

11Software Release Delay Version Accepted. Closed

12 Pilot will increase calls requesting assistance with issues and questions. This potentially could cause problems for project staff due to time constraints for other responsibilities. The team is not turning the help desk function over until at least the end of July.

July 2007 Update:The Help Desk has transitioned to first level support.

Closed

13 Secure Socket Layering (SSL) solution is still pending.

September update:SSL is in and scheduled for installation.

Closed

21


No. Issue Action(s) Taken Status14 Resources are limited. Resources will continue to be an issue, but

Donna Dossey was allowed to assign other bureau personnel to the project.Additional staff are assigned to the conversion team whenever possible and requested. The team made it through, regardless.

Closed

15 Pilot will increase calls requesting assistance with issues and questions. This potentially could cause problems for project staff due to time constraints for other responsibilities. The team is not turning the help desk function over until at least the end of July.

July Update: Being turned over to the Help Desk, but still concerns for adequate Help Desk staff for call volume. Suggested ancillary assistance when required. No mention of this during July. Refer to #17 below.

Closed

16. Production and maintenance costs after August 30, 2007 have not been funded. While this is not a project issue, it does have an effect on the ability of the Bureau to pay for services and particularly have software support for the modules. This is expensive software and an expensive project. The product must be protected.

Management is aware of the budget shortfalls. While it is not resolved, this does not impact the success of the EVitals project. This issue will be closed for purposes of the project reviews.

Closed

17 Help Desk transition of the system. Due to staffing shortages the project team has served as the customer service staff to report and resolve issues and questions from Pilot participants. This was expected to continue through July, 2007. That has changed in that notification was received that the Help Desk will now service all systems, but project management is concerned that there is still a lack of adequate resources to handle the call volume.

Transition was concluded on July 30, 2007. Closed

22


No. Issue Action(s) Taken Status18 The Department of Health

does not have a performance stress test environment available to support new and existing information systems. The Department should consider investing in a performance stress test environment for major applications. The size and importance of the Department warrant this type of investment.

August/September Update: This is discussed in the EVitals Project Lessons Learned document. It is highly recommended that ITSD be supported with the funds to purchase the hardware and software to be utilized by software/hardware implementation projects in the future.

Open

19ITSD server and communications caused the EVitals production environment to be moved to the fail-over server at the Bureau of Vital Statistics. This had to be performed manually because the replication server for production has not yet been functional during the project.

August/September 2007 Update

ITSD is working on the DOH network issues and is beginning to identify all issues. At this point replication isn’t possible, so it is recommended that the replication process be functional as soon as normal configuration is restored. It is anticipated that this will not occur during the final month of the project, but is being monitored by both ITSD and the Bureau. This will be an ongoing issue until the final, intended EVitals infrastructure is realized.

Open

23


Attachment E – Evitals Application Support Plan

Note that this is a ‘living document’ and will require completion and updating.

24


Attachment F – Evitals Conversion Report

25