DeltaV Version 10.3.1 Upgrade in 24hrs

DeltaV Version 10.3.1 Upgrade in 24hrs

Version 8.4.2 – Version 10.3.1

PresentersPresenters

� Gus Dionne, Automation ARI

� Dan DiMatteo, IS ARI

� Josh Gangl, OIS Platform Lead

� David Maglaya, Emerson LBP

IntroductionIntroduction

DeltaV Version Upgrade

� Planning the Upgrade

– Objectives, Deliverables, Schedule, Resources

� Executing the Plan

– Developing a Team, Direct and Manage the Execution

Monitoring and Controlling� Monitoring and Controlling

– Verify scope, Integrate Changes

� Closing the Project

– Lessons Learned

These are all requirements of a successful upgrade.

ARI Mfg. Site 75 Acre FootprintARI Mfg. Site 75 Acre Footprint

At a site of this size, planning is key to success

Planning and Defining Our ScopePlanning and Defining Our Scope

� Our goal was to upgrade two (2) DeltaV Systems with

minimum impact to users

� Our constraints were Time (24hrs) , Resources, and

Space

� We needed to upgrade not only the Software Application We needed to upgrade not only the Software Application

but we needed to upgrade most of the hardware that it

would run on

� And we also needed to successfully complete these goals

in the middle of a major project, which had extensive

software impact

Additional ScopeAdditional Scope

� 8 of our 18 Servers would need to be updated from Dell

2850s to R710 (18 servers in the environment– 16 DV

servers plus 2 dedicated domain controllers which were

updated to PE2950 hardware)

� 70 out of 92 controllers would need to be upgraded from

MD’s to MD Plus and MX modelsMD’s to MD Plus and MX models

� All of our 65 OITs would need to be upgraded from 5 and

6 year old models to Optiplex 780s

� Add an additional 19th server to manage Critical Data

Backup Application (CDBA) as part of the upgrade

Tools and ResourcesTools and Resources

� We would need high performance tools to complete the

upgrade in a 24hr window. The tool bag would have the

following +

– Imaging tool for OITs

– A Broadcast Server for deploying images

– Racks for Server buildup and testing– Racks for Server buildup and testing

– Spare OIT hardware for testing

– SI Dongles to turn all OITs to Development stations

– A team of Emerson Support Engineers

– Hard copy checklist for sequential steps

Tools and Resources (additional)Tools and Resources (additional)

– A (Time smart) detailed DeltaV Excel Add-in spreadsheet for

saving and restoring critical controller online parameters

– A controller map depicting model, new Mac address of hardware

and installation location

– A team of hands on Tech’s and Engineers was created– A team of hands on Tech’s and Engineers was created

– A timed schedule was created with each task for each person on

the team

The DeltaV Excel Add In Tool exampleThe DeltaV Excel Add In Tool example

Having a spreadsheet with macros simplifies work and

reduces risk.

The Controller Tracking ID ToolThe Controller Tracking ID Tool

Controller Tracker

The Simple Checklist goes a long wayThe Simple Checklist goes a long way

Organized tasks with priorities are a must for capturing

essentials and ensuring nothing is missed.

The Early ExecutionThe Early Execution

� Offline Migration Testing from 8.4.2 to 10.3.1 at LBP office

� 60+ new OITs would be deployed with the 8.4.2 image

prior to the upgrade day

� 10.3.1 Images would be created and tested in plans for

broadcasting to the new OITs on the upgrade day

� Each new controller to be deployed was labeled and Mac � Each new controller to be deployed was labeled and Mac

address recorded for its intended new location. This saved

the light flashing process at each location for the new

controllers during commissioning.

The Early Execution cont. The Early Execution cont.

� A disk was pulled from the Domain Controller and used to

copy the domain users onto the new DC box, (this was

done for offline code development in the Dev

environment, we used the live DC to migrate the Active

Directory during the outage)

� Critical parameters that needed to be restored would be � Critical parameters that needed to be restored would be

listed in Time Smart XL add in tool

� Server class boxes would be pre-built, racked, stacked

and configured as far as possible prior to execution and

left powered on until upgrade

� The Pro Plus Servers had Database imported at Version

8.4.2 and then the 10.3.1 upgrade wizard run

The Execution prep(the night before) The Execution prep(the night before)

� Critical parameters that needed to be restored would be

uploaded to the XL add in tool with an individual time

record for each time-sensitive value

� Controllers with outputs that needed to be maintained

would have the manual field overrides enabled

� 70 of the 92 controllers would be decommissioned and � 70 of the 92 controllers would be decommissioned and

replaced with the new pre-labeled controllers

The ExecutionThe Execution

� The execution started at 7:00am with the team

concurrently working the plan from a central automation

lab room, server room, and plant floor

� Servers previously racked and stacked are placed online

starting with the Pro Plus boxes

– Legacy DeltaV 8.4.2 PPS servers were “failed”

• Metadata cleanup, FSMO roles seized by dedicated DC• Metadata cleanup, FSMO roles seized by dedicated DC

– New PPS R710 hardware racked & stacked

• DCPROMO / DNS config, FSMO roles transferred to new PPS

The ExecutionThe Execution

� 10.3.1 Images are concurrently broadcasted to multiple

OIT stations seven (7) at a time

� A team of IS engineers are running OIT configuration and

Sys Prep at pre-scheduled OIT locations

The Execution continuedThe Execution continued

� As OITs became available controllers were commissioned

with no field assistance due to Mac address reservation,

then flashed, then downloaded

� Each OIT coming on line was used to RDP to field OITs to

multiply the Flashing capacity of the automation lab (SI)

� Critical online controller parameters were restored using

the DeltaV Time Smart Add-in tool. Online values that were

time sensitive were updated from within Excel, based on

the time the value was stored and the current time of

restoration.

The ClosingThe Closing

� As Systems came back online they were turned over to

Validation then manufacturing

� A Lessons Learned session was held shortly after

Lessons learnedLessons learned

Some Good Points

� Have a well defined plan and schedule

� Trained resources

� Pre building and Server Prep is key to saving time

� RAID 1 (mirroring) with hot spare disk setup to support DC

transfer (easily export Active Directory from dedicated transfer (easily export Active Directory from dedicated

domain controller – Pro Plus and other DeltaV servers use

standard RAID10 config)

� DeltaV Excel Add-in was key in restoring online values

quickly without manual calculations

� Broadcasting of OIT image saved time over upgrade

wizard

Lessons learned cont.Lessons learned cont.

Some Bad Points

� We were overstaffed the morning of outage. Staffing

schedule could be improved, (Pro Plus delay due to Active

Directory DCPROMO issue added to this)

� VX dongles were needed and not known right awayVX dongles were needed and not known right away

� Some user accounts got corrupted due to Schema mis-

match

� Servers pulled from service should have been left in racks

for a week or two since some data configuration recovery

was required

Lessons learned cont.Lessons learned cont.

Some more Bad Points

� Implementing an upgrade with concurrent complex

software changes is extremely difficult

� (Requirements were changed by people outside of core

project team – caused problems with keeping plan locked project team – caused problems with keeping plan locked

down) – point here is to get everyone to commit to

providing requirements up-front and early in the planning

Business Results AchievedBusiness Results Achieved

1. There were no Safety related incidents due to any

upgrade associated actions

2. The DeltaV System went from running version 8.4.2 one

evening to running version 10.3.1 the following evening

3. 18 Servers, 65 OITs, and 70 Controllers were upgraded

(8 R710 replacements and 8 PE2950s were upgraded in

place using Upgrade Wizard, 2 DCs were upgraded to place using Upgrade Wizard, 2 DCs were upgraded to

PE2950 hardware)

4. There was no loss of production outside the planned

outage due to the upgrade

5. There was no loss of utility access due to the upgrade

SummarySummary

� We have done our best to share how we planned,

managed, and executed a successful DeltaV software and

hardware upgrade within a large scale Bio-Tech

manufacturing Facility

� Our benefits are faster, better performing, and more

capable software and hardware to the end user as well as capable software and hardware to the end user as well as

a more robust system for the administrators

QuestionsQuestions

I’d like to take this time to ask if anyone has any questions

that pertain to the project upgrade?

FeedbackFeedback

If any of you would not mind sharing any feedback you might

have we would all like to know if you enjoyed the

presentation or not?

Thank you all so much for attending.

Where To Get More InformationWhere To Get More Information

� Other training sessions

� List books, articles, electronic sources

� Consulting services, other sources