neorsd process control system standards and conventions … standards...sfc sequential function...

NEORSD Process Control System Standards and Conventions Manual

Volume 1 – Introduction, Definitions, Policies and Procedures

Revision 1.5

December 2019

This page intentionally left blank.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Volume 1 Contents

i Revision 1.5 – December 2019

Volume 1 Contents Volume 1 Contents ........................................................................................................................ i

1.0 Revision History ................................................................................................................ 1

2.0 Introduction ....................................................................................................................... 1

3.0 Abbreviations .................................................................................................................... 2

4.0 PCS Revision Control ....................................................................................................... 4 4.1 Overview ......................................................................................................................... 4

4.2 Responsibilities ............................................................................................................... 4

4.3 Software Updates ........................................................................................................... 4

4.4 Testing Requirements .................................................................................................... 5

4.5 Deployment .................................................................................................................... 5

4.6 Recordkeeping ............................................................................................................... 6

5.0 Controller Firmware Revision ............................................................................................ 6 5.1 Overview ......................................................................................................................... 6


5.3 Firmware Version Requirements .................................................................................... 6

5.4 Firmware Updates .......................................................................................................... 6

5.5 First Version Testing Requirements ............................................................................... 7

5.6 Deployment .................................................................................................................... 8

5.7 Recordkeeping ............................................................................................................... 8

6.0 Communications Module Firmware Revision .................................................................... 8 6.1 Overview ......................................................................................................................... 8


6.3 Firmware Version Requirements .................................................................................... 9

6.4 Firmware Updates .......................................................................................................... 9

6.5 First Version Testing Requirements ............................................................................. 10

6.6 Deployment .................................................................................................................. 10

6.7 Recordkeeping ............................................................................................................. 11

7.0 Programming Software Revision ..................................................................................... 11 7.1 Overview ....................................................................................................................... 11

7.2 Responsibilities ............................................................................................................. 11

7.3 Controlled Software ...................................................................................................... 11

7.4 Recordkeeping ............................................................................................................. 12


ii Revision 1.5 – December 2019

8.0 Applications Management of Change ............................................................................. 12 8.1 Overview ....................................................................................................................... 12


8.3 Management of Change Approval Requirements ........................................................ 12

8.4 District Test Environment Requirements ...................................................................... 13

8.5 District Test Bed Requirements .................................................................................... 13

8.6 PLC Modifications ......................................................................................................... 13

8.7 OIT Modifications .......................................................................................................... 14

8.8 HMI Configuration Modifications ................................................................................... 14

8.9 Alarm and Event Modifications ..................................................................................... 15

8.10 Other Software Modifications ....................................................................................... 15

8.11 Contractor Check-out ................................................................................................... 15

8.12 Deployment Requirements ........................................................................................... 15

8.13 Recordkeeping ............................................................................................................. 16

9.0 Network Management of Change ................................................................................... 16 9.1 Overview ....................................................................................................................... 16


9.3 District Test Bed Requirements .................................................................................... 16

9.4 Management of Change Approval Requirements ........................................................ 17

9.5 Recordkeeping ............................................................................................................. 18

10.0 District Programming Laptop Request ............................................................................ 18 10.1 General ......................................................................................................................... 18

10.2 Procedure ..................................................................................................................... 18

11.0 Standards Deviation Request ......................................................................................... 18 11.1 General ......................................................................................................................... 18

11.2 Procedure ..................................................................................................................... 19

12.0 Management of Virtual Local Area Networks .................................................................. 20 12.1 General ......................................................................................................................... 20

12.2 Procedure ..................................................................................................................... 20

13.0 Applications Backup and Recovery Policy ...................................................................... 20 13.1 General ......................................................................................................................... 20

13.2 Backup Storage Requirements ..................................................................................... 20

13.3 Recovery Procedures ................................................................................................... 21

13.4 Recovery Testing Requirements .................................................................................. 22

13.5 Documentation Requirements ...................................................................................... 22


iii Revision 1.5 – December 2019

14.0 Controlled Software List .................................................................................................. 22

15.0 Responsibilities Matrix .................................................................................................... 23

16.0 Reference Documents .................................................................................................... 25

17.0 Forms .............................................................................................................................. 25 NEORSD PCS Request for Shutdown Form ........................................................................... 26

NEORSD PCS Request for Shutdown Form - EXAMPLE ...................................................... 27

NEORSD PCS Contractor Extended Program Check-out Request Form ............................... 28

NEORSD PCS Contractor Extended Program Check-out Request Form-EXAMPLE ............. 29

NEORSD PCS Change Request Form ................................................................................... 30

NEORSD PCS Change Request Form-EXAMPLE ................................................................. 32

NEORSD PCS Change Request Form Appendix A Back-out Plan ....................................... 34

NEORSD PCS Change Request Form Appendix A Back-out Plan-EXAMPLE ..................... 35

NEORSD PC&A Programming Laptop Request Form ............................................................ 36

NEORSD PC&A Programming Laptop Request Form-EXAMPLE .......................................... 37

NEORSD Standards Deviation Request Form ........................................................................ 38

NEORSD Standards Deviation Request Form-EXAMPLE ...................................................... 39

NEORSD Network Modifications Management of Change Form ............................................ 40

NEORSD Network Modifications Management of Change Form - EXAMPLE ........................ 42

Tables

Table 14-1 Controlled Software .................................................................................................. 22 Table 15-1 Responsibilities Matrix .............................................................................................. 23 Table 16-1 Reference Documents .............................................................................................. 25


iv Revision 1.5 – December 2019

This page intentionally left blank.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Revision History

1 Revision 1.5 – December 2019

1.0 Revision History Revision

Revision Date

Author Update Section

Description

0 02/17/2017 HDR All DRAFT Modification as part of APM Program.

Restructured to 5 volumes and update to current industry best practices.

New Volume with Policies and Procedures. Section 2 – Updated from ASCM Section 3 – 8, 9, 11-12: New Policies and

Procedures Section 10 – Minor updates from ASCM

1 05/05/2017 HDR All FINAL1.1 10/27//2017 HDR All Added new section 11 – Management of

Virtual Local Area Networks. Added Responsibilities Matrix Added startup/commissioning requirement

for existing equipment to section 12.2. Updated Forms Added example forms Added Network Management of Change

Section. Relocated all forms to section 17.0

1.2 4/25/2018 HDR 8.2 DRAFT Storage requirements for configurable

devices (VFDs, etc) 1.3 12/12/2018 HDR 11.1 Updated deviation requirements for PLCs.1.4 4/16/2019 HDR 8.6 Added paragraph to this section prohibiting

protection or lockout of code preventing access by NEORSD.

1.5 12/30/2019 NEORSD/PJM All Added Revision and Date to the footer.

2.0 Introduction This manual is a compilation of The Northeast Ohio Regional Sewer District’s (NEORSD) standards for Process Control System (PCS) programming, configuration and design. It is intended to be applied by the District’s contractors, consultants, and in-house personnel when developing or modifying any portion of the District’s PCS. The PCS is a utility-wide system of hardware and software that spans the wastewater collection system and all three wastewater treatment plants. This document must be treated as both requirements and guidance for PCS work.

This manual is divided into five main volumes:

Volume 1 contains an introduction to the Process Control System Standards and Conventions Manual. It also comprises District policies and procedures that apply to the use and management of the PCS, including approvals and practices for applying and documenting changes to hardware and software, code changes, alarm management requirements,

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Abbreviations


contractor’s responsibilities related to work performed on the PCS and other related topics. This volume also contains the standards deviation request form.

Volume 2 addresses practices for development and programming of control processors, primarily programmable logic controllers (PLCs) of various types, including requirements for programming software, databases, alarm processing, networking, control loops, control logic structure, I/O layout and signal processing, tagging and naming conventions.

Volume 3 includes operator interfaces and the PCS subsystem that provides Human-Machine Interface (HMI) functionality. Volume 3 includes the requirements for programming and configuration of the Area Control Stations (ACSs), historians, servers and the related network infrastructure. Standards for programming and display development for Operator Interface Terminals (OITs), industrial operator interfaces typically located in the field PLC panels, are included in this volume.

Volume 4 includes databases and other PCS interconnected software,

Volume 5 includes design standards for PCS including panel design, Process and Instrumentation Diagrams (P&ID), network architecture and security.

Compliance with the standards and conventions outlined in the manual are required for all projects that add to or modify the PCS. If a contractor or District staff member identifies a need for an addition or modification to the standards for a specific project, or if in the course of the work it is found that some part of the standards cannot be adhered to, a Standards Deviation Request may be submitted in accordance with the requirements of that section.

Refer to Section 16.0 for a complete list of all NEORSD standards and forms, and industry standards and best practices referenced in this manual.

3.0 Abbreviations The following is a list of applicable acronyms and definitions which may be utilized throughout the Process Control System Standards and Conventions Manuals.

A2ALMDB Wonderware Alarm and Event Database

ACS Area Control Stations

AOI Add-On Instruction (RSLogix PLC)

AOS Application Object Server (Wonderware)

APM Automation Program Management

BTL Base Template Library (Wonderware Scripting)

CAD Computer-Aided Drafting

CAT Category - relates to communication cable types such as CAT5, CAT6, etc.

CIP Capital Improvement Project

CLX Allen-Bradley ControlLogix PLC

CPU Central Processing Unit

DAS Data Acquisition Server (Wonderware)

DASABCIP Data Acquisition Server – Allen Bradley IP Driver

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Abbreviations


DASABTCP Data Acquisition Server – Allen Bradley TCP driver

DASMBTCP Data Acquisition Server – Allen Bradley Modbus TCP driver

E&C Engineering and Construction

EMSC Environmental and Maintenance Services Center

FBD Function Block Diagram

HART Highway Addressable Remote Transducer (Communication protocol feature on some instrumentation)

HMI Human Machine Interface

ID Identification

IO or I/O Input/Output - refers to process signals or signal processing equipment

IP Internet Protocol

ISA International Society of Automation

IT Information Technology

JSR Jump Subroutine

LAN Local Area Network

MOC Management of Change

NEORSD Northeast Ohio Regional Sewer District

O&M Operation and Maintenance

ODMS Operational Data Management System (See PDMS)

OEM Original Equipment Manufacturer

OIT Operator Interface Terminal

OPL One Point Lesson

P&ID Process and Instrumentation Diagram

PC&A Process Control and Automation

PCD Process Control Description

PCS Process Control System

PDF Portable Document Format (.pdf file format)

PDMS Process Data Management System (replaces ODMS)

PID Proportional-Integral-Derivative (Used in control output algorithm)

PLC Programmable Logic Controller

RFI Request for Information

RTU Remote Terminal Unit

SFC Sequential Function Chart

SCADA Supervisory Control And Data Acquisition

SLC Allen-Bradley small PLC model

SMC System Management Console

SMP Standard Maintenance Procedure

SNMP Simple Network Management Protocol

SOP Standard Operating Procedure

SQL Structured Query Language

UDT User Defined Data Type

UPS Uninterruptible Power Supply

USB Universal Serial Bus

VLAN Virtual Local Area Network

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures PCS Revision Control


WAM Oracle’s Work and Asset Management Application Software

WAN Wide Area Network

WWTP Wastewater Treatment Plant

4.0 PCS Revision Control

4.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Revision Control Policy is intended to establish requirements for PCS Software version and patch application. The NEORSD PCS consists of numerous highly integrated software packages. This Policy and Procedure addresses policy for software versions including version change and/or application of software patches, and testing requirements to promote stability of the PCS.

This Policy applies to all software and operating systems on District PCS Workstations, Servers, and Programming Laptops. Refer to Table 14-1 in Section 14.0 of this document for a complete list.

4.2 Responsibilities To ensure that the PCS Revision Control Policy is implemented and maintained in accordance with the procedures listed in this policy, the identified NEORSD Employees and contractors have the responsibilities identified in Table 15-1.

4.3 Software Updates No device shall be configured to receive automated updates. All updates and version changes shall be manually initiated and tested on the District test environment prior to deployment on the live production system.

4.3.1 Operating Systems All server, workstation and VM operating systems shall have automatic updates disabled. Service packs shall be applied by PC&A on a monthly basis or upon recommendation of a PCS software vendor, but only after validation with PCS Software vendors that the latest patches and service packs are supported. Upgrading to new versions of Operating Systems will occur on an as-needed basis but no earlier than 6-months after Control Software OEM vendors confirm support of the Operating System.

4.3.2 Control Software (Logix, Wonderware, Historian, etc.) All control software shall have patches applied by PC&A on a quarterly basis or as required due to identified critical flaw or vulnerability. Major revision changes shall be applied annually but no earlier than 3-months after OEM vendor release of latest version. Control Software major revision changes are defined as incrementing a full revision number (e.g. Archestra 2012 to Archestra 2014; or Rockwell ControlLogix 5000 Version 20 to version 21).

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures PCS Revision Control


4.3.3 Anti-Virus Definitions Anti-Virus definitions shall be updated no less than weekly via manual update by PC&A. OEM automatic updates shall be disabled, and will not function properly due to PCS network isolation from internet connections. Conflicts with virus definition updates will be handled on a case by case basis via technology WAM work order.

4.4 Testing Requirements All major and minor software revisions shall be installed and validated utilizing the NEORSD’s test environment. Software revisions must run hands free without error for 7-days for service packs and version changes and 3-days for minor patches (those not contained in a full service pack release) in the test environment prior to deployment on the live production system.

District Test environment has a limited number of concurrent seats available for use. Person(s) responsible for update must submit an emailed schedule request to District PC&A manager or assigned contact a minimum of two (2) weeks in advance of the required testing date to confirm availability. NEORSD will confirm schedule via email or coordinate alternate dates with requesting person(s) if required. Confirmation will be at least one week prior to requested date.

Person(s) responsible for update are required to document the following prior to beginning the update process:

1. Verification of software interoperability/compatibility a. Review software release notes to verify that software versions support

compatibility without conflict. Rockwell Product compatibility shall be verified utilizing the online compatibility tool (http://compatibility.rockwellautomation.com/Pages/MultiProductSelector.aspx?crumb=111)

2. Verification of proper licenses to support installation of latest software

Person(s) completing the update of software are required to perform the following:

1. Follow requirements of PCS Change Management Policy 2. Document back-up and recovery plan 3. Create backups of current running system (for software running on VM, archive current

VM) 4. Clear current error logs for appropriate software 5. Install Software and Restore Applications 6. Return Software to running state and check error logs 7. Verify proper operation via documented testing plan

4.5 Deployment This section contains the requirements and procedures for deploying fully tested and accepted changes into the live production environment.

An implementation plan shall be submitted for District approval prior to modification of the live production environment. At a minimum, the plan shall include the following:

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Controller Firmware Revision


Copy of Approved Applications Management of Change (MOC) Form (See Section 17.0) Written back-out plan for Approval (See Section 17.0 Appendix A) One Point Lesson (OPL) – Highlight of functional changes. Instructions and required

forms are located on the District Enterprise SharePoint under SOP/OPL/SMP Section. District Request for Shutdown Form (See Section 17.0)

District Request for Shutdown form is available in editable format on District Enterprise SharePoint site or by request from District Project Manager or PC&A Manager. Form shall be submitted to District Engineering whom will facilitate operations sign-off.

All changes must be implemented from PC&A Programming Laptop, see Section 10.0 and request form in Section 17.0 for details.

4.6 Recordkeeping Software updates must be documented by the person(s) performing the updates following the procedures defined in Section 8.0 - Applications MOC Policy.

5.0 Controller Firmware Revision

5.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Controller Firmware Policy is intended to establish requirements that promote stability of the PCS, consistency of firmware deployment and testing requirements for firmware updates.

This Policy applies to all District plants and locations where PCS Controllers exist. For purpose of this policy, PCS controller is defined as any PLC CPU module including but not limited to Allen-Bradley Logix Series Controllers.

5.2 Responsibilities To ensure that the Firmware Policy is implemented and maintained in accordance with the procedures listed in this policy, the identified NEORSD Employees and contractors have the responsibilities identified in Table 15-1.

5.3 Firmware Version Requirements The NEORSD has standardized on Allen-Bradley brand of PLCs. Firmware versions must be consistent throughout the District. It is required that the firmware of all new or replacement PLC Controllers installed match the most current version installed by the District. The District’s current versions are available on PC&A page of District Enterprise SharePoint or via email request to District PC&A Manager.

5.4 Firmware Updates Firmware version updates will be performed on an annual basis or as required by manufacturer to address vulnerabilities or other critical flaws. PC&A manager is responsible for approval and notification of adopting new firmware versions.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Controller Firmware Revision


The first application of any new firmware version on each controller type shall undergo testing on the District Test Bed requirements defined in Section 5.5, First Version Testing Requirements.

District Test Bed has a limited number of concurrent seats available for use. Person(s) responsible for update must submit an emailed schedule request to District PC&A manager or assigned contact a minimum of two (2) weeks in advance of the required testing date to confirm availability. NEORSD will confirm schedule via email or coordinate alternate dates with requesting person(s) if required. Confirmation will be at least one week prior to requested date.

Person(s) responsible for update of firmware version are required to document the following prior to beginning the update process:

1. Verification of compatibility with the District’s current RSLogix 5000 software a. Any firmware version updates that require software update must have PC&A

manager approval in writing 2. Verification of hardware compatibility for all cards in local and remote racks with

Rockwell’s compatibility matrix. (http://compatibility.rockwellautomation.com/Pages/MultiProductSelector.aspx?crumb=111)

a. Any component identified as non-compatible must have written plan for migration to compatibility during update process.

5.5 First Version Testing Requirements The first application of any new firmware version on each controller type shall undergo testing on the District Test Bed prior to deployment.


The District Test Bed shall be utilized to validate compatibility of firmware with the various I/O and communications modules that exist in NEORSD PCS. A test plan must be developed and submitted to PC&A manager that details equipment arrangements (rack layouts) that will be tested to ensure all types of I/O modules and communications modules are tested with the newly updated controller firmware.

All tests are required to run hands-off without error for a minimum of 100 hours. PLC Fault tables and HMI error logs shall be validated throughout the test; any PLC fault or HMI faults related to loss of communications will result in test failure. All AOIs shall be tested and confirmed to operate as defined after firmware upgrade.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Communications Module Firmware Revision




Copy of Approved Applications MOC Form (See Section 17.0) Written back-out plan for Approval (See Section 1230 Appendix A) OPL – Highlight of functional changes. Instructions and required forms are located on

the District Enterprise SharePoint under SOP/OPL/SMP Section. District Request for Shutdown Form (See Section 17.0)

District Request for Shutdown form is available in editable format on District Enterprise SharePoint site or by request from District Project Manager or PC&A Manager. Form shall be submitted to District PC&A Manager whom will facilitate operations sign-off.

Person(s) completing the update of firmware version are required to perform the following steps in addition to following Rockwell’s detailed migration documentation.

1. Follow requirements of PCS – Section 8.0 – Applications MOC 2. Create online (AssetCentre) and offline copies of RSLogix 5000 application files 3. Remote network connections are forbidden. All firmware version updates are required to

be completed utilizing District PC&A laptop locally at PLC Controller. After panel has been removed from service, connect directly to PLC from laptop to perform version update.

4. Upon restoration of service; a. PLC fault tables shall be checked and documented to show no compatibility or

other faults generated. b. Connectivity shall be verified between PLC & HMI, and PLC & OIT.


5.7 Recordkeeping Controller firmware updates must be documented by the persons(s) performing the updates following the procedures defined in the Section 8.0 - Applications MOC Policy.

6.0 Communications Module Firmware Revision

6.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Communications Module Firmware Policy is intended to establish requirements that ensure stability of the PCS, consistency of firmware deployment and testing requirements for firmware updates.



This Policy applies to all District plants and locations where PCS Communications Modules exist. For purpose of this policy, Communications Modules are defined as PLC modules utilized for network communications or additional processing not handled within the CPU.

6.2 Responsibilities To ensure that the Firmware Policy is implemented and maintained in accordance with the procedures listed in this policy, the identified NEORSD Employees and contractors have the responsibilities identified in Table 15-1.

6.3 Firmware Version Requirements The NEORSD has standardized on Allen-Bradley brand of PLCs. Firmware versions must be consistent throughout the District. It is required that all firmware versions of new or replacement Communications Modules installed match the most current version installed by the District. The District’s current versions available on PC&A section of District Enterprise SharePoint or via email request to District PC&A Manager.

6.4 Firmware Updates Firmware version updates will be performed on an annual basis or as required by manufacturer to address vulnerabilities or other critical flaws. PC&A manager is responsible for approval and notification of adopting new firmware versions.

The first application of any new firmware version on each communications module model shall undergo testing on the District Test Bed, requirements defined in Section 5.5, First Version Testing Requirements.


Person(s) responsible for update of firmware version are required to document the following prior to beginning the update process:

1. Verification of compatibility with District current RSLogix software a. Any firmware version updates that require software update must have PC&A

manager approval in writing, reference related policy for appropriate hardware type to be upgraded for requirements.

2. Verification of module firmware version compatibility with Controller firmware version by utilizing Rockwell’s compatibility matrix

a. If upgrade of communications module requires newer controller firmware, refer to Section 5.0 - Controller Firmware Revision for upgrade policy/procedure

Person(s) completing the update of firmware version are required to perform the following steps in addition to following Rockwell’s detailed migration documentation.



1. Follow requirements of Section 8.0 Applications MOC Policy 2. Create online (AssetCentre) and offline copies of RSLogix 5000 application files 3. Remote network connections are forbidden. All firmware version updates are required to

be completed utilizing District PC&A laptop locally at communications module. After panel has been removed from service, connect directly to the communications module from laptop to perform version update.

4. Upon restoration of service; a. PLC fault tables shall be checked and documented to show no compatibility or

other faults generated. b. Connectivity shall be verified between PLC & HMI. c. For communications devices, connectivity to ALL end devices shall be confirmed.

6.5 First Version Testing Requirements The first application of any new firmware version on each control module type shall undergo testing on the District Test Bed prior to deployment.

The test bed shall be utilized to validate compatibility of communications module firmware with the control processor and where possible any end devices for a communications module. A test plan must be developed and submitted to PC&A manager that details equipment arrangements (rack layouts) that will be tested with the newly updated controller firmware.

All tests are required to run hands-off without error for a minimum of 100 hours. PLC Fault tables and HMI error logs shall be validated throughout the test; any PLC fault or HMI faults related to loss of communications will result in test failure.



Copy of Approved Applications MOC Form (See Section 17.0) Written back-out plan for Approval. (See Section 17.0 Appendix A) OPL – Highlight of functional changes. Instructions and required forms are located on

the District Enterprise SharePoint under SOP/OPL/SMP Section. District Request for Shutdown Form (See Section 17.0)



NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Programming Software Revision


6.7 Recordkeeping Communication module firmware updates must be documented by the person(s) performing the updates following the procedures defined in the Section 8.0 - Applications MOC Policy.

7.0 Programming Software Revision

7.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Software Revision Control Policy is intended to establish a standard approach for implementation and documentation of software modifications. The NEORSD PCS consists of numerous highly integrated software packages and custom software for process control and monitoring. This Policy and Procedure addresses current version, policy for version change and/or application of software patches, and testing requirements to promote stability of the PCS.

This Policy applies to PLC Programming software packages.


7.3 Controlled Software The software applications defined in this section are controlled by Section 4.0 PCS Revision Control Policy.

7.3.1 PLC Programming Software (Rockwell Automation – RSLogix Suite) NEORSD maintains all Rockwell Automation RSLogix platform revisions to be consistent between plants. It is the responsibility of the contractor to confirm approved software revisions prior to proceeding with code modifications or development. Current approved versions are available on PC&A section of District Enterprise SharePoint or via email request to District PC&A Manager.

Rockwell “Product Resiliency” releases of programming software are NOT approved for use in the District. This includes version 20.03 of RSLogix 5000, as well as 21.03 and similar potential future versions of Studio 5000. Saved projects are not compatible with earlier software minor revisions. Projects downloaded to controllers cannot be edited with earlier software minor revisions. Refer to Rockwell KB Articles 565204 and 582891 for more information.

Studio 5000 version 23 will not be supported at the District, and shall not be used.

7.3.2 HMI Software NEORSD maintains a single Wonderware Galaxy Server which deploys objects to the servers located at each WWTP and at EMSC for collections. It is the responsibility of the contractor to confirm approved software revisions prior to proceeding with HMI modification or development.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Applications Management of Change


Current approved versions are available on PC&A section of District Enterprise SharePoint or via email request to District PC&A Manager.

7.3.3 OIT Software It is the responsibility of the contractor to confirm approved software revisions prior to proceeding with OIT modification or development. Current approved versions are available on PC&A section of District Enterprise SharePoint or via email request to District PC&A Manager.

7.4 Recordkeeping Software updates must be documented by the person(s) performing the updates following the procedures defined in the Section 8.0 - Applications MOC Policy.

8.0 Applications Management of Change

8.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Applications MOC is intended to establish requirements for PCS Change Management Policies and Procedures. The NEORSD PCS consists of numerous highly integrated software packages. This Policy and Procedure addresses requirements for change management including review/approval process, documentation and usage of AssetCentre.

This Policy applies to all PLC, HMI, OIT, and Historian modifications for NEORSD PCS Assets.


8.3 Management of Change Approval Requirements ALL modifications to PCS must be reviewed and approved prior to deployment on the live production system.

A change modification request shall be submitted to District for Approval utilizing the form in Section 17.0. Change request shall include the following:

1. Description of Change 2. Location of Change (e.g. PLC Name, Subroutine and rungs; HMI screen name, etc.) 3. Name, Company Name, and Contact Phone number for person implementing change 4. Copy of completed test plan from District Test Environment or District Test Bed 5. Copy of proposed field test verification plan 6. Back-out Plan. Back-out plan shall include details on how to revert to previous known

good installation.



No change shall be implemented without written approval from District. District PC&A manager is responsible for assigning appropriate approval authority and coordinating with District Operations.

Exceptions – Emergency work may be completed under direction of Operations supervisor or PC&A manager. Upon completion of emergency work, a retroactive change request form shall be submitted for record purposes.

Change management forms are to be stored on District Enterprise SharePoint site.

8.4 District Test Environment Requirements NEORSD requires all software version and patches to be validated in District Testing Environment prior to deployment on live production system

District Test environment has a limited number of concurrent seats available for use. Person(s) responsible for update must submit an emailed schedule request to District PC&A manager or assigned contact a minimum of two (2) weeks in advance of the required testing date to confirm availability. NEORSD will confirm schedule via email or coordinate alternate dates with requesting person(s) if required. Confirmation will be at least one week prior to requested date.

Person(s) responsible for modifications are required to document the testing and facilitate District witness and approval. A test plan shall be submitted to District and approved prior to scheduling testing.

8.5 District Test Bed Requirements NEORSD requires all modifications of PCS to be implemented in District Test Bed prior to deployment on live production system. The District Test Bed is not the same as District Test Environment, the Test Bed system is utilized to validate application programs (PLC/HMI/OIT/others) prior to deployment on the live production system.



All application programs shall be loaded into District AssetCentre at time of deployment into District Test Bed.

8.6 PLC Modifications ALL PLC Programs are saved in District AssetCentre. PLC Modifications must be made utilizing district provided PC&A Programming Laptop (Refer to Section 10.0 of this document).



AssetCentre shall be utilized to maintain versions via check-in/check-out process. AssetCentre transaction comments are required to be completed and must contain initials of person making the change, company name and approved MOC form file name.

Changes to logic (whether ladder logic, function block, or other programming language) must be commented and include:

Date of change Initials of person making the change Company that employs the person making the change Reason and description of the change Example: 12/02/16 – ABC – Company Name – Logic changes associated with VFD

(TAG) upgrade. New VFD ramp time 60-seconds at VFD instead of only PLC controlled ramp.

All changes made following system commissioning should be documented in a similar manner. Information including the date of programming change, initials and company of author, work order number or change requestor as appropriate, and description of the change shall be appended to the documentation on the first rung.

Password protection or lockout of code that prevents NEORSD to access, modify and/or configure is prohibited.

Example: 12/02/16 – ABC – CompanyName – PCD Update to document VFD upgrade for (TAG). VFD Ramp time modified to 60-seconds. Rate of change alarms modified to reflect VFD controlled ramp.

8.7 OIT Modifications ALL OIT Programs are saved in District AssetCentre. OIT Modifications must be made utilizing district provided PC&A Programming Laptop. All programmers assigned a PC&A Programming Laptop will be given access privileges for AssetCentre. AssetCentre transaction comments are required to be completed and must contain initials of person making the change, company name and approved MOC form file name.

AssetCentre shall be utilized to maintain versions via check-in/check-out process.

8.8 HMI Configuration Modifications Wonderware Templates are backed up in District AssetCentre. Additionally, a Galaxy backup is created every morning on the Galaxy server. The backups can be used to control change management.

Wonderware modifications must be made utilizing a district provided PC&A programming laptop or directly on the Galaxy server using VMware vSphere Client. Only programmers that request a PC&A programming laptop with a Wonderware development license can access the Wonderware development environment called ArchestrA IDE.



Changes made that modify an entire process area must be validated in the Testbed. Wonderware modifications are subject to the Application MOC Policy.

The Galaxy has been configure to require revision notes to be applied upon check-in of an object instance and must include:

Date of change Initials of person making the change Company that employs the person making the change Reason and description of the change Example: 12/02/16 – ABC – Company Name – Modified object instance to disable open

and close parameter events, this is handled through other indication per accepted process control description.

8.9 Alarm and Event Modifications ALL alarm and event modifications including setpoint changes are subject to both Application MOC Policy and Alarm Management Policy (Reference Volume 5, Section 10).

8.10 Other Software Modifications Variable Frequency Drive Programs files are saved in District AssetCentre. Variable Frequency Drive Modifications must be made utilizing district provided PC&A Programming Laptop. All programmers assigned a PC&A Programming Laptop will be given access privileges for AssetCentre. AssetCentre transaction comments are required to be completed and must contain initials of person making the change, company name and approved MOC form file name.

AssetCentre shall be utilized to maintain versions via check-in/check-out process.

8.11 Contractor Check-out Contractors may request to check-out a program file for modification off-site during CIP projects. During this time programming modifications may be required to maintain the operating facility, every effort will be made to copy contractors on MOC paperwork, however it is the contractor’s responsibility to verify transmitted check-out version versus current for changes prior to deployment of contractor developed code in District Test Environment.

8.12 Deployment Requirements This section contains the requirements and procedures for deploying fully tested and accepted changes into the live production environment.


Copy of Approved Applications MOC Form (See Section 17.00) Written back-out plan for Approval. Back-out plan shall include details on how to revert

to previous known good installation.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Network Management of Change


OPL – Highlight of functional changes. Instructions and required forms are located on the District Enterprise SharePoint under SOP/OPL/SMP Section.

District Request Form for Shutdown (See Section 17.0)



8.13 Recordkeeping In addition to MOC requirements stated in the above sections, it is the implementer’s responsibility to update all District PCS Documentation affected by the change including but not limited to:

Process Control Description (PCD) Databases – alarms, historian, ODMS P&IDs Loop diagrams Panel drawings Standard object libraries (if creating new objects) Reports Historian change log to PDMS Owner (add/change/delete)

9.0 Network Management of Change

9.1 Overview The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Network MOC is intended to establish requirements for PCS Network Change Management Policies and Procedures. The NEORSD PCS network consists of numerous highly integrated components. This Policy and Procedure addresses requirements for PCS network change management including review/approval process, documentation and usage of Zabbix.

This Policy applies to all network equipment modifications for NEORSD PCS Assets.


9.3 District Test Bed Requirements NEORSD requires all modifications to the network switches to be implemented in District Test Bed prior to deployment on live production system. The District Test Bed is not the same as

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Network Management of Change


District Test Environment, the Test Bed system is utilized to validate application programs (PLC/HMI/OIT/others) prior to deployment on the live production system.



All of the switch configurations shall be loaded into Zabbix at time of deployment into District Test Bed.

9.4 Management of Change Approval Requirements ALL modifications to PCS must be reviewed and approved prior to deployment on the live production system.

A change modification request shall be submitted to District for Approval utilizing the Network Modifications Management of Change form in Section 17.0. Change request shall include the following:

1. Description of Change 2. Location of Change (e.g. Equipment name and asset tag) 3. Name, Company Name, and Contact Phone number for person implementing change 4. Copy of completed test plan from District Test Environment or District Test Bed 5. Copy of proposed field test verification plan 6. Change justification 7. Impact of not implementing 8. Affected Parties and Systems 9. Implementation Process (written plan) 10. Detailed technical steps 11. Start/Finish Times (Estimated time to complete) 12. Back-out Plan. Back-out plan shall include details on how to revert to previous known

good installation.

No change shall be implemented without written approval from District. District PC&A manager is responsible for assigning appropriate approval authority and coordinating with District Operations.

Exceptions – Emergency work may be completed under direction of Operations supervisor or PC&A manager. Upon completion of emergency work, a retroactive change request form shall be submitted for record purposes.

Change management forms are to be stored on District Enterprise SharePoint site.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures District Programming Laptop Request


9.5 Recordkeeping In addition to MOC requirements stated in the above sections, it is the implementer’s responsibility to update all District PCS Documentation affected by the change including but not limited to:

P&IDs Panel drawings Network diagrams IP and VLAN Assignment Lists

10.0 District Programming Laptop Request

10.1 General NEORSD requires all PCS changes to be implemented from a District provided PC&A laptop. This is intended to provide consistent software versioning, use of AssetCentre, and to promote PCS system security.

10.2 Procedure The process to request a PC&A laptop from the NEORSD is as follows:

1. Fill out and submit the NEORSD PC&A Programming Laptop Request Form. The form must be filled out electronically and submitted via e-mail to the designated NEORSD Project Manager or other designated contact. NEORSD will supply the blank electronic form on request. A copy of the form is included in Section 17.0.

2. Transmit supporting documentation, such as assigned programmer name, project number/description, expected duration, and required software.

3. NEORSD Staff review of the PC&A Programming Laptop Request will result in the District assigning one of the following statuses:

Accepted – If the request is acceptable to the District, NEORSD will notify the requester in writing and will note any limitations placed and anticipated laptop transfer date.

Rejected – If the request is rejected the District will notify the requester in writing.

The District will return a written decision within 10 working days under normal circumstances. Laptops will generally be available for transfer 10 working days after approval. The District will work with the requester if further explanation or discussion is needed to reach a timely decision.

11.0 Standards Deviation Request

11.1 General The NEORSD PCS Standards and Conventions are intended to provide consistent, reliable controls within all plants. However, it is recognized that there may be specific instances in which a contractor or supplier may not be able to fully comply with the standards for software,

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Standards Deviation Request


equipment, cabling and other control system components supplied or developed under a contract or other procurement method, or may not be able to program or configure the system in conformance with these standards. When these cases arise, the contractor or supplier must submit a request for deviation from NEORSD Standards using the NEORSD Standards Deviation Request Form included in Section 17.0. The District will review the issues documented on the form and any supplemental documentation that is supplied with the request, and if compelling reasons are presented, the District may grant the deviation. The Standards Deviation Request must meet the following criteria:

1. It must clearly indicate those sections of the standards that are relevant to the proposed deviation(s)

2. It must verify that compliance with the standards from which the deviation(s) is requested is, under the circumstances, not feasible

3. It must explain how the deviation(s) meets the fundamental intent of the Standards and the project requirements

4. It must include a specific description of the proposed alternative to the requirements in the NEORSD Standards and include supporting documentation. Documentation may include references demonstrating successful use by other public utilities

5. It must show that such deviations are not contrary to the public interest, health and safety

6. It must verify that the deviation(s) will not compromise safety, functionality, public health, esthetics, and maintainability

7. It must verify that all applicable codes will be met.

For deviations from the District standard for PLCs the contractor must also provide the PLC program, documentation and drawings along with the Standard Deviation Request Form.

11.2 Procedure The process to request a deviation from the NEORSD PCS Standards and Conventions is as follows:

1. Fill out and submit the NEORSD Standards Deviation Request Form. The form must be filled out electronically and submitted via e-mail to the designated NEORSD Project Manager or other designated contact. NEORSD will supply the blank electronic form on request. A copy of the form is included in Section 17.0.

2. Transmit supporting documentation, such as drawings, mark-ups, photographs, sketches, vendor-supplied product information, etc. with the Deviation Request e-mail. If attachments exceed mailbox limits then submission should occur via USB Flash Drive or other means of sharing large files.

3. Additional information may be requested by the District and/or an alternative solution(s) presented by the requester. A meeting may be requested by the deviation requester to explain the issues to the District Project Manager and project team members.

4. NEORSD Staff review of the Standards Deviation Request will result in the District assigning one of the following statuses:

Accepted – If the requested deviation is acceptable to the District, NEORSD will notify the requester in writing and will note any limitations placed on the allowable deviation.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Management of Virtual Local Area Networks


Rejected – If the requested deviation is rejected the District will notify the requester in writing and the work must proceed without delay in accordance with the Contract or other pertinent procurement requirements.

The District will return a written decision within 10 working days under normal circumstances. However, complex issues may require additional time for analysis and preparation of a response. The District will work with the requester if further explanation or discussion is needed to reach a timely decision.

Appeals of the District’s decision shall be submitted within ten (10) calendar days of the date of the decision and shall be submitted in writing to the District’s Project Manager.

12.0 Management of Virtual Local Area Networks

12.1 General Each processing site contains two virtual local area networks (VLANs); the primary VLAN is utilized for all PCS communications. A secondary VLAN is installed for network switch management.

12.2 Procedure IP address and VLAN assignment requests shall be submitted to the PC&A manager.

13.0 Applications Backup and Recovery Policy

13.1 General The Northeast Ohio Regional Sewer District’s (NEORSD) PCS Recovery and Restore Policy is intended to establish requirements to promote efficient and accurate system restoration after a failure has occurred. The NEORSD PCS consists of numerous highly integrated software packages, which undergo continual change for plant maintenance, operational improvements and projects. This Policy and Procedure addresses requirements to promote known good system backups; and procedures to identify and utilize the backups and asset management systems during recovery.

This Policy applies to all NEORSD PCS Software Assets.

13.2 Backup Storage Requirements NEORSD has deployed Rockwell AssetCentre for version control and file storage for PCS Software applications including PLC Applications, OIT Applications, network application files and Wonderware Galaxy Standard Object Templates.

If the application programmer does not have access to the District AssetCentre then the application programmer shall request a data transfer for the following:

Galaxy backup PanelView Template

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Applications Backup and Recovery Policy


AOI Library

Additionally, there are software applications which are not stored within AssetCentre that require additional policies and procedures for proper file backup and storage as part of recovery and restoration procedures including VSphere server configuration and Wonderware Historian.

The following applications shall be stored in AssetCentre at initial installation and after any modification has been deployed in the production environment:

PLC Program Files including network configuration files (DNT, ENT) OIT Application Files Cisco 2960X and 3850 series network switch configuration files Allen-Bradley Stratix series network switch configuration files

All contractors completing on-site startup/commissioning modifications to operating equipment are required to complete change documentation and upload files to AssetCentre at the end of each shift. If the contractor or application programmer does not have access to AssetCentre, they shall submit all files the District PM to upload to AssetCentre at the end of each shift.

Wonderware System Platform backups are stored on the Galaxy Repository. The Galaxy repository is automatically backed up to GR Node on a daily basis.

Software application files not stored in AssetCentre and AssetCentre itself should be backed up no less than weekly. Backups shall be stored on a separate physical server in-plant AND at EMSC. Software not included in AssetCentre subject to this policy includes:

VSphere Server Image Historian Server Image and Database Backup Files Terminal Server Image

Note: Cisco Stratix 8000 series network switches have been installed including a Compact Flash memory card for configuration storage.

13.3 Recovery Procedures PC&A should be immediately notified upon identification of a PCS system or component failure and PC&A representatives, or designated contract support resources, shall be responsible for following recommended recovery procedures and verifying restoration of system stability.

13.3.1 AssetCentre Recoveries In the event of a component failure where backup files are stored within AssetCentre and the AssetCentre server has not failed; the assigned resource shall replace the failed hardware and utilize AssetCentre to push the last known application backup to the end device.

If the component’s backup file is stored on AssetCentre and AssetCentre server failure has occurred, follow 13.3.2 to restore AssetCentre Server and then follow normal recovery procedure listed above.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Controlled Software List


13.3.2 AssetCentre Server Failure Recovery In the event of AssetCentre failure, the AssetCentre must be restored from most current image file stored on EMSC server. Since backups are only taken on a weekly basis, changes implemented since last backup will not be in the backup file. The resource responsible for failure recovery will need to review approved Applications MOC Forms to determine any manual changes that must occur to validate backup files. AssetCentre should be utilized to pull files from any asset which has had documented changes since last server image.

13.3.3 Non-AssetCentre Related Recovery Known good backups will be stored both on-site and at EMSC. Site backups shall be primary source for software restoration. A review of the Applications MOC forms should be reviewed against application backup dates to ensure that no in progress work was lost between last backup image and restoration of the resource.

13.4 Recovery Testing Requirements PC&A shall complete recovery tests for each of the recover methods stated in Section 13.3 no less than quarterly. Each test shall follow the stated recovery method to restore associated equipment and applications within the District test bed.

Failure of any backup during recovery process shall result in a manually initiated and immediate backup for the identified associated live production environment equipment. The manually initiated backup shall also be tested to guarantee a known good and known valid backup. Backup policies and procedures shall be reviewed to determine and resolve issue creating backup recovery failure.

13.5 Documentation Requirements Applications MOC Forms shall be completed retroactively to document the recovery procedures followed including exact file names and dates for application backups used for the restoration.

14.0 Controlled Software List The software listed in Table 14-1 shall be considered controlled software and subject to the revision control, version control and MOC policies of this document.

Table 14-1 Controlled Software

Software Name

Wonderware System Platform

RSLogix 5000

RSLogix 500

RSLogix 5

RSStudio 5000

RSLinx

Rockwell AssetCentre

RSNetworx for Ethernet

RSNetworx for DeviceNet

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Responsibilities Matrix


Software Name

VSphere

VMWare

LANSweeper

15.0 Responsibilities Matrix Table 15-1 contains the responsibilities for NEORSD Employees and contractors as described in the sections above.

Table 15-1 Responsibilities Matrix

Manager of PC&A PC&A Staff Engineering/Individuals Supervising Contractors

PCS Revision Control

Overall implementation of this Policy for the NEORSD;

Coordinate the review/update of this Policy as needed, at software version change, due to PCS changes, or at least every three (3) years and;

Coordinate with E&C and subcontractors for software version and patching requirements during capital improvement projects (CIP).

Assist PC&A Manager with overall implementation of this Policy;

Coordinate and/or perform training sessions for personnel and contractors;

Assist with review/update of this Policy as needed, due to PCS changes, or at least every three (3) years.

Ensure the requirements of this Policy are implemented when contractors work on the PCS.

Controller Firmware Revision


Coordinate the review/update of this Policy as needed, at transition to new firmware release, due to PCS changes, or at least every three (3) years and;

Coordinate with E&C



Assist with review/update of this Policy as


NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Responsibilities Matrix



and subcontractors for firmware requirements during CIP projects.

needed, due to PCS changes, or at least every three (3) years.

Communications Module Firmware Revision


Coordinate the review/update of this Policy as needed, at transition to new firmware release, due to PCS changes, or at least every three (3) years and;

Coordinate with E&C and subcontractors for firmware requirements during CIP projects.





Programming Software Revision


Coordinate the review/update of this Policy as needed, at software version change, due to PCS changes, or at least every three (3) years and;

Coordinate with E&C and contractors for software version and patching requirements during CIP projects.





Applications Management of Change


Coordinate the review/update of this


Coordinate and/or


NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Reference Documents



Policy as needed, due to PCS changes, or at least every three (3) years and;

Assignment of approver for E&C and CIP project modification requests.

Approval of change requests submitted by PC&A staff

perform training sessions for personnel and contractors;


16.0 Reference Documents Table 16-1 contains a complete listing including live links to all referenced District standards and forms, industry standards, and best practices.

Table 16-1 Reference Documents

Document Title

Control of Hazardous Energy (lockout/tagout) – 29 C.F.R. § 1910.147

ISA101, Human-Machine Interfaces Standard

NEORSD Asset Tag Abbreviations File

NEORSD Process Control Description Design Standard

NEORSD Standard Object Library Programming Guideline

One Point Lesson (OPL)

Selection and Use of Work Practices – 29 C.F.R. § 1910.333

17.0 Forms NEORSD forms are provided on the following pages.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Request for Shutdown Form


NEORSD PCS Request for Shutdown Form Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600

Contract Number

Requested by:

When available:

When required:

Early Start Date:

Latest Start Date:

Estimated Man Hours:

Sketch/Drawing:

Affected Building:

Job Description:

Safety Precautions and Instructions

Approvals Required:

District Operations:

District Engineering:

Scheduled Shutdown Date:

Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Request for Shutdown Form - EXAMPLE


NEORSD PCS Request for Shutdown Form - EXAMPLE Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600

Contract Number 1381

Requested by: W. White

When available:

When required:

Early Start Date: 01-20-2008

Latest Start Date: 01-27-2008

Estimated Man Hours: 40

Sketch/Drawing: See Attached

Affected Building: Solids Handling

Job Description: Replacing the existing Incinerator PLC 5

Safety Precautions and Instructions

Approvals Required:

District Operations:

District Engineering:

Scheduled Shutdown Date:

Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Contractor Extended Program Check-out Request Form


NEORSD PCS Contractor Extended Program Check-out Request Form

Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600

Date of Request: Project Number:

Name of Requester: Name of Firm:

Phone No.: E-mail address:Anticipated Check-out Date:

Anticipated Check-in Date:

Attach additional sheets as needed to provide the information required below:

List of Program Files Requested:

Fully explain reason for extended request:

Supporting documentation attached? Yes No

Requester’s Signature Date

THIS AREA FOR NEORSD USE ONLY

Date Received:

Date Returned:

Reviewed by:

Authorized NEORSD Signature

District Decision: Accepted Rejected

Comments/Limitations:

Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Contractor Extended Program Check-out Request Form-EXAMPLE


NEORSD PCS Contractor Extended Program Check-out Request Form-EXAMPLE


Date of Request: 01-20-2008 Project Number: 1381

Name of Requester: W. White Name of Firm: A1A Integration

Phone No.: 216-555-5555 E-mail address: [email protected]

Anticipated Check-out Date: 01-21-2008

Anticipated Check-in Date: 01-31-2008


List of Program Files Requested:

Westerly PLC W20 Application

Fully explain reason for extended request:

Complete logic rewrite for optimization as part of CIP Project “EXAMPLE PROJECT”

Supporting documentation attached? Yes X No

W. White

01-20-2008



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Change Request Form


NEORSD PCS Change Request Form Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600



Phone No.: E-mail address:


Full description of change:

PLC Name(s):

Subroutine(s):

HMI Screen(s) Affected:

Historian Modifications:

Fully explain reason change:

Explain or Attach Back-out Plan:

Supporting documentation attached? Yes NoTest Bed sign-off attached? Yes NoRetroactive for Emergency Change? Yes NoProduction hold required? Yes No



Hold Date/Time Period:



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Change Request Form-EXAMPLE


NEORSD PCS Change Request Form-EXAMPLE Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600





Full description of change:

Addition of new OOS AOI per approved PCD.

PLC Name(s):

Southerly S46_INC1

Subroutine(s):

All control routines

HMI Screen(s) Affected:

Incinerator 1

Historian Modifications:

None

Fully explain reason change:

Addition of Out of Service AOIs to support alarm management plan.

Explain or Attach Back-out Plan:

Full revert to pre-modifications back-up file dated 1/20/2008 stored in AssetCentre.

Supporting documentation attached? Yes X NoTest Bed sign-off attached? Yes X NoRetroactive for Emergency Change? Yes X NoProduction hold required? Yes X No

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures


Hold Date/Time Period:

W. White

01-20-2008



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0


Appendix A Back-out Plan


NEORSD PCS Change Request Form Appendix A Back-out Plan






Outage and/or Testing Time Requested:

Maximum Allowable Time before acceptance or back-out:

Back-out Plan:

(Describe the specific steps to restore system to pre-change state)

Emergency Contact Name:

Emergency Contact Phone:



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PCS Change Request Form Appendix A Back-out Plan-EXAMPLE


NEORSD PCS Change Request Form Appendix A Back-out Plan-EXAMPLE






Outage and/or Testing Time Requested: January 27th – Feb, 2nd, 2008. One (1) week

Maximum Allowable Time before acceptance or back-out: Three (2) days

Back-out Plan:

Revert logic in PLC_Name by reset of test bit named STANDARDS_TEST located at top of Main routine. STANDARDS_TEST = 0 (original code), STANDARDS_TEST = 1 (new code)

Emergency Contact Name: W. White

Emergency Contact Phone: 216-555-5555

W. White 01-20-2008



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PC&A Programming Laptop Request Form


NEORSD PC&A Programming Laptop Request Form Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600

Date of Request: Project

Number:


Phone No.: E-mail

address:


Name of Programmer(s):

Expected Duration of Use:

Required Software Application(s):



Date Received: Asset:

Date Returned:

Reviewed by:




Rev. 1.0

The Requestor assumes responsibility for the asset assigned to them and must replace the asset if it is lost or stolen. The sharing of laptops,

passwords and login information is prohibited.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD PC&A Programming Laptop Request Form-EXAMPLE


NEORSD PC&A Programming Laptop Request Form-EXAMPLE Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600

Date of Request: 01-20-2008Project

Number: 1381


Phone No.: 216-555-5555E-mail

address: [email protected]


Name of Programmer(s):

W. White

Expected Duration of Use:

Two (2) Months

Required Software Application(s):

RSLogix5000

W. White 01-20-2008



Date Received: Asset:

Date Returned:

Reviewed by:




Rev. 1.0

The Requestor assumes responsibility for the asset assigned to them and must replace the asset if it is lost or stolen. The sharing of laptops, passwords and login information is prohibited.

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD Standards Deviation Request Form


NEORSD Standards Deviation Request Form Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600





Standards(s) for which a deviation is requested (list document name, section(s), subsection(s):

Fully explain reason for deviation request:

Proposed alternative implementation:

Supporting documentation attached? Yes No

Signing below indicates that the Requestor has verified compliance with the requirements of the Standards Deviation Request procedure included in the NEORSD Automation Standards and Conventions Manual



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD Standards Deviation Request Form-EXAMPLE


NEORSD Standards Deviation Request Form-EXAMPLE Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600





Standards(s) for which a deviation is requested (list document name, section(s), subsection(s):

Use of Structure Text for advanced calculations

Fully explain reason for deviation request:

Advanced calculations required by approved PCD would require an excessive number of rungs to complete in ladder logic. The resulting logic would be difficult to troubleshoot, use of structured text will make the logic readable and easy to compare to the approved PCD.

Proposed alternative implementation:

None

Supporting documentation attached? Yes X No

Signing below indicates that the Requestor has verified compliance with the requirements of the Standards Deviation Request procedure included in the NEORSD Automation Standards and Conventions Manual

W. White

01-20-2008



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD Network Modifications Management of Change Form


NEORSD Network Modifications Management of Change Form Northeast Ohio Regional Sewer District 3900 Euclid Ave. Cleveland, OH 44115 216-881-6600




Title Change:

URGENCY: Emergency (NO delay possible) High (Delay has significant Risk) Medium (Delay has modest risk) Low (Delay is only inconvenient)

Justification (Justify why the proposed changes should be implemented):

Coordinators(s)

Name: Project Role: Phone Number:

Impact of not implementing (Explain the impact if not implemented):

Description (Describe the proposed change):

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD Network Modifications Management of Change Form


Affected Parties and Systems (Worst case scenario):

Implementation Process:

Detailed Technical Steps:

Start/Finish Times (Estimated time to complete):

Back Out Method:



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures NEORSD Network Modifications Management of Change Form - EXAMPLE


NEORSD Network Modifications Management of Change Form - EXAMPLE





Title Change:

REF Switch Upgrade

URGENCY: Emergency (NO delay possible) High (Delay has significant Risk) Medium (Delay has modest risk) XLow (Delay is only inconvenient)

Justification (Justify why the proposed changes should be implemented):

Many of the switches at REF are end of life. They will not be supported by Cisco and are at risk of failing. This type of failure occurred with S21 at the 2nd Stage Lift station and caused several weeks of network instability.

Coordinators(s)

Name: Project Role: Phone Number:

Welch, Mark <[email protected]>; Lead RoviSys 440-667-4782

Scott Sander <[email protected]> Lead NEORSD 216-789-6342

Bretmersky, Jason <[email protected]>;

Coordinating other SFO work

Impact of not implementing (Explain the impact if not implemented):

Support for network switches unavailable. Old hardware end of life and prone to failure



Description (Describe the proposed change):

Remove several pairs of switches in 5 locations and replace them with new models with stacking features. The stack feature allows for better management and improved port usage.

Affected Parties and Systems (Worst case scenario):

S47, S57, S67, S38, S51. REF communications internal and external. A spanning tree event will occur at the beginning of this change. This will likely stop all network traffic for 2 to 5 minutes.

WORST CASE SCENARIO:

Total loss of SCADA system, total loss of historian and PLC data.

Implementation Process:

Confirm there are no other changes that may interfere. Make required notifications for start Call the Shift Manager and any others directly impacted Move SCADA polling to primary servers (if applicable) Verify systems are functioning and note any issues before work starts Perform any backups Confirm any staging required is in place

o Lift Station at Southerly fail safe is disable and location is manned o Hand readings are being made in case of system loss o Safety precaution are taken and PPE is used as required o Ensure access to equipment and keys or passwords are available

PERFORM THE CHANGE AS PLANNED. NOTE ANY DEVIATIONS Call the Shift Manager and any others directly impacted. Make required notifications for finish Reply to shutdown notification to document completion Note any lessons learned



Detailed Technical Steps:

Before Change Load install angry IP to PCA laptops Run angry IP to verify it will not have issues Verify spanning root tree. Note Per vLAN spanning tree to be used.

During Change

1. Run angry IP to capture up nodes 2. GET SUPERVISOR APPROVAL TO START WORK (GATE) 3. Move polling to one server like PMs and patching 4. Tie two networks 5. Run angry IP again to capture up nodes 6. Cut over copper REF Bldg Rm 301 (Server Room) to S47_NET_BLDG Stack Cisco 2960X 7. Cut over router SRREF1 (192.168.30.190) 8. VLAN trunk port S39_NET_COR - SPANNING TREE EVENT 9. Move the following switch ports over:

a. S57_NET_BLDG REF Bldg Rm 301 (Server Room) i. Run angry IP to capture up nodes

b. S47_NET_BLDG REF Bldg MCC Room (Biosolids) i. Run angry IP to capture up nodes

c. S67_NET_BLDG Steam Turbine i. Run angry IP to capture up nodes

d. S38_NET_BLDG Sludge Storage i. Run angry IP to capture up nodes

e. S51_NET_BLDG Skimmings Building i. Run angry IP to capture up nodes

10. Pull power from old switches. 11. Run angry IP to capture up nodes 12. Play what’s in my hand for any connections not known and described in the switches starting

with trunk ports. 13. Document what IP shows up for each unknown drop 14. Run angry IP again (last time) to capture up nodes

After Change

Add the new switches to monitoring systems and decommission old. WAM, GestioIP, LANsweeper etc.

Update all switch configs and documentation. Share this with APM.

Start/Finish Times (Estimated time to complete):

START: 10/03/2017 @ 9AM FINISH: 10/04/2017 (approx. 24 hours). We will work two 12 hour days and pause work in the evening as needed.

Back Out Method:

We can restore the old switches with a physical reconnection. The change is set up run partially in parallel with the old system.



NOTES: OLD Asset Names NEW Asset Names

S47_NET1_A S47_NET1_B S47_NET2_A

S47_NET2_B S47_BLDG5A S47_BLDG5B

Becomes S57_NET_BLDG

S47_NET3_A S47_NET3_B S47_NET4_A

S47_NET4_B

Becomes S47_NET_BLDG

S67_NET_A S67_NET_B Becomes S67_NET_BLDG S51_NET_SKM1 S51_NET_SKM2 Becomes S51_NET_BLDG S38_NET_SS1 S38_NET_SS2 Becomes S38_NET_BLDG

W. White

01-20-2008



Date Received:

Date Returned:

Reviewed by:




Rev. 1.0

NEORSD PCS Manual | Volume 1: Introduction, Definition, Policies, and Procedures Forms


End of Volume 1

3900 Euclid Ave Cleveland, OH 44115 216.881.6600

neorsd.org