7x24 nor cal om best practices presentation rev4 9-1-2011

8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011

1/41

Presented by:

Terry Rodgers, CPE, CPMP

Vice President, Sustainable Operations

Primary Integration Solutions, Inc.

[email protected]

DATA CENTER O&M BEST

PRACTICES


2/41

www.primaryintegration.com

Data Center O&M Best PracticesPerspectives

Designing for Maintainability

Commissioning, Construction, Startup & TestingO&M Staffing and Organization

Operations & Maintenance Processes

Existing Facilities

Agenda

2


3/41


Data Center O&M Best Practices

As the robustness & complexity of critical facilities has increased for

improved fault-tolerance and concurrent maintenance, the importanceof equally robust O&M practices has become apparent.

As much as 75% or more impact events where critical missions

have been compromised are associated with human Activity.

3


4/41

www.primaryintegration.com 4

A&E Programming, engineer & design, submittals & RFIs, CA

6 to 9 months of intense effort then some minor CA support

GC & Contractors Propose, purchase, construct, punch/commission9 to 15 months with minor follow-up effort

Mfgrs & Vendors Submittals/shop dwgs, assemble, startup & commission3 to 6 months and a 12 month warranty + SLAs

CxA Design, construction, startup, acceptance, training & documentation9 to 15 months plus 10 month pre-warranty return effort

Owner Design, construct, commission, transition, sustained operations15 months design/construction, 12 months transition, 24 yearsoperations

Perspectives


5/41


The A&Es, GC & Contractors, Manufacturers and Vendors, andCommissioning Agents completion date (i.e., project close-out)

is the Owners start date (i.e., Day-1 of mission critical operations)

Dont let Sustainable Operations be an after-thought!

Perspectives


6/41




7/41


Include facility O&M considerations in the programming phase

Define facility O&M needs and expectations in the Basis-of-Design

Specify the associated O&M deliverables in the Construction Documents


7


8/41


Include O&M considerations in the programming phase via the Owners Project Requirements(OPR) document

O&M Organization, Reporting Structure, & Budget

O&M Staffing (24x7 Coverage, Trades, Quals, etc.)

In-House, Outsourced, SLA, or a combination

Training Program (Day-1, New Hire, & Recurring)

O&M Programs & Processes

CMMS (new or expand existing)

Asset Management System (inventory control)

Configuration Management System

Comprehensive As-Built Documentation & Doc Control

Facilities Management space-planning (offices, command center, shops, etc.)


8


9/41


10/41


CDs (Drwgs & Specs) should define O&M deliverables as part of the overall construction project

Equipment, System, & Space naming and color conventions

Signage, Bar-coding, RFID, etc.

A vs. B plants (mechanical & electrical) should be easily differentiated

Color code systems (mechanical, electrical, controls, critical vs. non-critical, etc.)

Valve and switch tagging

Drawings should include Valve and Switch schedules (tables)

Specifications should dictate size, material, location, ease-of-view, etc.

Maintenance and access requirements

Shown on drawings, permanently identified in field

Access panels with labels where covered/closed-in

Heavy equipment transport including truck/forklift routes, overhead rigging, pick-points

Emergency lighting (response vs. egress)

Storage and spare parts requirements


10


11/41


Most (if not all) Data Centers are designed with the expectation they will be expanded or modifiedduring their lifespan

The design must accommodate this expectation

Appropriate means to isolate infrastructure between production from expansion

Avoid loss of redundancy schemes during construction

These situationsincrease the risk of impact events occurring and are when quality O&M

processes are most important

O&M staff well trained and practiced at reconfiguring systems and isolating equipment and

distribution paths

Updated and accurate as-built docs and operating procedures

Critical work authorization review and approval processes

Routine means to test/verify isolation points (valve leak tests, breaker tests, etc.)


11


12/41


Specifications should include defining close-out documentation As-Built (vs. redlines) and exact content and formats

Redlines maintained throughout construction

Formal Record Drawings (As-Built) in hardcopy & electronic

Field verified for accuracy prior to acceptance

O&M Training details, deliverables, content, effort, and quality

Systems Operations & Maintenance Manuals (vs. Manufacturers O&M manuals)


12


13/41


Systems Operations & Maintenance Manuals (SOMMs) should be organized by category(HVAC, FP, Security, BMS, etc.), system, & sub-system

Include for each system:

Descriptive narrative

Single-line diagrams (one for each operating mode and/or configuration)

Sequences-of-Operations, setpoints, alarm thresholds, safeties, etc. BMS points lists (DIs, DOs, AIs, AOs, etc.)

Standard Operating Procedures (Normal, Maint., Emergency, Recovery, etc.)

Approved submittals

Commissioning scripts including baseline data

Spare Part Lists (including on-site inventory)

Warranties

Service Level Agreements


13


14/41


Specifications should define O&M Training Curriculum, Scope, & Content Venues (classroom, shop, on-the-job)

Instructor qualifications

Duration and times (i.e., shift coverage)

Testing and/or certifications

Safety considerations (OSHA, Arc Flash, PPE, etc.)

Operating procedures (normal, maintenance, emergency, & recovery modes)

Equipment level, Systems level, and Integrated Systems level scenarios

Maintenance procedures (inspections, preventive, predictive, corrective, replacement, etc.)

NOTE: Commissioning affords excellent training opportunities


14


15/41


Commissioning, Construction, Startup & Testing


16/41


The overarching goal is to deliver the site, staff, and processes totally constructed andprovisioned on Day-1 to support 24x7 mission uptime for the life of the facility.

Commissioning Process Design Phase

Commissioning Process Strategy

Use the Commissioning Process to Provision for O&M

Transition from Construction to Production

ASHRAE Guideline 0-2005 The Commissioning Process is an excellent reference and

user-friendly guide geared towards new construction. A sister guideline is currently

being developed for commissioning existing facilities.


16


17/41


Commissioning process (Cx is not a procedure) Design Phase

Ensure the OPR captures construction QC, performance validation, and facility O&M

considerations

Ensure the BOD complies with the OPR and defines high-level requirements

Ensure the CDs comply with the BOD and include clear and concise specifications

Ensure the CDs capture the requirements and effort associated with Commissioning

(including documentation & training)

Develop the Commissioning Plan to ensure the built product can be operated,

maintained, and tested to validate performance


17


18/41


Commissioning Process Strategy Identify issues and discrepancies as early as possible so they can be resolved with least

impact on schedule, resources, and costs

Factory Witness Tests identify issues prior to shipment

Field Component Verification identifies issues upon receipt and ensure proper

storage & care prior to installation Systems Construction Verification (progress inspections) identify issues prior to

start-up and functional testing

Site Acceptance Testing identifies issues at the component, equipment, and system

level before testing interfaces

Integrated Systems Testing identifies issues at system interfaces and facility-widelevel prior to going live.


18


19/41


Use the Commissioning Process to provision for O&M Use the Commissioning design review process to ensure the OPR, BOD, and CDs capture

O&M needs

O&M staff participation in commissioning (FWTs, inspections, and testing) affords in-depth

ops training and establishes contacts with the EOR, mfgrs/vendors, and contractors

Submittals, Field Component Verification, equipment startup forms, and completed test

scripts can capture and document data for input to the CMMS, AMS, & CMS and spare

parts collection & inventory

Functional Tests and Integrated Systems Tests set the original baseline for performance


19


20/41


Use the Commissioning Process to provision for O&M The Commissioning Process includes ensuring the built facility is thoroughly documented

System Operations & Maintenance Manuals should be developed concurrently with the

construction phase

SOMMs are a compilation of data and documents from:

OPR, BOD, and CDs

Record Drawings (As-Builts)

Approved Submittals

Final Commissioning Report & completed test scripts

Warranties, SLAs, etc.


20


21/41


Use the Commissioning Process to provision for O&M Training Start with a comprehensive Training Plan

Develop a Training Program that is site and client specific

Embed training in the construction and acceptance testing phases

Utilize the site SOMMs to support and supplement training materials

Design training to match O&M staff position descriptions

Integrate project specific technical training with Corporate policy & procedures training

Training material should be editable so it can be updated and kept current

Training delivery should be recorded for future use (new-hires, remedial training, periodic

re-training, and refresher training prior to infrequently performed activities)


21


22/41


O&M Staffing and Organization

Remember the following should be addressed during the programming

phase and should influence how the facility is designed and built!


23/41


Use an Enterprise reporting structure Critical Facilities O&M should be a department within the IT division

Fundamentally eliminate the silos

Recognize the mutual support and dependence of IT and Facilities on each other

Combined goals drives collaboration and planning (sink or swim together)

Eliminate competition for resources (staff & $$$)

Combine NOC and Facilities Command Center operations into an Enterprise

Command & Control Center

Formalize and communicate load vs. capacity tracking, trending, and management

O&M Staffing & Organization

23


24/41


Staff the O&M department to meet site specific needs Start with a site specific O&M Job Analysis (dont confuse jobs with positions)

Compile list of Jobs (tasks) required to O&M the site infrastructure

Differentiate between operations and maintenance tasks

Develop Job Descriptions that group tasks and assess what requisite skills and

knowledge are required

Identify all that pertain to critical facilities

Consider what qualifications, certifications, or other credentials are necessary where

appropriate

Decide how critical vs. non-critical aspects will be structured Decide how operations vs. maintenance aspects will be structured


24


25/41


Staff the O&M department to meet site specific needs - continued Develop Position Descriptions

Position Descriptions describe the jobs and associated tasks, duties and

responsibilities expected for a specific staff position

Assemble with the following considerations:

Combined O&M or separate Operations from Maintenance

Jack-of-all-trades or Trade-specific (mechanic, electrician, controls tech, etc.)

True 24x7 coverage or Business Day + On-Call for off hours

Compile a list of duties and responsibilities based on the tasks and associated skills

and knowledge from the Job descriptions Decide how to staff each position (Direct employees, in-house contractors,

SLAs, or a combination)


25


26/41


Staff the O&M department to meet site specific needs - continued Develop an Organization and Reporting Structure

Enterprise level organization

Facilities O&M level organization

Critical Facilities vs. Non-Critical Facilities

Operations with shift coverage

Maintenance broken down by trades

Electrical (Med Volt, Low Volt, Gensets, UPS, etc.)

Mechanical (Central Plant, HVAC, Chem Treatment, etc)

Monitoring & Controls (BMS, EPMS, calibrations, etc.) Vendors, Contractors, SLAs, etc.

Administration (CMMS, procurement, work approvals, etc.)


26


27/41




28/41


Start with well defined Facilities Management mission Develop a mission or Vision statement

Create a Strategic Plan with well defined goals and tactical objectives

Develop programs and processes first, then supporting policies &

procedures


28


29/41


Establish Operations Program and Processes

Work authorization process (routine, PM, emergency, etc.)

Event notification & escalation protocols

Define Operations Key Performance Indicators (KPIs)

What defines operational excellence and how is it measured?

Critical vs. Non-critical boundary and operating protocols

Practice (dry-runs)

Double verification

Mimic boards or other real-time configuration status

Valve and switch position status tags

Lock-out/Tag-out and other Safety Programs


29


30/41


Establish Operations Procedures

Specific to each system

Normal, maintenance (configuration), emergency, and recovery procedures

Include pre-requisites and cautions

Work authorization requirements

Safety considerations (lock-out/tag-out, PPE, etc.)

BMS and alarm lock-outs

Verification of status of redundant or backup systems, etc.


30


31/41


Establish Operations Procedures - Continued

Provide step-by-step procedure including action to be taken, expected result (alarm,

indication, action, etc.), and means to collect resulting data (actual alarm, reading, action,

etc.).

Embed digital photos of equipment, components, and especially displays and control

panels

Annotate with arrows and relate to specific steps, actions, readings, etc.

Consider using a flowchart format

Include contingency and direction if expected does not match predicted

Include completion/closeout checklist to ensure all aspects of system are restored to

normal operating status (reactivate BMS, remove switch tags, lockouts, notify NOC, etc.)


31


32/41


Establish Maintenance Program and Processes

Define Maintenance Key Performance Indicators (KPIs)

What defines maintenance excellence and how is it measured?

Develop programs to manage assets, configurations, and documentation

Asset Management System (AMS)

Bar-code equipment and components

Computerize inventory, location, and who is authorized to pull stock

Include shelf-life, PMs (rotate shafts, etc.), and other storage considerations

Source, vendors, warranties, spec sheets, etc.


32


33/41


Establish Maintenance Processes - Continued

Configuration Management System (CMS)

Computerized database of deployed assets & interconnectivity

Sum loads and capacities to track & alarm at defined thresholds

Can be integrated with Work Flow processes so asset deployment, relocation, or

decommissioning automatically generates updates to CMS Can combine both IT, Network, and Facilities infrastructure

Utilize actual real-time metering (BMS, branch-circuit monitoring, Tivoli, HP Openview,

etc.) where available

Use eDiscovery for auto audits where possible

Set thresholds low enough to ensure redundancy and fail-over strategies remain valid


33


34/41



Computerized Maintenance Management System (CMMS)

Implementation tool for the Maintenance Program

Include install date, warranty duration, lifespan/replacement date

Provides input to develop and maintain 5-year & 10-year Capital Plans

Meet or exceed manufacturer recommendations

Stagger dates for work load leveling, seasonal considerations, & freeze windows

Tie to Maintenance Procedures

Separate daily, weekly, monthly, quarterly, etc

Develop logical ID convention that facilitates filtering, searching, and reporting ease

(from design phase!!!)


34


35/41



Types of Maintenance Programs

Corrective run to failure

Preventive routine based on frequency or run-hours

Predictive On-line condition monitoring with thresholds; extrapolate trends to predict

needs (predict failures) Vibration analysis (snap-shot or continuous via multiplex)

Thermography (IR Scans) and temperature sensors

Tribology (lubricant analysis)

Laser Alignment

Computerized Balancing Combination of all above

Input field notes and create equipment histories and review routinely


35


36/41


Establish Maintenance Procedures

Specific to each system and equipment

Separate procedure for daily, weekly, monthly, quarterly, etc.

Include pre-requisites and cautions (similar to Ops procedures)

Provide step-by-step procedure (similar to Ops procedures)

Include acceptance criteria, tolerances, settings, etc.

Include access considerations (ladder, confined space, PPE, key#)

Include ability to take field notes, comments, technician ID, etc.

Include completion checklist to ensure all aspects of system are restored to normal

operating status (reactivate BMS, remove switch tags, lockouts, etc)


36


37/41


Implement BIMS(Building Information Modeling System)

BIMS products are still relatively new to the industry

BIMS software uses databases to link electronic data, records, and documents including

AutoCAD drawings

BIMS includes virtual 3D models in layers such that you can simulate the construction and

identify conflicts, fouls, and sequence conflicts

BIMS can link AutoCAD objects to associated specs, quantities, suppliers/vendors,

submittals, warranties, inventory, ordering info, maintenance history, and almost any other

electronic record

BIMS can link CDs to CMMS, AMS, CMS, etc.

BIMS implementation, use, and upkeep requires significant pre-planning and resource

allocation


37


38/41


Existing Facilities

Build / improve programs to support management, staff, and infrastructure using the

Sustainable Operations Services (SOS) Methodology

The SOS Methodology:

1) Define the requirements and establish acceptance criteria

2) Evaluate / Assess actual conditions and develop a gap report3) Make recommendations to bridge the gap4) Build the needed program(s) based on client/site-specificBest Practices5) Build the associated processes and procedures6) Train management and staff on the program(s), processes, & procedures7) Implement the program(s), processes, and procedures

8) Test, verify, audit the results9) Reassess and apply Continuous Process Improvement


38


39/41


Data Center O&M Best Practices

As with Critical Facility design, there are few absolute right and wrong

solutions. The O&M Best Practice for a given facility depends upon

many (often conflicting or competing) variables.

What is most important, is plan ahead, select a solution, and then

execute in a high quality manner.


40/41


Sustainable Operations

Data Centers will undergo change during their lifespan (expansions,

infrastructure upgrades, & technology refresh, etc.)

Industry Best Practices will change and evolve

Long-term Sustained Operations require programs with integral

performance measurement and Continuous Process Improvement


41/41