The Road to Predictive Operations

Brandon Porco

Network Operations Manager

www.NorthropGrumman.com

Northrop Grumman Corporation

Northrop Grumman is a leading global security company providing innovative systems, products, and solutions in aerospace, electronics, information systems, shipbuilding and technical services to U.S. and international military, government and commercial customers worldwide.

Operations in all 50 states and 25 countries

120,000 employees

$17 B Market Cap

$33.8 B in Sales

The Need for IT Transformation at Northrop Grumman

Greater enterprise focus: bring balance between sectors and enterprise

Service Catalog vs. custom

Enterprise architecture vs. local solutions and one-offs

Limited information architecture

Re-use vs. re-building

Implement technological advances within our service offerings

Complex funding model

Funding managed almost entirely by sectors

No line of sight to financial plan or cost of services

Driving Factors

Historic reporting

Current operational status

Management by Excel

But that‟s not how you drive… right?

(and no, this isn‟t my car)

Driving Towards Predictive Operations

When we drive, we look ahead. We look behind. We look at the current operating status… and we steer. It‟s real-time situational awareness and decision making.

Predictive Operations is a Shift in Thinking

It‟s not about tools…yet

It‟s about avoiding downtime

Always On requires 100% uptime, not 99%

Customers expect 100%, soon they will demand it

99% availability won‟t be good enough

It isn‟t about high-availability networking

Reactive metrics, such as % Uptime, Cost of Downtime will become irrelevant

Cost of Downtime Avoided/month will become a more meaningful metric to show the value of predictive operations.

To achieve this, you have to know about a problem before it becomes downtime – hence, Predictive Operations

It‟s A Change From Reaction to Prediction

Reactive Predictive

REGION SECTOR UNIT LOCATION% WAN

AVAILABILITY

UTILIZATION

%

LATENCY

milisecondsROOT CAUSE?

ROOT CAUSE

CATEGORYPLANNED? ACTION OWNER

West NGTS TSG Albuquerque 99.97 35.77 77

East NGTS LCOE California 99.25 17.9 60 AT&T circuit problem Circuit No Corrected

East NGTS TSG Columbus 99.97 12.21 38

Central NGTS SSG Fort Polk 100 0.26 21

East NGTS SSG Ft. Eustis 99.99 18.45 56

West NGTS SSG Ft. Irwin 100 46.11 102 No Response

Central NGTS TSG Grand Prairie 100 5.74 15

East NGTS LCOE Hagerstow 99.99 32.18 21

East NGTS TSHQ Herndon 99.99 8.41 45

East NGTS LCOE Indianapolis 99.95 18.17 47

Central NGTS LCOE Jacksonville 98.93 3.32 41 Unknown Unknown No Corrected

Central NGTS TSG Killeen 99.99 13.8 67

Central NGTS LCOE Lake Charles 100 57.1 17

West NGTS SSG Lakewood 100 19.8 76

West NGTS SSG Lakewood 100 43.75 96 No Response

East NGTS LCOE Lanham 99.99 2.18 10

West NGTS TSG Layton 98.73 48.98 108 Power Outage - Weather Power No Corrected

Central NGTS TSG Leavenworth 100 26.94 31

East NGTS LCOE Macon 99.82 6.65 40

East NGTS LCOE McGuire 100 1.76 53

East NGTS LCOE Middle River 99.66 33.8 33 Power Outage - Weather Power No Corrected

Central NGTS TSG Montgomery 100 7.07 36

East NGTS LCOE Norfolk 99.96 27.97 89 No Response

West NGTS LCOE Oak Harbor 100 12.41 72

East NGTS LCOE Oceana 99.78 12.08 52 Power Outage Power No Corrected

Central NGTS LCOE Oklahoma 99.99 42.06 14

East NGTS TSG Radcliff 100 30.67 70

West NGTS LCOE San Diego 99.99 33.31 72

West NGTS LCOE Sierra Vista 99.99 30.26 33

East NGTS TSG Suffolk 99.97 67.13 67

West NGTS LCOE Travis 99 5.15 56 Unknown Unknown No Corrected

East NGTS LCOE Warner Robins 100 63.1 50

East NGTS LCOE Warner Robins 100 51.69 31

East NGTS LCOE Warner Robins 100 31.75 61

East NGTS LCOE Warner Robins 99.68 69.22 52 Circuit Problem Circuit No Corrected

East NGTS LCOE White Marsh 99.99 12.13 16

Effective Predictive Operations Define „Normal‟ First

The Road to Predictive Operations

Observe and Learn before you buy any tools

Drive Simplicity & Transparency into your operations

Drive Standardization (This is KEY)

Search for Meaningful Metrics (we‟ll talk about that in a bit)

Remember, it‟s a Process – not a tool

1/3 Tool, 2/3 Prep (when it finally is time to look at a tool)

Meaningful Metrics Answer Relevant Questions

Week Ending Week Ending Issue Resolution Status

1/8/2006 B 2/5/2006 B

1/15/2006 B 2/12/2006 B

1/22/2006 B 2/19/2006 Y New installation had logging set incorrectly

1/29/2006 B 2/26/2006 B

3/6/2006 B

3/13/2006 B

3/20/2006 B

3/27/2006 B

Legend

B 100%

G 95% to 99%

Y 85% to 94%

R Below 85%

After – “What types of tickets are there?”

Before – “Are we closing our tickets?”

Meaningful Metrics Are Self-Evident

Before After

Equipment EOL

0

1000

2000

3000

4000

5000

6000

7000

Sep-06 Mar-07 Apr-07 May-07 Jun-07 Jul-07 Aug-07 Sep-07 Oct-07 Nov-07 Dec-07

Devices

OK EOSA EOS/EOSW EOCR EOL Other0

5

10

15

20

25

30

2007 2008 2009 2010 2011 2012

Number of Devices Reaching End of Life per Year (EOCR/EOL)

Meaningful Metrics Are Actionable

Before After

0

50

100

150

200

250

300

350

400

Sep-07

Oct-07

Nov-07

Dec-07

Jan-08

Feb-08

Mar-08

Apr-08

May-08

Jun-08

Jul-08

Aug-08

Sep-08

Oct-08

Nov-08

Dec-08

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09

Aug-09

Sep-09

<40 <80 <120 >120

Meaningful Metrics Highlight Trends

Before After

Our Tools

ITIL Service Framework Software

Request-to-Production Lifecycle management

Automated Network Change & Configuration Management

Integrated event management and automatic remediation of mis-configured elements

Network modeling prior to configuration changes

Network mapping linked to ITIL database

Performance analysis, near real-time trends/health data

Automating manual workflow processes

DNS, DHCP, IPAM

Firewall management & rule set modeling

Integrated governance and risk compliance

Correlate and analyze real-time streaming data from disparate and complex systems into a cohesive view

Key Takeaways

Shift your thinking

It‟s not a tool problem

This is a process and will take time

1/3 tool, 2/3 prep

Standardization is key

Use meaningful metrics as a guide (and eventually to predict)

Events and thresholds need to be defined anyways, so do it early in the process

Don‟t get lost in the bells & whistles

Keep your focus – predict issues to avoid downtime

Research Resources

PNL Site: http://predictiveanalytics.pnl.gov/about.stm

Gartner‟s PBS article link: http://www.gartner.com/technology/research/reports/pattern-based-strategy.jsp

SPC example: http://www.infinityqs.com/espc.html

Ionix Datasheet: http://www.emc.com/collateral/software/data-sheet/h5743-it-perf-report-ds.pdf

Splunk Site: http://www.splunk.com/?r=header

Opnet link: http://www.opnet.com/solutions/application_performance/riverbed.html

Article on Predictive Ops: http://www.thefreelibrary.com/Bright+future+for+predictive+network+management.(Database+&+Network...-a0107928301

Thank You

Brandon Porco

Network Operations Manager

www.NorthropGrumman.com