brksan-2891

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

1

© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKSAN-289114573_05_2008_c1 2

Advanced MDS SAN Management

BRKSAN-2891


2

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3BRKSAN-289114573_05_2008_c1

Session Objectives

Cisco Fabric Manager Overview

Fabric Manager Deployment Best Practices

Review Install Options

SAN Monitoring Details

Overview of Performance Monitoring and Reporting Features


Fabric Manager Overview


3


Fabric Manager Evolution

Client and Server in a

Single Application

Performance Collection and Web

Client

Oracle 10g

Option

JBOSS, FM Standalone,

HSQLDB Replaced with

PostgreSQL DB

FM 1.0December 2002

FM Split into Client and Server

FM 1.3December 2003

FM 2.0October 2004

FM 3.1January 2007

FM 3.2October 2007

FM 3.3March 2008

Performance Enhancements

Releases Representing Major FM Architecture Changes


Cisco Fabric Manager Components

Fabric Manager serverCentralized services

Fabric Manager clientFabric topology view

Fabric management

Device Manager clientDevice specific view

Device management

Web clientHistorical performance reports

Operational views

Cisco Fabric Manager Standalone

Cisco FabricManager Client

(FM/DM/Web Client)

Cisco Fabric Manager Server


4


Fabric Manager Server

Centralized management services

Continuous health monitoring

Multiple fabric management

Performance monitoring

Prediction analysis

Roaming user profiles

Published database schema


Fabric Manager Client

Real-time fabric topology views

Fabricwide configuration wizards

Fibre Channel troubleshooting tools

Health and configuration analysis tools


5


Device Manager

Standalone application communicates directly with the device over SNMP

Graphical representation of the switch chassis

Configure device-specific functionality

Real-time statistics


Fabric Manager Server License Package

Multiple fabric management

Historical performance monitoring

Thresholds based on performance monitoring

Performance prediction

Summary and drill-down reports

Continuous health and event monitoring

Roaming user profiles

Cisco Fabric Analyzer integration

FM Server License Provides Additional Functionality with No Further Software Installation Required


6


Web Client

Requires FMS license to be fully operationalOperations viewHistoric performance report viewsThresholds based on collected performance dataPrediction capabilities based on collected performance dataInventorySyslog collectorCustom reportsSchedule report generation


Fabric Manager Standalone:3.2 and Later

Fabric Manager server and client run in a single process

Switch events collected only while the application is running

FMS license is not required, but if present, multiple fabrics can be opened in the FM client

No historic performance statistics collection

No Web client

Optimizes host resources Cisco Fabric Manager FM/DM Clients


7


FMS Architecture 3.2 and Above

Red Hat JBoss—provides clustering, load balancing, distributed deployment features, and future release of FMS plans on taking advantage of these capabilitiesService Oriented Architecture (SOA)

Application built as services (J2EE)Loosely coupled software components

PostgreSQL/Oracle 10g (RDBMS)Web Server

Role-Based Access Control and Licensing

Transport : Telnet, SSH, SNMP

Discovery

Event Management

SME

Inventory

Performance Collection

ACL

Interfaces

Users and Roles

Security...

Componentization of FM Server J2EE Services

RDBMS


FM Deployment Best Practices


8


Fabric Manager Server (FMS) Deployment

Dedicated serverWindows 2000, Windows 2003, Windows XP, Solaris 8/10, Red Hat Enterprise Linux AS Release 4 (2.6 Kernel), VMware Server 1.0 (Windows)

2 GHz or above processor with 2 GB RAM10 GB storageDual NIC—private/public networkManagement network connectivity—dedicated VLAN preferredSNMP proxyFMS server to be deployed in proximity to the MDS switches

AAA Server

Data Center/Private Network

Corporate/ Public Network


(FM/DM/Web Client)



FMS Sizing Parameters

Monitoring DiscoveryConfigurationEvents

Performance collectionFlowsErrors and discards

Number of Ports Managed by an FM Sever Instance

10,0005000Under 1000Monitoring + Performance Collection

10,00010,000Under 1000Monitoring Only

FM3.3PostgreSQL/Oracle 10g

FM3.2PostgreSQL/Oracle 10g

FM1.0–FM3.1


9


FMS Deployment Scenarios

A fabric to be managed by a single dedicated server

For fabrics spanning multiple data centers, a single instance ofFMS server at one of the locations, and good network connectivity to minimize SNMP packet loss

Currently FM does not support performance collection and monitoring to be split across servers

FMS Servers Recommended Based on Number of Ports Managed

FMS Server BFMS Server A5001–10,000 Ports

Single FMS ServerUp to 5000 Ports

Fabric BFabric APorts per Fabric


AAA

Staring with SAN-OS 3.1(1) Fabric Manager has a user database independent of the MDS users

FM users can be “network-admin”or “network-operator”

FM User ID is required to log into FM Client and Web Client starting with 3.1(1)

Device Manager login uses switch credentials

FM users can be authenticated by a AAA server, similar to how MDS switch users are authenticated by AAA

MDS can be used as a proxy

Data Center/Private Network

Corporate/ Public Network


(FM/DM/Web Client)


AAA Server


10


Fabric Manager—Communications

FM Client <-> FM Server : Java RMI

FM Client <-> MDS : SNMP

SNM

P

Device Manager

Web Client

FM Client

FM Client <-> MDS : SNMP


Web Client <-> FM Server : HTTP

DM communicates directly with MDS using SNMP

FM/DM perform all configuration changes in real time talking to the switch directly using SNMP

FM Client talks to FM Server using Java RMI

Web Client is primarily a read only tool and talks to the FM server over HTTP


SNMP Proxy

In firewall situations, enable “SNMP Proxy” so that all SNMP calls to the MDS from Device Manager and Fabric Manager are tunneled via the FM Server

When SNMP Proxy is enabled, software upgrade option on FM Clients working from behind the firewall will not work, as that functionality depends on FM client establishing a direct connection with the switch, and CLI output parsing


SNM

P

Device Manager

FM Client Cisco Fabric Manager Server Also Acts as

SNMP Proxy

FM Client SNMP Calls to MDS

DM SNMP Calls to MDS

Firewall


11


SNMP Proxy

Firewalls require SNMP proxy

Configure “SNMP proxy” so that SNMP calls to the MDS from Device Manager and Fabric Manager are tunneled via the FM Server

Configure the MDS management port to receive “SNMP” traffic only from the designated FM server—prevents unauthorized instances of Fabric Manger servers


SNM

P

Device Manager

FM Client Cisco Fabric Manager Server

with SNMP Proxy

FM Client SNMP Calls to MDS

DM SNMP Calls to MDS


MDS/Fabric Manager—Protocols

UDP (Random)/9198 TCP—SNMP ProxySNMP

Available Port 1163–1170 (UDP)SNMP TrapDM Client

Available Port 19199–19399 (TCP)JAVA RMI

UDP (Random)/9001 TCP—SNMP ProxySNMPFM Client

FM Server

FM Server

9099, 9100 (TCP)JAVA RMI

UDP (Random)/9198 TCP—SNMP ProxySNMP

2162 (UDP)SNMP Trap

514 (UDP)Syslog

69 (UDP)TFTP

80 (TCP)HTTP

23 (TCP)Telnet

22 (TCP)SSH

Port(s) UsedCommunication Type


12


FMS Behind a Firewall—3.1 and Earlier

Protocols and ports used for FM client and FM server to communicate

FM Server MDS

FM Client 2

9099

9100

9101

91029103

9198

9198

9099

Open RMI

Close 9099Open 9100

Open 9101for Events

Open SNMP Proxy

Use 9100

Open RMI

Close 9099

Open 9102

Open 9103for Events

Open SNMP Proxy

Use 9102

9198

9198

161

2162 SNMP TRAPS

Firewall

SNMP

TelnetSSHTFTPSyslogHTTP

23226951480

FM Client 1


FMS Behind a Firewall—3.2 and Above

FM Server MDS

1612162 SNMP TRAPS

Firewall

SNMP

TelnetSSHTFTPSyslogHTTP

23226951480

Java RMI Port

RMI Object PortServer Bind Port

AJP Connector

HTTP ConnectorWeb Service Port

Server Bind PortServer Bind Port

FM Server Bind Port

10989099

44444445800980928093

FM Client initiates communication with FM Server on port 9099 for Java Naming Directory and Interface (JNDI) lookup

FM Server directs client to 1098, JBoss directs the request to the appropriate service

Ports 4444, 4445, 8009, 8092, and 8093 are used by JBoss to monitor connectivity and for JMS to send messages to the FM Client

SNMP proxy uses port 9198

Web Service Port 8083 used for Web Services API, XML over HTTP

HTTP port 80

SNMP Proxy 9198808380


13


Aliases

Device Aliases Are Distributed to All Switches in a Fabric Using the Coordinated Distribution Mechanism Using the Cisco Fabric Services (CFS)

Recommend Using Device Aliases

Device Alias

Unique name across the entire physical fabric

User-friendly name for a port WWN that can be used in all configuration commands like FCNS, Zone, FC Ping, FC Trace Route, and IVR

Limited to a VSAN

Part of the zoning configuration and limited to zone configuration

FC Alias/Zone Alias


Enclosure

Enclosures is a Fabric Manager functionality and is not supported by SAN-OS

Devices with multiple HBAs may be represented as individual devices by Fabric Manager

By default, enclosures names are generated by the FM Server based on the alias or WWN OUI

Enclosure name provides the ability to group end devices in a single enclosure to have them represented by a single icon on the Fabric Manager topology view

An enclosure name can be manually created based on the alias name, by selecting one or more rows and using the “Alias Enclosure” option

Enclosure Names Auto Generated by FMS


14


Enclosure Example—Step 1Assign Device Aliases


Enclosure Example—Step 2

Select Alias Enclosure button on top of the End Devices table—an enclosure is generated based on the Alias name using Java Regular Expressions

For more details about Java Regular Expressions, please refer to: http://java.sun.com/docs/books/tutorial/essential/regex/


15


Enclosure Example—Step 3

Enclosure name generated based on Alias

All Disks for Storage Represented as Single Entity in Topology Map


FM Server Trap Registration

MDS can forward events up to 10 destinations

At the time of fabric discovery, FM Server registers with each MDS in the fabric as a recipient of SNMP events

Device Manager can also be a recipient of SNMP events

SNMP events can be forwarded to a NOC, and destinations can be configured via GUI or CLI


Device Manager

FM Client


16


FM Client—No Traps

Status Message “No Traps” Would Appear Under Two Conditions:

FM Server failed to register with the switch as the list of 10 destinations is full

FM Server failed to query its IP address, this can be addressed by providing an IP address in the Web client under Admin/Configure/Preferences/trap.registeraddress, this a rare occurrence


Clean Up Trap List

Identify Valid Destinations and Delete Unwanted Entries from GUI/CLI

# show snmp host

(config)# no snmp host 20.1.1.2 traps version v1


17


Manage Continuously

FM Server loads a fabric information the first time a client opens a fabricWhen the last client closes, FM Server closes the fabric, and any events received from the switches will be ignoredFor FM Server to not close the fabric information when the last client disconnects from the server, mark “Manage Continuously” next to the fabric in the Server Admin dialogPrior to release 3.2 this was called “Monitor Continuously”3.2 and above release provides three options

Unmanage: Stop managing the fabricManage: Keep fabric open as long as client is connectedManage Continuously: Always keep the fabric information open


FM Backup—3.1 and Before

Regular backup of FM data highly recommended

List of files to backupC:\Program Files\Cisco Systems\MDS 9000\*.properties

C:\Program Files\Cisco Systems\MDS 9000\*.log

C:\Program Files\Cisco Systems\MDS 9000\bin\*

C:\Program Files\Cisco Systems\MDS 9000\conf\*

C:\Program Files\Cisco Systems\MDS 9000\db\*

C:\Program Files\Cisco Systems\MDS 9000\log\*

C:\Program Files\Cisco Systems\MDS 9000\pm\*

C:\Program Files\Cisco Systems\MDS 9000\reports\*


18


PostgreSQL DB—FMS Backup

By default, FM 3.2 and above uses PostgreSQL DBFM relies on vendor tools to backup and restore FMS database

Contents of pgbackup.bat file, under C:\Program Files\Cisco Systems\MDS 9000\binset PGDIR=C:\Program Files\PostgreSQL\8.2. set DBNAME=dcmdbset DBUSERNAME=admin "%PGDIR%\bin\pg_dump.exe" -c %DBNAME% -U %DBUSERNAME% > %1%Usage Examplepgbackup.bat ciscoFMSData-apr-10-2008

In addition to completely restore FMS, the following files need to be backed up

C:\Program Files\Cisco Systems\MDS 9000\*.log C:\Program Files\Cisco Systems\MDS 9000\bin\* C:\Program Files\Cisco Systems\MDS 9000\conf\* C:\Program Files\Cisco Systems\MDS 9000\db\* C:\Program Files\Cisco Systems\MDS 9000\logs\* C:\Program Files\Cisco Systems\MDS 9000\pm\db\*

PostgreSQL DB Location

By Default, Set to Admin Update if Needed

Do Not Change This

Provide Backup File Name to the Script


PostgreSQL DB—Restore

Prior to restoring FMS database, please stop the FM serverContents of pgrestore.bat file under C:\Program Files\Cisco Systems\MDS 9000\bin

set PGDIR=C:\Program Files\PostgreSQL\8.2

set DBNAME=dcmdb

set DBUSERNAME=admin

echo "You will delete all existing data and restore db with file %1%"

echo "Please stop FMServer before db restore"

set /p ans=Are you sure you want to continue? [Y/N]

IF /i %ans% EQU Y ("%PGDIR%\bin\psql.exe" -U %DBUSERNAME% %DBNAME% < %1%)

Usage Example

pgrestore.bat ciscoFMSData-apr-10-2008


19


Run CLI Commands from FM

Execute Cisco SAN-OS CLI commands on multiple switchesScreen output is captured from each switch and copied over to the FM Client desktop“Run CLI Commands” dialog can be accessed from the Fabric Manager Tools menu

Set Location to Save Switch Output


Installation


20


FM Server Install (1 of 3)

License is not required to have a standalone FM Server instance

Cisco Fabric Manager Server License enables additional functionality

Fabric Manager Standalone

Customers with Cisco FMS License can open multiple fabrics

FMS (Licensed)



Device Alias—unique name across all VSANS in a fabric

FC Alias—Unique name across a VSAN

Earlier releases called for “Use Device Aliases in place of FC Aliases”, with default check mark, new install rewords the same option to “Use FC Aliases as Fabric default”, and the option is unchecked by default.

Recommendation: Go with default option (Devices Aliases recommended)

FM 3.1 install, default admin password was set to password. With new install, no default admin password is defined, administrator configures password as part of install.

1

21

1

2


21



Oracle 10g option offered starting with FM 3.1

Oracle 10g is not packaged with FM, and has to be installed prior to installing the FM Server

Starting with 3.2(1) release, HSQLDB has been replaced with PostgreSQL as default option

No significant performance difference between Oracle 10g and PostgreSQL


FM Client Install

FM Client can be installed from the Web client or using the URL

http://server-ip-address/download.do

By default, only FMS users with “network-admin” credentials are allowed to download the FM Client

To allow FMS users with “network-operator” role to download the client, enable the property

web.allowDownload4All=true

in server.properties file located in directory C:\Program Files\Cisco Systems\MDS 9000\conf

Click on Download to Install FM Client


22


User Login and Fabric Discovery

Starting with FM 3.1Step 1: FM User login authentication is a separate step from fabric discovery

Step 2 If FM server has open fabrics, the open fabric window is presented for user to select a fabric; if no open fabrics, then new fabric discovery window is presented

1 2

3


FMS—Multifabric Management

Ability to manage multiple fabrics simultaneouslyFabrics need to be discovered by the same FM Server instance

Each Fabric topology displayed in its own tab, as shown below


23


SAN Monitoring


SAN Monitoring

SNMP events

Syslog messages

Thresholds

Callhome

Web client health analysis reportsMultipath

Host to storage connectivity

Storage to host connectivity

Zone discrepancy

Configuration analysis

Switch health


24


SNMP Events

MDS has over 125+ MIBs that generate a large number of events

SNMP vents are grouped by functionality that can be enabled or disabled

YesNoZoneYesYesVRRPNoNoSNMP Authentication

NoYesNoNoNoNoNoNoNo

Default

NoRSCNYesLicenseYesFSPFNoFDMINoFabric Configuration Services (FCS) YesName ServerYesFC DomainNoFCCYesEntity FRU

RecommendedTrap Group


Forwarding Events to NOC

MDS has over 125+ MIBs that could potentially generate a large number of events

NOC operates are trained to flag hardware failures and critical failures

Working with customers Cisco has identified a subset of events that are of interest to a NOC

Cisco recommends customers to implement the identified subset as a phase one approach

The document Cisco MDS Event Monitoring v1.pdf details the identified events

Scripts have been developed by Cisco for the most common used NOC applications—HP OpenView and IBM NetView

Syslog messagesEvery release of MDS SAN-OS publishes the Syslog messages supported by that release, every effort is made to keep the Syslog messages backward compatible

Syslog messages can be forwarded from MDS to any application that can parse the messages and populate a monitoring application


25


Syslog Messages

2200+ Syslog messages

Syslog messages are published with each release of SAN-OS

Syslog messages are classified into eight security levels (1—emergency, 8—debug); syslog messages from MDS can be forwarded to FM Server

Syslog messages can be viewed via Web Client

Non MDS Syslog messages can be forwarded to FM Server, by setting “syslog.promiscuous = true” property via the Web Client

Default max rows in FM server database is set to 10K; when limit is reached, logs are copied to a log file on the server


DM—Syslog (1 of 2)

Syslog messages can be forwarded to a maximum of three syslog servers

Security level (1–8) can configured per functionality/module


26


DM—Syslog (2 of 2)

Syslog messages can be sent to console by selecting “ConsoleEnable”

Message severity level to be forwarded can be configured too


RMON—Threshold Monitoring

Device CPU, memory

FC services threshold per VSAN

Interfaces thresholds

Default is 100, need to configured to 512

32-bit alarms—512 per box

64-bit alarms—512 per box


27


DM—Configured Threshold Monitors

Recommend using 64-bit counters over 32-bit counters for monitoring link counters


MDS Callhome

MDS Callhome is independent of OSM Callhome

Fixed set of predefined alerts and trigger events on the switch

Multiple message format options—short text, plain text, XML

Up to 50 e-mail destination addresses for each destination profile

Multiple message categories including system, environment, switching module hardware, supervisor module, hardware, inventory, syslog, RMON, and test


28


Health Analysis—Multipath

Find devices with no redundancy or inactive paths


Health Analysis—Connectivity

Storage to host

Host to storage


29


Health Analysis—Zone Discrepancy

Not in VSAN

Not in fabric

Single member zone

Full zone distribution off

Default permit on

Only initiators in zone


Health Analysis—Switch Health

In-depth switch health analysis verifies the status of all critical switches, modules, ports, and Fibre Channel services; over 40 conditions are checked


30


Health Analysis—Configuration Analysis

Compares the configurations of the switch to a policy file

Define what functions to check and what type of checks to perform

Looks for mismatched values, and missing or extra values; over 200 checks performed


Performance Collection


31


DM—Summary Tab

Real-time performance metrics

Charting option

Quick switch health

Filter by VSAN


DM—Logging Real-Time Performance Data

Real-time stats greatly help with debugging

Log file saved under “logs” directory as:

“switch_name_summarylog.txt”

Logging happens only while the summary tab window is open


32


FM—Real Time ISL Stats


Historic Performance—Flows (1 of 4)

Flows provide important information about SAN traffic patterns (top talkers)

FM Performance Collection collects flow statistics based on source destination combination

Source and destination can be on different switches

First generation line cards support about 1000 flows per module, second generation line cards support 2000 flows per module

As flow collection is based on source and destination FCIDs, recommend persistent FCIDs enabled


33



Flow configuration is based on active zone database

Flows are configured in hardware

Flow statistics frame count and bytes are incremented in real time

Flows cannot be reset—need to delete and add again

CLI Output of Flows Configured on an MDS

Source FCID

Destination FCID

VSAN



Flow Performance Collection is a two-step process, and need to be configured from the FM Client; flow configuration setup has to be repeated for each VSAN

The option “Type” refers to the collection process; recommend that flows are collected both ways; FMS server is smart to consolidate the data at the time of reporting

Checking the “Clear old flows on modified switches” option purges all old data related to the FCID; please select this option with careful consideration; once selected, no way to restore lost the data

The option “Create flows on all cards,” gives flexibility for physical port to be moved to another module on the same switch; this option available on SAN-OS 3.1 and later


34



Second step of the Flow wizard lists all flows for the VSAN

Manually remove flows of not interest

Finish configures the selected flows on the corresponding MDS switches and FM Server

FM Server need to be restarted for it to pick the new flow configurations


Performance Data Collection

FM Server collects flow information once every 5 minutes; this interval cannot be changedPerformance data collected every 5 minutes is saved in flat files with an “rrd” extension under the…pm\db directoryEvery hour a background process consolidates the data to the databaseTo limit the size of database, performance data collected is retained for a finite period time; the length of data retention can be configured to customers needs; longer retention periods of data will require larger disk space; default configurations are:

5 minute interval samples for 48 hours30 minutes samples for 14 days120 minute samples for 2 months1-day samples for 300 days

Using default retention periods, each flow takes 115 KB of disk spaceBy default ISL interfaces statistics are also collected once every 5 minutes; the time interval can be modified to as low as once every 30 seconds


35


Database Sizing

To limit the size of database, performance data collected is retained for a finite period timeFM Server collects flow information once every 5 minutes; this interval cannot be changedISL statistics can be configured to collect in 30 second intervals; default, once every 5 minutesConfigure data retention periods based on need; longer retention periods of data will require larger disk space; default configurations are:

5 minute interval samples for 48 hours30 minutes samples for 14 days120 minute samples for 2 months1-day samples for 300 days

Performance data collected every 5 minutes is saved in flat files with an “rrd” extension under the…pm\db directoryEvery hour a background process consolidates the data to the databaseUsing default retention periods, each flow takes 115 KB of disk space


FMS Performance Monitoring—Summary

SAN summary

Link utilization summary

Drill down


36


Performance Monitoring—Flows

Top talkers

Clicking on the flow generates chart


Performance Prediction

Based on past data, and acceptable threshold utilization percentage, predict future traffic growths


37


FMS—Threshold

Configure threshold monitoring based on absolute values or based on past performance (week/month/year)


Custom Reporting

Customized report configuration

On demand/scheduled report generation


38


FMS Server—Fabric Monitoring

Events, syslog messages being forwarded to FM server?

Performance collection problems?


FMS—MIB OID Monitoring

FM server provides an ability to poll on any MIB OID once every 5 minutes

In Web Client, under Admin/Configure/Others, select Add, and provide the OID to be polled for

Can be configured for all switches in a fabric, or select switches


39


FM Server Properties

File located “MDS 9000”; default windows location

C:\Program Files\Cisco Systems\MDS 9000\server.properties

Changes require Fabric Manger Server restart

Make backup copy prior to making any changes

Changes can be made from Web Client


Key Takeaways

Cisco Fabric Manager deployment

Switch and fabric health analysis tools

SAN monitoring

Performance collection

FMS reporting


40


Q and A


Recommended Reading

Continue your Cisco Live learning experience with further reading from Cisco Press

Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store


41


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.

Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.


brksan-2891

Documents