Articles from IT WorldVeritas Volume Manager (VxVM) command lineexamples2012-06-15 15:06:02 Yannick

Table Of Contents1. Preamble2. Dynamic Multipathing (DMP)3. Basic commands

1. Disk groups2. Volumes3. Plexes4. Subdisks

4. Performance monitoring5. References

PreambleBefore entering in Veritas Volume Manager (VxVM) details we must go in fine detailof our hardware configuration. Let’s first confirm at Linux level that I see my 2 fiberchannel (FC) cards with each 2 FC ports:

[root@server1 ~]# lspci | grep -i fibre09:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)09:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)0c:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)0c:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)[root@server1 ~]# ll /sys/class/fc_host/host*/devicelrwxrwxrwx 1 root root 0 Jun 5 17:49 /sys/class/fc_host/host0/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.0/host0lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host1/device -> ../../../devices/pci0000:00/0000:00:03.0/0000:09:00.1/host1lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host2/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.0/host2lrwxrwxrwx 1 root root 0 Jun 5 17:51 /sys/class/fc_host/host3/device -> ../../../devices/pci0000:00/0000:00:05.0/0000:0c:00.1/host3

We will see this later but we need correspondence between SCSI chain numberand its World Wide Name (WWN), here it is:

[root@server1 ~]# for host in `ls /sys/class/fc_host/`; do echo "$host WWN: `cat /sys/class/fc_host/$host/port_name`"; donehost0 WWN: 0x50060b0000c35e0chost1 WWN: 0x50060b0000c35e0ehost2 WWN: 0x50060b0000c35e08host3 WWN: 0x50060b0000c35e0a

So if everything is fiber wired (FC cards/ports to FC switches) I should see any diskarray (enclosure) device/LUN as 4 equivalent name. Please note that this isabsolutely not mandatory, what’s really important in my configuration is to have eachdisk visible through my 2 FC cards (so by one port minimum so each disk/LUN willhave 2 names).

Please note that this post is more for DBAs to see and understand what have beendone by you system teammates i.e. I’m not entering in the creation/modification ofVxVM disk/disk group/controller and so on…

All commands have been executed on a Red Hat Enterprise Linux Server release5.5 (Tikanga) server configured with Veritas Volume Manager release VRTSvxvm-5.1.132.000-SP1RP2_RHEL5.

To avoid putting real server name I’m using the below tips for my promptcustomization:

export PS1="[\u@server1 \W]# "

Dynamic Multipathing (DMP)Veritas Dynamic Multi-Pathing (DMP or VxDMP) is the VxVM functionalityimplemented to support disk availability from multiple paths. Even if it looks complexto start with this functionality it is unfortunately mandatory before doing the moresimple things as it is based configuration on which all other components will becreated (subdisks, plexes, volumes).

Configured disk arrays or enclosures accessible from this server:

[root@server1 ~]# vxdmpadm listenclosureENCLR_NAME ENCLR_TYPE ENCLR_SNO STATUS ARRAY_TYPE LUN_COUNT=======================================================================================emc0 EMC 000290102696 CONNECTED A/A 54emc1 EMC 000290105264 CONNECTED A/A 1other_disks OTHER_DISKS OTHER_DISKS CONNECTED OTHER_DISKS 1

I/O policy default configuration of this enclosure:

[root@server1 ~]# vxdmpadm getattr enclosure emc0ENCLR_NAME ATTR_NAME DEFAULT CURRENT============================================================================emc0 iopolicy MinimumQ MinimumQemc0 partitionsize 512 512emc0 use_all_paths - -emc0 failover_policy Global Globalemc0 recoveryoption[throttle] Nothrottle[0] Nothrottle[0]emc0 recoveryoption[errorretry] Timebound[300] Timebound[300]emc0 redundancy 0 0emc0 dmp_lun_retry_timeout 0 0emc0 failovermode - -

Disk default naming convention of VxVM:

[root@server1 ~]# vxddladm get namingschemeNAMING_SCHEME PERSISTENCE LOWERCASE USE_AVID============================================================Enclosure Based Yes Yes Yes

List of configured disk on your system:

[root@server1 ~]# vxdisk -e listDEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTRcciss/c0d0 auto:none - - online invalid cciss/c0d0 -emc0_00f3 auto:cdsdisk - - online sdbg stdemc0_3d6b auto:cdsdisk vgp1148_01 vgp1148 online clone_disk sdbn srdf-r1emc0_3d7b auto:cdsdisk vgp1148_03 vgp1148 online clone_disk sdbp srdf-r1emc0_3d73 auto:cdsdisk vgp1148_02 vgp1148 online clone_disk sdbo srdf-r1emc0_3ebb auto:cdsdisk vgp1118_02 vgp1118 online clone_disk sdca srdf-r1emc0_3eb3 auto:cdsdisk vgp1118_01 vgp1118 online clone_disk sdbx srdf-r1emc0_3ecb auto:cdsdisk vgp1318_01 vgp1318 online clone_disk sdcc srdf-r1emc0_3ec3 auto:cdsdisk vgp1118_03 vgp1118 online clone_disk sdcb srdf-r1..

Disks of a disk group (please note it makes relationship between VxVM disks andthe ones you see at system level):

[root@server1 ~]# vxdisk -g vgp1417 -e listDEVICE TYPE DISK GROUP STATUS OS_NATIVE_NAME ATTRemc0_084b auto:cdsdisk vgp1417_04 vgp1417 online clone_disk sdax srdf-r1emc0_084f auto:cdsdisk vgp1417_05 vgp1417 online clone_disk sdaz srdf-r1emc0_085b auto:cdsdisk vgp1417_08 vgp1417 online clone_disk sdbd srdf-r1emc0_526f auto:cdsdisk vgp1417_09 vgp1417 online clone_disk sdbh srdf-r1emc0_527b auto:cdsdisk vgp1417_10 vgp1417 online clone_disk sdbi srdf-r1emc0_527f auto:cdsdisk vgp1417_11 vgp1417 online clone_disk sdbj srdf-r1emc0_0853 auto:cdsdisk vgp1417_06 vgp1417 online clone_disk sdbb srdf-r1emc0_0857 auto:cdsdisk vgp1417_07 vgp1417 online clone_disk sdbc srdf-r1

Main problem with above command is that you don’t see multipathing details ofVxVM disk, to go deeper in detail use:

[root@server1 ~]# vxdisk list emc0_084bDevice: emc0_084bdevicetag: emc0_084btype: autohostid: server1disk: name=vgp1417_04 id=1325653859.61.server1group: name=vgp1417 id=1318435685.29.server1info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3flags: online ready private autoconfig noautoimport imported clone_diskpubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3guid: -udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000site: -version: 3.1iosize: min=512 (bytes) max=1024 (blocks)public: slice=3 offset=65792 len=70539872 disk_offset=0private: slice=3 offset=256 len=65536 disk_offset=0update: time=1335608893 seqno=0.17ssb: actual_seqno=0.0headers: 0 240configs: count=1 len=51360logs: count=1 len=4096Defined regions: config priv 000048-000239[000192]: copy=01 offset=000000 enabled config priv 000256-051423[051168]: copy=01 offset=000192 enabled log priv 051424-055519[004096]: copy=01 offset=000000 enabled lockrgn priv 055520-055663[000144]: part=00 offset=000000Multipathing information:numpaths: 2sdax state=enabledsdh state=enabled

Remark:You may also use the opposite command:

[root@server1 ~]# vxdisk list sdaxDevice: emc0_084bdevicetag: emc0_084btype: autohostid: server1disk: name=vgp1417_04 id=1325653859.61.server1group: name=vgp1417 id=1318435685.29.server1info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3flags: online ready private autoconfig noautoimport imported clone_diskpubpaths: block=/dev/vx/dmp/emc0_084bs3 char=/dev/vx/rdmp/emc0_084bs3guid: -udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000site: -version: 3.1iosize: min=512 (bytes) max=1024 (blocks)public: slice=3 offset=65792 len=70539872 disk_offset=0private: slice=3 offset=256 len=65536 disk_offset=0update: time=1335608893 seqno=0.17ssb: actual_seqno=0.0headers: 0 240configs: count=1 len=51360logs: count=1 len=4096Defined regions: config priv 000048-000239[000192]: copy=01 offset=000000 enabled config priv 000256-051423[051168]: copy=01 offset=000192 enabled log priv 051424-055519[004096]: copy=01 offset=000000 enabled lockrgn priv 055520-055663[000144]: part=00 offset=000000Multipathing information:numpaths: 2sdax state=enabledsdh state=enabled

Using vxdmpadm command you can also display complete multipathing informationand obviously also confirm that each VxVM disk is accessible through 2 paths:

[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=emc0_084bNAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS================================================================================sdax ENABLED(A) - c2 EMC emc0 -sdh ENABLED(A) - c0 EMC emc0 -[root@server1 ~]# vxdmpadm -v getdmpnode nodename=sdaxNAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME SERIAL-NO ARRAY_VOL_ID========================================================================================================emc0_084b ENABLED EMC 2 2 0 emc0 960084B000 084B[root@server1 ~]# vxdmpadm getdmpnode enclosure=emc0NAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME==============================================================================emc0_00f3 ENABLED EMC 2 2 0 emc0emc0_3d6b ENABLED EMC 2 2 0 emc0emc0_3d7b ENABLED EMC 2 2 0 emc0emc0_3d73 ENABLED EMC 2 2 0 emc0emc0_3ebb ENABLED EMC 2 2 0 emc0emc0_3eb3 ENABLED EMC 2 2 0 emc0emc0_3ecb ENABLED EMC 2 2 0 emc0emc0_3ec3 ENABLED EMC 2 2 0 emc0..[root@server1 ~]# vxdmpadm list dmpnode dmpnodename=emc0_084bdmpdev = emc0_084bstate = enabledenclosure = emc0cab-sno = 000290102696asl = libvxemc.sovid = EMCpid = SYMMETRIXarray-name = EMCarray-type = A/Aiopolicy = MinimumQavid = 084Blun-sno = 960084B000udid = EMC%5FSYMMETRIX%5F000290102696%5F960084B000dev-attr = srdf-r1###path = name state type transport ctlr hwpath aportID aportWWN attrpath = sdax enabled(a) - FC c2 c2 7b-b 50:06:04:84:52:a7:6a:36 -path = sdh enabled(a) - FC c0 c0 10b-b 50:06:04:84:52:a7:6a:39 -[root@server1 ~]# vxdmpadm getlungroup dmpnodename=sdaxNAME STATE ENCLR-TYPE PATHS ENBL DSBL ENCLR-NAME=========================================================================emc0_084b ENABLED EMC 2 2 0 emc0[root@server1 ~]# vxdmpadm getsubpaths dmpnodename=sdaxNAME STATE[A] PATH-TYPE[M] CTLR-NAME ENCLR-TYPE ENCLR-NAME ATTRS================================================================================sdax ENABLED(A) - c2 EMC emc0 -sdh ENABLED(A) - c0 EMC emc0 -

Remark:Please note the two controllers that are linked to the FC card we have identified inprevious chapter:

[root@server1 ~]# vxdmpadm getsubpaths enclosure=emc0 ctlr=c0NAME STATE[A] PATH-TYPE[M] DMPNODENAME ENCLR-NAME CTLR ATTRS================================================================================sdo ENABLED(A) - emc0_00f3 emc0 c0 -sdz ENABLED(A) - emc0_0837 emc0 c0 -sdh ENABLED(A) - emc0_084b emc0 c0 -sdi ENABLED(A) - emc0_084f emc0 c0 -sdj ENABLED(A) - emc0_0853 emc0 c0 -sdk ENABLED(A) - emc0_0857 emc0 c0 -sdl ENABLED(A) - emc0_085b emc0 c0 -..

The vxdisk command can also display the multipathing information:

[root@server1 ~]# vxdisk path -g vgp1417

SUBPATH DANAME DMNAME GROUP STATEsdax emc0_084b vgp1417_04 vgp1417 ENABLEDsdh emc0_084b vgp1417_04 vgp1417 ENABLEDsdaz emc0_084f vgp1417_05 vgp1417 ENABLEDsdi emc0_084f vgp1417_05 vgp1417 ENABLEDsdbd emc0_085b vgp1417_08 vgp1417 ENABLEDsdl emc0_085b vgp1417_08 vgp1417 ENABLEDsdbh emc0_526f vgp1417_09 vgp1417 ENABLEDsdp emc0_526f vgp1417_09 vgp1417 ENABLEDsdbi emc0_527b vgp1417_10 vgp1417 ENABLEDsdq emc0_527b vgp1417_10 vgp1417 ENABLEDsdbj emc0_527f vgp1417_11 vgp1417 ENABLEDsdr emc0_527f vgp1417_11 vgp1417 ENABLEDsdbb emc0_0853 vgp1417_06 vgp1417 ENABLEDsdj emc0_0853 vgp1417_06 vgp1417 ENABLEDsdbc emc0_0857 vgp1417_07 vgp1417 ENABLEDsdk emc0_0857 vgp1417_07 vgp1417 ENABLED

Let’s come back to the controller and display the ones configured on my system(each enclosure is configured with two controllers):

[root@server1 ~]# vxdmpadm listctlr allCTLR-NAME ENCLR-TYPE STATE ENCLR-NAME=====================================================c0 EMC ENABLED emc0c2 EMC ENABLED emc0c0 EMC ENABLED emc1c2 EMC ENABLED emc1c360 OTHER_DISKS ENABLED other_disks

We can re-conciliate WWN identified in previous chapter with VxDMP commands:

[root@server1 ~]# vxdmpadm getctlr c0LNAME PNAME VENDOR CTLR-ID=============================================================================================c0 c0 Qlogic-Corp. 50:06:0b:00:00:c3:5e:0c[root@server1 ~]# vxdmpadm getctlr c2LNAME PNAME VENDOR CTLR-ID=============================================================================================c2 c2 Qlogic-Corp. 50:06:0b:00:00:c3:5e:08

And confirm, as anticipated, that each of my FC card is wired with only one FCcable. Each of my FC or Host Bus Adapter (HBA) cards has two ports and only oneis used:

[root@server1 ~]# vxddladm list hbasHBA-ID DRIVER FIRMWARE DISCOVERY STATE ADDRESS======================================================================================c1 8.03.07.03.05.07-k 5.03.16 (496) fabric Offline 50:06:0b:00:00:c3:5e:0ec3 8.03.07.03.05.07-k 5.03.16 (496) fabric Offline 50:06:0b:00:00:c3:5e:0ac360 - - - Online -c0 8.03.07.03.05.07-k 5.03.16 (496) fabric Online 50:06:0b:00:00:c3:5e:0cc2 8.03.07.03.05.07-k 5.03.16 (496) fabric Online 50:06:0b:00:00:c3:5e:08

To obtain the configured ports:

[root@server1 ~]# vxddladm list portsPORT-ID HBA-ID STATE ADDRESS==========================================c0_p3 c0 Online 50:06:04:84:52:a7:6a:39c0_p2 c0 Online 50:06:04:84:52:a7:6a:08c0_p1 c0 Online 50:06:04:84:52:a9:ec:09c2_p3 c2 Online 50:06:04:84:52:a7:6a:36c2_p2 c2 Online 50:06:04:84:52:a7:6a:07c2_p1 c2 Online 50:06:04:84:52:a9:ec:06[root@server1 ~]# vxddladm list targetsTARGET-ID ALIAS HBA-ID STATE ADDRESS=====================================================c0_p3_t0 - c0 Online 50:06:04:84:52:a7:6a:39c0_p2_t0 - c0 Online 50:06:04:84:52:a7:6a:08c0_p1_t0 - c0 Online 50:06:04:84:52:a9:ec:09c2_p3_t0 - c2 Online 50:06:04:84:52:a7:6a:36c2_p2_t0 - c2 Online 50:06:04:84:52:a7:6a:07c2_p1_t0 - c2 Online 50:06:04:84:52:a9:ec:06[root@server1 ~]# vxddladm list targets hba=c0TARGET-ID ALIAS HBA-ID STATE ADDRESS=====================================================c0_p3_t0 - c0 Online 50:06:04:84:52:a7:6a:39c0_p2_t0 - c0 Online 50:06:04:84:52:a7:6a:08c0_p1_t0 - c0 Online 50:06:04:84:52:a9:ec:09

To understand which devices are using which ports and the name of the VxVMdriver:

[root@server1 ~]# vxddladm list devicesDEVICE TARGET-ID STATE DDL-STATUS (ASL)===============================================================cciss/c0d0 - Online -sdg c0_p3_t0 Online SKIPPEDsdi c0_p3_t0 Online CLAIMED (libvxemc.so)sdf c0_p3_t0 Online SKIPPEDsdh c0_p3_t0 Online CLAIMED (libvxemc.so)sde c0_p3_t0 Online SKIPPEDsdd c0_p3_t0 Online SKIPPEDsdr c0_p3_t0 Online CLAIMED (libvxemc.so)..[root@server1 ~]# vxddladm list devices target=c0_p1_t0DEVICE TARGET-ID STATE DDL-STATUS (ASL)===============================================================sdbl c0_p1_t0 Online CLAIMED (libvxemc.so)sdbk c0_p1_t0 Online SKIPPED

To get the list of supported enclosures:

[root@server1 ~]# vxddladm listsupport allLIBNAME VID PID=================================================================================================libvxpurple.so SUN T300libvxveritas.so VERITAS Alllibvxhillsannet2.so DotHill SANnet II FC, SANnet II SCSI, SANnet II SATA, SANnet II U320libvxhds.so HITACHI Alllibvxpp.so EMC, DGC AlllibvxFJTSYe8k.so FUJITSU Alllibvxfje3k4ka.so FUJITSU E3000, E400Alibvxibmds6k.so IBM 1750libvxsun7x10.so SUN Sun Storage 7410, Sun Storage 7310, Sun Storage 7210, Sun Storage 7110, ZFS Storage 7120, ZFS Storage 7320, ZFS Storage 7420, ZFS Storage 7720libvxfsc.so FSC FibreCAT_SX1libvxxp1281024.so HP Alllibvxlsiall.so IBM, SUN, STK, SGI Alllibvxnipnyis.so NEC iStorage 1000, iStorage 2000, iStorage 4000libvxtsbaf.so TOSHIBA AF_AF3500, AF_AF1500, AF2_AF7000, AF2_AF2000, AF3_AF7500, AF3_AF2500libvxfusionio.so FIO ioDrivelibvxhpalua.so HP, COMPAQ HSV101, HSV111 (C)COMPAQ, HSV111, HSV200, HSV210, HSV300, HSV400, HSV450libvxxp12k.so HP Alllibvxhitachi.so HITACHI DF350, DF400, DF400F, DF500, DF500Flibvxpillaraxiom.so Pillar Axiom 300, Axiom 500, Axiom 600libvxFJTSYe6k.so FUJITSU E6000libvxxiv.so XIV, IBM NEXTRA, 2810XIVlibvxxp256.so HP Alllibvxramsan.so TMS RamSan 400libvxhdsalua.so HITACHI DF600, DF600-V, DF600F, DF600F-Vlibvxhpmsa.so HP MSA VOLUMElibvxhds9980.so HITACHI Alllibvxhdsusp.so HITACHI AlllibvxCLARiiON.so DGC Alllibvxnetapp.so NETAPP Alllibvxeqlogic.so EQLOGIC 100E-00libvxemc.so EMC SYMMETRIXlibvxddns2a.so DDN S2A 9550, S2A 9900, S2A 9700libvxfjtsye2k.so FUJITSU E2000, ETERNUS_DXLlibvxmsa2kfc_sa.so HP MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSI

libvxmsa2kfc_sa.so HP MSA2312fc, MSA2324fc, MSA2012sa, MSA2312sa, MSA2324sa, MSA2312i, MSA2324i, P2000 G3 FC, P2000G3 FC/iSCSI, P2000 G3 SAS, P2000 G3 iSCSIlibvxdothill.so DotHill R/Evo 2730-2R, R/Evo 2530-2R, R/Evo 2330-2R, R/Evo 2130-2RX, R/Evo 2130-2J, R/Evo 5730-2Rlibvxcompellent.so COMPELNT Compellent Vollibvxibmds8k.so IBM 2107libvxmsa2k.so HP MSA2012fc, MSA2212fc, MSA2012ilibvxxiotechE5k.so XIOTECH ISE1400libvxcopan.so COPANSYS 8814, 8818libvxibmsvc.so IBM 2145, 2062libvxhuawei.so HUAWEI, HS S5300, S5500, S5600, S2300, S2100, V1800, V1500, VIS6000, S2300E, S2600, S6800E, S8000-I, S8000libvx3par.so 3PARdata VVlibvxhpsvsp.so HP HSVX740libvxshark.so IBM 2105

And more information (Active/Active mode) on the one I use:

[root@server1 ~]# vxddladm listsupport libname=libvxemc.soATTR_NAME ATTR_VALUE=================================================================================================LIBNAME libvxemc.soVID EMCPID SYMMETRIXARRAY_TYPE A/AARRAY_NAME EMC

Basic commandsDisk groupsHow to determine if a disk is associated to a disk group:

[root@server1 ~]# vxdisk -s list sdaxDisk: emc0_084btype: autoflags: online ready private autoconfig noautoimport imported clone_diskguid: -udid: EMC%5FSYMMETRIX%5F000290102696%5F960084B000site: -diskid: 1325653859.61.server1dgname: vgp1417dgid: 1318435685.29.server1hostid: server1info: format=cdsdisk,privoffset=256,pubslice=3,privslice=3

Disk groups list:

[root@server1 ~]# vxdg listNAME STATE IDvgp1417a enabled,cds 1318432411.19.server1vgoem3 enabled,cds 1320851456.27.server1vgptws02a enabled,cds 1318847636.93.server1..

More information of a disk group:

[root@server1 ~]# vxdg list vgp1417Group: vgp1417dgid: 1318435685.29.server1import-id: 1024.62flags: cdsversion: 160alignment: 8192 (bytes)ssb: onautotagging: ondetach-policy: globaldg-fail-policy: dgdisablecopies: nconfig=default nlog=defaultconfig: seqno=0.1556 permlen=51360 free=51325 templen=17 loglen=4096config disk emc0_084b copy 1 len=51360 state=clean onlineconfig disk emc0_084f copy 1 len=51360 state=clean onlineconfig disk emc0_085b copy 1 len=51360 state=clean onlineconfig disk emc0_526f copy 1 len=51360 disabledconfig disk emc0_527b copy 1 len=51360 disabledconfig disk emc0_527f copy 1 len=51360 disabledconfig disk emc0_0853 copy 1 len=51360 state=clean onlineconfig disk emc0_0857 copy 1 len=51360 state=clean onlinelog disk emc0_084b copy 1 len=4096log disk emc0_084f copy 1 len=4096log disk emc0_085b copy 1 len=4096log disk emc0_526f copy 1 len=4096 disabledlog disk emc0_527b copy 1 len=4096 disabledlog disk emc0_527f copy 1 len=4096 disabledlog disk emc0_0853 copy 1 len=4096log disk emc0_0857 copy 1 len=4096

Free space of a disk group:

[root@server1 ~]# vxdg -g vgp1417 freeDISK DEVICE TAG OFFSET LENGTH FLAGSvgp1417_04 emc0_084b emc0_084b 56623104 13916768 -vgp1417_05 emc0_084f emc0_084f 64389600 6150272 -vgp1417_09 emc0_526f emc0_526f 17592064 52947808 -vgp1417_10 emc0_527b emc0_527b 0 70539872 -vgp1417_11 emc0_527f emc0_527f 0 70539872 -

Remark:So how much at the end ? For vgp1417_04 (emc0_084b) it is 13916768*512 (sectorsize)/(1024 * 1024) MB i.e. 6795MB free (I have already seen more user friendlycommand output)…

Even if not written in official VxVM documentation there is an alternative commandwith a much better output:

[root@server1 ~]# vxassist -g vgp1417 maxsizeMaximum volume size: 109616431104 (104538Mb)

Complete information of a disk group (subdisks, plexes, …):

[root@server1 ~]# vxprint -g vgp1417TY NAME ASSOC KSTATE LENGTH PLOFFS STATE TUTIL0 PUTIL0dg vgp1417 vgp1417 - - - - - - dm vgp1417_04 emc0_084b - 70539872 - - - -dm vgp1417_05 emc0_084f - 70539872 - - - -dm vgp1417_06 emc0_0853 - 70539872 - - - -dm vgp1417_07 emc0_0857 - 70539872 - - - -dm vgp1417_08 emc0_085b - 70539872 - - - -dm vgp1417_09 emc0_526f - 70539872 - - - -dm vgp1417_10 emc0_527b - 70539872 - - - -dm vgp1417_11 emc0_527f - 70539872 - - - - v lvol1 fsgen ENABLED 6291456 - ACTIVE - -pl lvol1-01 lvol1 ENABLED 6291456 - ACTIVE - -sd vgp1417_04-01 lvol1-01 ENABLED 6291456 0 - - -..

VolumesVolumes of a disk group:

[root@server1 ~]# vxprint -hvt -g vgp1417V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPEPL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODESD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODESV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODESC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODEDC NAME PARENTVOL LOGVOLSP NAME SNAPVOL DCOEX NAME ASSOC VC PERMS MODE STATE

v lvol1 - ENABLED ACTIVE 6291456 SELECT - fsgenpl lvol1-01 lvol1 ENABLED ACTIVE 6291456 CONCAT - RWsd vgp1417_04-01 lvol1-01 vgp1417_04 0 6291456 0 emc0_084b ENA v lvol2 - ENABLED ACTIVE 20971520 SELECT - fsgen..

More information of a particular volume:

[root@server1 ~]# vxprint -g vgp1417 -t lvol1V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE v lvol1 - ENABLED ACTIVE 6291456 SELECT - fsgen

PlexesPlexes of a disk group:

[root@server1 ~]# vxprint -pt -g vgp1417PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE pl lvol1-01 lvol1 ENABLED ACTIVE 6291456 CONCAT - RWpl lvol2-01 lvol2 ENABLED ACTIVE 20971520 CONCAT - RWpl lvol3-01 lvol3 ENABLED ACTIVE 2097152 CONCAT - RWpl lvol4-01 lvol4 ENABLED ACTIVE 6291456 CONCAT - RWpl lvol5-01 lvol5 ENABLED ACTIVE 4194304 CONCAT - RWpl lvol6-01 lvol6 ENABLED ACTIVE 14680064 CONCAT - RWpl lvol7-01 lvol7 ENABLED ACTIVE 44040192 CONCAT - RWpl lvol8-01 lvol8 ENABLED ACTIVE 169869312 CONCAT - RWpl lvol9-01 lvol9 ENABLED ACTIVE 12582912 CONCAT - RWpl lvol10-01 lvol10 ENABLED ACTIVE 27262976 CONCAT - RWpl lvol11-01 lvol11 ENABLED ACTIVE 41943040 CONCAT - RW

More information of a plex:

[root@server1 ~]# vxprint -g vgp1417 -l lvol1-01Plex: lvol1-01info: len=6291456type: layout=CONCATstate: state=ACTIVE kernel=ENABLED io=read-writeassoc: vol=lvol1 sd=vgp1417_04-01flags: busy completemediatype: hdd

SubdisksSubdisks of a particular disk group:

[root@server1 ~]# vxprint -st -g vgp1417SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODESV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODESC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE sd vgp1417_04-01 lvol1-01 vgp1417_04 0 6291456 0 emc0_084b ENAsd vgp1417_04-02 lvol9-01 vgp1417_04 6291456 12582912 0 emc0_084b ENAsd vgp1417_04-03 lvol10-01 vgp1417_04 18874368 27262976 0 emc0_084b ENAsd vgp1417_04-04 lvol6-01 vgp1417_04 46137344 10485760 4194304 emc0_084b ENAsd vgp1417_05-01 lvol5-01 vgp1417_05 0 4194304 0 emc0_084f ENAsd vgp1417_05-02 lvol7-01 vgp1417_05 4194304 31457280 0 emc0_084f ENAsd vgp1417_05-03 lvol11-01 vgp1417_05 35651584 16155104 0 emc0_084f ENAsd vgp1417_05-04 lvol7-01 vgp1417_05 51806688 12582912 31457280 emc0_084f ENAsd vgp1417_06-01 lvol2-01 vgp1417_06 0 20971520 0 emc0_0853 ENAsd vgp1417_06-02 lvol11-01 vgp1417_06 20971520 25787936 16155104 emc0_0853 ENAsd vgp1417_06-03 lvol8-01 vgp1417_06 46759456 23780416 114269008 emc0_0853 ENAsd vgp1417_07-01 lvol8-01 vgp1417_07 0 56312048 0 emc0_0857 ENAsd vgp1417_07-02 lvol8-01 vgp1417_07 56312048 14227824 138049424 emc0_0857 ENAsd vgp1417_08-01 lvol3-01 vgp1417_08 0 2097152 0 emc0_085b ENAsd vgp1417_08-02 lvol4-01 vgp1417_08 2097152 6291456 0 emc0_085b ENAsd vgp1417_08-03 lvol6-01 vgp1417_08 8388608 4194304 0 emc0_085b ENAsd vgp1417_08-04 lvol8-01 vgp1417_08 12582912 57956960 56312048 emc0_085b ENAsd vgp1417_09-01 lvol8-01 vgp1417_09 0 17592064 152277248 emc0_526f ENA

More information of a subdisk:

[root@server1 ~]# vxprint -g vgp1417 -l vgp1417_04Disk: vgp1417_04info: diskid=1325653859.61.server1assoc: device=emc0_084b type=autoflags: autoconfigdevice: path=/dev/vx/dmp/emc0_084bs3devinfo: publen=70539872 privlen=65536mediatype: hdd

Performance monitoringWhen having I/O issues the standard command you would use could be iostat:

[root@server1 ~]# iostat -mdxLinux 2.6.18-274.3.1.el5 (server1) 06/07/2012 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %utilcciss/c0d0 6.45 40.26 3.54 25.84 0.05 0.17 15.73 0.82 28.01 5.13 15.07cciss/c0d0p1 0.00 0.00 0.00 0.00 0.00 0.00 31.97 0.00 21.14 14.42 0.00cciss/c0d0p2 6.45 40.26 3.54 25.84 0.05 0.17 15.73 0.82 28.01 5.13 15.07sda 0.00 0.00 0.00 0.00 0.00 0.00 3.98 0.00 1.87 1.87 0.00sdb 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 4.51 4.51 0.00sdc 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 1.83 1.83 0.00sdd 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 1.66 1.66 0.00sde 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.36 2.36 0.00sdf 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.51 2.51 0.00sdg 0.00 0.00 0.00 0.00 0.00 0.00 9.71 0.00 2.34 2.34 0.00..

Remark:As stated in man pages if you do not specify interval and count parameter displayedstatistics are since server boot time so not really relevant…

[root@server1 ~]# iostat -md -p sda 1 2Linux 2.6.18-274.3.1.el5 (server1) 06/07/2012 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtnsda 0.00 0.00 0.00 7 0 Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtnsda 0.00 0.00 0.00 0 0

You can also use nmon.

With the command we have seen in previous chapter you are now able to identify towhich disk group/volume/VXVM disks it is associated…

But even more interesting you can use VxVM command to get detailed informationof disk using Operation System disk names:

[root@server1 ~]# vxdmpadm iostat show pathname=sdax cpu usage = 374381us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)PATHNAME READS WRITES READS WRITES READS WRITESsdax 1182902 4756775 328093320 294705430 5.10 20.82[root@server1 ~]# vxdmpadm iostat show dmpnodename=sdax cpu usage = 375891us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)PATHNAME READS WRITES READS WRITES READS WRITESsdax 1183412 4762187 328098810 295062539 5.10 20.82sdh 1133886 4773058 324904565 294251856 5.35 20.75[root@server1 ~]# vxdmpadm iostat show pathname=sdax interval=2 count=2

cpu usage = 374396us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)PATHNAME READS WRITES READS WRITES READS WRITESsdax 1182902 4756777 328093320 294705543 5.10 20.82 sdax 0 0 0 0 0.00 0.00[root@server1 ~]# vxdmpadm -e iostat show pathname=sdax cpu usage = 374509us per cpu memory = 1114112b ERROR I/OsPATHNAME READS WRITESsdax 0 0

As well as using VxVM disk names:

[root@server1 ~]# vxdmpadm -q iostat show dmpnodename=emc0_084b cpu usage = 374484us per cpu memory = 1114112b QUEUED I/Os PENDING I/OsDMPNODENAME READS WRITESemc0_084b 0 0 0[root@server1 ~]# vxdmpadm -z iostat show dmpnodename=emc0_084b cpu usage = 374558us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)PATHNAME READS WRITES READS WRITES READS WRITESsdax 1182910 4756814 328093384 294707962 5.10 20.82sdh 1133430 4767643 324899821 293884532 5.35 20.76[root@server1 ~]# vxdmpadm -u bytes -z iostat show dmpnodename=emc0_084b cpu usage = 374888us per cpu memory = 1114112b OPERATIONS BYTES AVG TIME(ms)PATHNAME READS WRITES READS WRITES READS WRITESsdax 1183012 4760612 480699392 707400192 5.10 20.82sdh 1133494 4771329 3140286976 287247360 5.35 20.76

You can display information of controllers to see if your multipathing policy isworking well:

[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c0 cpu usage = 374524us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)CTLRNAME READS WRITES READS WRITES READS WRITESc0 135753949 218251362 3572140375 2841606614 7.62 7.10[root@server1 ~]# vxdmpadm iostat show groupby=ctlr ctlr=c2 cpu usage = 374541us per cpu memory = 1114112b OPERATIONS BLOCKS AVG TIME(ms)CTLRNAME READS WRITES READS WRITES READS WRITESc2 135349009 215663711 3357968941 2737547261 7.96 6.92

Those interesting information can also be display at disk group level:

[root@server1 ~]# vxstat -u m -g vgp1417 OPERATIONS BLOCKS AVG TIME(ms)TYP NAME READ WRITE READ WRITE READ WRITEvol lvol1 827072 413478 17059m 29032m 1.51 7.34vol lvol2 761459 357294 3465m 2781m 4.46 10.80vol lvol3 9518191 7008657 175801m 56452m 0.26 0.02vol lvol4 610864 15518552 259528m 176116m 1.86 0.19vol lvol5 170534 303104 886m 998m 4.50 8.68vol lvol6 2646674 2379866 69607m 16601m 1.19 1.66vol lvol7 16854588 20012885 206030m 119118m 0.10 0.16vol lvol8 79042007 128500851 899927m 689835m 0.04 0.02vol lvol9 301556 4788315 2090m 74546m 3.00 0.83vol lvol10 980251 4487076 287228m 193539m 3.20 0.62vol lvol11 82761 10693 338m 82m 2.00 11.79

Remark-d, -p, -s and -u m options are interesting to display information at differentgranularity level. -r option can also be used to reset statistics and focus on currentperiod only (to remove boot time statistics or reorganization operations)…

There is also vxtrace command to trace operations on volumes but I have, mostprobably, not really understood its added value as the output is too much verboseand does not contains any synthetic information:

[root@server1 ~]# vxtrace -g vgp1417 -o disk lvol857 START write disk emc0_0857 op 0 block 50782432 len 1657 END write disk emc0_0857 op 0 block 50782432 len 16 time 358 START write disk emc0_0857 op 0 block 195779 len 158 END write disk emc0_0857 op 0 block 195779 len 1 time 159 START write disk emc0_0857 op 0 block 1246640 len 1659 END write disk emc0_0857 op 0 block 1246640 len 16 time 2..

Remark:I/O statistics are gathered by default but when you system is stable or when youhave high I/O demand it may be interesting to deactivate it to maximize throughput.To check if statistics are gathered:

[root@server1 ~]# vxtune vol_stats_enable1

Then to de-activate them, as well as for DMP part (not possible to check if there areactivated or not):

[root@server1 ~]# vxtune vol_stats_enable 0[root@server1 ~]# vxdmpadm iostat stop

If you have slow response from a device group it may be also interesting to check ifyou have running operation in background:

[root@server1 ~]# vxtask -h listTASKID PTID TYPE/STATE PCT PROGRESS

Even if default VxVM parameters suit most workloads you can change them usingvxdmpadm command. To display tunable VxDMP parameters use:

[root@server1 ~]# vxdmpadm gettune all Tunable Current Value Default Value------------------------------ ------------- -------------dmp_cache_open on ondmp_daemon_count 10 10dmp_delayq_interval 15 15dmp_enable_restore on ondmp_fast_recovery on ondmp_health_time 60 60dmp_log_level 1 1dmp_low_impact_probe on ondmp_lun_retry_timeout 0 0dmp_path_age 300 300dmp_pathswitch_blks_shift 9 9dmp_probe_idle_lun on ondmp_probe_threshold 5 5dmp_queue_depth 32 32dmp_restore_cycles 10 10dmp_restore_interval 300 300dmp_restore_policy check_disabled check_disableddmp_retry_count 5 5dmp_scsi_timeout 20 20dmp_sfg_threshold 1 1dmp_stat_interval 1 1dmp_monitor_fabric on ondmp_monitor_osevent on ondmp_native_support off off

Remark:You may also want to use vxbench available in VRTSspt package…

ReferencesVeritas Volume Manager documentation

Veritas Volume ManagerVeritas VXDMPBasic I/O Monitoring on LinuxUnderstanding Linux Load Average – Part 1Understanding Linux Load Average – Part 2Understanding Linux Load Average – Part 3