f 1 - vcs hardware troubleshooting & linux os (1)

8/11/2019 f 1 - VCS Hardware Troubleshooting & Linux OS (1)

1/65

VCS Hardware Troubleshooting &Boot Process

6/17/201

Vernon !e"ee


2/65

#b$ecti%es

'(ter this lesson) learners will be able to*

+ecall the ,ost co,,on (or,s o( hardware (ailure on the

Cisco VCS

!eter,ine i( one o( these hardware (ailures is ta-ing "lace

+ecall the i,"ortant ste"s o( the boot "rocess on the

Cisco VCS and use this -nowledge to troubleshoot issues

with VCS booting


3/65

Phase 1 . VCS Hardware

H!!

C

'S


4/65

VCS Harddri%es

sda. Pri,ar !ri%e3 ( this (ails) the VCS will not boot at

all3 This is ,ounted to the root and /tandberg

sdb 4 Secondar !ri%e3 ( this (ails) the VCS will act 5odd3

This is ,ounted to /,nt/harddis-3


5/65

S!' ailure*

The VCS will ne%er begin loading the #S3 So,ething

si,ilar to this ,a be (roen on the console*


6/65

S!B (ailure*

S!B can (ail in se%eral was*

138 9hen the sste, boots) sdb will ne%er be detected

238 sdb will go unreachable a(ter the sste, has booted

38 sdb ,a be detected during sste, boot) but bounce

bac- and (orth between o"erating and not o"erating

"ro"erl


7/65

Testing sdb* ls

:ood*

ls 4l /de%/sd;

brw4rw4444 1 root root *2> /de%/sda

brw4rw4444 1 root root *2> /de%/sda1



brw4rw4444 1 root root 201241241= 1>*2> /de%/sda>



brw4rw4444 1 root root *2> /de%/sda /de%/sdb

brw4rw4444 1 root root *2> /de%/sdb1

brw4rw4444 1 root root *2> /de%/sdb2


8/65

Testing sdb* ls

Bad*

ls 4l /de%/sd;





brw4rw4444 1 root root 201241241= 1>*2> /de%/sda>



brw4rw4444 1 root root *2> /de%/sda3=0 2010410416 r1


10/65

Testing sdb* s,artctl

Bad*

? @ s,artctl 44all /de%/sdb

s,artctl >3=0 2010410416 r1


11/65

Testing sdb* s,artctl

Bad*

? @ s,artctl 44all /de%/sdb

s,artctl >3=0 2010410416 r1


12/65


13/65

Testing sdb* d(

Bad*

? @ d( M gre" sdb

? @

Sdb is not ,ounted


14/65

Testing Ithernet Ports

:ood*

? @ i(con(ig 4a M gre" eth

eth0 Lin- enca"*Ithernet H9addr 00*10**1A*'0*B=

eth1 Lin- enca"*Ithernet H9addr 00*10**1A*'0*B>

eth2 Lin- enca"*Ithernet H9addr 00*10**1A*'0*B6

eth Lin- enca"*Ithernet H9addr 00*10**1A*'0*B7


15/65

Testing Ithernet Ports

Bad*

? @ i(con(ig 4a M gre" eth

eth0 Lin- enca"*Ithernet H9addr 00*10**1A*'0*B=



'll = Ithernet "orts ,ust show


16/65

Serial u,ber +ange (or Potential

C ssues ( the serial nu,ber is in the range >2'1A2674>2'2>>10(or VCS and >>'000014>>'0001 (or Conductor) the VCS

is at ris- (or a (ailed C3

Iact Serial u,bers can be seen on the ecel doc athtt"*//172313107/"atches/SN20listN20o(

N20"otentialN20CN20issueD%er23ls


17/65

an ssues

' VCS can run (ine with 2 (ailed (ans

Process +' i( reOuested b custo,er e%en i( onl one

(an (ailure3 Custo,er will get a re(urbished VCS that ,a

ha%e ,ore se%ere "roble,s3 VCS running > and 6 will re"ort a (an alar, e%en i( the

(an is now o"erating at a "ro"er s"eed3 'lar,s can be

,anuall reset3

Current VCS code will onl raise (an alar, i( 2 or ,ore(ans ha%e (ailed3


18/65

VCS :enerations

>2'0 . Brage1

>2'1)>2'2 . Brage2

Brage1 is Ouite old and has reached the e"eced H!! li(e

(or sdb


19/65

Phase 2 . VCS Boot Process

:rub

nit scri"ts


20/65

VCS Boot Process 4 :rub

The VCS boots o(( o( grub on sda13 :rub will then ha%e

the VCS boot either o(( o( sda> Fi,age18 or sda6

Fi,age283

The acti%e i,age is ,ounted onto /3


21/65


22/65

VCS Boot Process . rc3ssinit

/etc/rc3ssinit calls all o( the startu" scri"ts in /etc/init3d/3

Scri"ts starting with I are called (irst) a(ter this scri"ts

starting with S are called3 Scri"ts are called in nu,berical

order3

Ia,"le* I00bootlogd is called (irst and SAA%,toolsd is

called last3

/etc/init3d/ scri"ts are called with an argu,ent o( either

start) sto") or restart3


23/65

VCS Boot Process . ounting o(

other Partitions /etc/init3d/I26,ount reads /etc/"artitions3con( to get thelocation o( the ro and rw "artitions3

The scri"t will then detect which "artition is ,ounted as

root and ,ount the a""ro"riate rw "artition on /tandberg3 +o "artition 1 is /de%/sda>

+w "artition 1 is /de%/sda7

+o "artition 2 is /de%/sda6 +w "artition 2 is /de%/sda

oa-Dcrashlogs F"re47328 / crashlogsF732[8

3tar3g (ile o( recent crashes ote an "rocess on the bo can crash . so,e serious) so,e not

VCS so(tware F5a""8 will auto,aticall restart) but call and registration statewill be lost

5net is har,less's are 5linustatus3" or 5,anage,ent(ra,ewor-3" F"rior to 7328

5winbindd could necessitate a restart

Loo- inside crash L du,"s to see i( the are re"etiti%e

Load into htt"s*//103>031>23110to see i( theUre -nown

Please chec- the out"ut ,a-es sense based on s,"to,s/logs( in doubt)please do check with us!
http://10.50.152.110/http://10.50.152.110/


36/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential 6

clusterdb

Tetual du,"s o( the Cluster!B tables

Those with a serial nu,ber su((i* sharded F"eer4s"eci(ic8 records

i3e3 records under co,"lete control o( that "eer

on4sharded Fglobal8 tables

'n "eer can edit

Status is t"icall sharded


37/65


harddis-logs

'll logs in logs/ subdirector

Prior to 7* ou need to untar the logs3tar3g (ile too

Logs are on a single "artition on the VCS . not %ersion4s"eci(ic

u"grade3log

Logs u"grades on the bo3 'llows correlation o( u"grade trans(or,s) reboots)ser%ice and load changes) etc3

tandberg4webcon(ig3log

Logs all con(iguration changes3 Ver use(ul3

a"ache;log

Logs web ser%er accesses and errors) including TS "robes3


38/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential #2#Q%c-/li/sFQ/l/nod!ex90Rusc01!cisco!comQ &otocolFQU3&Q LevelFQ1QUTCTimeFQ2012-0-20 09:49:4)E24S

essages include*

+egistration +eOuested M 'cce"ted M +e$ected E

Search 'tte,"ted M Co,"leted M Cancelled E

Source 'liases +ewritten

Call 'tte,"ted M Connected M +e$ected M !isconnected E

essage Sent M +ecei%ed E FH322> Signalling8

+eOuest Sent M +ecei%ed E FSP Signalling8


53/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >

harddis-logs . networ-Dlog

Logs actual networ- ,essages3 Ia,"le H322>*

2012-0-2)T1#:9:1K01:00 usc01-vcs1 tvcs: UTCTimeFQ2012-0-2)12:9:1E410Q ModuleFQnetwo8!h#2#Q LevelFQN57Q: 3st-ipFQ10!0!1"1!2#Q3st-potFQ120Q M 3et/ilFQ%endin@ >!22 %etup >!#2#v" (/ndwidth:")8ps3estli/s:h/ll/m2!ex90Rusc01!cisco!com 3estC%dd:V&v4VVTC&VV10!0!1"1!2#:120VW Jendo:TN3(.=+Q

Ia,"le SP3 Trace with Call4!*2012-0-2)T1":00:19K01:00 usc01-vcs1 tvcs: UTCTimeFQ2012-0-2)1:00:19E#Q ModuleFQnetwo8!sipQ LevelFQN57Q: %c-ipFQ10!0!1"!1Q%c-potFQ20#Q 3et/ilFQ=eceive =euest MethodFNJT.E =euest-U=Fsip:usc01-1-ed@/stonRcisco!comE C/ll-3Fddd1"#9ce)/1dR10!!)0!22Q

2012-0-2)T1":00:19K01:00 usc01-vcs1 tvcs: UTCTimeFQ2012-0-2)1:00:19E4Q ModuleFQnetwo8!sipQ LevelFQN57Q: 3st-ipFQ10!0!1"!1Q3st-potFQ20#Q 3et/ilFQ%endin@ =esponse CodeF100E MethodFNJT.EToFsip:usc01-1-ed@/stonRcisco!comE C/ll-3Fddd1"#9ce)/1dR10!!)0!22S

ote* uni(iedDlog G networ- [ ,essages [ de%elo"erDlog


54/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >=

!iagnostic Logging

Pro%ides a si,"le wa to get "rotocollogs and other "ertinent log ,essages(ro, the VCS

Select aintenance ] !iagnostics ]!iagnostics Logging

+e"laces 5netlog (ro, 730 onwards

Set 5networ- to !IBK: FeOui%alent to5netlog 28

'lso set 5interwor-ing to !IBK:

unless ouUre sure no interwor-ing isgoing on

9ill a((ect what is in sna"shot Flog ista-en (ro, uni(iedDlog8

Logging is started/sto""ed acrosswhole cluster) but ,ust be downloaded


55/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >>

ew Sna"shot eatures in 732

'lar,s (older logging acti%e alar,s

Sensors logs now logs 5a""stats . the +esourceKsage out"ut Fcallsand registrations in use at that "oint in ti,e8

Res!urce"sa#e $tem&1&'

(a))s $tem&1&'

Traversa) $tem&1&'

(urre*t $tem&1&'0/(urre*t'

+a, $tem&1&'-/+a,'

T!ta) $tem&1&'680/T!ta)'

/Traversa)'

.!*Traversa) $tem&1&'

(urre*t $tem&1&'0/(urre*t'

+a, $tem&1&'8/+a,'

T!ta) $tem&1&'1099/T!ta)'

/.!*Traversa)'


56/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >6

Ti"s) tric-s and co,,on issues^


57/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >7

9or-ing with !iagnostic Logs

Tracing SP call

Loo- (or the VTI

Then trace the Call4!

Beware o( (or-ing

!ebugging nterwor-ing

#"enLogicalChannel F#LC8 onl sent on ,edia

Kse the 5LCList out"ut (or ,onitoring logical channels

Con(iguration

5Status sna"shot is ,ore co,"lete than stat/con(


58/65


59/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential >A

+'

Should onl be used (or hardware (ailure

See earlier diagnosis o( so,e issues

See +' :uide on Cisco website FVCS !ocs/Troubleshooting8*

htt"*//www3cisco3co,/en/KS/docs/tele"resence/in(rastructure/%cs/troubleshooting/+'DProceduresD(orDCiscoDVCSDandDTelePresenceDConductorD'""liances3"d(

actor reset scri"t Flog is as root) run 5(actor4reset8 reinstalls the last%alid i,ageR suitable (or ,an so(tware issues
http://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/troubleshooting/RMA_Procedures_for_Cisco_VCS_and_TelePresence_Conductor_Appliances.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/troubleshooting/RMA_Procedures_for_Cisco_VCS_and_TelePresence_Conductor_Appliances.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/troubleshooting/RMA_Procedures_for_Cisco_VCS_and_TelePresence_Conductor_Appliances.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/troubleshooting/RMA_Procedures_for_Cisco_VCS_and_TelePresence_Conductor_Appliances.pdf


60/65


Securit Vulnerabilities

5VCS ails scan (or reason

Chec- out grwi-i list (or (ies in later VCS %ersions*

htt"*//grwi-i/inde3"h"/VCS/SecuritDoti(ications

There are a nu,ber o( (alse "ositi%es Fe3g3 PHP C:) 9eb!'V8

( ou ha%e a CVI nu,ber (or a third "art a""lication Fe3g3 '"ache)o"enssl8) chec- out the re"ort e3g3*

htt"*//c%e3,itre3org/cgi4bin/c%ena,e3cgiXna,eGCVI420124001

This will t"icall sa which %ersions are %ulnerable

Then chec- against the %ersion o( so(tware running on the ,ost recentVCS %ersion

or certi(icate errors) (a,iliarise and "oint towards VCS Certi(icateCreation and Kse :uide FCisco3co,) VCS Con(iguration :uides8*

htt"*//www3cisco3co,/en/KS/docs/tele"resence/in(rastructure/%cs/con(igDgu
http://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Certificate_Creation_and_Use_Deployment_Guide_X7-2.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Certificate_Creation_and_Use_Deployment_Guide_X7-2.pdfhttp://grwiki/index.php/VCS/Security_Notificationshttp://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0031http://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Certificate_Creation_and_Use_Deployment_Guide_X7-2.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Certificate_Creation_and_Use_Deployment_Guide_X7-2.pdfhttp://www.cisco.com/en/US/docs/telepresence/infrastructure/vcs/config_guide/Cisco_VCS_Certificate_Creation_and_Use_Deployment_Guide_X7-2.pdfhttp://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0031http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-0031http://grwiki/index.php/VCS/Security_Notificationshttp://grwiki/index.php/VCS/Security_Notifications


61/65


62/65


63/65


64/65

Y 2010 Cisco and/or its a((iliates3 'll rights reser%ed3 Cisco Con(idential 6=

#ther co,,on issues

Stuc- calls in status) "re4732

Call status snc between 5stat and what was seen on the web had issuesbe(ore 732 . rewritten in 732 and issues resol%ed FCSCtr2


65/65

WuestionsX

f 1 - vcs hardware troubleshooting & linux os (1)

Documents