Download - Paging Per LAC
For internal use
1 © Nokia Siemens Networks 2013 CEF HR CoE C&B PMI / 2014-04-13 -- Version: 1.0 / Life Cycle: ATP 2014
Case StudyNA05747502 N121_Paging Per LAC%.
Author Name: Mobily Care Team_ Xiao XiongfeiDate:25, Dec 2014.
2 © Nokia Siemens Networks 2013
For internal use
CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013
Case Study Reference
Reference ID: NA05747502 :Mobily:mcRNC :RU40Author Reference: Xiao Xiongfei
Case Study TypeTechnical Problem Investigation:
Case study prepared to investigate mcRNC network problem and presents its relevant workaround, solution and recommendations
Case Study Status:
Case Study Audience:Direct audience: Care engineers for mcrnc technical support.
Related case studies (documents):
Change note of mcrnc 3.0 3.0
Status Date
In-progress last reviewed on Dec 25, 2014.
Case Study team:
Role Name
Author Xiao Xiongfei
Contribute 1 Zhang Liming
Contribute 2
mcRNC3.0 3.0 Documentation Set v1.4.zip
3 © Nokia Siemens Networks 2013
For internal use
CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013
Case Study Problem Description
Mobily reported that RNC N121 have a severe paging degradation on this RNC for last 3 days. Cell Updates due to RLC unrecoverable errors have increased on many sites without any obvious reason for last 3 days. There was also an increase noticed from 12/13/2014 in Cell Updates due to RLC unrecoverable errors on some sites. From geolocation, there is no change in RF. PCH load on RNC is also acceptable
ImpactM1006C40 and M1006C215 increased.Cell PCH paging also got degraded.HSPA accessibility due to UE fails (M1002C415, M1002C423) have increased and HSDPA CREDIT reduction counters (M5000C176, 177 & 178) are showing increase
Problem description symptoms
Case symptoms are under RNCTS server: ftp://[email protected]/mnt/array1/rncts/users/Xiao/NA05747502 N121_Paging Per LAC%/
Symptoms based problem description
We are seeing a severe paging degradation on this RNC for last 3 days. Cell Updates due to RLC unrecoverable errors have increased on many sites without any obvious reason for last 3 days,
4 © Nokia Siemens Networks 2013
For internal use
CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013
Case Study Troubleshooting
1, Someone did RNW plan download and active on 21.12.2014 15:48, after that, significant increase of this two counters can be seen.2, Collect fishing log with two affect counters.3, as required by PL, collect Megamon log with below conditions at the same time:USCP with default conditionsCSCP with SR:((OFAM=9B6,9B7)AND(NOT(NUM=1,2)))4, most of the affected BTSs linked with CSPU-2, and based on the megamon logs. CSCP-2.MGI mcf_ul_ccch_s load_status is 1 , but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) this means CSCP-2 unit cell cch load is too much high.5, to recovery this issue, need switch or reset CSPU-2 unit.
Attachments
Investigation details:
IllustrationsPaging shows improvement after CSPU2 unit switchover. .
Investigation Task Explanation
1. Collect mcrnc symptom log2. Collect fishing log based on affected counters.3. Collect megamon logs required by PL4. enable/disable minor alarm which definde the error code5. need switch or reset CSPU-2 unit
To check alarms and unit stateTo check the call drop on which phaseTo check detials of RRA,RRB message. To double confirm this is know issue which fixed on mcrnc3.0 3.0To recovery this problem
N121_Alarm History.zip
N121_M1006C251.rar
5 © Nokia Siemens Networks 2013
For internal use
CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013
Analysis of troubleshooting
most of the affected BTSs linked with CSPU-2, and based on the megamon logs.CSCP-2.MGI mcf_ul_ccch_s load_status is 1 , but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) this means CSCP-2 unit cell cch load is too much high. 1, This looks the same issue as NA05697607 for that case correction included in mcRNC 3.0 3.0, DSP of CCH handling by CSUP inside of CSPU.to double confirm, please enable minor error and provide syslog.Step-1:-Enabling of minor error in csup-2 unit.ilcliru.sh omu "/opt/nokiasiemens/SS_QNTools/bin/umw_diag -- loglevel -C csup-2 -a on -u on -m off"output will come in syslog only. That you can collect as part of symptom report.step-2:-After getting the expected output over the time, please disable the minor error by executing following cmd.ilcliru.sh omu "/opt/nokiasiemens/SS_QNTools/bin/umw_diag -- loglevel -C csup-2 -a on -u on -m on" 2,to recovery this issue, please switch or reset CSPU-2 unit.
Analsis Summary:and based on the megamon logs.CSCP-2.MGI mcf_ul_ccch_s load_status is 1 , but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) Based on syslog we can see 0x30B40023/ 0x30B40024
[root@CFPU-0(RNC-549) /srv/Log/log]# cat syslog | grep -i "30B400"Dec 3 22:58:17.148244 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:17:148156|TEXT:APP_ERROR: |DATA:0xD0000003 0x30B40024 0x00000000|APP_DUMP: |0x0000 0x0220 0x76D0 0x7B31 0x0000 0x0064 0x0000 0x0017 0x009A 0x1463 Dec 3 22:58:17.382285 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:17:382198|TEXT:APP_ERROR: |DATA:0xD0000002 0x30B40023 0xFFFF0000|APP_DUMP: |0x0000 0x0064 0xFFFF 0xFF7C 0x0000 0x0018 Dec 3 22:58:25.617304 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:25:617200|TEXT:APP_ERROR: |DATA:0xD0000004 0x30B40024 0x00000000|APP_DUMP: |0x0000 0x0222 0x0000 0x8E22 0x0000 0x0064 0x0000 0x0018 0x008F 0x22C5 Dec 3 22:58:26.041940 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:26:041853|TEXT:APP_ERROR: |DATA:0xD0000001 0x30B40023 0xFFFF0000|APP_DUMP: |0x0000 0x0064 0xFFFF 0xFF7C 0x0000 0x0019
6 © Nokia Siemens Networks 2013
For internal use
CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013
Case Study ResultCase study conclusion:
1, First step, to check alarm history and unit status to check if any operation which may casue this issue.2, sometimes fishing log not trigger the call trace since mcrnc PRB located on different unit, you have to check with PL for different counter. 3, find what is the common issue for the affect site, if it is located in the same EIPU? CSCP unit.
Recommendations
Check what is the common issue for the affect site, if it is located in the same EIPU? CSCP unit.
Risks and impact:
Action items and next steps if any
Upgrade to mcrnc 3.0 cd3.0
- KPI have been affected
Solution steps:CN-id: mcRNC30_0269Description of the correction:Load control mechanism has been changed:- Load control is now per UP server in CSPU unit.- Load control algorithm has been made generic and to handle exceptional situations.
Lessons learned
RE N121_Paging Per LAC%.zip