paging per lac

For internal use

1 © Nokia Siemens Networks 2013 CEF HR CoE C&B PMI / 2014-04-13 -- Version: 1.0 / Life Cycle: ATP 2014

Case StudyNA05747502 N121_Paging Per LAC%.

Author Name: Mobily Care Team_ Xiao XiongfeiDate:25, Dec 2014.

2 © Nokia Siemens Networks 2013

For internal use

CEF HR CoE C&B PMI / 2013-01-28 -- Version: 1.0 / Life Cycle: ATP 2013

Case Study Reference

Reference ID: NA05747502 :Mobily:mcRNC :RU40Author Reference: Xiao Xiongfei

Case Study TypeTechnical Problem Investigation:

Case study prepared to investigate mcRNC network problem and presents its relevant workaround, solution and recommendations

Case Study Status:

Case Study Audience:Direct audience: Care engineers for mcrnc technical support.

Related case studies (documents):

Change note of mcrnc 3.0 3.0

Status Date

In-progress last reviewed on Dec 25, 2014.

Case Study team:

Role Name

Author Xiao Xiongfei

Contribute 1 Zhang Liming

Contribute 2

mcRNC3.0 3.0 Documentation Set v1.4.zip


For internal use


Case Study Problem Description

Mobily reported that RNC N121 have a severe paging degradation on this RNC for last 3 days. Cell Updates due to RLC unrecoverable errors have increased on many sites without any obvious reason for last 3 days. There was also an increase noticed from 12/13/2014 in Cell Updates due to RLC unrecoverable errors on some sites. From geolocation, there is no change in RF. PCH load on RNC is also acceptable

ImpactM1006C40 and M1006C215 increased.Cell PCH paging also got degraded.HSPA accessibility due to UE fails (M1002C415, M1002C423) have increased and HSDPA CREDIT reduction counters (M5000C176, 177 & 178) are showing increase

Problem description symptoms

Case symptoms are under RNCTS server: ftp://[email protected]/mnt/array1/rncts/users/Xiao/NA05747502 N121_Paging Per LAC%/

Symptoms based problem description

We are seeing a severe paging degradation on this RNC for last 3 days. Cell Updates due to RLC unrecoverable errors have increased on many sites without any obvious reason for last 3 days,


For internal use


Case Study Troubleshooting

1, Someone did RNW plan download and active on 21.12.2014 15:48, after that, significant increase of this two counters can be seen.2, Collect fishing log with two affect counters.3, as required by PL, collect Megamon log with below conditions at the same time:USCP with default conditionsCSCP with SR:((OFAM=9B6,9B7)AND(NOT(NUM=1,2)))4, most of the affected BTSs linked with CSPU-2, and based on the megamon logs. CSCP-2.MGI mcf_ul_ccch_s load_status is 1 ， but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) this means CSCP-2 unit cell cch load is too much high.5, to recovery this issue, need switch or reset CSPU-2 unit.

Attachments

Investigation details:

IllustrationsPaging shows improvement after CSPU2 unit switchover. .

Investigation Task Explanation

1. Collect mcrnc symptom log2. Collect fishing log based on affected counters.3. Collect megamon logs required by PL4. enable/disable minor alarm which definde the error code5. need switch or reset CSPU-2 unit

To check alarms and unit stateTo check the call drop on which phaseTo check detials of RRA,RRB message. To double confirm this is know issue which fixed on mcrnc3.0 3.0To recovery this problem

N121_Alarm History.zip

N121_M1006C251.rar


For internal use


Analysis of troubleshooting

most of the affected BTSs linked with CSPU-2, and based on the megamon logs.CSCP-2.MGI mcf_ul_ccch_s load_status is 1 ， but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) this means CSCP-2 unit cell cch load is too much high. 1, This looks the same issue as NA05697607 for that case correction included in mcRNC 3.0 3.0, DSP of CCH handling by CSUP inside of CSPU.to double confirm, please enable minor error and provide syslog.Step-1:-Enabling of minor error in csup-2 unit.ilcliru.sh omu "/opt/nokiasiemens/SS_QNTools/bin/umw_diag -- loglevel -C csup-2 -a on -u on -m off"output will come in syslog only. That you can collect as part of symptom report.step-2:-After getting the expected output over the time, please disable the minor error by executing following cmd.ilcliru.sh omu "/opt/nokiasiemens/SS_QNTools/bin/umw_diag -- loglevel -C csup-2 -a on -u on -m on" 2,to recovery this issue, please switch or reset CSPU-2 unit.

Analsis Summary:and based on the megamon logs.CSCP-2.MGI mcf_ul_ccch_s load_status is 1 ， but for other CSCP units it is 0.asn_data_mcf_ul.load_status = 1 (0x0001) Based on syslog we can see 0x30B40023/ 0x30B40024

[root@CFPU-0(RNC-549) /srv/Log/log]# cat syslog | grep -i "30B400"Dec 3 22:58:17.148244 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:17:148156|TEXT:APP_ERROR: |DATA:0xD0000003 0x30B40024 0x00000000|APP_DUMP: |0x0000 0x0220 0x76D0 0x7B31 0x0000 0x0064 0x0000 0x0017 0x009A 0x1463 Dec 3 22:58:17.382285 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:17:382198|TEXT:APP_ERROR: |DATA:0xD0000002 0x30B40023 0xFFFF0000|APP_DUMP: |0x0000 0x0064 0xFFFF 0xFF7C 0x0000 0x0018 Dec 3 22:58:25.617304 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:25:617200|TEXT:APP_ERROR: |DATA:0xD0000004 0x30B40024 0x00000000|APP_DUMP: |0x0000 0x0222 0x0000 0x8E22 0x0000 0x0064 0x0000 0x0018 0x008F 0x22C5 Dec 3 22:58:26.041940 warn CSPU-5 umwsup: RU:CSUPPXY-5|PNAME:QNSUPPRB|LPID:3967|PID:0xA1F 0x0 0x0|TIME:2014-12-03 22:58:26:041853|TEXT:APP_ERROR: |DATA:0xD0000001 0x30B40023 0xFFFF0000|APP_DUMP: |0x0000 0x0064 0xFFFF 0xFF7C 0x0000 0x0019


For internal use


Case Study ResultCase study conclusion:

1, First step, to check alarm history and unit status to check if any operation which may casue this issue.2, sometimes fishing log not trigger the call trace since mcrnc PRB located on different unit, you have to check with PL for different counter. 3, find what is the common issue for the affect site, if it is located in the same EIPU? CSCP unit.

Recommendations

Check what is the common issue for the affect site, if it is located in the same EIPU? CSCP unit.

Risks and impact:

Action items and next steps if any

Upgrade to mcrnc 3.0 cd3.0

- KPI have been affected

Solution steps:CN-id: mcRNC30_0269Description of the correction:Load control mechanism has been changed:- Load control is now per UP server in CSPU unit.- Load control algorithm has been made generic and to handle exceptional situations.

Lessons learned

RE N121_Paging Per LAC%.zip

paging per lac

Documents