michael monaghan - evolution of new feature verification in 3g networks
TRANSCRIPT
Evolution of new feature verification in 3G Networks
Eurostar
Stockholm, December 2009
Michael Monaghan, LM Ericsson (Ireland) Ltd
Short overview of LM Ericsson Founded 1876, Stockholm, Sweden
Presence in 175 countries
> 75,000 employees
~24,000 patents
> 40% of all mobile traffic passes through Ericsson networks
Ericsson supports networks which serve > 1 billion subscribers
Business Units– Networks– Multimedia– Global Services– Sony Ericsson
In this presentation, we will visit the 3G Mobile Network Development area– 3G refers to 3rd Generation, a family of standards for wireless communication– A glossary of terms appears at the end of this presentation
Feature Verification in 3G Networks
The scenario– Very complex systems– Very complex features– Very difficult to recreate real world environment– Many levels of test from block test to complex network test– Many tools needed
The aim– To verify that features behave according to requirements– To make a quality statement about new features, so a decision can
be made about whether to release them– To satisfy Time to Market and cost goals for the feature
Feature Verification in 3G Networks
Elementmanagement
OSS-RC
RNCRBS RXI EM
Operations Support SystemRadio & CoreRadio Network ControllerRadio Base StationRadio Network IntegratorElement Manager
EM
WCDMA: Wideband Code Division Multiple Access, an air interface standard
HSPA EvolutionUplink
20-40 Mbps
12 Mbps
5.8 Mbps
1.4 Mbps
0.4 Mbps
2 ms TTI
16QAM
Downlink
3.6 Mbps
14 Mbps
21 Mbps 28 Mbps
42 Mbps
80-160 Mbps
15 codes
2x2 MIMO64QAM
Multi Carrier
Multi Carrier64QAM + MIMOHigher ModulationCombinations
Multi Carrier
Refer to glossary for explanation of terms...
Major Advances in Verification of 3G Network Features
Evolution from Function Based to Feature Based Verification
More generic development leads to verification savings
Evolution of Feature Verification
Looking to the Future
Summary
Advances in RNC Feature Verification
2004 2005 2006 2007 2008 2009 2010
Generic Design/Test
StrategyFT to FV
MoveFunction
Test
ParallelTesting
Future
FT: Function TestFV: Feature Verification
Function Test
RNC verification was done from a functional viewpoint
Several functional teams triggered the same test case and then checked one part of the result
A test case most often consists of– Setting up a particular type of call (Control Plane)– Sending data (User Plane)– Moving the phone (Mobility)
Function Test
2004 2005
Function TestFunction
Test
2004 2005
Feature1
Feature2
Feature3
Feature4
Feature5
Mobility Team Control Plane Team User Plane Team
Mobility
Mobility
Mobility
Mobility
Control Plane
Control Plane
Control Plane
Control Plane
User Plane
User Plane
User Plane
Function Test
Small area of expertise for each engineer
Little feature overview knowledge in verification
Led to rigid test organisations and verification projects
Function Test
2004 2005
Feature Based Verification
Verification based on Use Cases
Verify features end to end
Verify each Use Case in the most suitable environment
Many challenges had to be overcome
FT to FV Move
2006 2007
Feature Verification
Feature1
Feature2
Feature3
Feature4
Feature5
Mobility Team Control Plane Team User Plane Team
Mobility
Mobility
Mobility
Mobility
Control Plane
Control Plane
Control Plane
Control Plane
User Plane
User Plane
User Plane
FT to FV Move
2006 2007
Feature Based Verification
Feature Verification has led to significant cost savings
Very responsive and flexible test organisation
An enabler for a feature based release strategy
FT to FV Move
2006 2007
Feature Based Verification
Feature Verification, an evaluation:
Feature #TCs Cost per Test Case (hours)
Total Cost (hours)
#TCs Cost per Test Case
Total Cost (hours)
Feature1 168 9 1512 34 8-20 (Avg 14)
476
69% saving!!
FT to FV Move
2006 2007
Function Test Feature Verification
RAB (Radio Access Bearer) Concept
Speech
SpeechData
Data
Speechplus data
Speechplus data
Generic Design/ Test
Strategy
2005 2006
Generic Design/Test Strategy
Market pressure for many radio connection combinations – e.g. different interactive packet RAB rates and combinations
Development began to be based on generic code units which made it easier to introduce new RAB rates and combinations
In P6 the number of RAB combinations exploded : (13 in P3, 18 in P4, 39 in P5), > 70 in P6, Not possible to verify all transitions
CS Streaming
UDI (CS64)PS Streaming+Int 8/8
SP0
SP64
UDI+Int. 8/8
Int. 64/128
Int. 64/384
64/64
Int. FACH
Speech
SRB
Other int. RAB state
SRB
Int. 128/128
Int. 128/384
128/64
Int. 384/128
Int. 384/384
384/64
HS
SPHS
Generic Design/ Test
Strategy
2005 2006
Generic Test Strategy
Test strategy was to verify the generic code once and then run a greatly reduced test scope on new RABs/RAB Combinations going forward
Reduced scope testing based on RAB type groups
2005 2006
Generic Design/ Test
Strategy
Generic Test Strategy Savings
Feature Pre-Generic(including all error and exception TCs)
Generic Test Principle 1(Removal of error and exception TC)
Generic Test Principle 2 (Rab type groups)
Generic Test Principle 3 (Some functions moved outside FV)
Generic Test Principle 4(Functional based changes costs removed)
Feature2 ~ 196 TC~ Exe cost 1764 hours
96 TC (-100)~ Exe cost 864 hours
88 TC (-8)~ Exe cost 792 hours
58 TC (-30)~ Exe cost 522 hours
43 TC (-15)~ Exe cost 387 hours
79% saving!
2005 2006
Generic Design/ Test
Strategy
Generic Test Strategy Effectiveness
Feature #TCs run in RNC Feature Verification
Faults found by RNC Feature Verification
Faults found by IODT (Mobile phone Verification)
Faults found by RAN I&V
Slipthrough analysis to RAN I&V
Feature3 102 34 8
Analysis: Same faults would have been found in RNC but testing was done in parallel
2
Actual TRs on feature
2
Analysis: 1 due to tools issues, 1 due to intro of fault after test case had been run.
Generic test is very effective when applied correctly
2005 2006
Generic Design/ Test
Strategy
More Demanding Environment
2 releases per year
Market demand for many and more complex features (Mobile Broadband)
Continuous improvements in Time To Market required
Need for continuous improvements in operational excellence
Need to streamline verification further:– Introduce more parallel testing– Minimize repetition in testing at all levels– Maximise test effectiveness
ParallelTesting
2007 2008
Parallel Testing
RNC node test work more closely with higher layer test organisations
Form a RAN Level Joint Test team with participation from network nodes (RNC, RBS) and RAN
Make Phased Deliveries for key features
Split scope of feature test between RAN and nodes
ParallelTesting
2007 2008
Node/Subsystem I&V Integration
Subsystem I&V is a very useful, low cost and under-utilised test environment
Was Best Effort/Happy Test
Integration of test activities means node level TCs are passed in a Subsystem I&V environment
Improves verification lead times at RNC node level
Reduces repetition between Subsystem and RNC test documentation and execution
Need to minimise quality risk to RNC and Subsystem
Node/Subsystem
I&VMerge
2008 2009
Maximise Test Effectiveness Merge of RNC node feature verification with RAN feature verification
– Pass RAN level test cases in a subsystem verification environment– Remove RNC node verification as a layer of verification
Eurostar conference– Ericsson verification problems are industry-wide– We compare favourably with other companies in many respects– However there are some methodologies we should study more
Risk Based Test Case Selection– Each test case ranked by likelihood and impact– Trial was inconclusive
Estimate number of remaining faults– Tools exist which predict number of latent faults in a feature– Too much emphasis now on “slip-through”
Preventive Testing
Future
2009 2010
Summary
We have come a long way
Time pressure will continue to mount
Borders between layers of the development process are being broken down
Summary
2004 2010
Glossary EM - Element Manager
– A simplified management interface for RNC and RBS nodes
FT - Function Test– a test strategy where new features are verified by verifying the functions which realise the
features one by one
FV - Feature Verification– a test strategy where new features are verified as a unit
HSPA - High Speed Packet Access– a collection of protocols which provide high bandwidth and improved spectral efficiency in 3G
networks
Interfaces – The communication channel and protocol between network nodes– Iu - interface between the RNC and the core network– Iub - interface between the RNC and the Radio Base Station– Iur - interface between 2 RNC nodes– Mun – interface between other management systems and OSS-RC– Mur - operator management system interface towards RNC– Mut - operator management system interface towards RXI– Mub - interface from operator management system towards RBS– Uu - interface between the RBS and the Ue
Glossary (contd...) I&V - Integration and Verification
– a generic term in Ericsson to refer to all verification activities
MIMO - Multiple-input and multiple-output– use of multiple antennas at both the transmitter and receiver to improve communication
performance
Multi-Carrier– a method of allowing a mobile device listen to two HSPA data streams from different cells
simultaneously in downlink, thus facilitating higher bandwidth
OSS-RC – Operation Support System for Radio and Core– a network management node
P6 – Project or Phase 6– Ericsson RAN product releases are numbered P2...Pn
RAB - Radio Access Bearer– a UMTS service providing an application using the UMTS network with the ability to send
and receive data over the UMTS network with a specific Quality of Service (QoS).
RAN - Radio Access Network– part of a mobile telecommunications system which implements the radio access technology,
so the mobile equipment can communicate with the backbone networks
Glossary (contd...) RBS - Radio Base Station
– a 3G network node containing radio frequency transmitter(s) and receiver(s) used to communicate directly with the User Equipment
RNC - Radio Network Controller– a network node in a 3G network which controls the base stations which are connected to it.
It also carries out other functions including radio resource management and mobility management
RXI – An Access Network Aggregator– an ATM switch supporting handling of various network topologies and technologies
TC - Test Case– For the purposes of this presentation, a test case generally involves establish a specific type
of phone call (voice and/or data)
Ue - User Equipment– any device used directly by an end-user to communicate. It can be a hand-held telephone, a
card in a laptop computer, or any other device. It connects to the base station
WCDMA - Wideband Code Division Multiple Access– an air interface standard used in 3G networks
16/64 QAM -16/64 Quadrature amplitude modulation– methods of coding data for transmission over the air
Glossary (contd...) 2MS TTI - 2MS Transmission Time Interval
– Transmission Time Interval refers to the length of an independently decodable radio transmission
3G - 3rd Generation– a family of standards for wireless communications. Main characteristics are wide range of
services, high throughput and spectral efficiency