ttcn3 as another brick in the wall of test automation
DESCRIPTION
TTCN3 as Another Brick in the Wall of Test Automation. Authors: Gergely Horváth ([email protected]) Róber t Vastag ([email protected]) TTCN-3 User Conference 2011 7 – 9 June 2011 - Bled, Slovenia. Some Facts About Nokia Siemens Networks. Joint Venture of Nokia and Siemens - PowerPoint PPT PresentationTRANSCRIPT
TTCN3 as Another Brick in the Wall of Test Automation
Authors: Gergely Horváth ([email protected])Róbert Vastag ([email protected])
TTCN-3 User Conference 20117 – 9 June 2011 - Bled, Slovenia
Some Facts About Nokia Siemens Networks
•Joint Venture of Nokia and Siemens
•Started operations on April 1, 2007
•€12.7 billion net sales in 2010
•120+ years of telecom experience
•65,000+ employees
•45,000 service professionals (including externals)
•75 of top 100 operators worldwide
•150 countries
•3 billion mobile subscribers and ¼ of world’s voice households served
Agenda
User Plane Routing Development(what we do)
Agile and TTCN3(how we do it)
Test Automation with TTCN3(how TTCN3 is integrated into it)
TTCN3 Core and Fast Test Case Creation(how it works exactly)
User Plane Routing DevelopmentControl
Data
• Each interface of the SUT is captured• Radio access is simulated by the traffic
generator• MGWs are simulated by an executable
built from TTCN3 code that is specific to each test case
H.248
H.248
USER PLANE
RADIOACCESS
System Under Test: Main focus is on user plane routing functionalities
TTCN3 executable:Connected with the SUT through H.248 interface, simulating MGW functionalities
User Plane Routing Development(what we do)
Agile and TTCN3(how we do it)
Test Automation with TTCN3(how TTCN3 is integrated into it)
TTCN3 Core and Fast Test Case Creation(how it works exactly)
• Automated unit tests • Scheduled and automated SW builds• Automated test execution environment for regression (set of functional tests)
Agile and TTCN3
• Fast module test environment for "instant" checking of the new code• Nightly functional regression test set execution
• New feature increment implemented in every iteration• Scrum meetings• Incremental test case design (TTCN3)
Agile
ContinuousIntegration
FastFeedback
Iterative
FeatureRequirements
FeatureDeployment
• Evaluating TTCN3 Results• Fine tuning the test case
• MGW simulation by TTCN3 executable • Acquiring results
• Developing TTCN3 core functions to follow changes in user plane routing• Creating test cases using TTCN3
Agile and TTCN3
Test Design
Test ExecutionTest Planning
Verification
Specification
Implementation
Automate Tests & Add to Regression Set
Evaluation
Unit Test
Development& Unit Test
TTCN3 Core Development& Integration Test
FeatureRequirements
FeatureDeployment
User Plane Routing Development(what we do)
Agile and TTCN3(how we do it)
Test Automation with TTCN3(how TTCN3 is integrated into it)
TTCN3 Core and Fast Test Case Creation(how it works exactly)
Test Automation with TTCN3
TTCN3 Build Server
SVN Repository for Test Macros and TTCN3
Test Management System
Manager• Tester• Test Designer• Test Architect• TTCN3 Key User Test Automation Engine
Test Laboratory
Test Control Equipment Pool
Analysis Server Pool
Test Laboratory
Configurationdata and
commandsequence
MGW simulationwith TTCN3
Collectingexecution
results
LAB NETWORK
Test Control Equipment Pool
SUT POOL
Test datafrom SVN
Test Automation with TTCN3
TTCN3 Build Server
SVN Repository for Test Macros and TTCN3
Test Management System
Manager• Tester• Test Designer• Test Architect• TTCN3 Key User
Test Laboratory
Test Control Equipment Pool
Test Automation Engine
Analysis Server Pool
Test Automation Engine
• Resource usage report• Resource reservation• Test set execution report• Test execution reports• Archivation of results• Interface to test management system• Authorized access (Tester, Test Designer, Test Architect…)
Resource Usage ReportTest Set Execution Report
Test Execution Report
TTCN3 Result in XML
Test Automation with TTCN3
TTCN3 Build Server
SVN Repository for Test Macros and TTCN3
Manager•Tester•Test Designer•Test Architect•TTCN3 Key User
Test Laboratory
Test Control Equipment Pool
Test Automation Engine
Analysis Server Pool
Test Management System
• Traceability Support• Requirement Fulfillment Check• Maintenance of Test Documents• Management Reporting• Statistics
Test Automation with TTCN3
TTCN3 Build Server
SVN Repository for Test Macros and TTCN3
Manager•Tester•Test Designer•Test Architect•TTCN3 Key User
Test Laboratory
Test Control Equipment Pool
Test Automation Engine
Analysis Server Pool
Test Management SystemRoles & Responsibilities:
Tester Executes and analyzes test cases, writes incident reports
Test Designer Creates test cases according to the test plan
Test Architect Responsible for regression set creation and high level planning
TTCN3 Key User Responsible for the maintenance of the TTCN3 Core (common functions used for test case creation)
User Plane Routing Development(what we do)
Agile and TTCN3(how we do it)
Test Automation with TTCN3(how TTCN3 is integrated into it)
TTCN3 Core and Fast Test Case Creation(how it works exactly)
TTCN3 Core System. Why?• Resource Reservation• Resource Modification• Resource Removal• Service Change• Audit
H.248 messages have a difficult hierachy and convey many parameters
Building a template that can match a specific message is a difficult task, even
for an experienced tester.
Solution:
• New user plane control functionalities are implemented in TTCN3 core to make testers’ lives easier.• Only high level parameters need to be given
A Sample H.248 Message:
TTCN3 Core System. How?
• Parameterized functions that can construct templates for H.248 messages• Compliant with the H.248 standard• Each test step implements one H.248 transaction• It hides the complex internal logic from the tester
• Only high level knowledge is necessary for TTCN3 test case creation• Makes test case creation fast and efficient• Test cases are well structured• Global modifications in the functionalities of the SUT are easy to follow by updating the TTCN3 core (TTCN3 key users’ task)
ComplexTest Steps
MainParametersIn Testers’
Hands
Features of Test Steps:
…and Benefits of its Usage:Specific
MessageTemplate
Pass
Compare
ActualH.248
Message
FailExact location of the failure is shown!
=
+
TTCN3 Library Structure
TTCN core
Feature N
FEATURE_1_TEST_CASE_1FEATURE_1_TEST_CASE_2
.FEATURE_1_TEST_CASE_N
Feature 1
FEATURE_2_TEST_CASE_1FEATURE_2_TEST_CASE_2
.FEATURE_2_TEST_CASE_N
Feature 2
• Test cases of the same feature compile into the same executable• The executable has to be started with the test case ID as input parameter in order to execute the proper MGW simulation
TTCN3 Test Case Structuremodule FEATURE_1_TEST_CASE_1 {
import from TTCN_CORE all;
testcase tc_FEATURE_1_TEST_CASE_1() runs on CORE_TEST_COMPONENT system TEST_SYSTEM { SET_CONNECTIONS(); TEST_COMPONENT.start(TEST_SEQUENCE_1()); }
TEST_SEQUENCE_1() runs on TEST_COMPONENT { Init(); TEST_STEP_1(PARAM1,PARAM2); TEST_STEP_2(PARAM1,PARAM2,PARAM3,PARAM4); TEST_STEP_1(PARAM5,PARAM6); TEST_STEP_3(PARAM5,PARAM6,PARAM1,PARAM2); RESOURCE_REMOVAL; }}
Test step definitions
ModuleConnectivity settings
Actual Test Case Behaviour
Conclusion
• The TTCN3 core system makes it easier to cope with frequent changes in the software that we are testing
• It is thoroughly integrated with the automated test environment that makes its usage smooth and locating faults easy
• These characteristics aid it in supporting the agile development
• This way we can ensure efficient testing procedures and provide high quality software products
Thank you!
Your questions are welcome!