international telecommunication union geneva, 9(pm)-10 february 2009 providing testability for itu...
TRANSCRIPT
Geneva, 9(pm)-10 February 2009
InternationalTelecommunicationUnion
Providing testability for ITU Recommendations
Ostap Monkewich,OMCI
ITU-T Workshop on“New challenges for Telecommunication
Security Standardizations"
Geneva, 9(pm)-10 February 2009
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 2
Theme
“What is missing from your Recommendations that is needed for
testing?”
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 3
Why do we need to test?
To show the customer that the product meets the requirementsTo show that the product is likely to interoperate with other vendorsTo demonstrate that Recommendations were implemented inside the productTo improve the quality of ITU-T Recommendations
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 4
Need high-quality test results
Repeatable by any competent test laboratoryCan be used to compare with test results obtained for similar productsRecognized by all and accepted in all geographical market regions
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 5
What kind of testing?
Conformance Testingverifies if the implementation does what the Recommendation says it is supposed to do
Product Test Suite (Gold Standard)
Interoperability Testingchecks if two implementation communicate at functionalities level
• N = 6 products or 15 pairs• Each product is tested 15 times • N = 100 or ~5000 pairs• Each product is tested 5000
times
• Functionalities• Every pair
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 6
Frequently Asked Question
Why not do only interoperability testing?Answer: We don’t know what to do when two implementations do not interoperate:
If we change something – are we closer to the Recommendation or farther from it?
Non-conforming changes destroy interoperability with other vendors
Serious problems will not be discovered when observing functionalities only
example: turning lights on and off in a new house is a good “interoperability” or functionality test. The house may burn down at a later time because the wiring did not conform to the standard
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 7
Sources of Interoperability Problems
RecommendationsErrors in RecommendationsAmbiguities in natural languageUnverified or invalid behaviours described
Implementers’ different interpretations of text
Requirements in text, more than one meaningNo standardized questionnaire for supplier
Incompatible Recommendations/implementationsDifferent choices of optionsDevice incompatibilitiesDifferent host system configurations
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 8
In addition to Base RecommendationRequirements Clause
Extracted from text, need no interpretationImplementation Conformance Statement (ICS) Proforma
Supplier declares what pieces were implemented from Recommendation
Implementation eXtra Information for Testing (IXIT) Proforma
Identifies non-standardized items, how architecture, interfaces are packaged
Test Suite StructureLogically groups test cases
Test Purposesone test purpose/verdict per test case
Abstract Test SuiteSet of tests that covers the Recommendation
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 9
Recommendations to support testing
Bas
e R
ecom
men
dat
ion R1
R2
R3
Rn
.
.
.
TP1
TP2
TP3
TPn
.
.
.
TC1
TC2
TC3
TCn
.
.
.
I
CS
An
swer
s
TC1
TC3
TCn
.
.
.
RequirementsClause
TSS &TP ATSExecutionTest Cases
ICS
TSS & TP - Test Suite Structure and Test Purposes TP - Test PurposeATS: - Abstract Test Suite TC - Test CaseICS: - Implementation Conformance Statement R - Requirement
IX
IT I
nfo
rmat
ion
IXIT
IC
S Q
ues
tion
s
ICSProforma
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 10
Requirements – NGN Draft Rec. Y.2702
(R-40) Each user/subscriber associated with an application service subscription is required to be uniquely
addressable
(R-41) It is required that it be possible for the end user to access a service simultaneously multiple times and/or from multiple devices.
(R-42) It is required that it be possible to support multiple
subscription profiles for an individual end-user.
(O-1) It is an option that network access points supporting NGN TE and TE-BE support capabilities to allow the end user to uniquely identify the NGN provider.
(O-2) It is an option that network access points supporting NGN TE and TE-BE support capabilities to allow the user to
authenticate and authorize the NGN provider.
Sections 7.2.3 and 8.2.1 – NGN User authentication for service access and network attachment
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 11
Implementation Conformance Statement (ICS) Proforma
Index Text Status Ref. Val. Support
3.5.1
3.5.2
3.5.3
When User Equipment (UE) sends HTTP Request does Service Provider (SP) return http Response with <lib:AuthnRequest >
M
M
O
IV.2.1.2
1)
IV.2.1.2
2)
IV.2.1.2
x)
_Yes _No
_Yes _No
_Yes _No
On receipt of HTTP Request from UE does SP send a redirect http Response with <lib:AuthnRequest > in the URL
When UE sends HTTP Request does SP inform the Identity Provider (IdP) of the receipt
Appendix IV to NGN Draft Rec. Y.2702 on Identity management (IdM)
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 12
Examples of Test Purposes inSIP Security Testing
Ensure that the IUT, after sending the initial REGISTER request to the Registrar, ignores Registrar OK response by sending a second REGISTER request Ensure that the IUT re-sends the initial REGISTER request on receipt from the Registrar of a 401 Unauthorized response in which WWW-Authenticate header does not contain the nonce= field
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 13
Implementation Conformance Statement (ICS)
ICS = ICS Proforma with answersA list of requirements and options claimed to have been implemented
Used forShopping for products to match for interoperabilityselecting Test Cases from test suite for test execution
ICS
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 14
Test Suite Structure
Test Suite
Test Case Test CaseTest CaseTest Case
Test Group
Test GroupTest Group
Test Group
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 15
Test Case Structure
Test Case
Test Event Test EventTest EventTest Event
Test Step
Test StepTest Step
Test Step
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 16
Conclusion
First, Conformance testing then Interoperability testingHigh-quality Recommendations are neededBase Recommendations require additional parts to produce widely accepted test resultsMethodology Recommendations for testing are available – attached charts
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 17
Additional Slides
All you need to develop high-qualityRecommendations and Test Specifications
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 18
Conformance and Interoperability Testing Methodology Recommendations
X.290 - General ConceptsX.291 - Abstract Test Suite SpecificationX.292 - (Superseded by Z.160/170 series)X.293 - Test RealizationX.294 - Requirements on Test Laboratories and ClientsX.295 - Protocol Profile Test SpecificationX.296 - Implementation Conformance StatementsSupplement 1 to X.290 series - Generic approach to interoperability testingSupplement 2 to X.290 series - Interoperability testing framework and methodology
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 19
Testing and Test Control Notation (TTCN-3)
Z.161 (Z.140 Revised): Core Language Z.162 (Z.141 Revised): Tabular FormatZ.163 (Z.142 Revised): Graphical Format Z.164 (Z.143 Revised): Operational semanticsZ.165 (Z.144 Revised): Runtime interfaceZ.166 (Z.145 Revised): Control interfaceZ.167 (New): Using ASN.1 with TTCN-3Z.168 (New): The IDL to TTCN-3 mapping Z.169 (New): Using XML Schema with TTCN-3Z.170 (New): TTCN-3 documentation tags
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 20
Specification and Description Language
Z.100 Overview of SDL-2008Z.101 Basic SDL-2008Z.102 Comprehensive SDL-2008Z.103 Shorthand notation and annotation in SDL-2008Z.104 Data and action language in SDL-2008Z.105 SDL-2008 combined with ASN.1 modulesZ.106 Common Interchange Format for SDL-2008
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 21
Abstract Syntax Notation One (ASN.1)X.680 (07/02) Information technology - Abstract Syntax Notation One (ASN.1): Specification of basic notation
X.680-X.693 (07/02) Information Technology - Abstract Syntax Notation One (ASN.1) & ASN.1 encoding rules
X.681 (07/02) Information technology - Abstract Syntax Notation One (ASN.1): Information object specification
X.682 (07/02) Information technology - Abstract Syntax Notation One (ASN.1): Constraint specification
X.683 (07/02) Information technology - Abstract Syntax Notation One (ASN.1): Parameterization of ASN.1 specifications
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 22
Abstract Syntax Notation One (ASN.1)
X.690 (07/02)Information technology - ASN.1 encoding rules: Specification of Basic Encoding Rules (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules (DER)
X.691 (07/02) Information technology - ASN.1 encoding rules: Specification of Packed Encoding Rules (PER)
X.692 (03/02) Information technology - ASN.1 encoding rules: Specification of Encoding Control Notation (ECN)
X.693 (12/01) Information technology - ASN.1 encoding rules: XML encoding rules
Geneva, 9(pm)-10 February 2009InternationalTelecommunicationUnion 23
Supporting Recommendations
Z.120 Message Sequence Chart (MSC)Z.150 User Requirements Notation - Language Requirements and FrameworkZ.151 User Requirements Notation - Language Definition
Z.110 Application of Formal Description TechniquesZ.450 Quality aspects of protocol-related Recommendations