opnfv/functest: lessons learned and best practices for vnf test automation

June 20–23, 2016 | Berlin, Germany

Functest: lessons learned and best practices for VNF test automation

V.Boucher, Orange, M.Richomme Orange

Agenda

• Back to vIMS test case in Brahmaputra

• Lessons learned

• The future of VNF testing in Functest

Back to vIMSTest-case

Back to vIMS test-case

• vIMS Functest test-case in Brahmaputra => Get a first complex VNF test case

• vIMS test-case includes orchestration/VNF/Test

• vIMS test-case fully open source including both VNF and orchestrator

• vIMS fully integrated in CI pipeline

Back to vIMS test-case

vIMS Clearwater

architecture

Back to vIMS test-case

VNF Descriptor (Cloudify Blueprint) :

homestead:type: clearwater.nodes.homesteadproperties:private_domain: clearwater.local

relationships:- type: cloudify.relationships.contained_in

target: homestead_host- type: app_connected_to_bind

target: bind

homestead_host:type: clearwater.nodes.MonitoredServerinstances:deploy: 1

relationships:- target: base_security_group

type: cloudify.openstack.server_connected_to_security_group

- target: homestead_security_grouptype: cloudify.openstack.server_connected_to_security_group

See more : https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/

Back to vIMS test-case

vIMS Testcase step by step

Create user/tenantUpload OS ImagesChoose flavorsGet external network

1 Environment preparation

Back to vIMS test-case

vIMS Testcase step by step

Install orchestrator components

2.2 Orchestrator deployment

Create router/networkCreate security groupsCreate VM

2.1 Orchestrator deployment

Back to vIMS test-case

Create security groupsCreate VMCreate floating ip

3.1 VNF deployment

Install and configure VNF software

3.2 VNF deployment

Back to vIMS test-case

Create numbers

4.1 Test VNFSignaling testing

4.2 Test VNF

Back to vIMS test-case

Push results of deployment duration and test results into result api

5 Push results

DB

RESULT API

Back to vIMS test-case

Reporting status example

Lessons learned&

Best practices

Lessons learned

• Valuable test case to give a high level of confidence:• Creation of user/tenant/networks

• Access to Internet, public APIs

• Use of an orchestrator (Cloudify) to deploy a complex VNF based on Tosca descriptors

• Spawning of more than 5 VMs cross connected

• Association orchestrator/VNF makes sense for functional testing

• Realistic test suite for signaling

Brahmaputra lessons learned

• Success criteria• Orchestration + vIMS deployment

• Tests not part of the criteria due to some issues

• Not possible to run it on all the installers/scenarios• vIMS launched systematically

• Timeout adaptations needed to reduce test duration

• When previous tests failed (e.g. vPing) no need to try vIMS => improvement in Colorado

• Issues detected and reported (e.g. access to public API)

• Multi-tenant/Multi-sites would be also valuable

Future of VNF testing

Future of VNF testing

• Functest test slicing => 1 category dedicated to VNFs

• VNF part of the weekly loop i.e. not used anymore for scenario validation

• Not scenario specific – shall work on all the scenarios….

• Potentially long duration test

Future of VNF testing

• VNFs validation = 1 of the plugfest activity

• Reference/consider Open Source VNFs to enrich VNF category

• Goal develop test suite/ref open source VNFs to • Define ref baselines / harmonize test strategy

• Create a “VNF library”

• Initiate MANO stack testing

Future of VNF testing

Proprietary VNF

• Cannot be integrated to CI (license)

• Can be tested in Pharos Community labs

• Can be tested during Plugfests

• Open source test suite even towards proprietary VNFs should be encouraged

Thank you