at&t e n t e r t h e t i t l e o f y o u r o p e n s t a c ... · process backloads delivery...

E N T E R T H E T I T L E O F Y O U R O P E N S T A C K

P R E S E N T A T I O N

Q A A U T O M A T I O N I N A K U B E R N E T E S W O R L D

Elise Eiden –AT&TDoug Schveninger – AT&T

OpenStack Summit 2018 – 5/21/2018

2

A G E N D A

© 2018 AT&T Intellectual Property. All Rights Reserved. AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners.

Security QualityContainerization

Testing &Containerization

AT&T Network Cloud OSH

FutureTesting

3

A T & T N E T W O R K C L O U D



QUALITY THROUGHOUT

Kubernetes, Helm, OpenStack

Fast and Predictable

Deployments

Scaling and Reusability

Resiliency

Open source from the beginning

Designed for operational excellence

Evolves LCP security

Evolves existing AT&T test frameworks

AT&T’S MOVE TO A CONTAINERIZED PLATFORM

4

O P E N S T A C K H E L M ( O S H )



How principles work in OSH vs without Helm test

Ensure it’s implemented

If you use Armada, does it for you

OSH can also be used to test other software

Upstream keystone gates

AT&T internal CI/CD runs in and manages an

OSH cloud

OSH allows devs to run a real-life OpenStack on

their laptop

“Prod-like” environment(vs Devstack)

Resiliency testing

Test rolling upgrades

5

T E S T I N G T R A N S F O R M A T I O N



• Unit tests, system test, and end to end / acceptance testing once deployment is complete

• Testing later in the development process backloads delivery cycles with defects requiring further testing

• No Standard test runner to share tests

• Preliminary efforts to shift testing to the left included the development of the AQuA testing framework for earlier and more diverse test integration

PREVIOUS TESTING OVERVIEW TESTING TRANSFORMATION

7

A Q U A F R A M E W O R K



AIC Quality Automation (AQuA) represents an extensible test framework, that spans various dimensions of the delivery lifecycle required to have confidence in delivering a stable platform.

AQuA seeks to establish consistent and continuous testing across our organization.

AQuA principles include:

Utilize sanctioned open source test frameworks (Tempest, Shaker, Tox)

Extended and contribute to those frameworks to meet program testing needs

Integrated as quality gates in CI/CD pipeline

CSO compliant testing and certification mechanisms (i.e. least privilege)

Ensure that each execution cleans up its resources

Functional and integration tests call and assert meaningful results from APIs

Provide Development toolset/documentation for the development of tests

Principles behind AQuA AQuA Framework Components

Unit testing

Code Quality Analysis

API Testing

Resource Testing

Dashboard

Command Line Interface (CLI) Testing

Graphic User Interface (GUI) Testing

Infrastructure Testing

Security

Performance

Resiliency

In Use

Beta

POC

Ideation

8

O P E N S T A C K H E L M & A Q U A A P I



Work with the standardOpenStack Helm tempest

Chart

Custom Image override with needed Tempest plugins

& support framework

values.yaml override for test runner

Repository Model Custom Functionality

Mirrored Upstream Repos of Tempest &

Upstream Plugins

Internal Test Plugin Repos

Automated testing, package, delivery and execution

framework repo

Persistent requirements

Custom CLI to enhance the Tempest CLI

Image Build Process that build the image of the needed repo versions

10 © 2018 AT&T Intellectual Property. All Rights Reserved. AT&T, the Globe logo, Mobilizing Your World and DirecTV are registered trademarks and service marks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks are the property of their respective owners.


AQuA-Resource testing is an extensible, multi-phase test framework that allows tenants to orchestrate resources and create tests that exercise specific use cases, enabling early defect detection in development lifecycle.

Compute-1 Compute-2

Master_1 VM

Test Traffic (Ping, SCTP, TCP, UDP, etc.)

Shaker agent

Iperf3

Master_0 VM

Shaker agent

Iperf3

Servant_1 VM

Shaker agent

Iperf3

Servant_0 VM

Shaker agent

Iperf3

D2 IPE

Conexus

Shaker Controller

High Level AQuA-Resource Framework Shaker Architecture

Contrail

These steps provide a report that gives metrics for testing of resources like VMs, networks and volumes within an LCP’s data plane and defined execution boundaries

The test is orchestrated via Heat

Test executed via open source Shaker framework against the data plane resources created in step 1

Steps

V N F C O L L A B O R A T I V E T E S T I N G :

A Q U A - R E S O U R C E F R A M E W O R K

12

F U T U R E T E S T I N G



We want to make sure OSH testing has everything that operators need to feel confident in their deployment!

WE WANT TO HEAR FROM YOU

NEW FEATURES INCLUDE

TESTCUSTOMIZATION

TEST EXPANSION

Allow users to customize when and how tests are

triggered

Promote Custom Tools

Expand existing plugins test

Expand charts beyond OpenStack test suites like:

ResourceHorizon GUI

CLI

Providing additional templates and documentation

ADDITIONALDOCUMENTATION

13

C A L L I N G L A R G E O P E R A T O R S



LCOO Working Group Session

May 22nd, 2018 9:00 am – 10:30 am Room 215-216

at&t e n t e r t h e t i t l e o f y o u r o p e n s t a c ... · process backloads delivery...

Documents