quality assurance with etics – multi-node automated testing
DESCRIPTION
Quality Assurance with ETICS – multi-node automated testing. M.Zurek, A. A. Rodriguez, A. Aimar, A. di Meglio, L. Dini. CGW 09. CERN. Krakow, 12 October 2009. Contents. ETICS Overview Multi-Node Testing Q & A. ETICS Overview. The Problem. - PowerPoint PPT PresentationTRANSCRIPT
INFSO-RI-223782
Quality Assurance with ETICS – multi-node automated testing
CGW 09 M.Zurek, A. A. Rodriguez, A. Aimar, A. di Meglio, L. DiniCERN
Krakow, 12 October 2009
INFSO-RI-223782
2
Contents
• ETICS Overview
• Multi-Node Testing• Q & A
ETICS Testing Tools CGW09
INFSO-RI-223782
3
ETICS Overview
ETICS Testing Tools CGW09
INFSO-RI-223782
The Problem
4ETICS Online tutorials - Module 01
Distributed Computing, Distributed Development
INFSO-RI-223782
The ETICS System
5
ETICS Testing Tools CGW09
• Continuous software build, test and QA verification system• Easily extensible with additional plugins• Scheduled or on-demand build and test jobs• User computers or remote distributed infrastructures• Built-in connectors to distribute build and test jobs on different types of infrastructures from standard job management systems to the grid • Open repository of configuration metadata, packages and build, test and QA reports• Support for standard package management systems like YUM• It’s multi-platform and independent from any specific language, build or test tool
INFSO-RI-223782
Architecture
6
1
Build/TestArtefacts
ConfigurationWeb Service
Report/MetricsDB
ConfigurationDB
Execution Engines(Metronome, gLite, UNICORE, etc)
Command Line User Interface
RepositoryWeb Service
ETICS Infrastructure
Physical WorkerNodes
Web Portal
Virtual OS Images
CGW09ETICS Testing Tools
INFSO-RI-223782
The Configuration Web Client
7
ETICS Online tutorials - Module 01
INFSO-RI-223782
Command-line Client
8
ETICS Online tutorials - Module 01
INFSO-RI-223782
The Repository Web Application
9
ETICS Online tutorials - Module 01
INFSO-RI-223782
Examples of Plugins
10
ETICS Online tutorials - Module 01
Metrics Type Programming languages/
technologies
Tool ETICS Plugin
Complexity static Java
Python
Javancss JCcnPlugin
PyComplexityPlugin.py
Design quality static Java Jdepend JDependPlugin
N of potential bugs static C/C++
Python
Perl
PHP
Java
Flawfinder,
RATS
PMD
Findbugs
CFlawfinderPlugin
CPyPhpRatsPlugin
JPmdPlugin
JFindbugsPlugin
N of potential bugs dynamic C/C++ Valgrind CValgrindPlugin
Lines of code static All SLOCCount SLOCCountPlugin
Coverage dynamic Java Emma
Cobertura
JUnitemmaPlugin
JCoberturaPlugin
Unit tests success rate
dynamic Java
Python
JUnit
PyUnit
JUnitPlugin
JUnitreportsPlugin.py PyUnitPlugin.py
Compliance with standards
static IPv6
WSI
IPv6Plugin WSInteroperabilityPlugin
Profiling dynamic C/C++
Java
Jrat
Valgrind
JRatPlugin
CValgrindPlugin
INFSO-RI-223782
11
Multi Node
ETICS Testing Tools CGW09
INFSO-RI-223782
12
Multi-Node
ETICS Testing Tools
The ability to setup scenarios where multiple services are automatically deployed in multiple nodes.
These services must be able to work properly as if they would be installed manually by a user performing any required operation in the required order.
CGW09
INFSO-RI-223782
13
ChallangeService Synchronization
NODE1
NODE2
WEBSERVERSERVICE
TESTSUITESSERVICE
CLIENTSERVICE
1
32
NODE1
NODE2
WEBSERVERSERVICE
CLIENT1SERVICE
1
2
NODE4
TESTSUITESSERVICE
3
NODE3
CLIENT2SERVICE
2
ETICS Testing Tools CGW09
INFSO-RI-223782
14
ChallangeInformation Interchange
NODE1
TOMCAT1SERVICE
WEB APPLICATIONSERVICE
CONFIGURATIONPHASE
CONFIGURATIONPHASE
NODE2
MYSQL1SERVICE
INITPHASE
NODE2.MYSQL1.INIT.IP
MYSQL.INIT.IP
NODE1.TOMCAT1.CONFIG.CATALINA_HOME
TOMCAT.CONFIG.CATALINA_HOME
LOGGER = CONSOLE
RAM = 256MB THREAD = 300 CACHE = TRUE
REQUIRED (GLOBAL / NODE)
OPTIONAL(WITH DEFAULTS)
-
++
---
--
- + IN / OUT
= DESIGN-TIME
ETICS Testing Tools CGW09
INFSO-RI-223782
15
ChallangeFlow Monitoring
NODE1
TOMCAT1SERVICE
WEB APPLICATIONSERVICE
CONFIGPHASE
CONFIGURATIONPHASE
NODE1.TOMCAT1.CONFIG.CATALINA_HOME
TOMCAT.CONFIG.CATALINA_HOME
RAM = 256MB THREAD = 300
LOG
NODE1: START
NODE1.TOMCAT1: START
NODE1.TOMCAT1.CONFIG: RAM = 256MB && THREAD = 300
NODE1.TOMCAT1.CONFIG: START
NODE1.TOMCAT1.CONFIG: ENDCATALINA_HOME = /usr/share/tomcat/
NODE1.TOMCAT: END
NODE1.WA: START
NODE1.WA.CONFIGURATION: STARTCATALINA_HOME = /usr/share/tomcat/
NODE1.WA: FAILED: ERROR MESSAGE
NODE1: END
ERROR
/usr/share/tomcat/
ETICS Testing Tools CGW09
INFSO-RI-223782
16
The ETICS Solution
• ETICS-SET is used to publish key-value pairs or only keys to the central information system in order to make the information available to all the other nodes, services or phases.• ETICS-GET is used to get (blocking or not-blocking) the information published by some local or remote execution of ETICS-SET.• ETICS-LOG is used to log and for real-time monitoring.
ETICS Testing Tools
Server Node Client NodeCentral IS[ … INSTALL …]
[ … CONFIG … ]
[ … START … ]
ETICS-SET S.IP = x.x.x.x
ETICS-LOG “RUNNING”
ETICS-GET C.DONE
[… STOP AND EXIT …]
$IP = ETICS-GET S.IP
[ … USE $IP … ]
[… RUN TEST …]
ETICS-SET C.DONE
[… EXIT …]
S.IP = x.x.x.x
S: “RUNNING”
C.DONE
CGW09
INFSO-RI-223782
The PING Example (1/3)
ETICS Testing Tools CGW0917
INFSO-RI-223782
The PING Example (2/3)
ETICS Testing Tools CGW0918
INFSO-RI-223782
The PING Example (3/3)
ETICS Testing Tools CGW0919
INFSO-RI-223782
Challenges
• Deployment automation in the context of complex software development is a prerequisite for automatic system testing
• Complexity of service configuration• Developers should provide deployable configurations, or
parameterised configurations
• External dependencies: all dependencies must be deployable automatically• Knowledge of these “external” components is not
necessarily with the end-user
• Privileged deployment/configuration operations• Root / Super User actions• Mix of normal user-level and root level actions
20
ETICS Testing Tools CGW09
INFSO-RI-223782
Benefits
Automation of complex processes
Automatic setup of complex environment
Automatic generation of reports
Save on resources and time• More scenarios can be explored• Less human errors
Form the foundation for more advanced system tests
21
ETICS Testing Tools CGW09
INFSO-RI-223782
Thanks!
22
http://www.eticsproject.eu
ETICS Testing Tools CGW09