INFSOM-RI ETICS: the international software engineering service for the grid Alberto Di Meglio, Peter Couvares, Elisabetta Ronchieri, Eva Takacs, Marc-Elian Begin ETICS Project INFSOM-RI CHEP 2007, Victoria September Contents IntroductionIntroduction OverviewOverview Service ArchitectureService Architecture The Data ModelThe Data Model The Web ApplicationsThe Web Applications The CLIs and APIsThe CLIs and APIs The Metronome Execution EngineThe Metronome Execution Engine ETICS in ActionETICS in Action Conclusions and Future WorkConclusions and Future Work INFSOM-RI CHEP 2007, Victoria September ETICS in a nutshell ETICS stands for e-Infrastructure for Testing, Integration and Configuration of SoftwareETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software Project started on January 1 st, 2006 and has a duration of two years (an extension for two more years has been asked)Project started on January 1 st, 2006 and has a duration of two years (an extension for two more years has been asked) Goal is to provide:Goal is to provide: efficient automated tools for building and testing distributed software an international distributed facility for executing build and test jobs across three sites (CERN, INFN, University of Wisconsin) with extensions from University of Southampton, GARR and IN2P3 INFSOM-RI CHEP 2007, Victoria September The ETICS partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT The Condor batch system, distributed testing tools, service infrastructure, NMI INFSOM-RI CHEP 2007, Victoria September ETICS Features Summary Out-of-the-box automatic build/test systemOut-of-the-box automatic build/test system Multiplatform support for development and testingMultiplatform support for development and testing Powerful and flexible automatic dependency management (internal and external dependencies)Powerful and flexible automatic dependency management (internal and external dependencies) Trigger coding convention checks, unit test, documentation tools and collect and publish the resultsTrigger coding convention checks, unit test, documentation tools and collect and publish the results Possible to build everything from source or use pre-built binary packagesPossible to build everything from source or use pre-built binary packages Designed to support several Version Control SystemsDesigned to support several Version Control Systems Possible to register artefacts on a public repositoryPossible to register artefacts on a public repository Publishes run-time configuration information such as environment variables, runtime dependencies, etc Publishes run-time configuration information such as environment variables, runtime dependencies, etc Produce different package formats (rpms, tar, msi, debs)Produce different package formats (rpms, tar, msi, debs) Produces rich build and test reportsProduces rich build and test reports INFSOM-RI CHEP 2007, Victoria September Service architecture Build/Test Artefacts Web Application Report DB Project DB Metronome Execution Engine Clients Web Service Metronome Client Wrapper Via browser WNs ETICS Infrastructure INFSOM-RI CHEP 2007, Victoria September Data model Project: container for components and subsystemsProject: container for components and subsystems Subsystem: container for componentsSubsystem: container for components Component: a module configuration can also be made of sub-modules configurationsComponent: a module configuration can also be made of sub-modules configurations Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms)Configuration: different versions of the modules (e.g different CVS tags, dependencies, platforms) P S C Config INFSOM-RI CHEP 2007, Victoria September Data model Platform : represents an specific combination of architecture, operating system and C/C++ compilerPlatform : represents an specific combination of architecture, operating system and C/C++ compiler E.g. slc3_ia32_gcc323 Allows users to specify platform specific metadataAllows users to specify platform specific metadata VCS commands (e.g. CVS) Build commands Test commands Properties Environment variables Dependencies Config Platform P S C INFSOM-RI CHEP 2007, Victoria September Build and Test Web Application Build and Test Web Application (WA) allows users toBuild and Test Web Application (WA) allows users to Browse metadata Edit metadata Submit build and test jobs State-full and secure application using x509 certificatesState-full and secure application using x509 certificates Written in Java, jsp, javascript and JDuck (Engineering client/server web framework)Written in Java, jsp, javascript and JDuck (Engineering client/server web framework) Enables the development of web-based applications using a activity- page-tool model Provides a library of GUI widgets based on the MVC pattern Exploits AJAX technology for the synchronisation and communication between client and server models Currently being revised to use the Google Web Toolkit for increased flexibilityCurrently being revised to use the Google Web Toolkit for increased flexibility INFSOM-RI CHEP 2007, Victoria September Build and Test WA Browse Panel Configuration List View/Edit Panel INFSOM-RI CHEP 2007, Victoria September The Remote Build Form INFSOM-RI CHEP 2007, Victoria September Report and Repository Browsers The Report and Repository Browsers allow users to access build products, logs and test results (artefacts)The Report and Repository Browsers allow users to access build products, logs and test results (artefacts) Provides permanent and volatile artefact storageProvides permanent and volatile artefact storage Currently written in Java, jsp and javascript, uses a file system backend based on AFSCurrently written in Java, jsp and javascript, uses a file system backend based on AFS A complete digital library solution to manage the artefacts (packages, logs, documentation) using the Apache Jackrabbit framework is being developedA complete digital library solution to manage the artefacts (packages, logs, documentation) using the Apache Jackrabbit framework is being developed The new Repository application will allow users to manage their projects and upload/download artefactsThe new Repository application will allow users to manage their projects and upload/download artefacts APIs to build customized and federated repositories will be provided together with connectors for popular repositories like SourceForgeAPIs to build customized and federated repositories will be provided together with connectors for popular repositories like SourceForge INFSOM-RI CHEP 2007, Victoria September Reports INFSOM-RI CHEP 2007, Victoria September Registered artefacts INFSOM-RI CHEP 2007, Victoria September Command-line Client Build and Test Command-Line Interface client allows users toBuild and Test Command-Line Interface client allows users to Build and test locally Browse metadata Edit metadata Submit build and test jobs Develop custom management tools using the provided APIs State-full and secure application using x509 certificatesState-full and secure application using x509 certificates CLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisationCLI written in Python, using ZSI (SOAP toolkit for Python) for web service interfacing and serialisation INFSOM-RI CHEP 2007, Victoria September Example: local build Download project metadata: etics-get-project Download project metadata: etics-get-project Action: download project metadata Action: download project metadata Checkout and/or download software:Checkout and/or download software: etics-checkout c Action: download binary packages and dependencies; if available for the local platform, otherwise, fetch code from source packages or version control system Build the software: etics-build Action : build and unit test code hieratically, and generate packages and reportsBuild the software: etics-build Action : build and unit test code hieratically, and generate packages and reports INFSOM-RI CHEP 2007, Victoria September Editing Command-line Client can also be used for editing: etics-module [ ] [ ] etics-configuration [ ] [ Command-line Client can also be used for editing: etics-module [ ] [ ] etics-configuration [ ] [ Can be used for scripting or interactive sessionsCan be used for scripting or interactive sessions Provides the same features as the Web ApplicationProvides the same features as the Web Application INFSOM-RI CHEP 2007, Victoria September Client Plugin-Framework The Plug-in Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment by adding their tools into the ETICS build/test processThe Plug-in Framework allows users with investments in existing build, test, certification and/or software quality assurance tools to leverage this investment by adding their tools into the ETICS build/test process Users can select from the built-in set which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible in the reports (pyUnit, Junit, emma, jdepend, gcov, sloccount, IPv6 compliance, bug tracking connectors, etc)Users can select from the built-in set which metrics are important to them and have ETICS automatically calculate them during build and test, and make them visible in the reports (pyUnit, Junit, emma, jdepend, gcov, sloccount, IPv6 compliance, bug tracking connectors, etc) Promotes inclusion of contributions from the ETICS user communityPromotes inclusion of contributions from the ETICS user community INFSOM-RI CHEP 2007, Victoria September Build and Test Web Service Build and Test Web Service (WS) provides single secure interface to the ETICS backend systemBuild and Test Web Service (WS) provides single secure interface to the ETICS backend system Metadata (persisted in database) Execution engine Security (based on x509 certificates and role-based) Heart of the ETICS System, it performs complex data queries on behalf of the clients (WA and CLI) and manages the job submission to the execution engineHeart of the ETICS System, it performs complex data queries on behalf of the clients (WA and CLI) and manages the job submission to the execution engine Provides a document-literal/wrapped (WS-I compliant) WSDL document for interface definitionProvides a document-literal/wrapped (WS-I compliant) WSDL document for interface definition WS is a pure Java application, using Axis (SOAP) and JDBC (database interconnect)WS is a pure Java application, using Axis (SOAP) and JDBC (database interconnect) INFSOM-RI CHEP 2007, Victoria September Execution Engine ETICS uses Metronome and Condor as its Execution EngineETICS uses Metronome and Condor as its Execution Engine Users can submit build and test jobs without having the burden of managing the hardware resourcesUsers can submit build and test jobs without having the burden of managing the hardware resources In order to provide reproducible builds and tests, only artefacts built and tested using resources managed by the Execution Engine, can be registered in the RepositoryIn order to provide reproducible builds and tests, only artefacts built and tested using resources managed by the Execution Engine, can be registered in the Repository INFSOM-RI CHEP 2007, Victoria September Local and Remote Build and Test For users to trust the system, local (user) and remote (system) builds have to be executed as much as possible in identical environmentsFor users to trust the system, local (user) and remote (system) builds have to be executed as much as possible in identical environments Once a build or test job has landed on a Metronome worker-node (WN), the ETICS client is startedOnce a build or test job has landed on a Metronome worker-node (WN), the ETICS client is started The exact same build or test sequence is executed on the WN, as if done by a user on his/her local machineThe exact same build or test sequence is executed on the WN, as if done by a user on his/her local machine Setup Get project Checkout Build/Test The difference is that the WN is controlled and maintained by the ETICS experts and guaranteed to be a clean standard installation of one of the supported OS distributionsThe difference is that the WN is controlled and maintained by the ETICS experts and guaranteed to be a clean standard installation of one of the supported OS distributions This approach guarantees that over time, builds and tests will be reproducible, a important tenet in software quality standardsThis approach guarantees that over time, builds and tests will be reproducible, a important tenet in software quality standards INFSOM-RI CHEP 2007, Victoria September Metronome Web interface INFSOM-RI CHEP 2007, Victoria September ETICS sites CERNCERN SLC3, SLC4, RHES, Debian, FC (32- and 64-bits), WinXP, Mac OSX Univ. of Wisconsin, Madison, USAUniv. of Wisconsin, Madison, USA HP HPUX, IBM AIX, OSF1, SGI Irix, Sun Solaris, Linux (several flavours) INFN, Bologna, ItalyINFN, Bologna, Italy CentOS, SLC3, SLC4 (32- and 64-bits) INFSOM-RI CHEP 2007, Victoria September ETICS in Action Currently 28 projects registered and 181 usersCurrently 28 projects registered and 181 users Major projects using ETICS include EGEE/gLite, OMII- Europe, DILIGENTMajor projects using ETICS include EGEE/gLite, OMII- Europe, DILIGENT Preliminary prototypes done with Globus, UNICORE, Quattor, CASTORPreliminary prototypes done with Globus, UNICORE, Quattor, CASTOR First production system released in February 2007First production system released in February 2007 New functionality to be released by end of the year includesNew functionality to be released by end of the year includes Automatic deployment and execution of multi-node tests Digital Library Repository Automatic use of virtual machines images INFSOM-RI EGEE/gLite gLite uses ETICS to build packages for SLC4, SL5 and Debian on 32 and 64 bit architecturesgLite uses ETICS to build packages for SLC4, SL5 and Debian on 32 and 64 bit architectures Service metapackages are also built every night according to specific definitionsService metapackages are also built every night according to specific definitions The deployment of metapackages is tested using dynamically created yum and apt servers on all supported platformsThe deployment of metapackages is tested using dynamically created yum and apt servers on all supported platforms Work is now being done to port into the ETICS system existing system tests and automated their deployment and executionWork is now being done to port into the ETICS system existing system tests and automated their deployment and execution CHEP 2007, Victoria September INFSOM-RI OMII-Europe OMII-Europe uses ETICS to build its re-engineered middleware components on all platforms supported by the original providers (mainly SLC4, SL5, Debian)OMII-Europe uses ETICS to build its re-engineered middleware components on all platforms supported by the original providers (mainly SLC4, SL5, Debian) OMII-Europe also performs interoperability testing of services on ETICS to validate the compliance of such components with reference standards (BES/JSDL, SAML, RUS) and interoperability between different implementationsOMII-Europe also performs interoperability testing of services on ETICS to validate the compliance of such components with reference standards (BES/JSDL, SAML, RUS) and interoperability between different implementations OMII-Europe has a dedicated ETICS installation at Southampton and also uses the public ETICS distributed testbedOMII-Europe has a dedicated ETICS installation at Southampton and also uses the public ETICS distributed testbed CHEP 2007, Victoria September INFSOM-RI DILIGENT DILIGENT uses ETICS to build its digital library software (based on gLite) and to perform deployment tests of the servicesDILIGENT uses ETICS to build its digital library software (based on gLite) and to perform deployment tests of the services Full system tests execution is now being implementedFull system tests execution is now being implemented CHEP 2007, Victoria September INFSOM-RI CHEP 2007, Victoria September Conclusions ETICS defined, integrated and delivered a novel Configuration, Build and Test ServiceETICS defined, integrated and delivered a novel Configuration, Build and Test Service The service leverage the rich knowledge foundation of the consortium partners in the domain of software quality assurance, development, integration and test of complex Grid and distributed softwareThe service leverage the rich knowledge foundation of the consortium partners in the domain of software quality assurance, development, integration and test of complex Grid and distributed software The ETICS service has been used to build and test the ETICS service itself, providing rapid feedback to the teamThe ETICS service has been used to build and test the ETICS service itself, providing rapid feedback to the team A first production release is available and work is going on to add additional features following users feedbackA first production release is available and work is going on to add additional features following users feedback INFSOM-RI CHEP 2007, Victoria September Thanks