spi software process & infrastructure egee france - 11 june 2004 yannick patois...
TRANSCRIPT
SPI Software Process & Infrastructure
http://spi.cern.ch
EGEE France - 11 June 2004
Yannick Patois
Y.Patois SPI - Software Process & Infrastructure 2
SPI Services Overview
Provide General Services needed by each project• CVS repository, Web Site, Software Library• Mailing Lists, Bug Reports, Task Management, Collaborative Facilities
Provide solutions specific to the Software Development phases• Tools, Templates, Policies, Support, Documentation, Examples
Coding Analysis and DesignDevelopment
Release
SpecificationsTesting
Build systemsDeployment and
InstallationDocumentation
Quality Assurance
Software Development
General Services
CVS service
Collaborative Facilities Task Management Mailing Lists
Web Portal External Software
Y.Patois SPI - Software Process & Infrastructure 3
SPI Project Guidelines
• Have different and separated services- Simple solutions, easy to learn, commonly needed services- Leave any process for later
• Work with the users• Develop as little as possible• Establish simple deliverables• Everything is done starting from existing infrastructure
- LCG and LCG projects (Pool, Seal, etc)- LHC experiments- IT division - Big projects (G4, Root, etc)
• We did not start from tools for requirements, design, etc. • We started from development-related work
- repository, releases, testing, bug report, etc
The rest of the talk describes SPI services
Y.Patois SPI - Software Process & Infrastructure 4
SPI Services (April 2004)
• External Software
• Savannah Project Portal
• Testing Frameworks
• Development of LCG policies, templates
• QA checklists and reports
• Software Distribution
• LCG Software Configuration
• CVS server and AFS management for LCG App. Area
• Code Documentation (doxygen, lxr, viewcvs)
• Automatic Nightly Build (Nicos)
• Software Librarian, builds and releases (new, was Scram support)
• Documentation and LCG Workbook
Y.Patois SPI - Software Process & Infrastructure 5
SPI Web Site - http://spi.cern.ch
Y.Patois SPI - Software Process & Infrastructure 6
SPI External Software Service
• We install software needed by projects.
• Open Source and Public Domain software (libraries and tools) like:• Compilers (icc, ecc)• HEP made packages • Scientific libraries (GSL)• General tools (python)• Test tools (cppunit, qmtest)• Database software (mysql,
mysql++)• Documentation generators
(lxr, doxygen)• XML parsers (XercesC)
• There are currently 50 different packages, plus others under evaluation. For more than 300 installations
The platforms, are those needed by the users • Linux RedHat 7.3 and compilers
- gcc 3.2 and 3.2.3 (rh73_gcc32)
- icc 7.1 (rh73_icc71) - ecc 7.1 (rh73_ecc71)
• Windows - Visual Studio .NET 7.1:
(win32_vc7).• Mac OSX (osx103_gcc33)
• Platforms always been reviewed• We also provide configuration
and installation area• A unique AFS location• Standard structure
package_name/version/platform/package_ content
Y.Patois SPI - Software Process & Infrastructure 7
External Software – http://spi.cern.ch/extsoft
Y.Patois SPI - Software Process & Infrastructure 8
SPI Savannah Portal Service
• Functionality:• Bug tracking• Task management• Mailing lists, news, faqs• Access to CVS repository• Download area, etc
• The Web portal for software projects
• Customized from GNU (SourceForge as origin)
• Totally web based• Single entry point to all projects• Uniform access to project
information• Set up common web
infrastructure for a project without coding
• What SPI changed• installation from GNU, general
bug fixing and improvements• integration with AFS
authentication• Integration with standard services
already available
• What SPI does• administration (project approval) • maintenance (submitted bugs)• development (support requests)
• Status• >90 hosted projects• >650 registered users
Y.Patois SPI - Software Process & Infrastructure 9
Savannah Service – http://savannah.cern.ch
Y.Patois SPI - Software Process & Infrastructure 10
IntegrationTests
System Tests
Examples
Software testing
Integration test
System test
Acceptance test
Unit test
Software developer
Sw-testing team
LHC experiments
Use in Exp.
Work PackageTest
Automated testing
SPI Testing Services
• Software testing should be an integral part of the software development
• The goal was to provide something that can be run automatically as often as needed (releases, development, etc)
SPI provides• Test frameworks
• CppUnit, Oval
• Qmtest
• Test support• Test policies• Test doc
• Different platforms/compilers
Y.Patois SPI - Software Process & Infrastructure 11
Testing Support – http://spi.cern.ch/testing
Y.Patois SPI - Software Process & Infrastructure 12
Quality Assurance Service - http://spi.cern.ch/qa
• The main goal of QA activity help sw projects
• assess and improve the quality of the software • provide tools to collect useful metrics/statistics
which help toasses quality;
• generate reports;
• verify if project setup is correct with policies.
• QA Tools and Focus • Automatic reports
• Development/integration of automatic tools
Y.Patois SPI - Software Process & Infrastructure 13
Quality Assurance Activities
QA Checklist on each Release• Build the release • Run automatic tests• Statistics
• Test Inventory • Documentation/Examples Inventory • Savannah Statistics • Code Inventory • Rule Checker , Logiscope
• LCG Policies • Configuration of a build system • CVS directory structure
• Well-defined, transparent, open• clear rules and checklist of assessed
items• anybody at anytime may see statistics • create reports themselves• anybody may contribute
QA Reports
• Automatic reports• Generated at every release
• Published on the SPI web site
• Soon be generated by anybody from the web
• Evaluation and usage of external tools
• Rule Checker
• Logiscope, Test coverage
• SLOC, Valgrind, ignominy
Y.Patois SPI - Software Process & Infrastructure 14
QA Reports
Y.Patois SPI - Software Process & Infrastructure 15
SPI Software Distribution Service
• Simple solution to use• local installations (external sites,
laptops,...)• using simplest approach• python downloader + tar format• replicate the central AFS tree (in
a optimized way)• package dependency from
SCRAM
• ...until a complete, long-term solution available
• Looking into pacman as a suitable solution
• SPI will adopt what Grid Deployment decides to provide
• Simple tool to install• successful for users:
- POOL @ Karlsruhe- BNL nightly builds, CMS- developers at home, etc
• very easy to use and reliable
• Different use-cases should have different solutions• Our tool is adequate as a
temporary solution for LCG Application Area Distribution
• but long-term solutions must be investigated:
- pacman, LCFGng ....
Y.Patois SPI - Software Process & Infrastructure 16
Software Distribution – http://spi.cern.ch/lcgsoft
Y.Patois SPI - Software Process & Infrastructure 17
Summary
• The set of services is working and fully available• Savannah Project Portal• Software Testing• External Software Service• Quality Assurance and Policies• Software Distribution• …and many more
• We have followed a simple strategy• Work with the users• Ask their help • Develop as little as possible in order to have little maintenance• Provide simple and modular solutions
• We have commitment to provide a sustainable service • Most people moved on to new projects, as it was planned• The services are used by LCG projects, and many outside LCG (LHC
experiments, EGEE JRA1, IT department, etc.)
• The services are generic and to some extend customizable by SPI according to EGEE needs