U.S. ATLAS S&C Planning Meeting - June 2015 1

ATLAS Software Infrastructure :Requirements and Goals at Run 2

Period

Alex Undrus

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 2

Outline

Current status Plans for the next year Long term perspectives

This presentation reports for the following WBS items:

5.2.1 – Software Validation 5.2.2 – Librarian and Infrastructure

Services

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 3

Current Status

Some statistical data Tendencies U.S. contribution to ATLAS

infrastructure

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 4

Number of files in ATLAS offline release

C++

C/C++ h

eade

r

Pyth

onJa

vaXM

L

Fortr

an

Shel

l scr

ipt

0

5000

10000

15000

20000

25000

17678 20216 10531 835 1105 491 861

Calculated by cloc-1.62 for ATLAS nightly development release created on June 15, 2015 (cmt, InstalllArea, NICOS, platform

specific. genConf, dict areas excluded). External packages are not included.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 5

Number of lines in ATLAS offline release

0.00E+005.00E+051.00E+061.50E+062.00E+062.50E+063.00E+063.50E+06

3.06E+069.25E+051.18E+06

1.48E+051.53E+06

8.70E+04

5.30E+04

Calculated by cloc-1.62 for ATLAS nightly development release created on June 15, 2015 (cmt, InstalllArea, NICOS, platform

specific. genConf, dict areas excluded, comments and blank lines excluded). External packages are not included.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 6

Number of submitters and commits to offline SVN repository

05/16-06/15-2014

11/16-12/15-2014

05/16-06/15-2015

0 2000 4000 6000 8000 10000

4499

7227

7676

277

352

412

Number of submitters Number of commits

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 7

New and persistent submitters to offline SVN repository (period of 05/16 – 06/15,

2015)

259

153 New authors (not active 1 year ago)Authors active 1 year ago

412 - number of unique submitters during 1 month (05/16/15 – 06/15/15)

809 - number of unique submitters during 1 year (06/16/14 – 06/15/15)

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 8

ATLAS Nightly System at a glance

6

16

915

9

2 1 1

Nightly Branches in

December 2014

59 in total

8

15

8164

36

1123

Nightly Branches in June 2015 Major Branches

Migration

ASG

Patch Branches

Physics Anal-ysis

Athena Analysis

Exp. (e.g. CMAKE)

ROOT 6 integr. (NEW)

Athena Simula-tion (NEW)

MAC (RE-SUMED)Doxygen67 in total

NEW

NEW

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 9

ATLAS Nightly System at a glance (2)

2013 Mar

2013 Jun

2013 Oct

2014 Feb

2014 Jun

2014 Sep

2014 Dec

2015 Feb

2015 Jun

0 20 40 60 80 100

SLC5SLC6

Number of ATLAS nightly jobs. Record high number of 100 daily jobs is registered on 08/01/2014. As of 06/22/2015 the Nightly System run

83 daily jobs.

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 10

Statistics Interpretation

Athena development releases contain 7 million lines

The number of lines is not growing last 5 years, but they keep changing

Software development activity was always high and it is increasing since the Run 2 start 70% increase in SVN commits 50% increase of developers community Hundreds of new developers joined Many new nightly branches are opened (total number

reached the record high 67) Unprecedented rate of new stable releases: 1.4 per day

(~ 250 stable release Jan. 1, 2015 – June 24, 2015) New kinds of release for targeted use

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 11

Intensity of Run 2 Software Development

. . . is high It is considerable higher than at LS1 period:

Some important LS1 software projects continue: CMAKE, ROOT 6 integration

New kind of releases for targeted use Athena Simulation Athena Analysis AtlasP1Mon (for Tier 1 online monitoring)

RootCore based releases are actively developed

New compilers are probed (gcc 4.9, clang) Release configuration management is under

major change (cmake)

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 12

Infrastructure Challenges and U.S. Contribution

All areas (code configuration, documentation, externals, platforms&compilers, QA/QC, release building and distribution, code

repositories, information protection, etc.) must cope with increased workloads when the work force remains the same

Infrastructure needs both support and development (tools must be updated in sync with software they support)

ATLAS Software Infrastructure Team includes 19 persons contributing 7.5 FTE (with 50 % shortage: additional 3.7 FTE are needed for an adequate support) as of 04/02/2015

U.S. ATLAS contribution to SIT is currently 1 FTE as it appears in OTP (Alex Undrus and Shuwei Ye)

U.S. contributes to critical areas: the Nightly System (Alex Undrus), environment setup (Shuwei Ye), LXR service

Expertise Gain Influence on ATLAS-wide policies and

decisions Parallel and effective user support for

U.S. based physicists Librarian and User services at U.S. Analysis

Center, BNL PROOF farm and Tier I Center Capture innovations and new ideas

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 13

Rationale of U.S. Participation in Infrastructure Projects

14

Plans for Next Year

Permanent goal: create supportive environment for code development, data processing and analysis jobs across all ATLAS sites and file systems (local, afs, cvmfs)

Key areas: Nightly builds (details in the next slides) Build and run-time (details in Shuwei's

presentation) Criteria of success: users satisfaction and

absence of complaints Explore new innovative technologies and

tools Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 15

Plans for the Nightly System

Further improve ATLAS Nightly web and database services brought by successfully completed Nightly System LS1 upgrade

Add new on-demand functionality to the System Current system is for daily builds at fixed times Software coordinators increasingly request urgent nightly

builds Up to 100 nightly jobs are manually restarted monthly Some branches need irregular builds separated by few days New on-demand functionality will be demonstrated on July

9 at the Annual Nightlies Workshop (https://indico.cern.ch/event/387000)

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

NEW ADMIN PANEL

Administrative functions for privileged

users, authentication via

CERN SSO

Main task button (restart, cancel, etc.)

Current and detailed progress

information

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 17

Plans for the Nightly System (2)

Complete projects according to the schedule New Nightlies CVMFS server Hot spare for the Nightlies CVMFS server (greatly

improves the system reliability) ATLAS Nightly Mail Facility (personalized emails about

nightlies results) Adapt the System for cmake builds

Experimental cmake nightlies support compilations, no tests yet

Assess recent requests Expand doxygen documentation builds git repository support

Continue to encourage users to use ATLAS Nightlies DB for customized views (successful experience with Trigger developers) example on the next slide

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

New panel of Yasu Okumura with summary

of Trigger related problems across major

nightly branches

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 19

Plans for the Nightly System (technical work)

Update NICOS for new Tag Collector 3 Builds on CC7 (CERN CentOS 7) Optimization and testing new machines on

the nightly farm CERN IT plans to replace all real hardware

machines with VMs VMs performs differently, need a lot of testing

and optimization Key problem: I/O bottleneck prevents full CPU

usage New VM machines with SSD disks allow to

reach 60% CPU utilization (a success after some VM types could not be loaded above 20%)

20

Long Term Perspectives

Rising concerns: Unclear relationship between releases Confusion where essential parts of software

are located (simulation, digitization, reconstruction, derivations)

Dissatisfaction with standard release coordination bureaucracy, ways for development of new algorithms and techniques

Single platform support, essentially no software portability

All-inclusive offline releases ( installation size 12 GB, with few tens of externals )

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

21

Long Term Perspectives (2)

Increasing requests for smaller releases with targeted purpose (Simulation, Athena Analysis)

Ideas about software restructuring

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

Core

Event

Reco

Analysis

Core

Simulation

HLT

Analysis2

Analysis 1

Reco

???

22

Long Term Targets

Improved software structure Compact releases with targeted

purpose Expanded multiplatform support Decrease of centralized tag

validation bureaucracy and effort Discussion on July 2 at 13:30 -

15:00 – "Release build: technical session"

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015 23

Perspectives for the Nightly System

Expansion to new platforms (e.g. PowerPC) 90% of the Nightly System is already

portable Moving some operations to the GRID

Making and testing releases on sites where jobs run would bring better results and save human and machine efforts

Nightly testing is already available on the GRID but attracted few tests so far

Desirable: developing communality with CMS (CMSSW), LCG (Jenkins) nightly builds

Keep users and management happy with the System is always a priority

24

Summary

U.S. contributes to key areas of ATLAS software infrastructure Infrastructure

Despite thin manpower the ATLAS Nightly System and Environment Setup procedures sustain increasing load and demand successfully

Keeping abreast of new technologies trends is at the top of U.S. contributors priorities The Nightly System LS1 upgrade brought new

database and web technologies and greatly improved user experience

The Nightly System is proactively prepared for meeting demands of new platforms (PowerPC), builds tools (cmake) and software restructuring

Alex Undrus – U.S. ATLAS S&C Planning Meeting – June 2015