acctesting framework - motivation, overview and first experience
DESCRIPTION
Acctesting Framework - Motivation, Overview and First Experience . Kajetan Fuchsberger TE-MPE-TM, 2012-05-10. Thanks to: M . Galetzka, V.Baggiolini , R.Gorbonosov , M. Pojer, M. Solfaroli Camillocci, M. Zerlauth. Content. Content. LHC Hardware Comissioning. - PowerPoint PPT PresentationTRANSCRIPT
1
Acctesting Framework - Motivation, Overview and First Experience
Kajetan FuchsbergerTE-MPE-TM, 2012-05-10
Thanks to:M. Galetzka, V.Baggiolini, R.Gorbonosov, M. Pojer,
M. Solfaroli Camillocci, M. Zerlauth
K. Fuchsberger 2TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Content
1. Introduction/Motivation
2. System Overview
3. First Experience
K. Fuchsberger 3TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Content
1. Introduction/Motivation
2. System Overview
3. First Experience
K. Fuchsberger 4TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
LHC Hardware Comissioning
• About 7000 Tests on Magnet Circuits• Must be executed every year after Christmas
Stop (Potentially more after LS1)• Workflow:
1. ‘Execution’: Test Sequence is Executed on the Hardware Commissioning Sequencer.
2. ‘Analysis’: Test-Data (Measured Signals) is analysed (manually or partly automatic).
3. ‘Signing’: Experts have to ‘sign’ the test result (If manually analysed).
K. Fuchsberger 5TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Motivation I
Initial Project Description: “Create replacement for the p2n Web-Page
aka Alvaro’s pages”
• PHP, grown over time Hard to maintain
K. Fuchsberger 6TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Motivation II
• Some tests should be allowed to be executed even if previous analysis was not completed.
• Avoid starting of test if system is not ready.(e.g. locked)
• Avoid starting of tests on Circuits on same QPS controller.
• Force starting of tests on QF, QD at the same time
• ….. and some more …
New Requirements:
K. Fuchsberger 7TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Motivation III
The HWC Legacy System:
• Many interdependent Systems:• Sequencer• DB (Central Point)• web page• Daemon• Fesa Class• LabView• …
Decision: Design a new system to orchestrate the whole process (and can replace several parts).
K. Fuchsberger 8TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Content
1. Introduction/Motivation
2. System Overview
3. First Experience
K. Fuchsberger 9TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Overview
e.g.: LabView, Powering Server …
e.g.: HWC Sequencer,
LHC Sequencer orCustom Components
e.g.: Magnet Circuits, BIC/PIC …
K. Fuchsberger 10TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Acctesting Server
• Orchestrates the whole process:• Test Execution• Test Analysis
• Exclusively reads/persists data in the database.
• Notifies all the GUIs about changes.• Robust Design:
• Continuously persists relevant data to be able to recover in case of a crash.
• Gracefully handles unexpected behaviour of Execution- and Analysis Components.
K. Fuchsberger 11TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Workflow I
1. User “expresses his wish” to execute one (or many) tests.
K. Fuchsberger 12TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Workflow II
2. The Tests go to the Execution Basket (Server!)
3. The Scheduler (on the Server) will decide when to start which test(s).
K. Fuchsberger 13TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Why so complicated?
• Central Scheduling can respect all the conditions, even if requests come from different GUIs.
• When conditions are fulfilled later, the tests are started automatically (No delays).
• No Need for reservation of circuits anymore. More dynamic
K. Fuchsberger 14TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Test Phases
Tests within a test phase can be executed in arbitrary order.
TestPhase contains one or more Tests
K. Fuchsberger 15TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Preconditions
• Before: Only email exchange, that e.g. “Cryo is ready for a certain group of circuits”.• Hard to track (History)• No real constraint on test execution ( Many
wrongly started tests in previous years)• Now: ‘SignOnlyTest’ for each precondition.
Automated Tests
K. Fuchsberger 16TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Designed for Extension
• Server:• TestStepHandler (handle certain types of
SystemTests)• Constraints (Restrict Test Execution)• LockProvider (PIC, Db, …)• SystemInformationProvider (e.g. Issues)
• GUI:• TestResultsViewer (E.g. Powering Server)
Extension Points:
K. Fuchsberger 17TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Test Step Handlers
• Responsible for executing a specific TestStep (execution, analysis) for a certain type of tests.
• Is itself responsible for communicating with other systems, if required.
• Examples:• HwcTestExecutionHandler: Communicates with HWC
Sequencer to execute the tests• DaemoneAnalysisHandler: Communicates with
LabView system to retrieve analysis results• … Future: PicBicTestExecutionHandler?
K. Fuchsberger 18TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Constraints
• Simple Extension Point, which has to decide if one Test is allowed to be run together with another one. (Simple Yes/No decision)
• Used to formulate requirements like:“Only start one test on one of the four circuits on the same QPS controller.”
• Checked by the Scheduler, to decide if a certain test-configuration is allowed or not. See Michael’s presentation.
K. Fuchsberger 19TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Soft Migration
• Working System had to be in place for start up 2012, but:• Old System should still work.• The switch between the two Systems should be easy.
• Achieved by:• Using the old Db-Schema for new System (some
restrictions!) + some (careful) extensions• WebPage stayed operational.
K. Fuchsberger 20TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Content
1. Introduction/Motivation
2. System Overview
3. First Experience
K. Fuchsberger 21TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Some Statistics
4 big Campaigns:
Min 5 Tests launched.
K. Fuchsberger 22TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Test Statistics
Test failure rate did not decrease much !?
Average number of Tests per shift
Percentage of failed tests that timed out.
(Preconditions!)Efficiency increased.
K. Fuchsberger 23TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Time between Tests?
• Expectation: Average time between test executions on one circuit should decrease!? Not evident !?
In the shadow of Analysis Time!
Tim
e [h
]
(Data: X-Mas 11/12)
K. Fuchsberger 24TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Analysis Time
PIC2 and PNO.a1 automated (if successful)
K. Fuchsberger 25TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
HWC Campaign 2012
• System was operational from the beginning. No big problems. Never stopped progress.
• GUI was very well accepted (Lot of positive feedback).• Preconditions were useful, at least for tracking. Users
Point of view?• Constraints turned out to be very useful although
sometimes puzzling. Prevented many mistakes. Many additional constraints were added during the campaign.
• After some doubt, the automated scheduling was found to be very convenient.
K. Fuchsberger 26TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
To Improve
• Missing: Editing of Test Plans! (A lot of Db hacks were necessary )
• Constraints should be more dynamic (Currently Java Classes)… dynamic loading for plugins?
• Old page still used for statistics and quick lookups. Missing GUI features.
K. Fuchsberger 27TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Next Steps
• Database Migration (Old web page will die)• Improvements of GUI statistic features
(Summer Student)• Add test-plan editor to GUI.• Improve Scheduling Algorithm ( Michael)• Integrate other systems:
• Beam Commissioning as SignOnly tests. Migrate smoothly to partly automatized tests?
• BIC/PIC tests?• Others?
• More Automated Analysis (Powering Server)
K. Fuchsberger 28TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Conclusion
• System worked very well in 2011.• Some improvements (test plan editing, more
dynamic constraints, new GUI features)• A lot to improve in analysis part!• Next Steps important:
• Db migration.• Improvement of scheduling.• Improvement of automated analysis.• Integrate more systems.
K. Fuchsberger 29TE-MPE-TM, 2012-05-10, Acctesting – Motivation, Overview and First Experience
Thank you for your attention!