dymola testing library - modelica 7 testing tutorial presented on 15.05.2017 in prague slides...
TRANSCRIPT
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Tutorial
presented on 15.05.2017 in Prague
slides updated on 01.06.2017
by Marco Kessler, Dassault Systèmes
2
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Content
(4) Testing Modelica Libraries
(17) CSV Compare
(22) ModelManagement Library
(28) XogenyTest Library
(32) buildingspy
(40) PySimulator
(48) Commercial Testing Solutions
(52) Testing Library
(70) Summary of available testing solutions for Dymola users
(71) Feedback and more Information
(72) Literature
3
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Intention of this presentation
This presentation gives an overview about the available testing solutions for Modelica libraries
Every testing solution is described on a couple of slides, containing
Facts – who, what, how, where, up-to-date, documentation, highlights
Getting started – installation, test setup, running tests
Additional info – caveats, behavior in certain scenarios
Regarding Modelica tools and tools and operating systems
Everything was tested from the perspective of a Dymola user with Windows as operating system
However all solutions also work on other operating systems and/or in other Modelica tools
The getting started slides focus on Windows and Dymola, but they should also help to get the testing solution running with other tools and operating systems
The information for all tools have been gathered from the corresponding websites, documentations and publications. Some features might have been overseen or are not described correctly
e.g if they are undocumented
or if I, the author of this slides, just oversaw it or didn’t get it
4
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Daily bread of a Library Developer
Modelica Classes have many dependencies
Tools and their versions
Other libraries
Modelica language specification
Developer want’s to ensure that
All classes check
All Examples translate and simulate
Same results are produced
… with all configurations
5
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Possibilities to run checks for a Dymola Library developer
Within Dymola
Check button
Can check package content, but check aborts on error
Functions and .mos scripts
Tedious and not much fun (no debugging except print)
Automatization
.mos scripts
Python interface
Java interface
6
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
General Solution: In-house Implementation to run nightly tests
Automatization of Dymola via
.mos files
Python interface
Java interface
Common test cases
Check all classes
Translate examples
Simulate examples
Periodic testing with CI solutions like
Jenkins or Hudson
TeamCity
GitLab CI
7
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Test dependencies: Regression tests
To discover mistakes introduced later regression tests can be used:
Reference simulation results are generated
After any change simulation results are compared to reference
Possible changes: Tool type, tool version, library version, changed classes
More effort required to implement in-house tool
… but many Modelica implementations exist already
8
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Comparing signals
Common approach: definition of a tube
How to set the tolerance?
Relative?
Absolute?
… of course there is no general valid answer. One possibility: start with 0 tolerance and increase if necessary.
9
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Storing signals as references
By sampling the signal we loose information
Interpolation to recreate it
Which interpolation method?
10
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Comparing continuous with sampled signals
Sampled signals are problematic for continuous comparison and must be interpolated
11
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Comparing two sampled signals with different output interval
Common scale needed
Interpolation for intermediate points
12
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Comparing signals with small time delays
Large errors can occur if an event is triggered a little later
Easy solution for real signals: Filter
Better solution:
Compute a tube around reference
so a time delay is allowed
13
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Nomenclature
Unit test
test of a single component
For a defined input a defined output is expected
Integration test
Test if combination of several components gives the expected result
Test case
Contains everything to check a component for one scenario
Test suite
Collection of test cases
Test runner
Tool that identifies tests, runs them and checks the result
14
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Running tests in Modelica –
BuildingspyPySimulator
XogenyTest
CSV compare
Model Testing Toolkit
Model Management
MoUnit
RegressionTest
Testing Library
many options
15
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Regression tests: some fundamental questions
Test setup
With config files? In separate tool?
Test execution
Local or automated on server
Single models and batch mode
How to identify tests
How to specify tolerances
Effort to maintain tests
What happens on renames of class and variables?
Reference files
File size, possibility to view and maybe edit content
16
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Brief presentation of freely available solutions
… for Dymola users
17
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
CSV Compare - facts
who?
ITI GmbH
what?
Compare two csv files in specific format
Compare two directories with csv files
Compare FMU simulation result to csv file
how?
Standalone command line application
Using FMUComplianceChecker for FMU simulation
18
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
CSV Compare - facts
where?
https://github.com/modelica-tools/csv-compare
binaries available under releases
up-to-date?
Last commit 10 months ago
Last release 2015: v.2.0.0
documentation?
Online at https://github.com/modelica-tools/csv-compare
… or from the command line: compare.exe --help
highlights
Open Source (MIT)
HTML reports with svg graphics
Compatible with FMUchecker
cmd.exe
C:\>Compare.exe --help
19
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
CSV Compare – getting started
Download prebuild binary and extract• from https://github.com/modelica-tools/csv-compare/releases
Create csv file from Dymola simulation results
From loaded simulation results:
• Plot variable of interest
• Right click on the result in variable browser > Export Result > Only Plot Window … > filetype .csv > Save
From .mat file:
• Either Use alist.exe from C:\Program Files (x86)\Dymola <version>\bin\alist.exeset argument –e to extract single variables
• Or load result in Dymola and export as described above
From Dymola script :
• Use DataFiles.writeCSVmatrix
Call compare.exe with appropriate arguments
If alist.exe or Dymola created the results the delimiter must be set with --delimiter=, or –d ,
20
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
CSV Compare – comparing results
cmd.exe – compare
C:\CSVcompare>compare.exe --mode=csvFileCompare --verbosity=2 --delimiter=, --override -t 1e-2 --
reportdir=".\result" ClassA.csv ClassA_ref.csv
2017-05-08Z15:51:45 [ Debug ] Starting new result check @08.05.2017 15:51:45
2017-05-08Z15:51:45 [ Debug ] Parsing commandline options:
* --mode=csvFileCompare --verbosity=2 -d , --override -t 2e-2 --reportdir=.\result ClassA.csv ClassA_ref.csv
2017-05-08Z15:51:45 [ Debug ] Successfully parsed the following options:
2017-05-08Z15:51:45 [ Debug ] Operation mode is CsvFileCompare
2017-05-08Z15:51:45 [ Debug ] Tolerance is 2e-2
2017-05-08Z15:51:45 [ Debug ] Delimiter to be used to parse csv has been explicitly set and is ","
2017-05-08Z15:51:45 [ Debug ] Logfile is empty
2017-05-08Z15:51:45 [ Debug ] Column "x" seems to be text, this is good.
2017-05-08Z15:51:45 [ Debug ] Column "x" seems to be text, this is good.
2017-05-08Z15:51:45 [ Debug ] Exiting with exit code "0".
2017-05-08Z15:51:45 [ Information ] Comparing "C:\CSVcompare\ClassA.csv" to "C:\CSVcompare\ClassA_ref.csv"
2017-05-08Z15:51:45 [ Debug ] The resolution of the base x-axis is good.
2017-05-08Z15:51:46 [ Warning ] x is invalid! 450 errors have been found during validation.
2017-05-08Z15:51:46 [ Information ] Report has been written to: C:\CSVcompare\result\ClassA_ref_report.html
2017-05-08Z15:51:46 [ Information ] Writing meta report to C:\CSVcompare\result\2017-05-08-index.html
2017-05-08Z15:51:46 [ Information ] Metareport has been written to: C:\CSVcompare\result\2017-05-08-index.html
21
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
CSV Compare – behavior
Refactoring
Simulated class renamed
Filename changes -> must adapt cmd call
Variables or components renamed
Behavior: Test is skipped, Warning in cmd, nothing in report
Required action: ref csv must be updated manually with new name
Comparison
Tolerances
Argument –t to set tube size in x and y direction (time and abs. value)
doc says its only for discontinuities in x direction, but tests showed that y direction is also affected
Changed simulation settings
Number of result points
Test is ok, no info
Simulation time:
Act is shorter: Test is ok, no warning
Act is longer: Test fails (seems like limits are hold)
Result file
Format, Size, View
.csv files created by user from .mat files or with simulation tool
22
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – facts
who?
Dassault Systemes
what?
Regression Test
Compare Statistics
Style check
Class coverage
Condition coverage
how?
User calls function ModelManagement.Check.checkLibrary
MM creates references in working directory
.mat file with full simulation results
.mo file with statistic
.log file with some info
Design.Validation is used to compare results
23
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – facts
where?
Shipped with Dymola
up-to-date?
Current v1.1.6 released with Dymola 2017
No major changes in recent Dymola versions
documentation?
Dymola User Manual Volume 2: Chapter "4.4 Model and library checking“
ModelManagement.Check.UsersGuide
highlights
Part of every Dymola installation
Check of model statistics
Coverage analysis
HTML reports
24
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – getting started
Open ModelManagement library in Dymola
Define experiments
Set annotation experiment(StopTime=…) in every model that shall be checked
Load additional library with test cases if needed
Create references
Call ModelManagement.Check.checkLibrary() with generateReference=True
Compare
Call ModelManagement.Check.checkLibrary() with referenceFileDirectory=“path/to/references”
Dymola Commands – Create References
import ModelManagement.Check.checkLibrary
= true
checkLibrary(classCov=false, condCov=false, styleCheck=false,
name="MyLib", generateReference=true,
referenceFileDirectory="C:/MM/ref",
generateReferenceTranslationStructure=true);
Collecting classes in 'MyLib'
Test cases collected from 'MyLib'
Found 5 test case(s) in 'MyLib'
...
Done
25
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – compare results
Dymola Commands – Run test
import ModelManagement.Check.checkLibrary
= true
checkLibrary(name="MyLib",
referenceFileDirectory="C:/MM/ref");
Collecting classes in 'MyLib'
Found 148 classes
Checking all classes
Log of check generated. Path:
'C:/MM/wd/MyLib_StyleCheckLog.html'
Done checking 'MyLib'
Collecting classes in 'MyLib'
Test cases collected from 'MyLib'
Found 5 test case(s) in 'MyLib'
= true
= true
= true
Log generated. Path:
'file:///C:/MM/wd/MyLib_HTML_log.html'
Done
MODEL VALIDATION
Settings for validation
Model Name: MyLib.Examples.LossyGearDemo
Result couplings
Variable: Inertia2.phi, Tolerance: 0.001
Variable: Inertia2.w, Tolerance: 0.001
Variable: gear.mode, Tolerance: 0.001
Integrator Setup
Integrator: Dassl
Time interval -> Start time: 0, Stop time: 0.5
Tolerance: 0.0001
Advanced Setup Settings
Validation file: C:/MM/ref/MyLib.Examples.LossyGearDemo_ref.mat
Scalar Criteria check
Variable: Inertia2.w has scalar criteria 0.00214119 larger
than tolerance 0.001
26
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – behavior
Refactoring
Renamed classes and variables
Behavior
• Error: Reference is not found
Required Actions:
• Create new reference with renamed model
• Old references stay in Reference directory -> delete manually
Comparison
Tolerances
Absolute tolerance of 1e-3 (not changeable)
Changed simulation settings
Result points
No warning
Simulation time:
Act shorter: Test is ok
Act longer: Test is ok
Result file
Format, Size, View, Edit
Dymola .mat result files
All variables are stored > potentially large files
Can be viewed with SDF Editor
Other tool (like e.g. Matlab) needed to edit
27
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
ModelManagement – caveats
All state variables are compared
Big reference files
Errors for other variables are not detected!
To test single components like a sine block a state must be introduced
Test Identification
All classes with experiment(StopTime=…) are treated as test cases
Not possible to ignore an example
e.g. for similar demo examples in a library
No error if an experiment does not exist anymore – old references must be removed by hand
Storing default simulation settings with Dymola GUI not working as expected
Default settings are not stored, but removed (nothing set = default)
The annotation must be added by hand to get e.g. an experiment with StopTime=1
28
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
XogenyTest – facts
who?
Xogeny / Michael Tiller
what?
Proof of concept only!
No visual elements, no connectors, only models and functions
Run unit tests with models and functions
Contains functions to check single values
Contains models to check trajectories
how?
By calling Modelica functions
By instantiating models and setting parameters and inputs in text layer
Expected trajectories are defined as 2D table
29
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
XogenyTest – facts
where?
https://github.com/xogeny/XogenyTest
up-to-date?
Last commit on GitHub was 4 years ago
documentation?
Inside Models and Functions
highlights
Open source
100% Modelica – should work in every tool
30
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
XogenyTest – getting started
Clone or download from GitHub
Create test model in Modelica tool
Instantiate test component from library XogenyTest
Set parameters and input for test component
Optionally set TestCase annotation (has no effect yet)
Simulate model / call function
check simulation log
Notepad – CheckSuccess.mo
model CheckSuccess
Real x = time^2;
XogenyTest.AssertTrajectory check_x(
actual=x,
expected=[0,0; 1,1; 2,4; 3,9]);
annotation(
TestCase(
action="simulate", result="success"),
experiment(StopTime=4));
end CheckSuccess;
31
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
XogenyTest – behavior
Refactoring
Renamed classes and variables
Tool can take care of it
Renamed variables
Tool can take care of it
Comparison
Tolerances
Abs. tolerance is user choice
Interpolation
Not needed (tests run only for given values in trajectories record, not in between)
Changed simulation settings
Result points
No influence (comparison during simulation with events at specified time)
Simulation time:
Depends on test (some not affected, asserts for others)
Result file
Format, Size, View
user choice
32
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – facts
who?
Lawrence Berkeley National Laboratory
what?
Run regression tests for whole librariesby comparing selected variables with references
Simulate models
Plot
how?
Calls dymola.exe with .mos scripts
Test setup: one .mos-script for each model to test
References as .txt
33
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – facts
where?
http://simulationresearch.lbl.gov/modelica/buildingspy/ (official website)
https://github.com/lbl-srg/BuildingsPy (code)
https://pypi.python.org/pypi/buildingspy/1.6.0 (install via pip)
up-to-date?
Last release: Version 1.6.0, March 29, 2017
documentation?
http://simulationresearch.lbl.gov/modelica/buildingspy/
Python functions are well documented
Brief how-to on https://github.com/lbl-srg/modelica-buildings/wiki/Unit-Tests
highlights
Open Source (revised BSD) and hosted on GitHub
Automatic reference file generation; user is asked to accept new results
Supports Dymola and OpenModelica
34
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – installation
Install python 2.7
Either use a scientific python distribution
• Like anaconda: https://www.continuum.io/downloads
Or plain python installation
• from https://www.python.org/
Install numpy, scipy and matplotlib• Scipy on windows without scientific python distribution:
see https://www.scipy.org/install.html#windows-packages
Make Dymola available to buildingspy
Add dymola.exe to PATH environment variable
Optional: Install htmltidy for HTML syntax validation• From: http://tidy.sourceforge.net/
• Bit version must match with your python installation bit version
• Extract tidy.dll and add it to PATH environment variable
cmd.exe – install steps with anaconda
C:\Anaconda2>conda create -n buildingspy python=2.7
C:\Anaconda2>activate buildingspy
(buildingspy) C:\Anaconda2>conda install numpy
...
(buildingspy) C:\Anaconda2>conda install scipy
...
(buildingspy) C:\Anaconda2>conda install matplotlib
...
(buildingspy) C:\Anaconda2>pip install buildingspy
...
Successfully installed buildingspy-1.6.0 future-0.16.0
gitdb2-2.0.0 gitpython-2.1.3 pytidylib-0.3.2 smmap2-
2.0.1
35
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – test setup
Create test model in Modelica tool and simulate
Create a .mos script containing
simulateModel with simulation settings
createPlot for each variable to compare
Store the .mos in
<library-dir>\Resources\Scripts\Dymola
Creating the .mos file with Dymola:
Simulate model, plot variables of interest
File > Generate Script > Plot setup > OK > Store file
remove everything from file but createPlot()
Add simulateModel()
Notepad – Sine.mos
// .mos script for unit test of MCL.Sine
simulateModel(
"MCL.Sine",
resultFile="Sine")
createPlot(
id=1,
position={15, 10, 584, 416},
y={"sine.y"},
range={0.0, 3.0, -1.5, 1.5},
grid=true,
colors={{28,108,200}});
36
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – running tests
Python – C:\MyLib\run_tests.py
import buildingspy.development.regressiontest as r
import os
def main():
rt = r.Tester(executable='dymola', cleanup=True)
myLib = r"C:\path\to\library"
rt.setLibraryRoot(myLib)
# r.setSinglePackage('MyLib.Examples')
rt.batchMode(False)
rt.setNumberOfThreads(4)
rt.run()
if __name__ == '__main__':
main()
cmd.exe
(buildingspy) C:\MyLib>python run_tets.py
Using 4 of 8 processors to run unit tests.
Number of models : 8
blocks : 0
functions: 1
Generated 4 regression tests.
Script that runs unit tests had 0 warnings
and 0 errors.
See 'simulator.log' for details.
Unit tests completed successfully.
Execution time = 10.588 s
done.
Process finished with exit code 0
37
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – behavior
Refactoring
Renamed classes
Either rename in tool and update .mos scripts manually
Or use buildingspy.development.refactor.move_class
Renamed variables
.mos scripts must be updated manually
Comparison
Tolerances
Fixed to 1e-3 for absolute and relative error
No time delay
Interpolation
Simulation results are decimated to 101 points with linear interpolation
Changed simulation settings
Result points
Warning that variable has fewer points than in reference
Simulation time:
Warning is given, plot is shown, results can be updated
Result file
Format, Size, View
.txt files, only with variables user wants to compare
Small, diff possible
Up to 101 equally spaced data points
38
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – caveats 1/2
Results trajectories are decimated to 101 values
For comparison and for reference file generation
Small reference files
Be careful with fast changing signals
Variable _nPoi in buildingspy.development.regressiontest.Tester
e.g. for example on previous slide: rt._nPoi = 201
For each test the complete library is copied to a temporary directory
On Windows 7: C:\Users\<user-name>\AppData\Local\Temp
39
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
buildingspy – caveats 2/2
Make sure that Dymola does not save the startup directory!
Script will hang until Dymola.exe is killed
timeOut can be set for buildingspy.simulate.Simulator, but apparently not for Tester
Dymola 2018: Edit -> Options -> Settings -> Do not save Startup directory
Multicore: add if__name__=='__main__': to your script
… as shown in the previous example
Otherwise infinite processes are spawned, filling your hard disk with temporary files
40
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – facts
who?
DLR Institute of System Dynamics and Control
what?
Simulation Environment with many plugins
For different Modelica tools
For analysis of models and simulation results
Tester plugin included
HTML report generated
how?
Simulation: Calling dymola.exe with .mos scripts
Comparison: all variables in the result files of the selected folders are checked
41
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – facts
where?
https://github.com/PySimulator/PySimulator
up-to-date?
Last commit and latest release 5 months ago
documentation?
None in the tool
2 papers published
See last slide of this presentation
and https://github.com/PySimulator/PySimulator/wiki
highlights
Support for many tools
Can be easily extended
GUI included, but also usage of python functions possible
42
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – installation
Get latest release from https://github.com/PySimulator/PySimulator/releases
For Windows users with no Python installed
Execute Install PySimulator.bat
The .bat file installs Python(x,y)
• A python distribution like Anaconda
• Hosted https://python-xy.github.io/downloads.html
• No releases and commits since 2 years
• Not recommended to combine with other Python distributions
• First Installation Note on Python(x,y) Website: “It is recommended to uninstall any other Python distribution before installing Python(x,y)”
If you already use e.g. Anaconda and want to run PySimulator in a virtual python environment
Manual installation is described on https://github.com/PySimulator/PySimulator/wiki/Installation
See remarks on next slide
For Linux users
See https://github.com/PySimulator/PySimulator/wiki/Linux-Setup-(Ubuntu-16.10-x64)
43
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – remarks for manual installation on Windows
Required packages for manual installation listed on
https://github.com/PySimulator/PySimulator/wiki/Installation
You have to download all .whl files and install them one by with pip
Easier alternative:
Install Anaconda https://www.continuum.io/downloads
Download source of latest PySimulator release https://github.com/PySimulator/PySimulator/releases
Extract source code to C:\pysim\src
Create a virtual environment and follow these steps ------------------->
Remarks:
tested with PySimulator 0.62 on Windows 7 64bit with Anaconda2
Python setup.py install failed on the first execution, but on the 2nd try it worked…
If you try to start PySimulator from src the wrong module will be started. Leave the directory first (hence the cd .. In the example on the right)
Only the steps for the Dymola plugin are described here. Other plugins need more python packages. See the wiki on GitHub for details
cmd.exe – Install PySimulator
C:\pysim>conda create –n PySim python=2.7
C:\pysim>activate PySim
(PySim) C:\pysim>conda install chaco
(PySim) C:\pysim>conda install scipy
(PySim) C:\pysim>conda install Matplotlib
(PySim) C:\pysim>conda install tornado
(PySim) C:\pysim>conda install PIL
(PySim) C:\pysim>conda install H5py
(PySim) C:\pysim>conda install pyside
(PySim) C:\pysim>conda install kiwisolver
(PySim) C:\pysim>cd pysim src
(PySim) C:\pysim\src>python setup.py install
(PySim) C:\pysim\src>cd ..
(PySim) C:\pysim\>python -m PySimulator
44
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – test setup
Notepad – setup.txt
# Setup file for simulation of several models by PySimulator
#
# modelFile modelName subDir tStart tStop tol stepSize nInterv includeEvents
# ------------------------------------------------------------------------------------------
"C:/MyLib/package.mo" MyLib.ClassA "" 0.0 2.0 1e-6 10 500 true
"C:/MyLib/package.mo" MyLib.ClassB "" 0.0 1.0 1e-6 10 500 true
"C:/MyLib/package.mo" MyLib.ClassC "" 0.0 2.5 1e-6 10 500 false
Create references
Execute your setup.txt with Plugins > Testing > Simulate List of Models
Choose the appropriate simulator (Dymola, FMUSimulator, etc.)
Every model will be simulated and the result files (.mat for Dymola) are stored in the selected results directory
45
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
report.html
PySimulator – running tests Given Error Tolerance: 1e-3
Disk space of all used result files: 0.2 MB
Total number of compared files: 2 against 2 baseline files
Total number of Compared Variables: 68 (66 passed, 2 failed)
Disk space of full report directory: 0.0 MB
Generated: 12:21PM on May 11, 2017 by PySimulator
Time Taken: 00h:00m:00s
Result Files Status act Dymola
1 / 1 1 passed / 1 failed
50% 50% passed
Baseline
MCL.Sine 0 / 1 9 9 / 9 / 2 [1.8e-1]
MCL.SwitchRC 1 / 0 59 59 / 59 [0.0e+0]
46
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – behavior
Refactoring
Renamed classes
Test fails, no info about missing signal
Tests missing
Only files which are found in both the ref and the act folders are compared, no info if something is missing
Comparison
Tolerances
Deviation measure (abs. and rel. integrated error) For details: see the paper “PySimulator – A Simulation and Analysis Environment in Python with Plugin Infrastructure”
Changed simulation settings
Result points
Test is performed, no specific info
Simulation time:
Act shorter: Test fails
Act longer: Test fails
Result file
Format, Size, View
.mat, .mtsf
47
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
PySimulator – caveats
Settings are stored in PySimulator.ini
On Windows 7: C:\User\<user-name>\.config\PySimulator\PySimulator.ini
e.g. needed to change the path to Dymola.exe
Pip version is not the most recent
https://pypi.python.org/ (used by pip) has PySimulator 0.61 listed, whereas 0.62 was released on GitHub
Starting PySimulator if installed manually
From cmd: python –m PySimulator
48
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Modelon Model Testing Toolkit1
who?
Modelon AB
what?
Regression Test with HTML reports
Supports Several tools
Dymola, OpenModelica and OPTIMICA Compiler Toolkit
how?
GUI and Python Scripts
Local execution and on Jenkins Server
where?
http://www.modelon.com/products/model-testing-toolkit/
Check the following blog posts to get an impression:
• http://www.modelon.com/blog/articles/regression-testing-as-an-enabler-for-excellence-in-model-development/
• http://www.modelon.com/blog/articles/the-model-testing-toolkit-applied-to-modelon-library-development/
1: The tool was not tested for this tutorial. Instead all info and the picture are taken from http://www.modelon.com/products/model-testing-toolkit/
49
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Claytex Regression Test Tool1
who?
Claytex Services Limited
what?
Check all classes in library
Regression test (translation log, sim results)
Verify encryption (regression test with .moe)
how?
Standalone tool that controls Dymola
Experiments are simulated (models with simulation setting annotations)
where?
http://www.claytex.com/products/regressiontest-tool-for-dymola/
1: The tool was not tested for this tutorial. Instead all info and the picture are taken from http://www.claytex.com/products/regressiontest-tool-for-dymola/
50
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Regression tests in other Modelica Tools
MoUnit – Extension for OneModelica
Samlaus, Roland, Mareike Strach, Claudio Hillmann, and Peter Fritzson. "MoUnit--A Framework for Automatic Modelica Model Testing." In Proceedings of the 10 th International Modelica Conference; March 10-12; 2014; Lund; Sweden, no. 96, pp. 549-556. Linköping University Electronic Press, 2014.
http://www.onewind.de/ - Website down on 08.05.2017 and still on 01.06.2017
OpenModelica
https://test.openmodelica.org/
Wolfram System Modeler
https://www.wolfram.com/mathematica/new-in-10/integrated-unit-testing/
https://reference.wolfram.com/language/guide/SystematicTestingAndVerification.html
51
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
The new Testing Library
… which will be included in Dymola
52
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Why another implementation for Modelica Testing?
User convenience: making testing easier
All settings within Dymola
Available for all Dymola users
More control
Set tolerances for every variable
Define which signals to store and how to store
Low effort when library is refactored
53
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Library - facts
who?
Dassault Systemes
what?
A Modelica Library which allows to create and run tests within Dymola
how?
Running tests during simulation by comparing signals
Using features which ship with Dymola
• 95% pure Modelica
• 2.5% ModelManagement Library and DymolaCommands (for Test runners)
• 2.5% SDF Library (for reference results)
54
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Library - facts
where?
Not available yet, included in future Dymola versions
Demo-version was handed out at the Testing tutorial at the Modelica Conference 2017
up-to-date?
… v1.0 still under active development
documentation?
User’s guide
Documented classes
highlights
Everything inside Dymola
No config files, updated paths when refactoring
Run tests within Dymola
55
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Library
Includes models to
generate and read references
perform checks
Provides functions to
Create references
Check library
Run test models
56
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Usual Workflow
Develop Dymola model
Build internal test models
Plot results and check by hand
Commit
57
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Workflow with Testing Library
Develop Dymola model
Build test model with Testing Library
Provide reference manually or generate reference as sdf
Commit
Run Test periodically
58
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Creating a Test Model
Build models as usual
Add Setup block for global settings
Add compare blocks
Supply or generate references
Simulate and check messages manuallyor let a test runner do the job
59
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Creating References
Build test model with Setup component
Specify setup.filename
extend Base.TestCase for Dymola selection annotation
Connect signals to log block
Call Testing.Functions.createRefs
60
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Creating References – createRefs()
createRefs does the following for the selected model:
Check if Testing.Base.TestCase is extended
Get the value of setup.filename
By simulating 0 seconds with setup(writeFilename=true)
and reading the file Testing.filename in the working dir
Check if destination exists
argument replace must be true to continue if file exists
Read the experiment and output annotations
Set the output setup
equidistant time grid, doublePrecision
Simulate the model
With the experiment settings and setup.Mode=createRefs
Convert the result to sdf
Copy the sdf file to setup.filename
In short:
It simulates your model with the experiment settings stored inside the model and copies the result to the specified path.
Dymola Commands – Create References
Testing.Functions.createRefs("MyLib.Example", replace=true);
> result file copied to
C:/MyLib/Resources/References/Example.sdf
<path-to-file>
t=0 Testing.filename
<path-to-file>result
t=StopTime resultresult
61
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Creating References
Dymola selections
are used to reduce the result file size
TestCase
Sets selection annotation
Log Block
Sets tag for Dymola selection
Extend TestCase and using Log blocks
Signals connected to the inputs will be in group refs in the SDF file
Modelica – Testing.Base.TestCase
class TestCase "Base class to set Dymola selections"
annotation (
__Dymola_selections={
Selection(
name="ref",
match={
MatchVariable(
tag="refs",
name="*u[*]|*u",
newName="%name%")})},
...);
end TestCase;
Modelica – Testing.Refs.LogReal
model LogReal "Set tag to store signal in group refs"
extends Icons.AboveName;
Modelica.Blocks.Interfaces.RealInput u
annotation (
...,
__Dymola_tag={"refs"});
...
end LogReal;
62
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Creating References – reducing file size
Unfortunately Dymola stores not only selections but also
All parameters
All states
Remove unnecessary signals with SDF editor
Open the file, make sure that all groups are collapsed
Select everything with Ctrl+A
Deselect the .sdf file itself at the top, refs and Time
Delete and save
63
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Compare Models
Signal comparison for Boolean, Integer, Real
Compare signal with reference trajectory
Timed compare
Specify in a table the expected value for a point in time
Within Constant limits
Signal must be inside specified limit
64
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Compare Block RealSig
Two Inputs
actual and reference value
Prints result to message window
Optional: Writes JSON files on hard disk
Contain expected and actual result
Can be used for automatization
e.g. by reading the results with a python script
65
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Simulation results in Message Window
Error message when signals are outside tolerance
Summary at end
Results stored in working directory
If setup.writeJSON=true
For every test block a JSON file is created in<working directory>\TestingLib
Output directory can be changed with setup.resultDir
66
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Test runner – checkPackage()
… does the following for the selected package:
Get names of all classes
Check every class and print the result
Skip classes with the annotation __Dymola_interactive=true
Dymola Commands – checkPackage
Testing.Functions.checkPackage("MyLib");
------------------------------------------------
> Checking all classes in MyLib
(i) started at 17:15:34
#ok: MyLib.ClassA
#fail: MyLib.ClassB
#skip: MyLib.ClassC (__Dymola_interactive=true)
#ok: MyLib.ClassD
(i) check finished at 17:15:39
(i) Summary
4 classes
2 ok
1 fail
1 skipped
------------------------------------------------
= true
67
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Test runner – runTests()
… does the following for the selected package:
Get names of all classes
Simulate all classes with name starting with Test_
With the settings stored in experiment and output annotations
Activate JSON output and set the output dir
To <wd>\TestingLib\<class-path>
Get the check result from the JSON files in the dir
Print the result
Dymola Commands – runTests
Testing.Functions.runTests("MyLib.SubPackage");
------------------------------------------------
> Running all tests in MyLib.SubPackage
(i) started at 17:8:53
# ok: MyLib.SubPackage.Test_ClassA
tests: 3 ok: 3 skip: 0 fail: 0 not_run: 0
#fail: MyLib.SubPackage.Test_ClassB
tests: 4 ok: 2 skip: 0 fail: 1 not_run: 1
# ok: MyLib.SubPackage.Test_ClassC
tests: 2 ok: 2 skip: 0 fail: 0 not_run: 0
(i) check finished at 17:9:22
(i) Summary
3 classes
2 ok
1 fail
------------------------------------------------
= true
68
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Testing Library - behavior
Refactoring
Renamed classes
Automatically updated by Dymola if test models are loaded
Renamed variables
Automatically updated by Dymola if test models are loaded
Comparison
Tolerances
Absolute, relative and time delay can be set
Changed simulation settings
Result points
Ref from file: Interpolation with SDF tables
Simulation time:
Tests are marked as not_run where applicable
Ref from file: depending on chosen SDF extrapolation (hold, none, linear, etc.)
Result file
Format, Size, View
sdf, editable with SDF Editor
size can be controlled with simulation settings
69
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Automatization
writes
calls
reads
runs
junit.xml <check>.json
report.html
70
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Summary of available projects for Dymola users
Lib Author Environment License Supports Test Identification Configuration
buildingspy Berkeley Lab Python revised BSD Dymola, OM .mos files .mos files
PySimulator DLR Python LGPLDymola, OM, SimulationX, FMUWolfram System Modeler
Config files Config files
XogenyTest Mike Tiller Modelica CC3 Modelica - Modelica models
CSV compare ITI Command line BSD / MIT csv files - arguments
Model Testing Toolkit Modelon Standalone, Python Proprietary Dymola, OM, OPTIMICA, FMU .mo and .py files In Tool
Model Management Dassault Dymola Proprietary Dymola annotations Function args
MoUnit OneWind Java Non-Commercial OneModelica / Java Config files Config files
RegressionTest Claytex Standalone Proprietary Dymola annotations In Tool
Testing Library Dassault Dymola Proprietary Dymola Model Name Modelica models
71
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Feedback and more Information
Please contact us for more information or if you want to give feedback about this slides or the Testing Library!
72
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7
Literature on testing Modelica classes
Tiller, Michael M., and Burit Kittirungsi. "UnitTesting: A Library for Modelica Unit Testing." In Proceedings of the 5th International Modelica Conference, Vienna. 2006.
Describes a commercial unit test library by Emmeskay, Inc (acquired by LMS in 2010), which does not seem to be available anymore.
Samlaus, Roland, Mareike Strach, Claudio Hillmann, and Peter Fritzson. "MoUnit--A Framework for Automatic Modelica Model Testing." In Proceedings of the 10th International Modelica Conference, Lund. 2014.
Presents MoUnit, a test framework that can be used in OneModelica. Unfortunately the OneModelica Website is currently down.
Tilly, Anders, Victor Johnsson, Jon Sten, Alexander Perlman, and Johan Åkesson. "Optimica Testing Toolkit: a Tool-Agnostic Testing Framework for Modelica Models." In Proceedings of the 11th International Modelica Conference, Versailles. 2015.
Optimica Testing Toolkit is now released as Modelon Model Testing Toolkit
Asghar, Adeel, Andreas Pfeiffer, Arunkumar Palanisamy, Alachew Mengist, Martin Sjölund, Adrian Pop, and Peter Fritzson. "Automatic Regression Testing of Simulation Models and Concept for Simulation of Connected FMUs in PySimulator."In Proceedings of the 11th International Modelica Conference, Versailles. 2015.
Describes regression testing in PySimulator
73
3DS
.CO
M©
Das
saul
t Sys
tèm
es |
6/1
/201
7