testing virtual reality - software testingsast.se/q-moten/2016/q3/2016_q3_9_emil_alegroth.pdf ·...
TRANSCRIPT
![Page 1: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/1.jpg)
CONTACTEmil Alégroth – [email protected]
WEBwww.auqtus.com
Testing Virtual Reality (And other complex software environments!)
Dr. Emil AlégrothAuqtus | Blekinge Institute of Technology | Chalmers University of Technology
![Page 2: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/2.jpg)
• We delivering state-of-art testing solutions to industry with focus on Visual GUI Testing (VGT).
• Founded by academics and technical experts in the area of automated testing.– Combined; over 50 years of academic experience– 100s of academic publications– World-leading researchers on VGT (Founded the term)
Contributors to the talk:
Dr. Emil Alégroth Prof. Robert Feldt Dr. Simon Poulding
![Page 3: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/3.jpg)
What is Virtual Reality?
• Oculus rift and Google Cardboard• Leisure applications
• Games• Communication
• Learning applications
![Page 4: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/4.jpg)
Our application: VRHangout
![Page 5: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/5.jpg)
Unity
• Game engine
• All in-game fixtures treated as objects– Makes scripting easy
• C# scripting– Quick and powerful
– Same syntax as java
• Many plug-ins– Google cardboard!
• Main limitation– Standard components are generally slow!
Unity game: Hearthstone
![Page 6: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/6.jpg)
My experiences with Unity (5h vs 5 days)
![Page 7: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/7.jpg)
Unity
VR is only the beginning!
![Page 8: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/8.jpg)
Augmented reality
• Leisure applications (e.g. Minecraft)• Life-critical applications
• Medical systems
Safety requirements willrequire stringent qualityassurance!
![Page 9: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/9.jpg)
How do we test in this environment?
• Low-level:– Automated unit tests and integration tests– C# components (in Unity)
• High-level:– Manual testing
• Why?– The real-time, virtual environment, is (often) non-
deterministic and complex to test– Complexity: Computationally heavy to test, difficult to find
oracles, etc.
![Page 10: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/10.jpg)
A complex environment requires complex testing
• Search-based software testing (SBST)– Test case generation, based on a (*)fitness function, that
can automatically determine if the output of the generated test case is (**)correct/faulty/interesting to/for the tester.
• What is a (*)?– Domain model that defines rules for how to, for instance,
generate suitable test cases (A heuristic). Can be modified to learn during execution (Machine learning).
• How do we determine (**) in a VR environment?– For instance: Kolmogorov complexity!
![Page 11: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/11.jpg)
G
Test case generation with Meta-Heuristics
• Meta-heuristic– A description of a possible solution to generate test cases– For instance: Genetic algorithms
T1T2
T2 T1 T2
T1
T2
T1 T2T1Determine best child with a fitnessfunction (best fit)
Gen 1 seed
Gen 1
Gen 2 seed T2T1T1
![Page 12: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/12.jpg)
Oracle?
• Oracle– A means of determining test case outcome– Examples: Value, image, strings or other output
• Kolmogorov complexity allows you to determine a distance between any two objects!
• The measure gives a distance between two objects– Determine if a state is potentially correct/faulty/interesting
![Page 13: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/13.jpg)
Kolmogorov complexity example usage
![Page 14: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/14.jpg)
SBST + Kolmogorov
• Generate exploratory test cases, verify with the automated oracle and learn by testing!
• Small problem…
– Kolmogorov komplexity can NOT be computed!
– Theoretical concept...
• …BUT! It can be approximated with compression!
Compress((1,[1,3,4,5]) , (2,[1,4,5])
) = ([1,4,5],(1,[3]))
Compress( , )
= ( ,(255,0,0),(0,0,255))”Distance”
”Distance”
No tools do this yet butthey are on the way!
![Page 15: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/15.jpg)
VR and AR are tomorrow. We need tools/techniques today!
GUI
GUI model
System architecture
System components /Source code
System model
Multiple languages
Missinginterfaces
Too muchcode
Etc.
Encapsulateddesign
Distributedsystem
Etc.
Customcomponents
Limitedusability
Etc.Maintenance
costs
Speed Robustness Etc.
FAILURES
DEFECTS
CHALLENGES
MORE SUPPORT
STRONGER NEED
(e.g. Continuous
development)
![Page 16: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/16.jpg)
Visual GUI Testing (VGT): JAutomate
• Seamless testing of different applications• End-to-end testing• Advanced testing through the GUI• Non-functional testing (e.g. Performance)• Etc.
![Page 17: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/17.jpg)
VGT: Primarily System- and Acceptance Regression testing
• VGT-GUITAR– Automated Exploratory, model-based, GUI testing
– State space explosion problems– Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing:
an Empirical study
• What if we could provide the tester with a heads-up display to improve exploratory testing based on user domain knowledge?
• What if we could automatically record exploratory test cases for regression testing?
![Page 18: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/18.jpg)
Augmented exploratory testing
Circles: covered statesColor: Likelyhood of finding defect
Expand: Get more information
Fantastic!!! but is it a
Fantasy???
![Page 19: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/19.jpg)
Current state
![Page 20: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/20.jpg)
Neural networks combined with advanced search algorithms
Home
v
Neural net for Version X+1
• Each node has:• A screenshot of the state of the SUT• The interaction to move to the next node
COMING SOON!
If advanced model,automatic exploration!
![Page 21: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/21.jpg)
Conclusions
• VR and AR require more complex test techniques
• Search-based testing with Kolmogorov complexity
– Not available yet!
• Visual GUI Testing
• Augmented Exploratory Testing
![Page 22: Testing Virtual Reality - Software testingsast.se/q-moten/2016/q3/2016_Q3_9_Emil_Alegroth.pdf · Testing Virtual Reality (And other complex software environments!) Dr. Emil Alégroth](https://reader030.vdocuments.us/reader030/viewer/2022041101/5ed97e251b54311e7967a9ea/html5/thumbnails/22.jpg)
Thank you! Questions?
E. Börjesson, R. Feldt, “Automated System testing using Visual GUI Testing Tools: A Comperative Study in Industry”, International Conference on Software Testing, 2012.E.Alégroth, R.Feldt, H. Olsson, “Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study”, International Conference on Software Testing, 2013.E. Alégroth, M. Nass, H. Olsson, “JAutomate: a Tool for System- and Acceptance-test Automation”, International Conference on Software Testing, 2013. E. Alégroth, R. Feldt, L. Ryrholm, “Visual GUI Testing in Practice: Challenges, problems and limitations”, Empirical Software Engineering Journal, 2014E. Alégroth, Z. Gao, R. A.P. Oliveira, A. Memon, “Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing: An Empirical Study”, International Conference on Software Testing, 2015.E. Alégroth, R. Feldt, P. Kolström, “Maintenance of Automated Testing in Industry: An Empirical study on Visual GUI Testing”, Journal of Systems and Software (In submission).E. Alégroth, J. Gustafsson, H. Ivarsson, R. Feldt, “Replicating Rare Software Failures with Visual GUI Testing: An Industrial Success Story”, IEEE-Software.