test objects -- they just work
DESCRIPTION
Keynote: Google Test Automation Conference. September 8, 2006,London, England.TRANSCRIPT
![Page 1: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/1.jpg)
mVerify A Million Users in a Box ®
®
Test Objects -- They Just Work
September 8, 2006
Google Test Automation Conference
www.mverify.com
![Page 2: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/2.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 2
Overview
Motivation
MTS Goals
TTCN Influences
X-Unit Influences
MTS::TestObject
Demo
Q & A
![Page 3: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/3.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 3
The Mobile Testing Nightmare
Intense, high-stakes race to market
Configurations (platforms x devices x airlinks) increase exponentially
More testing necessary for competitive quality, reliability, performance
Ad hoc manual testing is slow, costly, ineffective
![Page 4: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/4.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 4
MTS Background
Advanced model-based, mobile-centric testing
Necessary for 21st century technology
It’s time for a change
Model-based, mobile-centric, reliability-maximizing
Advanced test strategy useless without
End to end and embedded observation & control
Scalable, Robust
Distributed
Broadly usable
![Page 5: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/5.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 5
MTS: Any App, Any Platform
Agent Host
Client Host Under Test
Agent HostConsole Host
MTS Console
MTS Test
Agent
MTS Test
Agent
Host Under Test may be
TEST RUN
REPORTS
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
Cell Phone
PDA
Desktop
Server
Embedded Processor
Network Equipment
Access Point
Base Station
Client Host Under Test
MTS Remote
Agent
Client
Under Test
Server Host Under Test
MTS Remote
Agent
Server
Under Test
![Page 6: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/6.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 6
Design Goals
Generate Test Objects from models
Generate models from IUT
Logical/Physical separation
Platform agnostic/robust
Channel agnostic/robust
Minimize IUT footprint
Distributed control
Works out of the box/Agile
Intuitive interaction
Minimize brittleness
Composable
![Page 7: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/7.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 7
About TTCN-3
Testing and Test Control Notation, international standard
Abstract specification
Structured/Procedural paradigm
Proven support for complex scenarios
Proven support for concurrency
ASN.1 data (packed, maps to XML)
Routine use in very high-reliability applications Rigorous conformance testing of protocol implemenations, from
single test spec
![Page 8: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/8.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 8
About TTCN-3
Point of Observation and Control Abstract Interface Specification
Each user provides physical binding for logical interface definition
Same test suites can be used on many implementations
Model-based Testing Generate TTCN from MSCs, FSMs, etc.
Compile to implementation language (C++, Java)
Compile & run implementation
![Page 9: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/9.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 9
Incremental Testing Framework
TestObject
TestCaseTestSuite -Contains
0..1 0..*
0..1
-Contains
0..*
TestRunner TestLog+Contains
1 1
MyTestSuite
-Uses
1 1
MyTestObject
«uses»
![Page 10: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/10.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 10
TTCN and ITF -- Complementary
MTS Design Goal TTCN ITF
Automate test object generation Yes NA
Logical/Physical separation Yes No
Platform agnostic/robust Yes No
Channel agnostic/robust Yes NA
Distributed control Yes No
Minimize IUT footprint NA No
Works out of the box/Agile No Yes
Intuitive interaction No Yes
Composable No Yes
Minimize brittleness No No
![Page 11: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/11.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 11
TTCN and ITF -- Limitations
TTCN Not executable – must be compiled
L2P implementation out of scope
Modular, but not composable
ITF TestSuite limited to static object list
Not control-composable
Tight coupling with IUT
Greedy
Language-centric
Developer-centric
![Page 12: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/12.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 12
MTS::TestObject
PPC 2002
YourTestObject
+getTestObjectClassName()
+getTestObjectHeadline()
+getTestObjectFileName()
+getTestObjectDir()
+getRemoteHostStatus()
+getTestRunRepeatCount()
+getTestRunIteration()
#require()
#setup()
#test()
#cleanup()
#ensure()
#check()
#run()
#submit()
#userLogEntry()
#userLogImage()
#evaluateCompareFrame()
#replayTestActionSequence()
+testObjectHeadline
MTS::TestObject
+require()
+setup()
+test()
+cleanup()
+ensure()
MyTestObject
MTS Framework
+connect()
+open()
+close()
+test()
+record()
Adapter
+getImage()
WM5
+getServiceWSDL()
+parseWSDL()
+queryForDictionary()
+queryForXML()
WebService
![Page 13: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/13.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 13
MTS::TestObject
Adapter: built-in remote host proxy; connection management, logging, exception handling
Control-composable TestObject::run <testobject>
Pre-conditions TestObject::require
Post-conditions TestObject::ensure
Built-in data-driven test from literal, variable, or function TestObject::submit <data_provider>
Pass/fail any Tcl expression: TestObject::check <expr>
XML test run log
Incr Tcl, any Tcl package, library extension, or procedure
Support stories, suites, cases, procedures …
![Page 14: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/14.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 14
Demo
Adapter selection
One-click Repeat
Smart Progress Bar
![Page 15: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/15.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 15
Notes
NeXTSTEP demo, Steve Jobs says “It just works -- they’re objects“ http://rixstep.com/1/1/20060814,00.shtml
Mobile Testing Nightmare Poster
Questions: [email protected]
![Page 16: Test Objects -- They Just Work](https://reader033.vdocuments.us/reader033/viewer/2022051514/5483ecddb47959d30c8b4a72/html5/thumbnails/16.jpg)
© 2006 mVerify Corporation A Million Users in a Box ® 16
Q & A