for testers - anztb ai and...relevant context testing neural net ranker personalized web search and...
TRANSCRIPT
Relevant Context
Testing Neural Net Ranker
Personalized Web Search and Chrome Test Automation
AI for Mobile Test Automation
AITesting.org
Testing: Apply Inputs, checking outputs.
https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they
AI: Apply Inputs, check outputs
Features
Complexity increases
exponentially as new features and
states interact with existing
featuresTests
Test coverage grows linearly
because tests can only be added
one at a time
Time
Co
mp
lexi
ty /
Co
vera
ge
COVERAGE GAP
Testing Needs AI
HOW A BOT LEARNS TO TAP, TYPE, AND SWIPE
Key Problems in Test Automation
1. Have to Code2. Element Selection
3. Test Step Sequencing
solved by AI systems
VS
Element Identification
Test.aiAuto Labeled
Quick Visual Custom Labeling
Selenium IDEPain Finding IdentifiersHave to write code. (brittle)
Old / Past
IDENTIFYING APP STATE
AI Auto-trains to recognize all elements and pages in your app
AI REPLACES HARD-CRAFTED TEST CASES
VS
Step Sequencing
Selenium IDEHave to code exact steps.Brittle when app changes.
Test.aiFamiliar Cucumber/GherkinOnly list labels of interesting
steps and verifications.Also, GUI Editor
Old / Past
Where’s the AI? Reinforcement learning
Robust testing
Instead of prescriptive tests, capture user intent
“Go to the shopping cart”
AIT - Open - Manual Too
VS
Test ExecutionOld / Past
VS
Test Reporting
Selenium IDEMany Tests Flaky/Break as app ChangesSimple pass/fail
Test.aiQ-Learning Test Step Generation
Element Classifiers at runtimeSee Test Steps.
Presentable.
Old / Past
Test Run Cross Platform, Cross App!
HARNESSING COLLECTIVE INTELLIGENCE FOR TESTING
How to test AI?
https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they
30
Testing AI: Test Boundary + InputTest Data outside the range of Training Data. Errors are at the margins.
What We Do31
Testing AI: Test Boundary InputTest Data outside the range of Training Data. Errors are at the margins.
What We Do32
Testing AI: Feature Correctness
AI “input” correctness● Features computed correctly● E.g. “#words on page”● Representative Sample?● Bad/Partial data?● Sample Temporal/Domain relevance?
Credit: https://www.researchgate.net/figure/224915935_fig2_Figure-1-Neural-Network-Architectures-Used-in-the-SimulationsTwo-different-neural33
Testing AI: Each new build--consider it a new implementation
Credit: https://www.wolfram.com34
Testing AI: “Drift”
Networks are randomly initialized
New network might be just as accurate as last networkON AVERAGE
Test for the texture change.
Mapping from network output -> product output functionally correct.
Our Story35
Testing AI: Action / Flow Graphs
Our Story36
Testing AI: App Generation and Traversal
Our Story37
Testing AI: Test Graph Generation and Traversal
What is Next?
What We Do39
Testing AI: “DARPA HARD”
What We Do40
Humans vs Bots
Humans Train Bots in Course of Work
Humans Remain to Personalize and Re-train bots.
Little testing that AI cannot do “soon”
What We Do41
Abstract Intent Language: AIT
Focus on Intent, not “how”
Human Readable
App-Independant
EXAMPLE: HOW TO DETECT A BUG
ML : Subjective or complex labels via Humans
AI for Testing: Input
Reduce input space
150 actions per page
35 steps -> 150^35 = 10^78 paths
#atoms in universe10^78...
ML: Page Label Training