smb2 model-based testing · nico kicillof, albert lee microsoft storage developer conference 2009
TRANSCRIPT
![Page 1: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/1.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
SMB2 Model-Based Testing
Nico Kicillof, Albert Lee
Microsoft
![Page 2: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/2.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
222 open protocol specifications tested
22,847 pages
~10,000 document bugs
66,962 person days (250+ years)
Tool dev center in Beijing
Vendor test teams
Hyderabad: 250
Beijing: 100
2
“BlueLine” Technical Document
Testing Project
![Page 3: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/3.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Formalization
Technical DocumentTest Suite
Normative
statementsAssertions
Checked against Windows
(Accuracy)
• Validate
consistency
• Simulate
implementation
process
Test-Driven Document Analysis
3
![Page 4: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/4.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Model-Based Testing
A Lightweight Formal Method
4
![Page 5: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/5.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Model
Implementation
Test OracleTest Sequences
Control
Verdict
Informal Spec/Reqs
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
5
![Page 6: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/6.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Requirements
Gathered from technical document alone Gatherers not protocol experts
Clean-room approach
Windows-specific behavior as separate requirements
Document testing Not implementation
Independent reviewers
Main input for model design
Technical Document
Requirements Spec
Model
Test Suite
Logs
Network Captures
6
![Page 7: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/7.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
The Value of Modeling
Model-Based Testing vs. traditional testing comparison
Empirical SE Research group assessed statistical significance
Uniform sample from one testing site
Modeling resulted in 42% productivity gain
Overall, modeling saved 50 person-years (12,547 days)
No formal methods background was required
Teams of junior engineers
Mostly vendors, fresh out of college
7
![Page 8: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/8.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Representation
8
Getting data on the wire and back
Known problem for protocol testing (e.g. TTCN-3)
Our Solution: Protocol Test Framework (PTF)
Test adapters written in a managed language (C#)
Custom support for dealing with protocols
Automatic data packet (de)serialization
Based on declarative data definitions
Protocol-specific logging capacity
Beacon packets
![Page 9: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/9.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Adapters
Adapter Interface
TCTest Cases(traditional or MBT)
TC TC
TC TC TC
Adapter Implementation
SUT(System under test)
•Abstracts SUT functionality
•Contract between teams
•Test case team
•Adapter team
•Pluggable
•Different server setups
•Different transports
9
![Page 10: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/10.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Spec Explorer
A Model-Based Testing Tool from Microsoft
10
![Page 11: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/11.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Features
Multiple modeling styles and languages
Programs, patterns, diagrams
Asynchronous & non-deterministic systems
State machine extraction from model program
Test code generation from state machine
Model composition
11
![Page 12: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/12.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Spec Explorer Usage
Internally used to test several Microsoft products
External Early Adopter Program
Shipping plans
MSDN DevLabs
Visual Studio 2010 Beta 2 timeframe
12
![Page 13: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/13.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
SMB2 Interoperability Test Suite
13
![Page 14: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/14.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Suite Scope
Validate the Technical Document against Windows for
interoperability
Accuracy
Consistency
Usability
Aimed at the Microsoft Communications Protocol
Program
Only server behavior is tested
14
![Page 15: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/15.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Approach (First Pass)
Assumptions
Single connection
No compounded message
No nested shares
Out of model scope
GSS-API authentication
Race conditions
Exhaustive flag combinations
121 scenarios
15
![Page 16: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/16.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Test Suite Architecture
Test Suite
Test generated by Model Traditional Test
MS-SMB2 Infrastructure SDK
SMB2 Server
SMB2 Adapter Interface
Request
Call
Response
Event
SMB2 Adapter Core
Traditional Adapter Interface
Traditional Adapter Core
Request
Call
Response
Event
Server
Config
Adapter
Interface
· Server
Config
Adapter
Core
16
![Page 17: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/17.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Demo
The SMB2 Test Suite
17
![Page 18: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/18.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Conclusions
Interoperability test suites developed for 200+
protocols
Thousands of document bugs filed and fixed
Test suite further usage
Regression testing
Third-party implementations (Plugfests)
SMB2 test suite (first pass)
Document improvement
80% requirement coverage
18
![Page 19: SMB2 Model-Based Testing · Nico Kicillof, Albert Lee Microsoft Storage Developer Conference 2009](https://reader030.vdocuments.us/reader030/viewer/2022011818/5e8a7e6018e71a1ef12dd9f5/html5/thumbnails/19.jpg)
Storage Developer Conference 2009
© 2009 Microsoft Corp. All rights reserved.
Questions
{nicok, albertle} @ microsoft . com
http://www.microsoft.com/protocols/
19