agile test management joins devops - testcon moscow · faster feedback for qa & devs integrate...
Post on 09-Aug-2020
2 Views
Preview:
TRANSCRIPT
agile test management joins DevOpsMartin Koch | 2019
It’s 2019 and test management needs to move faster than ever before.
How to combine classic manual testing, test automation, DevOps and smart test management for better results with reduced effort and headaches.
We are showing in an hands on approach, how modern QA and tooling can make your life easier.
Мы привезли подарок ...
Для того, чтобы принять участие врозыгрыше – пришлите текст
“Testcon2019”
На номер:
+7 929 646 8235
Победитель получит приз на стендеaqua в Зале№1 😃
2
who am i?
Martin Koch
Head of Product Management aqua
m.koch@andagon.com
aqua cloud GmbH (andagon)
Germany
andagon.com/en/aqua
3
aqua
• Test management software made in Germany
• Integrates very tightly with Jira
• Manages and simplifies test automation
• Powerful & flexible reporting
4
The challenge in QA
Businesses need to react fast to stay competitive
Do more with less.
5
The need for speed
Complexity of software systems: rising 📈
6
Systems
Complexity
time
The need for speed
Complexity of software systems: rising 📈
Number of Platforms: rising fast📈• OS x Web browsers
• mobile & smart devices
• # of IoT Devices and Platforms
• microservices architecture with multiple services
7
# of Platforms
Systems
Complexity
time
The need for speed
Complexity of software systems: rising 📈
Number of Platforms: rising fast📈• OS x Webbrowsers
• mobile & smart devices
• # of IoT Devices and Plattforms
• microservices architecture with multiple services
Number of testers ️♀️: rising at the same rate?
8
# of Platforms
Systems
Complexity
time
What do the Developers do?
• Shorter Release Cycles10 years ago: months – years
today: shorter and shorter,
multiple times a day
• Continuous Delivery & Continuous Integration
• DevOps
• Agile
9
How to we keep up this speed in QA?
How do we integrate withDevOps processes?
10
Integrate with your Development Pipeline
11
DevOps unwrapped
12
plan code build test release deploy operate monitor
Agile Development
Continuous Integration
Continuous Testing
Continuous Delivery
DevOps
We want Continuous Testing
Faster feedback for QA & devs
Integrate with DevOps pipeline
Needs automated tests• Unit Tests (Developer Tests)• Service Tests• UI Tests
We want
• environment-triggered tests
• traditional developer/tester-triggered tests
13
Diagram: environment-triggered test
Automated Testing ≠ Continuous Testing
14
Test Pyramid
Mike Cohn - Succeeding withAgile
15
End2End: Brittle, break easily unless build smart
Simulate a real user
slower
faster
more integration
more isolation
Are we building the right code?
Are we building the code right?
Test Pyramid in detail
16
Manual Tests
Automated Component Tests
Automated Unit Tests
Automated Integration Tests
Automated API Tests
Automated E2E Tests
Tools
17
Manual Tests
Automated Component Tests
Automated Unit Tests
Automated Integration Tests
Automated API Tests
Automated E2E Tests
Test management suite:
e.g. aqua
UI Tests: Selenium, Appium, Ranorex,
Managing E2E: aqua
xUnit, TestNG,
Mockito, Jasmine
SoapUI, FitNesse, Cucumber,
REST-assured, Postman
Granularity of tests
1. Write tests with different granularity
2. The more high-level you get, the fewer tests you should have
A failure in a high level test means
a) a bug in your functional code
b) a missing / incorrect unit test
Tip: If this bug gets fixed, write the unit test first to make sure it stays fixed.
18
How to build End-2-End tests
• Build End-2-End tests from smaller tests, stack them, combine them
• Reference test scripts, never copy them
For UI tests either:
a) Build simple UI tests that you can record very quickly and re-record them when they break
b)Build smart UI tests• Dedicated Object Repositories• Invest extra time in solid object recognition paths, IDs, etc.• Write code
and of course use a good test automation tool
19
Key take aways
For agile & DevOps speed: Automation!
Make good use of Unit Tests• Cheaper• Focused • Easier to maintain• Bug is fixed? -> Write another unit test
Good End-2-End & UI tests can be expensive (good software* can help enormously)
Build up End-2-End Tests from existing tests.
Combine multiple test levels
Invest in good test data
20
* aqua!
How can you manage all this?
aqua provides a way from manual to automated testing
aqua manages • Manual Tests• Unit Tests (Developer Tests)• Component Tests• Integration Tests• API Tests• End-2-End Test
aqua orchestrates your test automation
21
From manual to automated tests
Good practice:
1. Write a manual test case first
2. Automate this test case
3. Ideal: You manage manual and automated test & scripts in the same tool
4. Ensure traceability
22
user storymanual test
caseautomation
script
Picture: Manual test case in aqua
From manual to automated tests
23
Connect with your Continuous Integration (CI) Server
Moving to DevOps, Developers set up their CI Server and Build Pipelines
This is where the Unit Tests reside and get triggered.
Next up:
We want
• all our test results in one place
• environment-triggered tests
• traditional developer/tester-triggered tests
• Unit, Service, UI & End-2-End tests
Connect your test management solution with your CI server and DevOps pipeline!
24
Jenkins CI
Continuous Integration
Get Unit Test results in the your Test Management solution
25
aqua Jenkins Plugin Test Management
Unit Test results
Case 1 • Save unit test results to your test management suite
• Holistic Reporting • unit test results• manual test results• any automation test results
• Full traceability from user stories to all test results
Continuous Integration
Get Unit Test results in the your Test Management solution
26
aqua Jenkins Plugin Test Management
trigger test automation
Case 2• Start automation scripts managed by your
test managemet suite
• This can be complex End-2-End test scenarios using multiple technologies.
Example:
1. DB script inserts new products into DB2. Selenium creates new Customer3. Selenium buys 2 products4. SoapUI checks results over REST API
Continuous Integration
Get Unit Test results in the your Test Management solution
27
Jenkins CI Test Management
Trigger
Build Jobs / Unit Tests
Case 3 • Start build jobs
• Deploy new software to QA environment
• Run Unit Tests
Trigger an Test Scenario from your CI Server
Example:
• aqua Jenkins Plugin
• Test scenario contains multiple testcases
• All test cases can contain multiple automated test scripts and test data
28
DevOps example test caseThis aqua test case does:
1. Trigger Job in Jenkins
2. Checkout GIT
3. Build of application (SUT)
4. Execution of Unit-Tests
5. automatic Deployment on Tomcat-Server
6. Trigger Functional Test in aqua (TS000265 - Functional Test PetClinic)
7. Results are written to Test Szenario TS000265 -Functional Test PetClinic
8. Detailed results of the single test case can be found here: TC000264 - Petclinic Smoketest
9. Results from Jenkins are written to: TC000266 -Unit test result PetClinic
29
Agents – they make your life easier
• aqua agents manage test tools for you
• They
1. Wait for a task2. They fetch everything they need: test scripts, test data,
files3. Execute a test for you4. Monitor the execution5. Send live updates to the server and you6. They make your life easier
30
31
aqua | Integrations
aqua | Jira
32
Continuous Testing and all is good?
Not so fast...
... flaky tests
33
Tension between QA and development
• Cooperation with developers can be difficult
• Nobody likes to be told they're doing something wrong.
• Classic problem in many companies:
•"Works on my machine"
•"This can't happen."
•"Your test is broken."
34
More Tension!
Example:
• Developer checks in new source code
• Automated tests run on every check-in/build
• Test results show 1 to N failures
• Developer checks his code for errors
Failures from automated tests that are not defects.
Developers spend time searching and debugging just to notice that the test was wrong…
35
The false alarm
Loss of confidence in test results
Loss of confidence in QA
It's difficult to regain lost trust.
36
Flaky Tests!
What are flaky tests? Unreliable tests.
Flaky tests run sometimes green, sometimes red, then sometimes greenagain
… without changes to the SUT or the test scripts
Randomly occurring problems
Flaky tests are poison.
"Unreliable tests are worse than missing tests.”
Analyse them. If they aren’t real bugs: Fix your test automation scripts fast!
37
Test Data
• Don‘t skimp on good test data
• Invest time upfront in a good test data sets & pipeline• Saves a lot of time later!• Script: Write test data to environment • Run tests• Script: Tear down / Clean up used data or environment
How to get good test data?
a) Grab it from production• Secure sensitive parts & anonymize where needed
b) Generate it with dedicated tools• Make your test data smart• A few dynamic parts
• Don‘t underestimate the importance of good, available test data
38
Use agile in your day to day tasks!
• Adopt agile processes in QA
• Organize your QA tasks• Writing new test cases• Reviews• Automating• Updating & maintaining • Managing test environments• Preparing test data• Research new technologies and tools
• Match test cases to user stories & development tasks
39
Picture: aqua scrum board
International Projects?
Screenshots: aqua machine translation 40
41
top related