qemu development and testing automation using ms hck - anton nayshtut and yan vugenfirer, daynix

QEMU development and testing automation using MS HCK

Anton Nayshtut, [email protected] Yan Vugenfirer, [email protected]

Daynix Computing LTD

mailto:[email protected]

mailto:[email protected]


Agenda

What is WHQL/HCK?

Introducing VirtHCK

Using VirtHCK

Call to action

�2


What is HCK/WHQL?The Windows certification program (previously known as the Windows Logo Program) for devices and system

A comprehensive device\system test toolkit by Microsoft

Devices which pass HCK are certified and their binaries are digitally signed

�3


HCK test kit

Introduced with Windows 8 (previous kit know as WLK)

Predefined test suites for each device type

Ability to add your own tests for additional testing outside of the certification process

Reproducible scenarios

�4


Benefits for QEMU and Linux community

QEMU device testing (not only virtio)

Host subsystems testing, for example networking

Extensive coverage for defined device categories

Regression testing

Stress testing

�5


HCK PCI compliance testDevice under test compliance to PCI spec

Applicable for:

New QEMU chipsets

PCIe devices

Next generation of virtio devices

Existing PCI devices

�6


HCK NDIS (network) tester (1)

Comprehensive networking test suite

Send/receive,

Stress tests (MPE)

Offload, VLAN, priority, packet filtering

etc

Not once used to test the robustness of the host networking stack

�7


HCK NDIS (network) tester (2)Applicable for

QEMU network devices

New devices

New features

Host network components

Bridge

Open vSwitch

TAP

Kernel�8


HCK Windows OS tests

Device path exerciser

Identifies drivers that do not correctly handle the different IO calls

Driver verifier

Captures deadlocks, memory leaks, memory corruption

Verified correctness of kernel API usage

�9


HCK power management tests

The test cycling through different sleep states

There are variations of IO and PNP (disable/enable) during, before and after power transitions

Applicable for:

QEMU Devices power management support

SeaBIOS

�10


Our past experienceTransfer hangs with vhost.

Offload/VLAN related corner cases with network devices (virtio-net, vmxnet3, vmxnet).

vmxnet3 and pvscsi in QEMU

Open vSwitch out of order issues

RSC (Windows GRO)

PCI compliance

�11


HCK - great, but frightening

Cumbersome setup requirements

Complex network configuration

Hard to replicate setups

Setup conflicts

No power management tests automation support

�12


VirtHCK

�13


Introducing VirtHCK

Framework for automating HCK test execution

Based on easily configurable host and guest scripts

Creates isolated virtual network environments

Allows running different HCK scenarios in automated manner

OSS, distributed under BSD license

�14


Repository and WiKi

Get VirtHCK from https://github.com/daynix/bricklets.git

VirtHCK wiki with steps for deployment and usage: https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation

�15

https://github.com/daynix/bricklets.git

https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation


HCK issues solved by VirtHCK

Setup replication

Multiple setups sharing the same host

Setup isolation solves the following issues:

HCK controller controls all HCK clients in the network

Inability to use VM templates without manual VM reconfiguration

Automatic network topology setup during network device tests

�16


Setup types supported

Network (virtio-net, vmxnet3, vmxnet)

Storage (virtio-block, virtio-scsi, pvscsi)

Simple PCI (virtio-serial, virtio-balloon)

Easily extendable by small changes in configuration files

�17


Typical HCK deployment

WHQL control network

Control NIC

Test Client

HCK Controller

Control NIC - e1000

Control NIC

Test Client

Control NIC

Test Client

�18


VirtHCK setup

Disconnected from outside world network during test run

Isolated NICs under test network

Isolated WHQL control network

A different number of test clients can run according to test type

�19


VirtHCK setup for storage and other devices

WHQL control bridge

“World” bridge Physical NIC

Control NIC - e1000

Test VM 1

Studio VM

Outside world NIC

Control NIC - e1000

�20


VirtHCK setup for storage and other devices - running

WHQL control bridge


Control NIC - e1000

Test VM 1

Studio VM

Outside world NIC

Control NIC - e1000

X�21


VirtHCK setup for NICs

Test bridge

WHQL control bridge


Control NIC - e1000

Test VM 1

NIC under test

Control NIC - e1000

Test VM 2

NIC under test

Studio VM

Outside world NIC

Control NIC - e1000

�22


VirtHCK setup for NICs - running

Test bridge

WHQL control bridge


Control NIC - e1000

Test VM 1

NIC under test

Control NIC - e1000

Test VM 2

NIC under test

Studio VM

Outside world NIC

Control NIC - e1000

X�23


VirtHCK - quick startClone VirtHCK from github

Create VM images in /images directory (studio and test client)

Update STUDIO_IMAGE/CLIENT1_IMAGE/CLIENT2_IMAGE variables in hck_setup.cfg

Run VirtHCK - “sudo ./VirtHCK/hck.sh”

Install Microsoft HCK Studio and Clients software according to Microsoft documentation Windows HCK Getting Started and following Checklist for a new client VM guidelines.

�24

http://msdn.microsoft.com/en-us/library/windows/hardware/jj123537.aspx

https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_client_VM


Directory treeVirtHCK/ - Directory of all VirtHCK scripts, supposed to run in Linux environment

VirtHCK/guest_tools/ - Directory for scripts required for Windows guests

Images directory where VM's images should be located

�25


VirtHCK walkthrough - host

Configuration: network, VMs etc

Host scripts

Images

�26


hck_setup.cfg

VirtHCK setup configuration file

Environment configuration

VM start up parameters

Configured accordingly to required HCK setup type

Allows multiple setups on one host

�27


Running multiple setups

Use UNIQ_ID to configure multiple setups

UNIQ_ID will be used as a base for creation of:

VNC or Spice ports

Bridges names

Network interface names, MAC addresses and etc.

�28


Test device types

Network

Storage (also boot)

Balloon

Serial

More can be added

�29


Remote access

VNC or Spice

Spice is needed for power management tests

�30


Network infrastructure

Linux bridge or Open vSwitch can be used

�31


Images

Configure

Studio VM image

Test VMs images

�32


VM resources

Number of vCPUs

Memory

NICs for control channels

�33


Extra parametersQEMU binary

VHOST, snapshots, cache mode for fast install

ISOs for guest installation

BIOS

�34


Configuration summary

�35


Host scripts - hck.sh

Main script to run HCK enviroment

Dumps configuration

Creates network configuration for HCK setup

Runs VMs in loop (crucial during PM tests)

Cleans up all the configurations on exit

�36


VM execution scripts

Executed by hck.sh

run_hck_studio.sh - starts up Studio VM

run_hck_client.sh -starts up Client VM

Configuration according hck_setup.cfg

�37


Network scripts

Passed as parameter to QEMU to add network interface to appropriate bridge

hck_ctrl_bridge_ifup.sh

hck_test_bridge_ifup.sh

hck_world_bridge_ifup.sh

�38


Guests configuration

�39


Studio VM

Setup HCK according to Microsoft’s documentation

Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_studio_VM:

Do not run HCK studio directly

Use run_hck_studio.bat or run_hck_studio_8.1.bat located in guest_tools

The above scripts will disconnect the HCK Studio VM from outside network

�40


Test client VMs

Follow the checklist for test client - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_client_VM:

Install HCK client according to Microsoft’s documentation from \\hck-studio\HCKInstall\Client\setup.exe

�41

Daynix Computing LTD�42


What’s nextHCK automation on the guest

SVVP configuration

Xen support (other hypervisors?)

Adding more device types

Using external configuration

Using libvirt

More guest scripts to automate configuration of test clients

�43


HCK automation on the guest

Goal:

Run full test cycle for specific device type without user interaction.

�44


Adding more device types

Goal

Test all the devices in QEMU and not only virtio (PCI bridges, USB controllers and etc)

�45


More plans

Using external configuration

Multiple setups on one host without replicating VirtHCK and images

Use libvirt

Arbitrary number of test clients on one setup

Xen support

�46


Call to action

Test QEMU devices with VirtHCK (not only virtio devices)

Contribute

�47

Q&A�48


LinksWiKi - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation

Repository - https://github.com/daynix/bricklets.git

HCK kit download - http://msdn.microsoft.com/en-us/windows/hardware/hh833788.aspx

Windows HCK Getting Started - http://msdn.microsoft.com/en-us/library/windows/hardware/jj123537.aspx

Daynix - www.daynix.com

�49

https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation

https://github.com/daynix/bricklets.git

http://msdn.microsoft.com/en-us/windows/hardware/hh833788.aspx

http://msdn.microsoft.com/en-us/library/windows/hardware/jj123537.aspx

http://www.daynix.com

qemu development and testing automation using ms hck - anton nayshtut and yan vugenfirer, daynix

Technology

devices test vm

hck test kit

hck test execution

virthck daynix computing

test compliance

test cycling

hck pci compliance test

hck issues