![Page 1: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/1.jpg)
VT-d Posted Interrupts
Feng Wu, Jun Nakajima <Speaker>
Intel Corporation
![Page 2: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/2.jpg)
2
Agenda
• Motivation • Difference btw CPU-based and VT-d Posted Interrupts
• Architecture
• Implementation Details • Performance
• Summary
![Page 3: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/3.jpg)
3
• Interrupt virtualization efficiency
• Interrupt migration complexity
• Big requirement of host vector for different assigned devices
Motivation
![Page 4: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/4.jpg)
4
CPU-based Posted-Interrupt in Xen
VCPU0 running on
Physical CPU 0
External interrupt
VCPU1 running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR) of VCPU1
• External interrupt handling
VM Exit
Guest IPI: Notification Event
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
Xen
![Page 5: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/5.jpg)
5
Key Data Structures for CPU-based Posted-Interrupt Processing
• Posted-interrupt notification vector: − Send virtual interrupts to guests w/o VM exit − If physical vector == Posted-interrupt
notification vector (VMCS field)
• PIR (Posted-interrupt requests) − Set bits for guest vectors in advance
• ON (Outstanding Notification) − If this bit is set, there is a notification outstanding
for one or more posted interrupts
Virtual-APIC Page
Posted-interrupt notification vector
PI Desc. Address
PIR
Posted-interrupt Descriptor
ON
5
VMCS
![Page 6: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/6.jpg)
6
CPU-based Posted-Interrupt in Xen – cont’d
VCPU1 running on
physical CPU 1
Set bit for guest Vector in
Posted-interrupt requests (PIR) of VCPU1
• Virtual interrupts from QEMU
Guest IPI: Notification Event
Virtual Devices
QEMU Virtual interrupts - Virtual MSI - Virtual IOAPIC
Xen
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
![Page 7: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/7.jpg)
7
What’s new for VT-d Posted-Interrupts
VCPU1 running on
physical CPU 1
Direct-assigned Device
VT-d Engine with
Posted-Interrupt support
External Interrupts
Notification Event
- No need to kick VCPU1 - Notification Event is handled by
VCPU1 in non-root mode - vIRR is updated by H/W and
delivered by “virtual interrupt delivery” mechanism
- VM-Exit is eliminated
No VMM overhead AT ALL!
![Page 8: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/8.jpg)
8
VT-d Posted-Interrupts Architecture Virtual Processor A
VMCS vAPIC Page
Posted Descriptor
CPU 0 CPU 1
Virtual Processor B
VMCS vAPIC Page
Virtual Processor C
VMCS Posted Descriptor
Interrupt Remapping Table
IRTE for Interrupt Z
IRTE for Interrupt Y
IRTE for Interrupt X
IRTE for Interrupt N
External Interrupts
Notification Event
Notification Event
Posted Descriptor
vAPIC Page
Notification Event
CPU 0 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Notification Vector for vCPU B
Notification Vector for vCPU A
Other Vectors (e.g. Timer)
CPU 1 Host-Vectors
Other Vectors (e.g. Hyp. IPIs)
Notification Vector for vCPU C
Other Vectors (e.g. Timer)
![Page 9: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/9.jpg)
9
Xen Implementation Details:
• Update IRET according to guest’s modification to the interrupt configuration (MSI address, data)
• Interrupt migration during VCPU scheduling
![Page 10: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/10.jpg)
10
Xen Implementation: IRTE update
QEMU
Xen
VCPU
Interrupt configuration update (e.g. MSI/MSIx)
Hypercall XEN_DOMCTL_bind_pt_irq!
Update the guest information in IRTE - Guest vector - Posted-interrupt descriptor address
No changes needed for this part
Changes happen here
![Page 11: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/11.jpg)
11
Xen Implementation: VCPU Scheduling
RUNSTATE_runnable RUNSTATE_blocked
RUNSTATE_running
‘SN’ – Set ‘NV’ –Notification vector
‘SN’ – Clear ‘NDST’ – New physical CPU of the VCPU
‘SN’ – Set ‘NV’ –Wakeup vector
‘SN’ – Set ‘NV’ –Notification vector
‘SN’ – Set ‘NV’ –Wakeup vector
PIR [0 - 255]
Posted-Interrupt Descriptor
NV SN
NV – Host vector for Notification Event SN – If set, suppress Notification Event
![Page 12: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/12.jpg)
12
• VT-d Posted-interrupts advantages – External interrupts from direct-assigned devices are delivered to guest
running in non-root mode directly – Improve Interrupt virtualization efficiency, e.g. Less VM-Exits. – Simplify interrupt migration – Consume less physical interrupts
• Performance
• The Specification will be published very soon – Can be found in Intel website
Summary
![Page 14: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/14.jpg)
14
Back up
![Page 15: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/15.jpg)
15
• Interrupt-remap-table-entry (IRTE) enhanced as follows: - An existing reserved bit claimed to indicated Posted-interrupt (PST) - Software may choose to “remap” or “post” each interrupt independently
• IRTEs with ‘PST’ set are interpreted per below format
- New Fields • Descriptor Address: the address of the posted-interrupt descriptor • Virtual Vector: the guest vector of the interrupt • URG: indicates if the interrupt is urgent
- Other fields continue to have the same meaning
VT-d Posted-Interrupts Support
![Page 16: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/16.jpg)
16
Posted-interrupt Descriptor
![Page 17: VT-d Posted Interrupts-final - Linux Foundation Eventsevents17.linuxfoundation.org/sites/events/files/slides/VT-d Posted Interrupts-final .pdf- An existing reserved bit claimed to](https://reader030.vdocuments.us/reader030/viewer/2022040503/5e2b7a42c94517777121e851/html5/thumbnails/17.jpg)
17
VT-d: Steps for Interrupt Posting Read IRTE
Interrupt Remapping
‘PST’ == 1
Read Posted-interrupt descriptor (locking the cache-line)
N
Set guest virtual in PIR
(‘ON’ == 0) &&
(‘URG’ == 1 || ‘SN’ == 0)
Y
Set ‘ON’ and Generate notification event
Unlock cache-line N
Unlock cache-line
Y
Y