External guide– Internal guide-Mr. Kedar Sovani Prof Mukta TakalikarDREAMZ Group. PICT,Pune.

Group members–

Aditya Gadre B3054204 Keshav Darak B3054243 Kaustubh Kabra B3054279 Ashwin Vasani B3054365

HUGEPAGE SUPPORT IN LINUX BASED VIRTUALIZATION

1

PROBLEM DEFINITION

To design a memory management unit (MMU) based paging mechanism in Xen hypervisor which will add support for huge pages (2MB) such that 4 KB pages and huge pages will now co-exist for a paravirtualized domain kernel to improve performance.

2

3

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

HUGE PAGE

4

A.Huge page translation reduces the normal page walk by 1 level.

B. The Huge page TLB cache improves the performance by increasing the speed by15% to 20%.

 

PGD PMD Page OffsetPTE

PGD Page OffsetPTE

4 KB pageSingleton

2 MB pageHuge Page

21

3

0PS 1

Virtual address

Virtual address

Physical address

Physical address

3 memory acccess

additional to reach to a

byte

Only 2 memory access

additional to reach to a

byte

5

HUGE PAGE

6

A. Huge page translation reduces the normal page walk by 1 level.

B.The Huge page TLB cache improves the performance by increasing the speed by15% to 20%.

 

7

TLB Cache

512 dispersed 4 KB pages

2 MB512entries

1 entry

RAM

TLB Scenario VA PA

Bare Metal VS XEN

8

Hardware Hardware

XEN ??

9

10

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

11

12

13

14

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

DOMU(1)

DOM U(2)

4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB

Hypervisor

RAM

Requesting 4KB

Requesting 2MB

4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB4KB 4KB 4KB 4KB 4KB 4KB 4KB 4KB

Memory Request denied

15

16

PROBLEM DEFINITION

To design a memory management unit (MMU) based paging mechanism in Xen hypervisor which will add support for huge pages (2MB) such that 4 KB pages and huge pages will now co-exist for a paravirtualized domain kernel to improve performance.

17

18

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

19

• VA : Virtual Address in Kernel• PFN : Physical Frame Number• MFN : Machine Frame Number• P2M Mapping: Physical to Machine Mapping

Hypervisor

2 4 1 5 6 3

1 2 3 4 5 6 1 2 3 4 5 6

3 2 4 1 1 4 5 6 1 6 5 3

VA

PFN

VA_to_PFNVA_to_PFN

P2M

3 4 6 1 5 2 1 3 4 2 6 5

RAM

Domain Kernel Domain Kernel

MFN

Kernel CONCEPTS

DOMU(1)

Hypervisor

RAM

Requesting 2MB

Boot time reservatio

n of memory

IMPLEMENTATION OVERVIEW

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

4KB

2MB 2MB 2MB 2MB2MB

Requesting 4KB

4KB

Implementation steps :

1. Domain creation

2. Domain booting

3. Demand supply

22

1. Domain creation

2. Domain booting

3. Demand supply

Implementation steps :

Xen

MFNs

RAM

23

memory=900----------------------

----------------------

----------------------

hugepage_num=50

Domain’s Configstruct domain{

hugepage_num; hugepage_list;

};Domain Create

50

Xen’s Allocator

MFNs

Domain creation

24

1. Domain creation

2. Domain booting

3. Demand supply

Implementation steps :

25

Kernel Booting

Xen Hypervisor

Hypercall toGet hugepage count

struct domain{

hugepage_num; hugepage_list;};

50

PFNs

xen_extra_mem_start

800M 900M

Domain bootingKernel

26

1. Domain creation

2. Domain booting

3. Demand supply

Implementation steps :

27

Kernel

Xen Hypervisor

PFNs

xen_extra_mem_start

800M 900M

Hugepage Request

Xen

MFNs

RAM

struct domain{

hugepage_num; hugepage_list;};

Hypercall toAllocate hugepage MFN

MFN

PFN

P2M

DemandKernel

DOMU(1)

DOM U(2)

Hypervisor

RAM

Requesting 4KB

Requesting 2MB

2MB 2MB 2MB 2MB2MB 2MB 2MB 2MB2MB 2MB 2MB 2MB2MB 2MB 2MB 2MB

4K

4K

4K

4K

4K 2MB 2MB 2MB

2MB 2MB 2MB 2MB

FINAL VISION(Transparent Huge Pages)

28

29

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

-AMAZON

DEMO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

30

Threads =4 Threads =6

Threads =8 Threads =10

PERFORMANCE GRAPHS

X axis: Memory read and wrote (in MBs)Y axis: Time required (in seconds)

31

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

Challenge 1 :

Paging DOCUMENTATION

Xen Hypervisor

2 1 2 1

1 2 1 2

2 1 2 3 1 3

1 2 3 1 2 3

Read-Only Access to Guest Page Tables

Read-Only Access to Guest Page Tables

Challenge 2 : Handling read-only ptes in guest domains

33

2 1 11 1 3 4 8 7

Xen Hypervisor

1 2 3 4 7 8

MFN

PFN

512 aligned pfns for Huge Pages

Unaligned non- contiguous 512MFNs

Domain Kernel

Challenge 3 (1/2): Aligned PFNs and unaligned MFNs

34

2MB Boundary

Xen Hypervisor

MFN

PFN

1 2 5 8 5 7

2 1 11 1 5 8 9

Aligned (512 )MFN

Unaligned un-contiguous PFN

Domain Kernel

Challenge 3 (1/2): Aligned PFNs and unaligned MFNs

35

2MB Boundary

Hypervisor can complete request successfully

MFN

PFN

2 1 11 1 2 8 9

Aligned (512 )MFN

Domain Kernel

1 2 3 4 7 8

512 aligned pfns for Huge Pages

Our solution: Aligned PFNs and Aligned MFNs

36

2MB Boundary

2MB Boundary

37

X-XEN

ARCHITECTURE

REVIEWBUSINESS

VALUE

CURRENT SCENARIO

IMPLEMENTATION

PERFORMANCE

APPLICATIONS & FUTURE

ENHANCEMENTS

CHALLENGES

APPLICATIONS

38

FUTURE ENHANCEMENTS• Existing Live Migration implementations

need to be compatible with this implementation.

• Support for 1GB huge pages.

39

REFERENCES• http://linuxgazette.net/155/krishnakumar.html : HugeTLB -

Large Page Support in the Linux Kernel

• “Memory Management in Linux” by Abhishek Nayani, Mel Gorman & Rodrigo

• “XEN Memory Management (Intel IA-32)” by Andr´es Krapf.

• “Memory Overcommit without the commitment” by Dan Magenheimer, Oracle Corp. Xen Summit, June 2008

• “Xen and art of virtualization” By P. Barham, B.Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. In Proc. SOSP, 2003 40

CONTRIBUTION Patch submitted for review to Jeremy Fitizhardinge, Keir

Fraser and Xen-Devel community http://tinyurl.com/HugePagea2k2Patch1

www.github.com/a2k2/ Kernel-PV-hugepages Xen-hugepages

XEN memory: Enlightening memory management in XEN - Team a2k2 www.ashwinvasani.wordpress.com

Summarized documentation of the project www.xxen-docs.co.cc 41

AWARDS

PROJECT @ INTECHXICATION ,MIT

PAPER @ EXCELSIOR ,Sinhagad

PROJECT @ TESLA ,MITCOE

PAPER @ TESLA ,MITCOE

PROJECT @ BHARTEEYAM ,BVP

PAPER @ BHARTEEYAM ,BVP

PAPER @ TECHKRITI , IIT-Kanpur

1

23

POSTER @ TESLA ,MITCOE

PROJECT @ ZION ,DY PATIL

POSTER @ ZION ,DY PATIL

42PROJECT @ IMPETUS ,PICT

• 1st in Systems • 2nd in Open

Software• PLUG trophy

PAPER PRESENTED @

OTTAWA LINUX SYMPOSIUM

CONCEPTS-2011

ENLIGHTENERS• Kedar Sovani (Marvell & Former Maintainer of

Fedora- ARM)

• Prof. Mukta Takalikar (PICT)

• Jeremy Fitizhardinge (PV-Kernel maintainer)

• Keir Fraser (Xen maintainer)

• Dan Magenheimer (Tmem maintainer)

• Tim Deegan (Citrix)43

THANK YOU.

44

?