dynamic adaptivity in support of extreme scale pat teller, utep

9 June 2005 FastOS PI Meeting 1

Dynamic Adaptability in Support of Extreme Scale University of Wisconsin-Madison

University of Texas-El Paso

Dynamic Adaptivity

in Support of Extreme Scale

Pat Teller, UTEP




Outline

• Collaborators

• Overview

• Progress

• Plans




University of Texas at El PasoDepartment of Computer SciencePatricia J. Teller ([email protected])

University of Wisconsin — MadisonComputer Sciences DepartmentBarton P. Miller ([email protected])

International Business Machines, Inc.Linux Technologies CenterBill Buros ([email protected])

U.S. Department of EnergyOffice of ScienceFred Johnson ([email protected])

Partners

Lawrence Berkeley National Laboratory Leonid Oliker ([email protected])new partner




• UTEP team (Pat Teller)– Rodrigo Romero, Ph.D. (Post-doc)– Seetharami Seelam, Ph.D. candidate in CS– Luis Ortiz, Ph.D. candidate in CS– Jayaraman Suresh, Master’s candidate in CS– Brenda Prieto, Master’s candidate in ECE– Nidia Pedregon, Undergraduate in CS– Alejandro Castaneda, Undergraduate in CS

• U. Wisconsin-Madison team (Bart Miller)– Michael Brim, Ph.D. candidate– Igor Grobman, Ph.D. candidate

Teams




Outline

• Collaborators

• Overview– Goal– Challenges– Deliverables– Methodology

• Progress & Direction




Goal:Goal: Enhanced PerformanceEnhanced Performance

Generalized CustomizedCustomized resource management

Fixed Dynamically AdaptableDynamically Adaptable OS/runtime services




Determining

• WhatWhat to adapt

• WhenWhen to adapt

• HowHow to adapt

• HowHow to measure effects of adaptation

Challenges




1. Mechanisms to dynamically sense, analyze, and adjust

• performance metrics• fluctuating workload situations • overall system environment

conditions

Deliverables




2. Linux prototypes and experiments that demonstrate dynamic self-tuning / provisioning in HPC environments

Deliverables




3. Methodology for general-purpose OS adaptation

Deliverables




identify adaptationtargets

characterize workloadresource usage patterns

determine/redetermine feasible adaptation ranges

define/adapt metrics/heuristics to trigger adaptation

generate/adapt monitoring, triggering andadaptation code, and attach it to OS

potentially profitable adaptation targets

Methodology

KernInstmonitor application execution,

assessing performance (gain) and triggering adaptation as necessary

off line

off line/run time




InstrumentationTool

Client

KernInst APIKernInst Device

Linux Kernel

KernInst Daemon

IBM pSeries eServer 690

KernInst

dynamic monitoring, instrumentation, and adaptation of the kernel




Outline

• Collaborators

• Overview

• Progress & Direction– Tools– Infrastructure– Collaboration– Research




Tools Progress• KernInst –

– POWER4 port for Linux 2.4 and IA32 Linux 2.6• Modifications/Enhancements for DAiSES research

– POWER4 port for Linux 2.6 underway

• IOstat (coarse statistics) – POWER4 Linux 2.6

• Investigating complementary use of oprofile and kprobes – POWER4 Linux 2.6




KernInst• Intel IA32 port for Linux 2.4 and 2.6 for Pentium 3 and Pentium 4 processors • IBM POWER4 port for Linux 2.4• Supports stand-alone kernels and kernels that run under the Hypervisor virtual machine

layer• Hypervisor layer not transparent and requires explicit support• Little public documentation on this layer available




Outline

• Collaborators

• Overview





Infrastructure ProgressIBM SUR Grants, UTEP Star Award, UTEP PUF Funds

• Development of Experimental Platforms at UTEP– IBM eServer pSeries 690 (16 processors, 32GB, 2TB)

• Linux 2.4/2.6 partition for KernInst development• Linux 2.4 partition for DAiSES research• Linux 2.6 partition for DAiSES research• DS4300 RAID for DAiSES I/O-related research (1TB)

– Xeon workstations – Linux 2.4 and 2.6– IBM eServer p590 (24 processors, 64GB, 2TB)– IBM eServer p550 (4 processors, )

• Establishment of DAiSES Lab at UTEP




Outline

• Collaborators

• Overview





Build/Strengthen Collaborations - 1• April 14-16, 2004 – Seetharami Seelam, UTEP, attended

Paradyn/Condor week• October 24, 2004 – Barney MacCabe, UNM, visited UTEP to meet

with the DAiSES team and give a talk re: his team’s FASTOS research

• November 10, 2004 – Pat Teller, UTEP, participated in the FASTOS Birds-of-a-Feather meeting at SC2004 – this was the first public presentation of the DAiSES project

• November 2004 – Rodrigo Romero, Seetharami Seelam, and Pat Teller, UTEP, and Michael Brim, Igor Grobman, and Bart Miller, UW-Madison, promoted the DAiSES project at two SC2004 research exhibits, one shared by UTEP, UNM, New Mexico State University, and New Mexico Institute of Technology, and another of UW-Madison




Build/Strengthen Collaborations - 2• February 25, 2005 – Luis Ortiz, Rodrigo Romero, Seetharami

Seelam, and Pat Teller, UTEP, attended a half-day meeting at IBM-Austin with approximately nine members of the Linux Technologies team

• March 3, 2005 –Rodrigo Romero, Seetharami Seelam, and Pat Teller, UTEP, attended an all-day meeting at UNM with Barney Maccabe, Patrick Bridges, Kurt Ferreira, an Edgar Leon, UNM/CS, Orran Krieger, IBM, Ron Brightwell and Rolf Riesen, SNL, and Rod Oldehoeft, LANL/ACL

• March 20-24, 2005 – Igor Grobman, UW-Madison, visited UTEP and led a workshop re: the use of KernInst and Kperfmon to implement adaptations, in particular, in the process scheduler and I/O scheduler; Bill Buros, IBM-Austin, senior member of Linux Technologies team, attended for three days – resulted in modifications/enhancements to KernInst




Build/Strengthen Collaborations - 3• May 4, 2005 – after IBM Petaflops Tools Strategy Workshop at IBM

TJ Watson Research Center, Bart Miller, UW-Madison, and Seetharami Seelam (awarded $500 to attend the workshop) and Pat Teller, UTEP, met with Evelyn Duesterwald and Robert Wisniewski re: possible collaborations

• Weekly telecons with IBM-Austin team• Telecons and Access Grid meetings with UW-Madison team• Shared Enotebook to be launched shortly• Shared data repository with search tool to be launched shortly




Outline

• Collaborators

• Overview





Current Research Thrusts

• Dynamic Code Optimization• Low-hanging Fruit (i.e., opportunitistic

targets of adaptation)• Identification of Adaptation Targets via Self-

propelled Instrumentation• Other Directions




Dynamic Code Optimization• Investigation of dynamic code optimization strategies

– [Tamches and Miller] used dynamic reorganization of basic block layout in parts of SPARC Solaris kernel to improve performance via I-cache miss reduction

• Discussion in research community asks if such optimizations are – workload dependent and need to be done dynamically

or – mostly independent of workload and can be done

statically• Goal: to provide conclusive evidence either way




Low-hanging Fruit

I/O scheduling – extend work of IBM-Austin

process scheduling – extend published observations and address daemon controlvirtual memory management –

extend dissertation work page size – extend work of IBM TJ Watson Research Ctr.

I/O scheduler parameter selection via neural networks – extend work of IBM-Austin that uses genetic algorithms




I/O Scheduling (in progress)• WhatWhat to adapt: I/O scheduler

• Dynamic selection of “appropriate” I/O scheduler for observed “system state”

• WhenWhen to adapt• Change in “system state” (now identified via IOstat)• Below threshold related to number of queued I/O

requests• HowHow to adapt

• Linux 2.6 provides capability• I/O schedulers characterized w.r.t. “best” performance

for different “system states” [Pratt and Heger]• HowHow to measure effects of adaptation

• Execution time and throughput MB/s (for now)




I/O Scheduling By-products - 1

Enhancements to Linux I/O Scheduling,” to appearEnhancements to Linux I/O Scheduling,” to appearin in Proceedings of the Linux SymposiumProceedings of the Linux Symposium, Ottawa, , Ottawa, Canada, July 2005 (S. Seelam, R. Romero, P. Canada, July 2005 (S. Seelam, R. Romero, P. Teller, and W. Buros)Teller, and W. Buros)

• Reviews previous work of IBM-Austin characterizes Reviews previous work of IBM-Austin characterizes workloads best served by each of the four Linux 2.6 I/O workloads best served by each of the four Linux 2.6 I/O schedulers (can be selected at boottime or runtime)schedulers (can be selected at boottime or runtime)

• Presents cases where the Anticipatory Scheduler (AS) Presents cases where the Anticipatory Scheduler (AS) results in process starvation results in process starvation




I/O Scheduling By-products - 1 cont’d.

• Presents and demonstrates performance of UTEP CAS, Cooperative Anticipatory SchedulerCooperative Anticipatory Scheduler• extends anticipation to “cooperative” processes that extends anticipation to “cooperative” processes that

collectively issue synchronous requests to a close set collectively issue synchronous requests to a close set of disk blocksof disk blocks

• compares performance to current four schedulers compares performance to current four schedulers • shows order of magnitude performance improvement shows order of magnitude performance improvement

in cases where AS performs poorlyin cases where AS performs poorly




I/O Scheduling By-products - 2

In progress: demonstration of heuristically-guideddynamic selection of Linux 2.6 I/O schedulers (target: FAST)

• First step towards making I/O scheduling fully autonomic First step towards making I/O scheduling fully autonomic (Ph.D. dissertation topic: Seetharami Seelam)(Ph.D. dissertation topic: Seetharami Seelam)

• Selection based on observed system behavior, i.e., Selection based on observed system behavior, i.e., system (workload) I/O behavior, metric, in particular, I/O system (workload) I/O behavior, metric, in particular, I/O request sizerequest size




I/O Scheduling By-products - 2 cont’d.

• Using Using a prioria priori measurements of disk throughput under measurements of disk throughput under the various schedulers and request sizes to generate a the various schedulers and request sizes to generate a function that at runtime, given the current average function that at runtime, given the current average request size, returns the scheduler that gives the best request size, returns the scheduler that gives the best measured throughput for the specified diskmeasured throughput for the specified disk

• Identying adaptation interval, i.e., when it is not too Identying adaptation interval, i.e., when it is not too expensive to switch schedulers–based on number of expensive to switch schedulers–based on number of queued I/O requestsqueued I/O requests

• Future work: UW-Madison team will use KernInst to Future work: UW-Madison team will use KernInst to effect the adaptationeffect the adaptation




0

2

4

6

8

10

12

14

1 2 4 8 16 32 64 128 256 512 1024

Number of Requests

Ove

rhea

d (1

000

ms)

.

Deadline to AS

AS to Deadline

Time to Drain for Default nr_requests

Overhead of Draining I/O Queue




Comparison of Different I/O Schedulers on RAID-0

0

9

18

27

36

45

1KB 4KB 16KB 32KB 65KB 256KB 1MB

Read Size

Ban

dw

idth

(M

B/s

)

. CFQ

AS

Deadline

NOOPPoints of interests

AS

Deadline

CFQ

Microbenchmark Synchronous Reads





1

2

3

4

5

6

1KB 4KB 16KB 32KBRead Size

Ban

dw

idth

(MB

/s) .

CFQ

AS

Deadline

NOOP

30% difference between the best and the worst performing scheduler

Microbenchmark Synchronous Reads ZOOMED 1





0

5

10

15

20

25

30

35

40

45

50

32KB 65KB 256KB 1MB

Read Size

Bandw

idth

(M

B/s

) .

CFQ

AS

Deadline

NOOP

>20% difference between the best and the

worst performing scheduler

Microbenchmark Synchronous Reads ZOOMED 2





0

9

18

27

36

45

54

1KB 4KB 16KB 64KB 256KB 1MB

Read Size

Ban

dw

idth

(M

B/s

) .

CFQ

AS

Deadline

NOOP

MicrobenchmarkWrites




Linux Compilation Disk Accesses

Histogram of Size of Linux Object Files

0

500

1000

1500

2000

2500

Size

Nu

mb

er o

f F

iles

.

Histogram of the Size of Linux Source files

0

1000

2000

3000

4000

5000

6000

0K 1K 2K 4K 8K 16K 32K 64K 128K 256K 512K 1M 2M

Size

Nu

mb

er o

f F

iles

.


0

9

18

27

36

45

54

1KB 4KB 16KB 65KB 256KB 1MB

Read Size

Ban

dw

idth

(M

B/s

)

.

CFQ

AS

Deadline

NOOP

Points of interests

Deadline

CFQ


0

9

18

27

36

45

1KB 4KB 16KB 32KB 65KB 256KB 1MB

Read Size

Ban

dw

idth

(M

B/s

)

. CFQ

AS

Deadline

NOOPPoints of interests

AS

Deadline

CFQ




• Dynamic Adaptation– Uses IOstat information, which is after the fact– Read/Write prediction uses two-bit saturating

counter– For reads: size 1K - 32K use Deadline

size > 32K use AS

– For writes: size 1K - 64K use AS

size > 64K use Noop




Read/Write Patterns while compiling Linux source

0

1

1 26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 401 426 451 476 501 526 551 576 601 626 651 676 701 726 751 776 801 826 851

Time (sec)

Read

/Wri

te

Linux Compilation Read/Write Access




• Performance comparison of Linux Compilation (gmake –j 16; xeon 2.78GHz; source on RAID-0 with 4 IDE drives)

Scheduler Time (Sec)

AS 830

Deadline 850

CFQ 860

Noop 848

Adaptive 843

preliminary results – inconclusive




Lessons Learned• 80 scheduler switches• Switching is not “costing our life”• Switching has to be on a coarser granularity

– “Smaller” picture (focus: requests) is captured– “Bigger” picture (focus: workloads) needs to be

captured– Prediction does not have to be per-request– Example desired prediction: database workload

followed by streaming reads – should select deadline/cfq followed by AS




Applications/Benchmarks used for

I/O Scheduling Research - 1 • Flexible File System Benchmark – FFSB bench

– used to generate profile-driven, I/O workload with a characteristic I/O access pattern of a given type of server, e.g., web, file, email servers, and MetaData server.

• Microbenchmarks– Streaming writes and chunk reads– Streaming reads and chunk reads– Chunk reads




Applications/Benchmarks used for I/OScheduling Research - 2

• MADbench: I/O intensive (Borril, et al.)– based on MADCAP, an application for estimating the power

spectrum of cosmic microwave background radiation– retains computational intensity, operational complexity, and

system requirements of MADCAP and implements its three main processing steps1. builds signal correlation derivative matrices and it requires neither

read operations nor communication2. builds a subset of a data correlation matrix and then inverts it; does

not require writes3. reads a subset of the signal correlation matrices built in the first step

and performs matrix multiplication against the inverted matrix obtained in step two; does not require writes




Process Scheduling (stalled)

Several references indicate that the Linux 2.4 scheduling policy has an adverse effect on non-real-time, non-interactive applications—a description that fits the high-performance applications




Process Scheduling• WhatWhat to adapt: process scheduler

• Dynamic selection of “appropriate” process scheduler for observed “system state”

• WhenWhen to adapt• Change in “system state” • Below threshold related to time spent in scheduler or

length of “runnable” queue• HowHow to adapt

• Change process type to real-time, i.e., scheduler to round-robin

• A priori knowledge of process IDs• HowHow to measure effects of adaptation

• Execution time




Too much time in scheduler

Adaptation to round-robin

Hackbench 2.4 execution times for unmodified (blue) and adapted scheduler

0

20

40

60

80

100

120

140

160

20 40 60 80 100 120 140 160 180 200 220

Groups

seco

nd

s

2.4 Baseline

2.4 Queue Lengthadaptation

2.4 Time in sched.adaptation

Adaptation: timeshared to real time (round-robin) scheduling with fixed quantum and priority per process for the lifetime of the application




Hackbench Speedup wrt 2.4 unmodified kernel

0.000.501.001.502.002.503.00

Groups

2.4 Q. Length

2.4 Time in sched.




Learning-based, Heuristic-driven Dynamic Adaptation (longer term)

• I/O scheduler parameter selection• IBM-Austin: genetic algorithms• UTEP: investigate potentially lower cost hybrid

techniques that combine the use of neural networks, genetic algorithms, and fuzzy logic (possible dissertation topic: Luis Ortiz, UTEP)

• Master’s thesis: neural network approach to selecting parameters of the Anticipatory Scheduler of Linux 2.6 [Moilanen]




Near-term Research Directions – 1

• Adaptation target investigation via self-propelled instrumentation to obtain function-level traces from applications and the kernel [Mirgorodskiy and B. Miller]– Possible applications:

• LBMHD (plasma physics), PARATEC (material science), CACTUS (astrophysics), and GTC (magnetic fusion), which are able to fully utilize the performance of machines comparable to the Earth Simulator and Cray X1 [Oliker, et al.]

• Sweep3D (using for daemon control investigation)• SPECjAppServer2004 (Websphere, DB2)




Near-term Research Directions – 2

• Study of applications to determine profitability of – Dynamically adapting page size [Cascaval, et al.] – Daemon control (started) [D. Bailey and Hoisie, et al.]– Virtual memory management (just starting)– Dynamically adapting time quantum to control, e.g.,

process cache contention

• k-factor analysis to develop mathematical models to guide I/O parameter set selection




References – 1• Bailey, D., Private Communications, 2005.• Borril, J., J. Carter, L. Oliker, D. Skinner, and R. Biswas, “Integrated

Performance Monitoring of a Cosmology Application on Leading HEC Platforms,” Proceedings of the 2005 International Conference on Parallel Processing (ICPP-05), June 2005.

• Cascaval, C., E. Duesterwald, P. Sweeney, and R. Wisniewski, “Multiple Page Size Modeling and Optimization,” Proceedings of the Fourteenth International Conference on Parallel Architectures and Compilation Techniques (PACT-2005), September 2005.

• http://www.sourceforge.net/projects/ffsb




References – 2• Hoisie, A., D. Kerbyson, S. Pakin, F. Petrini, H. Wasserman, and J.

Fernandez-Peinador, “Identifying and Eliminating the Performance Variability on the ASCI Q Machine,” Technical Report LA-UR-03-0138, Performance and Architecture Lab, Los Alamos National Laboratory, January 2003.

• Mirgorodskiy, A., and B. Miller, “Autonomous Analysis of Interactive Systems with Self-propelled Instrumentation,” Proceedings of the Multimedia Computing and Networking Conference, 2004.

• Moilanen, J., “Genetic Algorithms in the Kernel,” http://kernel.jakem.net/, 2005.

• Oliker, L., A. Canning, J. Carter, J. Shalf, and S. Ethier, S. “Scientific Computations on Modern Parallel Vector Systems,” Proceedings of the 2004 ACM/IEEE Conference on Supercomputing, November 2004.




References – 3• Pratt, S., and D. Heger, IBM-Austin, “Workload Dependent

Performance Evaluation of Linux 2.6 I/O schedulers,” Linux Symposium, 2, July 2004, pp. 425-448.

• Tamches, A., and B. Miller, “Dynamic Kernel I-Cache Optimization,” Proceedings of the Workshop on Binary Translation, September 2001.

dynamic adaptivity in support of extreme scale pat teller, utep

Documents