advanced operating systems #1 - pflab...general information a bit of linux scheduler history from...
TRANSCRIPT
Advanced Operating Systems
#5
Shinpei Kato
Associate Professor
Department of Computer Science
Graduate School of Information Science and Technology
The University of Tokyo
<Slides download>http://www.pf.is.s.u-tokyo.ac.jp/class.html
Course Plan
• Multi-core Resource Management
• Many-core Resource Management
• GPU Resource Management
• Virtual Machines
• Distributed File Systems
• High-performance Networking
• Memory Management
• Network on a Chip
• Embedded Real-time OS
• Device Drivers
• Linux Kernel
Schedule1. 2018.9.28 Introduction + Linux Kernel (Kato)
2. 2018.10.5 Linux Kernel (Chishiro)
3. 2018.10.12 Linux Kernel (Kato)
4. 2018.10.19 Linux Kernel (Kato)
5. 2018.10.26 Linux Kernel (Kato)
6. 2018.11.2 Advanced Research (Chishiro)
7. 2018.11.9 Advanced Research (Chishiro)
8. 2018.11.16 (No Class)
9. 2018.11.23 (Holiday)
10. 2018.11.30 Advanced Research (Kato)
11. 2018.12.7 Advanced Research (Kato)
12. 2018.12.14 Advanced Research (Chishiro)
13. 2018.12.21 Linux Kernel
14. 2019.1.11 Linux Kernel
15. 2019.1.18 (No Class)
16. 2019.1.25 (No Class)
Process Scheduling
Abstracting Computing Resources
/* The case for Linux */
Acknowledgement:Prof. Pierre Olivier, ECE 4984, Linux Kernel Programming, Virginia Tech
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
General informationScheduling
� Scheduler: OS entity that decide which process should run,
when, and for how long
� Multiplex processes in time on the processor: enables
multitasking
.., Gives the user the illusion that processes are executing at the same
time
� Scheduler is responsible for making the best use of the resource
that is the CPU time
� Basic principle:.., When in the system there are more ready-to-run processes than
the number of cores.., The scheduler decides which process should run
General informationMultitasking
� Single core: gives the illusion that multiple processes are running
concurrently
� Multi-cores: enable true parallelism
� 2 types of multitasking OS:.., Cooperative multitasking
.., A process does not stop running until it decides to do so (yield the
CPU).., The operating system cannot enforce fair scheduling
- For example in the case of a process that never yields
.., Preemptive multitasking
.., The OS can interrupt the execution of a process: preemption
.., Generally after the process expires its timeslice
.., And/or based on tasks priorities
General informationA bit of Linux scheduler history
� From v1.0 to v2.4: simple implementation
.., But it did not scale to numerous processes and processors
� V2.5 introduced the O(1) scheduler.., Constant time scheduling decisions
.., Scalability and execution time determinism
.., More info in [5]
.., Issues with latency-sensitive applications (Desktop computers)
� O(1) scheduler was replaced in 2.6.23 by what is still now the standard Linux scheduler:
.., Completely Fair Scheduler (CFS)
.., Evolution of the Rotating Staircase Deadline scheduler [2, 3]
General informationScheduling policy - I/O vs compute-bound tasks
� Scheduling policy are the set of rules determining the choices
made by a given model of scheduler
� I/O-bound processes:.., Spend most of their time waiting for I/O: disk, network, but also
keyboard, mouse, etc..., Filesystem, network intensive, GUI applications, etc..., Response time is important
.., Should run often and for a small time frame
� Compute-bound processes:.., Heavy use of the CPU
.., SSH key generation, scientific computations, etc.
.., Caches stay hot when they run for a long time
.., Should not run often, but for a long time
General informationScheduling policy - Priority
� Priority
.., Order process according to their ”importance” from the scheduler
standpoint.., A process with a higher priority will execute before a process with a
lower one
� Linux has 2 priority ranges:.., Nice value: ranges from -20 to +19, default is 0
.., High values of nice means lower priority
.., List process and their nice values with p s ax - o p i d , n i , c m d
.., Real-time priority: range configurable (default 0 to 99).., Higher values mean higher priority.., For processes labeled real-time.., Real-time processes always executes before standard (nice)
processes.., List processes and their real-time priority using
p s ax - o p i d , r t p r i o , c m d
General informationScheduling policy - Priority (2)
� User space to kernel priorities mapping:
General informationScheduling policy - Timeslice
� Timeslice (quantum):.., How much time a process should execute before being preempted.., Defining the default timeslice in an absolute way is tricky:
.., Too long → bad interactive performance
.., Too short → high context switching overhead
� Linux CFS does not use an absolute timeslice.., The timeslice a process receives is function of the load of the
system.., it is a proportion of the CPU
.., In addition, that timeslice is weighted by the process priority
� When a process P becomes runnable:
.., P will preempt the currently running process C is P consumed a
smaller proportion of the CPU than C
General informationScheduling policy - Policy application example
� 2 tasks in the system:.., Text editor: I/O-bound, latency sensitive (interactive).., Video encoder: CPU-bound, background job
� Text editor:.., A. Needs a large amount of CPU time
.., Does not need to run for long, but needs to have CPU time available
whenever it needs to run
.., B. When ready to run, needs to preempt the video encoder
.., A + B = good interactive performance
� On a classical UNIX system, needs to set a correct combination of
priority and timeslice
� Different with Linux: the OS guarantee the text editor a specific
proportion of the CPU time
General informationScheduling policy - Policy application example (2)
� Imagine only the two processes are present in the system and run at the same priority
.., Linux gives 50% of CPU time to each
� Considering an absolute timeframe:.., Text editor does not use fully its 50% as it often blocks waiting
for I/O.., Keyboard key pressed
.., CFS keeps track of the actual CPU time used by each program
.., When the text editor wakes up:.., CFS sees that it actually used less CPU time than the video
encoder.., Text editor preempts the video encoder
General informationScheduling policy - Policy application example (3)
� Good interactive performance
� Good background, CPU-bound performance
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
Linux Completely Fair SchedulerScheduling classes
� CPU classes: coexisting CPU algorithms
.., Each task belongs to a class
� CFS: SCHED OTHER, implemented in k e r n e l / s c h e d / f a i r . c
� Real-time classes: SCHED RR, SCHED FIFO, SCHED DEADLINE
.., For predictable schedule
� s c hed c l a s s datastructure:
1
23
4
56
7
89
10
s t r u c t s c h e d _ c l a s s {
v o i d ( * e n q u e u e _ t a s k ) ( / * . . . * / ) ;
v o i d ( * d e q u e u e _ t a s k ) ( / * . . . * / ) ;
v o i d ( * y i e l d _ t a s k ) ( / * . . . * / ) ;v o i d ( * c h e c k _ p r e e m p t _ c u r r ) ( / * . . . * / ) ;
s t r u c t t a s k _ s t r u c t * ( * p i c k _ n e x t _ t a s k ) ( / * . . . * / ) ;
vo id ( * s e t _ c u r _ t a s k ) ( / * . . . * / ) ;v o i d ( * t a s k _ t i c k ) ( / * . . . * / ) ;/ * . . . * /
}
Linux Completely Fair Schedulers c h e d c l a s s hooks
� Functions descriptions:.., enqueue t a s k ( . . . )
.., Called when a task enters a runnable state.., dequeue t a s k ( . . . )
.., Called when a task becomes unrunnable.., y i e l d t a s k ( . . . )
.., Yield the processor (dequeue then enqueue back immediatly).., ch eck p r eemp t c u r r ( . . . )
.., Checks if a task that entered the runnable state should preempt the
currently running task.., p i c k n e x t t a s k ( . . . )
.., Chooses the next task to run.., s e t c u r r t a s k ( . . . )
.., Called when the currentluy running task changes its scheduling class
or task group to the related scheduler.., t a s k t i c k ( . . . )
.., Called regularly (default: 10 ms) from the system timer tickhandler, might lead to context switch
Linux Completely Fair SchedulerUnix scheduling
� Classical UNIX systems map priorities (nice values) to absolute
timeslices
� Leads to several issues:.., What is the absolute timeslice that should be mapped to a
given nice value?.., Sub-optimal switching behavior for low priority processes (small
timeslices)
.., Relative nice values and their mapping to timeslices.., Nicing down a process by one can have very different effects
according to the tasks priorities
.., Timeslice must be some integer multiple of the timer tick.., Minimum timeslice and difference between two consecutive
timeslices are bounded by the timer tick frequency
Linux Completely Fair SchedulerFair scheduling
� Perfect multitasking:.., From a single core standpoint
.., At each moment, each process of the same priority has received an
exact amount of the CPU time.., What we would get if we could run n tasks in parallel on the CPU
while giving them 1/n of the CPU processing power → not possible
in reality.., Or if we could schedule tasks for infinitely small amounts of time
→ context switch overhead issue
� 3 main (high-level) CFS concepts:
1
2
3
CFS runs a process for some times, then swaps it for the runnable
process that has run the least
No default timeslice, CFS calculates how long a process should run
according to the number of runnable processes
That dynamic timeslice is weighted by the process priority(nice)
Linux Completely Fair SchedulerFair scheduling (2)
� Targeted latency : period during which all runnable processes
should be scheduled at least once
� Example: processes with the same priority
� Example: processes with different priorities
� Minimum granularity : floor at 1 ms (default)
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
CFS implementation
� 4 main components:
1
2
3
4
Time accounting
Process selection
Scheduler entry point (calling the scheduler)
Sleeping & waking up
CFS implementationTime accounting
� s c h e d e n t i t y structure in the t a s k s t r u c t (se field)
e x e c _ s t a r t ;
sum_exec_ run t i me ;
v r u n t i m e ;
p r ev_sum_exec _ run t i me ;
/ * a d d i t i o n a l s t a t i s t i c s no t shown here * /
1 s t r u c t s c h e d _ e n t i t y
2 {
3 s t r u c t l o a d _ we i gh t l o a d ;
4 s t r u c t rb_node r u n _ n o d e ;
5 s t r u c t l i s t _ h e a d gro u p _ n o d e ;
6 uns igned i n t o n _ r q ;
78 u64
9 u64
10 u64
11 u64
12
1314 }
� Virtual runtime
.., How much time a process has been executed (ns)
CFS implementationTime accounting (2)
1
2
3
45
6
78
9
1011
12
1314
15
1617
18
s t a t i c vo id u p d a t e _ c u r r ( s t r u c t c f s _ r q * c f s _ r q )
{
s t r u c t s c h e d _ e n t i t y * c u r r =
c f s _ r q - > c u r r ;u64 now = r q _ c l o c k _ t a s k ( r q _ o f ( c f s _ r q ) ) ;
u64 d e l t a _ e x e c ;
i f ( u n l i k e l y ( ! c u r r ) )
r e t u r n ;
d e l t a _ e x e c = now - c u r r - > e x e c _ s t a r t ;
i f ( u n l i k e l y ( ( s 6 4 ) d e l t a _ e x e c <= 0 ) )
r e t u r n ;
c u r r - > e x e c _ s t a r t = now;
s c h e d s t a t _ s e t ( c u r r - > s t a t i s t i c s . e x e c _ m a x ,
m a x ( d e l t a _ e x e c , c u r r - > s t a t i s t i c s
. e x e c _ m a x ) ) ;
18
19
2021
22
23
2425
26
2728
29
30
31
3233
34
cu r r ->sum_ex ec_ ru n t ime += d e l t a _ e x e c ;
s c h e d s t a t _ a d d ( c f s _ r q - > e x e c _ c l o c k ,d e l t a _ e x e c ) ;
c u r r - > v r u n t i m e += c a l c _ d e l t a _ f a i r (
d e l t a _ e x e c , c u r r ) ;u p d a t e _ m i n _ v r u n t i m e ( c f s _ r q ) ;
i f ( e n t i t y _ i s _ t a s k ( c u r r ) ) {
s t r u c t t a s k _ s t r u c t * c u r t a s k
= t a s k _ o f ( c u r r ) ;
t r a c e _ s c h e d _ s t a t _ r u n t i m e ( c u r t a s k ,
d e l t a _ e x e c , c u r r - > v r u n t i m e ) ;c p u a c c t _ c h a r g e ( c u r t a s k , d e l t a _ e x e c ) ;
a c c o u n t _ g r o u p _ e x e c _ r u n t i m e ( c u r t a s k ,
d e l t a _ e x e c ) ;}
a c c o u n t _ c f s _ r q _ r u n t i m e ( c f s _ r q ,
d e l t a _ e x e c ) ;}
� Invoked regularly by the system timer, and when a process
becomes runnable/unrunnable
CFS implementationProcess selection
� Adapted from [1]
CFS implementationProcess selection (2)
� When CFS needs to choose which runnable process to run next:.., The process with the smallest vruntime is selected.., It is the leftmost node in the tree
1
23
4
56
7
89
s t r u c t s c h e d _ e n t i t y * p i c k _ f i r s t _ e n t i t y ( s t r u c t c f s _ r q * c f s _ r q )
{
s t r u c t rb _ n o d e * l e f t = c f s _ r q - > r b _ l e f t m o s t ;
i f ( ! l e f t )
re turn NULL;
re turn r b _ e n t r y ( l e f t , s t r u c t s c h e d _ e n t i t y , r u n _ n o d e ) ;
}
CFS implementationProcess selection: adding a process to the tree
� A process is added through enqueue e n t i t y :
1
2
3
4
56
7
89
10
1112
13
1415
16
1718
s t a t i c v o i d
e n q u e u e _ e n t i t y ( s t r u c t c f s _ r q * c f s _ r q ,
s t r u c t s c h e d _ e n t i t y * s e , i n t f l a g s )
{
b o o l renorm = ! ( f l a g s & ENQUEUE_WAKEUP)
| | ( f l a g s & ENQUEUE_MIGRATED);
b o o l c u r r = c f s _ r q - > c u r r == s e ;
i f ( r eno rm && c u r r )
s e - > v r u n t i m e += c f s _ r q - > m i n _ v r u n t i m e ;
u p d a t e _ c u r r ( c f s _ r q ) ;
i f ( r eno rm && ! c u r r )
s e - > v r u n t i m e += c f s _ r q - > m i n _ v r u n t i m e ;
u p d a t e _ l o a d _ a v g ( s e , UPDATE_TG);
e n q u e u e _ e n t i t y _ l o a d _ a v g ( c f s _ r q , s e ) ;
a c c o u n t _ e n t i t y _ e n q u e u e ( c f s _ r q , s e ) ;
u p d a t e _ c f s _ s h a r e s ( c f s _ r q ) ;
19
2021
22
2324
25
2627
28
2930
31
3233
i f ( f l a g s & ENQUEUE_WAKEUP)
p l a c e _ e n t i t y ( c f s _ r q , s e , 0 ) ;
c h e c k _ s c h e d s t a t _ r e q u i r e d ( ) ;
u p d a t e _ s t a t s _ e n q u e u e ( c f s _ r q , s e , f l a g s ) ;
c h e c k _ s p r e a d ( c f s _ r q , s e ) ;i f ( ! c u r r )
e n q u e u e _ e n t i t y ( c f s _ r q , s e ) ;
s e - > o n _ r q = 1 ;
i f ( c f s _ r q - > n r _ r u n n i n g == 1 ) {
l i s t _ a d d _ l e a f _ c f s _ r q ( c f s _ r q ) ;
c h e c k _ e n q u e u e _ t h r o t t l e ( c f s _ r q ) ;}
}
CFS implementationProcess selection: adding a process to the tree (2)
� enqueue e n t i t y :
1
2
3
4
56
7
89
10
1112
13
14
15
16
17
s t a t i c vo id e n q u e u e _ e n t i t y ( s t r u c t c f s _ r q
* c f s _ r q , s t r u c t s c h e d _ e n t i t y * s e )
{
s t r u c t rb_node * * l i n k = &cfs_ rq - >
t a s k s _ t i m e l i n e . r b _ n o d e ;s t r u c t rb_node * p a r e n t = NULL;
s t r u c t s c h e d _ e n t i t y * e n t r y ;
i n t l e f t m o s t = 1 ;
/ *
* Find t h e r i g h t p l a c e i n t h e r b t r e e :
* /
w h i l e ( * l i n k ) {
p a r e n t = * l i n k ;
e n t r y = r b _ e n t r y ( p a r e n t , s t r u c t
s c h e d _ e n t i t y , r u n _ n o d e ) ;/ *
*We dont care about c o l l i s i o n s .
Nodes wi th
*t h e same key s t a y t o g e t h e r.
* /
18
1920
21
2223
24
2526
27
28
2930
31
32
33
34
i f ( e n t i t y _ b e f o r e ( s e , e n t r y ) ) {
l i n k = & p a r e n t - > r b _ l e f t ;} e l s e {
l i n k = & p a r e n t - > r b _ r i g h t ;
l e f t m o s t = 0 ;}
}/ *
* Maintain a cache o f l e f t m o s t t r e e
e n t r i e s ( i t i s f r e q u e n t l y
* u s e d ) :
* /
i f ( l e f t m o s t )
c f s _ r q - > r b _ l e f t m o s t = &se -> run_no de ;
r b _ l i n k _ n o d e ( &s e - > r u n _ n o d e , p a r e n t , l i n k
) ;
r b _ i n s e r t _ c o l o r ( & s e - > r u n _ n o d e , &cfs_ rq - >
t a s k s _ t i m e l i n e ) ;}
CFS implementationProcess selection: removing a process from the tree
� dequeue e n t i t y :
1
2
34
5
6
7
8
9
1011
12
1314
s t a t i c v o i d
d e q u e u e _ e n t i t y ( s t r u c t c f s _ r q * c f s _ r q ,
s t r u c t s c h e d _ e n t i t y * s e , i n t f l a g s )
{
u p d a t e _ c u r r ( c f s _ r q ) ;
d e q u e u e _ e n t i t y _ l o a d _ a v g ( c f s _ r q , s e ) ;
u p d a t e _ s t a t s _ d e q u e u e ( c f s _ r q , s e , f l a g s
) ;
c l e a r _ b u d d i e s ( c f s _ r q , s e ) ;
i f ( s e ! = c f s _ r q - > c u r r )
d e q u e u e _ e n t i t y ( c f s _ r q , s e ) ;
s e - > o n _ r q = 0 ;
a c c o u n t _ e n t i t y _ d e q u e u e ( c f s _ r q , s e ) ;
15
16
1718
19
2021
22
23
24
i f ( ! ( f l a g s & DEQUEUE_SLEEP))
s e - > v r u n t i m e - = c f s _ r q - >min_v run t ime ;
r e t u r n _ c f s _ r q _ r u n t i m e ( c f s _ r q ) ;
u p d a t e _ c f s _ s h a r e s ( c f s _ r q ) ;
i f ( ( f l a g s & (DEQUEUE_SAVE |
DEQUEUE_MOVE)) == DEQUEUE_SAVE)u p d a t e _ m i n _ v r u n t i m e ( c f s _ r q ) ;
}
CFS implementationProcess selection: removing a process from the tree (2)
� dequeue e n t i t y :
1
23
4
56
7
89
10
11
s t a t i c vo id d e q u e u e _ e n t i t y ( s t r u c t c f s _ r q * c f s _ r q , s t r u c t s c h e d _ e n t i t y * s e )
{
i f ( c f s _ r q - > r b _ l e f t m o s t == &se -> ru n _ n o d e ) {
s t r u c t rb_node * n e x t _ n o d e ;
nex t_node = r b _ n e x t ( & s e - > r u n _ n o d e ) ;
c f s _ r q - > r b _ l e f t m o s t = n e x t _ n o d e ;}
r b _ e r a s e ( & s e - > r u n _ n o d e , & c f s _ r q - > t a s k s _ t i m e l i n e ) ;
}
CFS implementationEntry point: s c h e d u l e ( )
� The kernel calls s c h e d u l e ( ) anytime it wants to invoke the scheduler
.., Calls p i c k n e x t t a s k ( )1
2
3
4
5
67
8
9
10
11
1213
14
15
16
s t a t i c i n l i n e s t r u c t t a s k _ s t r u c t * p i c k _ n e x t _ t a s k ( s t r u c t r q * r q , s t r u c t
t a s k _ s t r u c t * p r e v , s t r u c t p i n _ c o o k i e
c o o k i e )
{
c o n s t s t r u c t s c h e d _ c l a s s * c l a s s = &
f a i r _ s c h e d _ c l a s s ;s t r u c t t a s k _ s t r u c t * p ;
i f ( l i k e l y ( p r e v - > s c h e d _ c l a s s == c l a s s &&
r q - > n r _ r u n n i n g == r q - > c f s .h _ n r _ r u n n i n g ) ) {
p = f a i r _ s c h e d _ c l a s s . p i c k _ n e x t _ t a s k ( r q
, p r e v , c o o k i e ) ;
i f ( u n l i k e l y ( p == RETRY_TASK))
go to a g a i n ;
i f ( u n l i k e l y ( ! p ) )
p = i d l e _ s c h e d _ c l a s s . p i c k _ n e x t _ t a s k (
r q , p r e v , c o o k i e ) ;re turn p ;
}
17
1819
20
2122
23
2425
26
27
28
a g a i n :
f o r _ e a c h _ c l a s s ( c l a s s ) {
p = c l a s s - > p i c k _ n e x t _ t a s k ( r q , p r e v ,
c o o k i e ) ;i f ( p ) {
i f ( u n l i k e l y ( p == RETRY_TASK))
go to a g a i n ;
re turn p ;
}
}
BUG(); / * t h e i d l e c l a s s w i l l a lways
have a runnable t a s k * /
}
CFS implementationSleeping and waking up
� Multiple reasons for a task to sleep:
.., Specified amount of time, waiting for I/O, blocking on a mutex, etc.
� Going to sleep - steps:
1
2
3
4
Task marks itself as sleeping
Task enters a waitqueueTask leaves the rbtree of runnable processes
Task calls s c h e d u l e ( ) to select a new process to run
� Inverse steps for waking up
� Two states associated with sleeping:.., TASK INTERRUPTIBLE
.., Will be awaken on signal reception
.., TASK UNINTERRUPTIBLE
.., Ignore signals
CFS implementationSleeping and waking up: wait queues
� Wait queue:
.., List of processes waiting
for an event to occur
1
2
3
4
t y p e d e f s t r u c t wai t_queue_head
wai t_queue_head_ ts t r u c t wai t_queue_head {
s p i n l o c k _ t l o c k ;
s t r u c t l i s t _ h e a d t a s k _ l i s t ; }
� Some simple interfaces used to go to sleep have races:.., It is possible to go to sleep after the event we are waiting for has
occurred.., Recommended way:
1
23
4
56
7
89
10
1112
/ * We assume t h e wa i t queue we want t o wa i t on i s a c c e s s i b l e through a v a r i a b l e q * /
DEFINE_WAIT(wait); / * i n i t i a l i z e a wa i t queue e n t r y * /
a d d _ wa i t _ q u e u e ( q , & w a i t ) ;
w h i l e ( ! c o n d i t i o n ) { / * e v e n t we a re w a i t i n g f o r * /
p r e p a r e _ t o _ w a i t ( & q , &wai t , TASK_INTERRUPTIBLE); i f ( s i g n a l _ p e n d i n g ( c u r r e n t ) )
/ * handle s i g n a l * /
s c h e d u l e ( ) ;
}
f i n i s h _ w a i t ( & q , & w a i t ) ;
CFS implementationSleeping and waking up: wait queues (2)
� Steps for waiting on a waitqueue:
1
2
3
4
Create a waitqueue entry (DEFINE WAIT())
Add the calling process to a wait queue (add w a i t q u e u e ( ) )
Call p r e p a r e t o w a i t ( ) to change the process state
If the state is TASK INTERRUPTIBLE, a signal can wake the task
up → need to check5
6
7
Executes another process with s c h e d u l e ( )
When the task awakens, check the condition
When the condition is true, get out of the wait queue and set the
state accordingly using f i n i s h w a i t ( )
CFS implementationSleeping and waking up: wake u p ( )
� Waking up is taken care of by wake u p ( ).., Awakes all the processes on a waitqueue by default
1
2
# d e f i n e wake_up(x) wake_up (x , TASK_NORMAL, 1 , NULL)
/ * type o f x i s wait_queue_head_t * /
� wake u p ( ) calls wake up common():
1
23
4
56
7
89
10
1112
13
s t a t i c vo id wake_up_common(wait_queue_head_t * q , uns igned i n t mode,
i n t n r _ e x c l u s i v e , i n t w a k e _ f l a g s , v o i d * k e y)
{
wa i t _ q u e u e _ t * c u r r , * n e x t ;
l i s t _ f o r _ e a c h _ e n t r y _ s a f e ( c u r r , n e x t , & q - > t a s k _ l i s t , t a s k _ l i s t ) {
uns igned f l a g s = c u r r - > f l a g s ;
i f ( c u r r - > f u n c ( c u r r , mode, w a k e _ f l a g s , k e y) && ( f l a g s
& WQ_FLAG_EXCLUSIVE) && ! - - n r _ e x c l u s i v e )break ; / * wakes up on ly a s u b s e t o f ’ e x c l u s i v e ’ t a s k s * /
}
}
� Exclusive tasks are added through
p r e p a r e t o w a i t e x c l u s i v e ( )
CFS implementationSleeping and waking up: wake up() (2)
e A wait queue entry contains a pointer to a wake-up function) include/linux/wait.h:
1
23
4
56
7
89
10
1112
typedefstruct wait_queue wait_queue_t;
typedefint (*wait_queue_func_t)(wait_queue_t *wait, unsigned mode, int flags, void *key);
int default_wake_function(*wait_queue_func_t)(wait_queue_t *wait, unsigned mode,
int flags, void *key);
/* ... */
struct wait_queue {
/* ... */
wait_queue_func_t func;
/* ... */
}
e default wake function() calls try to wake up() ...) ... which calls ttwu queue() ...) ... which calls ttwu do activate() (put the task back on
runqueue) ...) ... which calls ttwu do wakeup ...) ... which sets the task state to TASK RUNNING
CFS implementationCFS on multicores: brief, high-level overview
e Per-CPU runqueues (rbtrees)) To avoid costly accesses to shared data structures
e Runqueues must be kept balanced) Ex: dual-core with one large runqueue of high-priority processes,
and a small one with low-priority processes) High-priority processes get less CPU time than low-priority ones
) A load balancing algorithm is run periodically) Balances the queues based on processes priorities and their actual
CPU usage
e More info: [6]
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
Preemption and context switchingContext switch
e A context switch is the action of swapping the process currently running on the CPU to another one
) Performed by the context switch()function
) Called by schedule()
1
2
Switch the address space through switch mm()
Switch the CPU state (registers) through switch to()
e A task can voluntarily relinquish the CPU by calling schedule()) But when does the kernel check if there is a need of
preemption?) need resched flag (per-process, in the thread info of current)
) need resched is set by:
1
2
scheduler tick()when the currently running task needs to be
preempted
try to wake up() when a process with higher priority wakes
up
Preemption and context switchingneed resched, user preemption
e The need resched flag is checked:
1
2
Upon returning to user space (from a syscall or an interrupt)
Upon returning from an interrupt
e If the flag is set, schedule() is called
e User preemption happens:1
2
When returning to user space from a syscall
When returning to user space from an interrupt
e With Linux, the kernel is also subject topreemption
Preemption and context switchingKernel preemption
e In most of Unix-like, kernel code is non-preemptive:) It runs until it finishes
e Linux kernel code is preemptive) A task can be preempted in the kernel as long as execution is in a
safe state) Not holding any lock (kernel is SMP safe)
e preempt count in the thread info structure
) Indicates the current lock depth
e If need resched && !preempt count → safe to preempt) Checked when returning to the kernel from interrupt) need resched is also checked when releasing a lock and
preempt count is 0
e Kernel code can also call directlyschedule()
Preemption and context switchingKernel preemption (2)
e Kernel preemption canoccur:1
2
3
4
On return from interrupt to kernel space
When kernel code becomes preemptible again
If a task explicitly calls schedule() from the kernel
If a task in the kernel blocks (ex: mutex, result in a call to
schedule())
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
Real-time scheduling policiesSCHED FIFO and SCHED RR
e Soft real-time scheduling classes:) Best effort, no guarantees
e Real-time task of any scheduling class will always run before non-real time ones (CFS, SCHED OTHER)
) schedule() → pick next task() → for each class()
e 2 ”classical” RT scheduling policies (kernel/sched/rt.c):) SCHED FIFO
) Tasks run until it blocks/yield, only a higher priority RT task can
preempt it) Round-robin for tasks of same priority
) SCHED RR
) Same as SCHED FIFO, but with a fixedtimeslice
Real-time scheduling policiesOther scheduling policies
e SCHED DEADLINE:
) Real-time policies mainlined in v3.14 enabling predictable RT
scheduling) EDF implementation based on a period of activation and a worst
case execution time (WCET) for each task) More info: Documentation/sched-deadline.txt, [4], etc.
e SCHED BATCH: non-real-time, low priority background jobs
e SCHED IDLE: non-real-time, very low priority background jobs
Outline
1 General information
2 Linux Completely Fair Scheduler
3 CFS implementation
4 Preemption and context switching
5 Real-time scheduling policies
6 Scheduling-related syscalls
Scheduling-related syscallsScheduling syscalls list
e sched getscheduler, sched setscheduler
e nice
e sched getparam, sched setparam
e sched get priority max, sched get priority min
e sched getaffinity, sched setaffinity
e sched yield
Scheduling-related syscallsUsage example
1
23
4
56
7
89
10
1112
13
1415
16
1718
19
2021
22
23
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <sched.h>
#include <assert.h>
void handle_err(int ret, char *func)
{
perror(func);
exit(EXIT_FAILURE);
}
int main(void)
{
pid_t pid = -1;
int ret = -1;
struct sched_param sp;
int max_rr_prio, min_rr_prio = -42;
size_t cpu_set_size = 0;
cpu_set_t cs;
24 /* GetthePIDofthecallingprocess */
/* Gettheschedulingclass */
handle_err(ret,"sched_getscheduler");
*/
25 pid = getpid();
26 printf("Mypidis:%d¥n", pid);
27
2829 ret = sched_getscheduler(pid);
30 if(ret == -1)
3132 printf("sched_getschedulerreturns:"
33 "%d¥n", ret);
34 assert(ret == SCHED_OTHER);
3536 /* Getthepriority(nice/RT) */
37 sp.sched_priority = -1;
38 ret = sched_getparam(pid, &sp);
39 if(ret == -1)
40 handle_err(ret,"sched_getparam");
41 printf("Mypriorityis:%d¥n",
42 sp.sched_priority);
4344 /* Setthepriority(nicevalue)
45 ret = nice(1);
46 if(ret == -1)
47 handle_err(ret,"nice");
Scheduling-related syscallsUsage example (2)
46 /* Getthepriority */ 65 /* Getthepriority */
47 sp.sched_priority = -1; 66 sp.sched_priority = -1;
48 ret = sched_getparam(pid, &sp); 67 ret = sched_getparam(pid, &sp);
49 if(ret == -1) 68 if(ret == -1)
50 handle_err(ret,"sched_getparam"); 69 handle_err(ret,"sched_getparam");
51 printf("Mypriorityis:%d¥n", 70 printf("Mypriorityis:%d¥n",
52 sp.sched_priority); 71 sp.sched_priority);
5354
/* SwitchschedulingclasstoFIFOand7273
/* SettheRTpriority */
55 * thepriorityto99 */ 74 sp.sched_priority = 42;
56 sp.sched_priority = 99; 75 ret = sched_setparam(pid, &sp);
57 ret = sched_setscheduler(pid, 76 if(ret == -1)
58 SCHED_FIFO, &sp); 77 handle_err(ret,"sched_setparam");
59 if(ret == -1) 78 printf("Prioritychangedto%d¥n",
60 handle_err(ret,"sched_setscheduler"); 79 sp.sched_priority);
61 8062 /* Gettheschedulingclass */ 81 /* Getthepriority */
63 ret = sched_getscheduler(pid); 82 sp.sched_priority = -1;
64 if(ret == -1) 83 ret = sched_getparam(pid, &sp);
65 handle_err(ret,"sched_getscheduler"); 84 if(ret == -1)
66 printf("sched_getschedulerreturns:" 85 handle_err(ret,"sched_getparam");
67 "%d¥n", ret); 86 printf("Mypriorityis:%d¥n",
68 assert(ret == SCHED_FIFO); 87 sp.sched_priority);
Scheduling-related syscallsUsage example (2)
/* clearthemask */
85 /* GetthemaxpriorityvalueforSCHED_RR */
86 max_rr_prio = sched_get_priority_max(SCHED_RR);
87 if(max_rr_prio == -1)
88 handle_err(max_rr_prio,"sched_get_priority_max");
89 printf("MaxRRprio:%d¥n", max_rr_prio);
9091 /* GettheminpriorityvalueforSCHED_RR */
92 min_rr_prio = sched_get_priority_min(SCHED_RR);
93 if(min_rr_prio == -1)
94 handle_err(min_rr_prio,"sched_get_priority_min");
95 printf("MinRRprio:%d¥n", min_rr_prio);
9697 cpu_set_size = sizeof(cpu_set_t);
98 CPU_ZERO(&cs);
99 CPU_SET(0, &cs);
100 CPU_SET(1, &cs);
101 /* SettheaffinitytoCPUs0and1only */
102 ret = sched_setaffinity(pid, cpu_set_size, &cs);
103 if(ret == -1)
104 handle_err(ret,"sched_setaffinity");
105 /* GettheCPUaffinity */
106 CPU_ZERO(&cs);
107 ret = sched_getaffinity(pid,
108 cpu_set_size, &cs);
109 if(ret == -1)
110 handle_err(ret,
111 "sched_getaffinity");
112 assert(CPU_ISSET(0, &cs));
113 assert(CPU_ISSET(1, &cs));
114 printf("AffinitytestsOK¥n");
115 /* YieldtheCPU */
116 ret = sched_yield();
118 if(ret == -1)
119 handle_err(ret,
120 "sched_yield");
121 return EXIT_SUCCESS;
123 }
Additional documentation
e CFS:) http://www.ibm.com/developerworks/library/
l-completely-fair-scheduler/
) http://elinux.org/images/d/dc/Elc2013_Na.pdf
e Linux scheduling:) https://www.cs.columbia.edu/smb/classes/
s06-4118/l13.pdf
BibliographyI
1 Inside the linux 2.6 completely fair scheduler. https://www.ibm.com/developerworks/library/l-completely-fair-scheduler/.
Accessed: 2017-02-01.
2 The rotating staircase deadline scheduler. https://lwn.net/Articles/224865/.
Accessed: 2017-01-29.
3 Rsdl completely fair starvation free interactive cpu scheduler.
https://lwn.net/Articles/224654/.
Accessed: 2017-01-29.
4 Sched deadline: a status update. http://events.linuxfoundation.org/sites/events/files/slides/SCHED_DEADLINE-20160404.pdf.
Accessed: 2017-02-06.
5 Understanding the linux 2.6.8.1 cpu scheduler.https://web.archive.org/web/20131231085709/http://joshaas.net/linux/linux_cpu_scheduler.pdf.
Accessed: 2017-01-28.
[6]L OZI, J.-P., LEPERS, B., FUNSTON, J., GAUD, F., QUE MA, V., AND FEDOROVA, A.
The linux scheduler: A decade of wasted cores.
In Proceedings of the Eleventh European Conference on Computer Systems (2016), EuroSys ’16, ACM, pp. 1:1–1:16.