integrated real-time systems vibhooti verma m.tech. (cse) 05305016 advisor: prof. krithi ramamritham

Integrated Real-Time Integrated Real-Time SystemsSystems

Vibhooti VermaVibhooti VermaM.Tech. (CSE)M.Tech. (CSE)

0530501605305016

Advisor: Prof. Krithi RamamrithamAdvisor: Prof. Krithi Ramamritham

OutlineOutline• Introduction• Focus of the Work and Motivation• Contributions• SParK/SCOS: System Requirements• SParK: System Architecture• SParK: Design Issues

– Scheduling• Two-level Static cycle and RM based priority scheduling• Partition Inversion and Solution• Virtual Aperiodic Partition

– Interrupt handling• Virtual Interrupt Partition

– Timer Virtualization– I/O Virtualization– CPU Virtualization– Memory Virtualization and Inter- Partition Communication

• Implementation• Conclusions and Further Work

Introduction to Integrated Introduction to Integrated Real-Time Systems (IRTS)Real-Time Systems (IRTS)

• Applications with different levels of criticality– Hard real-time, Soft real-time, Non real-time

• Tasks with different arrival patterns– Aperiodic, Periodic

• Strong Partitioning– Spatial and Temporal partitioning

PartitionCriticality of

PartitionApplications

I High Update of field inputs & Control logic and output

II Low RS232 & Ethernet Communication

III Medium Diagnostics

• A Typical System:

Focus of Our WorkFocus of Our Work• Design and development of a Strongly Partitioned

real-time Kernel(SParK) which can host multiple RTOS (one of them being Safety Critical Operating System (SCOS)

MotivationMotivation• Fault Containment• Easy Verification and Validation• Application Specific Operating System• Local Schedulability Analysis• Efficient Utilization of Resources

ContributionsContributions• Identified system components and proposed

system architecture• Designed SParK ensuring strong partitioning

based on virtualization concepts• Proposed Rate Monotonic based partition

scheduler to achieve temporal partitioning• Identified the problem of Partition Inversion and

proposed solution for the same• Adopted Virtual Interrupt Partition to avoid

deadline miss due to interrupt storm• Implemented in Stand-Alone RTLinux:

– Two-level static cyclic scheduler– Two-level dynamic scheduler– Interrupt Server– Priority Inheritance– Serial Port driver

SParK: RequirementsSParK: Requirements• Strongly partitioned architecture

– Temporal and Spatial partitioning– Fault Containment and Isolation

• Multiplex system resources among different partitions

• Support partitions with different RTOS• Provide Inter-Partition Communication mechanism• Guarantee Interrupts to meet their deadlines • Simple design• Diskless system• Full Predictability

SCOS: RequirementsSCOS: Requirements• Interface to hardware• Task Management and priority based scheduling• Inter-task communication• Interrupt Management• Memory management with protection• Timer Services• I/O Services• Scalable, Portable• Support for embedded diskless target environment

(ROMable)

SParK: System ArchitectureSParK: System Architecture

• Salient Features– SParK layer between RTOS and Hardware– Virtual Aperiodic Partition (VAP)– Virtual Interrupt Partition (VIP)– Virtual Device Partition (VDP)

Virtualization and SParKVirtualization and SParK

• Similarities– Running multiple OS(RTOS)– Spatial Partitioning– Isolation and Fault containment– VMM (SParK) acts as resource manager– VMM (SParK) gives an illusion to guest OS that they have

exclusive access to resources

• Differences– Temporal partitioning– Inter-partition communication– Interrupt Guarantees without affecting tasks

Scheduling: IssuesScheduling: Issues

• Ensuring temporal partitioning• All real-time tasks of all the partitions should

meet their deadlines.• Mechanism to handle Partition Inversion• Minimizing partition switch overhead• Handling aperiodic tasks• Ensuring ‘n’ out of ‘m’ interrupts without affecting

other tasks

Scheduling: ApproachScheduling: Approach

• Two-level hierarchical scheduler– Lower level scheduler schedules partitions– Upper level scheduler schedules tasks within the

partition

• Lower level scheduler can adopt– Static cycle scheduling– RM based priority scheduling (proposed)– EDF based Dynamic Open environment

scheduling• Upper level can have any fixed priority

scheduling

Scheduling: TechniquesScheduling: Techniques

• SPIRIT’s Cyclic Scheduling [1]– Non-preemptable sections not handled in original

algorithm, proposed solution for same and modified schedulability bounds

– Separate partition for aperiodic tasks• Open Environment ‘s Dynamic-Priority-Driven (EDF)

Scheduling [2]– CBS and TBS for partition scheduling– Handling Aperiodic tasks not clear– Handling Blocking factor with TBS only

• RM Based Priority Scheduling– Handles non-preemtable section– Modified schedulability bounds to account for non-

preemptable sections– Least priority partition for aperiodic tasks

Static Cycle Driven Scheduling Static Cycle Driven Scheduling Algorithm at lower levelAlgorithm at lower level

• Allocates capacity and cycle time for every partition

• Creates a static schedule, uses Unique and Harmonic cycle approach

• Separate VAP with dynamic slot shifting to provide better responsiveness to aperiodic tasks

Schedulability ConditionSchedulability Condition

• If a partition has ‘n’ tasks with computation time Cj, deadline Dj and period Pj, and assigned processor capacity , then a task is schedulable if }{}/.....2,1:.....2,1|{ ijjji DTDlijlTHt

tT

tCtWCET

j

i

j k

jki

1

),(

)},({max)( tWCETt kiHtki i

)(min)(0 kinik

)1(

)(0

k

kk

Inactivity period of a task is

Minimum activity period of a partition is

A Partition is schedulable if:

1.It is schedulable at processor of speed k

2.Partition cycle

k

Example: Static cycle Example: Static cycle scheduling of partitionsscheduling of partitions

PartitionPartition Task(exec time, period)Task(exec time, period) Feasible assignment Feasible assignment

P1 (3,50)(4,170)(1,70)(7,120)

(0.6, 113)(0.45,75)

P2 (10,90)(16, 200)

(0.4, 109)(0.55,155)

Partition InversionPartition Inversion• Definition: scenario where a scheduled partition

gets blocked due to locks held by other partition(s)

P1:t11:-----

P1:t11:lock(S1)

P1 preempted (budget exhausted)

P2:t21:lock(S1)

P2 blocked

P1:t11:lock(S1)

P1:t12:lock(S2)

P1 budget exhausted but not preempted

P1:t11:-----

P1:t11:unlock(S1)

P1:t12:unlock(S2)

P2:t21:lock(S1)---granted

• Our solution: Let the partition finish all of its CS before partition switch

Partition Inversion: Modified Partition Inversion: Modified Schedulability BoundsSchedulability Bounds

• Maximum duration to complete a partition Pk’s critical section

)()(_1

n

iik TLCSPCSSum

Unique Partition cycle approach: every partition gets scheduled exactly once hence modified schedulability bounds

1)(min

)(

1

1

1

i

m

n

jj

m

i i

TLCS

P1 P1CS P2 P2CS P1 P1CS P2 P2CS

RM based Preemptive Priority RM based Preemptive Priority scheduling (RMscheduling (RMSParKSParK))

• If there are ‘m’ partitions with feasible for ith partition then all partitions are schedulable if

),( ii

)12( )/1(

1

mm

i i mWhere, lower the , higher the priority

)12(2 )2/1(2

1 i i

Partitions Feasible capacity and periods

P1 (0.4,80)(0.6,100)

P2 (0.4,109)

i

Handling Partition Inversion in RMHandling Partition Inversion in RMSParKSParK

)*()}(_{max nnnmiP

o PCSSumR n

)(__)*()}(_{max nnnnmiP

o PCSallSumPCSSumR n

)}(__)*{(*))(__()*()}(_{max )(1

jnnj

Pi

nhpjnnnnmiP

i PCSallSumR

PCSallSumPCSSumR n

n

• Current partition has to let other partitions finish their critical sections:– When it is scheduled for the first time in its period – When other higher priority partition preempts it and starts executing

Partition preemption: 1)Budget exhaustation 2)Arrival of higher priority partition

Using RMA exact analysis for partition scheduling taking blocking into account:

If Pn is highest priority partition

otherwise

Generalized formula for ith response time of a nth partition

P3 P3CS

P4 P4CS

P2 P2CS

P1 P1CS

P2 P2CS

P4 P4CS

P1 P1CS

P4 P5

Infinite Partition InversionInfinite Partition InversionP1 P2

t11: lock(S1)

t11:——–

t12:——–

t12:lock(S2)

t12:——-

t21:——

t21:lock(S1)—

blocked and hence lends budget to P1

t11:——

t11:——

t12:unlock(S2)

t12:—–

t12:finishes

t11:—–

t11:unlock(S1)

t11:finishes

t11:lock(S1)

t21:lock(S1) blocked again and budget lent was wasted

Interrupt HandlingInterrupt Handling

• Requirement: To guarantee min. ‘n’ number of interrupts, in a particular partition will be served with bounded delay at any given point of time

• Proposed Approach: Virtual Interrupt Partition(VIP)

– Separate partition which is assigned processor capacity and partition cycle according to interrupt characteristics

– Interrupts are served when VIP has budget otherwise deferred till next replenishment

– Static Cycle Scheduling– RM based priority scheduling

Timer VirtualizationTimer Virtualization• Issues

– Maintaining time in deactivated partition– Guest RTOSs sharing same timer device demand different

timer resolution

• Our Approach: Paravirtualization– No virtual interrupt generation for deactivated partition– Update real time in guest RTOS when scheduled– Timer interrupt delivery to active partition– Sudden increase in time but fine if tasks are schedulable – Simple with lesser overhead

Timer VirtualizationTimer Virtualization




– Timer Virtualization– CPU Virtualization– I/O Virtualization– Memory Virtualization and Inter- Partition Communication


CPU VirtualizationCPU Virtualization

• Challenges– Non –virtualizable CPU architecture

• Instructions which do not trap when run in unprivileged mode (POPF)

• Unprivileged instructions let the CPU access privileged state

• Techniques– Full Virtualization

• On-fly binary translation

– Paravirtualization• Replacement of problematic instruction

CPU VirtualizationCPU Virtualization• Our Approach: Paravirtualization

– Emulation for privileged instruction in SParK for guest RTOS

– Replacement of non-virtualizable instruction by hypercalls

– Trap source Identification• Memory Location Trekking• SParK in ring 0 and Guest OS in ring 1 and user application

in ring 3

I/O VirtualizationI/O Virtualization• Issues

– One device, multiple drivers results in inconsistent device state

– Device driver’s location• If in SParK, fault containment violated• If in guest RTOS, mechanism to service other partition's

– Accounting time for device access– I/O virtualization technique should also ensure

• Fault Containment• Isolation• Responsiveness • Unmodified Device Driver Reuse• SParK’s simplicity

I/O VirtualizationI/O Virtualization

• Our Approach: Split Driver Architecture

Memory VirtualizationMemory Virtualization

• Issues– Spatial Partition among partitions– No interference to SParK– Architecture specific

• Our Approach– Static memory allocation using segmentation at compile

time for all the partition

Inter-partition CommunicationInter-partition Communication

• Issues– Safe communication mechanism to satisfy isolation

among partitions.– Minimum SParK’s involvement to reduce overhead

• Our Approach– Asynchronous notification through event channel– Shared page for data transfer and information of grated

page kept in SParK – Budget accounted on requesting partition

ImplementationImplementation

• Priority inheritance (changes in synchronization primitives)

• Serial Port Driver• Interrupt Server• Two-level Hierarchical scheduling

– Static Cycle Driven• Budget Lending

– Dynamic EDF based

ConclusionConclusion• We presented complete design of SParK• We proposed:

– Modifications in virtual machine techniques for virtualizing resources to meet real-time constraints

– RM based preemptive priority partition scheduling– Modifications in integrated real-time scheduling

algorithms to consider non-preemptable section– Virtual Interrupt Partition – Virtual Aperiodic Partition

• We implemented in SARTL– Two-level static cyclic scheduler– Two-level dynamic scheduler– Interrupt Server– Priority Inheritance– Serial Port driver

Future WorkFuture Work

• Look into architecture specific issues for memory allocation and other hardware initialization

• Implementation of various hypercalls to act as interface between guest RTOS and SParK

• Porting guest RTOS to SParK by replacing problematic instructions in it by appropriate hypercalls

• Front-end device drivers for guest RTOSs • Ethernet driver for Stand-Alone RTLinux towards

building SCOS• Prototype Implementation of SParK

ReferencesReferences• Daeyoung Kim. Strongly Partitioned System Architecture For Integration Of Real-Time

Applications. PhD thesis, UNIVERSITY OF FLORIDA, 2001

• Zhong Deng, Jane W.-S. Liu, Lynn Zhang, Seri Mouna, and Alban Frei. An open environment for real-time applications. Real-Time Systems, 16(2-3):155{185, 1999

• Mendel Rosenblum and Tal Garnkel. Virtual machine monitors: Current technology and future trends. IEE Computer Magazine, May 2005

• Tullio Facchinetti, Giorgio Buttazzo, Mauro Marinoni, and Giacomo Guidi. Non-preemptive interrupt scheduling for safe reuse of legacy drivers in real-time systems. In ECRTS '05, pages 98-105, Washington, DC, USA, 2005. IEEE Computer Society

• Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Wareld. Xen and the art of virtualization. In SOSP '03, pages 164{177, New York, NY, USA, 2003. ACM Press

• Full Virtualization:– Virtual interrupt generation – VM Time tracker – Fast Catchup– Time synchronization tool when fast catchup is

impossible

Timer VirtualizationTimer Virtualization

GuestOS VMM Hosted I/O Bypass IDD

DD Location GuestOS VMM HostOSGuestOS and VMM

Dedicated partition

Isolation Guarantee

No Yes Yes Yes Yes

Fault Containment Guarantee

Yes No No Yes Yes

VMM Role No Large ModerateOnce for setup

Only for registration

Real-Time response

Fast Slow Medium Fast Medium

DD reuse Yes No Yes No Yes

OS MOdification

No No No Yes Yes

Special driverNo No No Yes (small) Yes

Device intelligence

No No No Yes No

Ease of impl Easy Moderate Easy Difficult Difficult

I/O VirtualizationI/O Virtualization

VM Environment Vs SParKVM Environment Vs SParK

VM Environment SParKStrict Temporal Partitioning

Not Required Necessary

Scheduling of Guest OS

Any scheduling RT Algorithm ensuring real-time constraints

Memory Virtualization Complex Simple

Memory Virtualization Techniques

Shadow paging Static allocation

Interrupt Handling Delay

Acceptable Not Acceptable

Separate Interrupt Server

Not Necessary Necessary

Inter-partition Communication

Not required Required

Under Utilization of CPU

Not Acceptable Acceptable

Main Motivation Cost Reduction Fault Containment

Comparison of RTOSComparison of RTOSFeatures SA-RTL RTLInux MicroC OS L4

Posix Compliance Y Y N N

Real-time primitives Y Not All Y N

Timer latency 7us 10 us 1ms 5ms

Finer timer resolution useconds useconds 1ms 10ms

Stand-Alone Y N Y Y

Large priority levels Y Y N Y

Task can have same priority Y Y N Y

Priority Degradation N N N N

PCP/PI N Y N N

Small size Very small Large Very small Small

Loadable kernel modules N Y N Y

Development Env. Difficult Easy Moderate Moderate

Code Modularity Y Y N Y

Portability X-86/ARM X-86 To many X-86/powerpc

Documentation Bad Good V. Good Moderate

Debugging Facility Not easy Easy Not easy Moderate

Comparison of various Comparison of various scheduling policiesscheduling policies

Cycle Driven

RM based Priority driven

Partition Scheduling Static Static

Lower level scheduler

Cyclic Priority

Handling Non-preemptable section

Modified by extra budget

Modified by extra budget

Hard aperiodic tasks VAP VAP

Soft aperiodic tasks Idle Time Least priority partition

Context Switch Overhead

Small Moderate

CPU Utilization Medium Lowest

Real-time Guarantees

Strict Strict

VIP implementation Difficult Easy

Flexibility of adding new partitions tasks

No Yes

EDF based open system Dynamic

EDF based

Total Bandwidth Server

No Provision

TBS

Large

High

Strict

NA

Yes

integrated real-time systems vibhooti verma m.tech. (cse) 05305016 advisor: prof. krithi ramamritham

Documents

system architecturespark

system requirementsspark

typical system

problem of partition

interrupt stormimplemented

realtime kernelspark

hardwaretask management

proposed solution