ittc iab 4/7/05 1 dr. douglas niehaus © 2005 information and telecommunication technology center...

ITTC IAB 4/7/05 1 Dr. Douglas Niehaus © 2005

Information and Telecommunication Technology CenterIndustrial Advisory Board Meeting

April 7, 2005

Douglas [email protected]

LibeRTOS: A Flexibly Configurable and Highly Capable Platform for

Industrial Automation


Overview• Linux for real-time and embedded systems

• KURT-Linux Origins• Time keeping

• Event scheduling

• Performance Evaluation

• LibeRTOS• Unified computation control

• Configurability

• Tool support

• Conclusions


Real-Time and Embedded Linux• Linux is strongly increasing in popularity for real-time

and embedded systems for several reasons

• Open source ensures all parts of the system are accessible to the system developers• Local archive of entire system

• Long term stability

• Simple licensing restrictions: GPL and LGPL• Application software unrestricted

• Low cost

• Mature programming tools: GCC, GDB, binutils, busybox


Real-Time and Linux• Two basic methods for real-time computation using Linux

• Two OS method

• Single OS method

• Two OS method establishes a RT OS on top of the hardware and then runs Linux under the RTOS control

• Linux is generally a non-real-time best effort computation

• RTLinux and RTAI use this approach

• Relatively simple way to provide accurate scheduling support for simple RT computations

• Drawback is that access to Linux services is more complex and RTOS services must all be implemented


Real-Time and Linux• Single OS approach modifies Linux internals to make it

appropriate for real-time• Harder to do but it permits use of all Linux system

services much easier• KURT-Linux was the first system to use this approach• Monta Vista and Timesys are commercial companies

using this architectural approach

• Several components of Linux need to consider RT• Interrupt handling, thread scheduling, concurrency

control, preemption, kernel computations

• Single OS solutions now approach two OS performance


KURT-Linux• Project started roughly eight years ago

• Initial areas of concentration were:• Time keeping

• Event scheduling

• Performance evaluation

• Simple programming model with explicit plan scheduling• Execution intervals for computations were explicitly

specified on a cyclically repeating time line

• Adequate for demonstrating microsecond resolution soft-real-time scheduling of fairly simple computations


KURT-Linux:Time Keeping• Vanilla Linux uses a periodic interrupt to keep time and

to control the occurrence of explicitly scheduled events

• KURT-Linux separated time keeping and event interrupts

• Uses x86 Time Stamp Counter (TSC) as time standard• TSC ticks with CPU clock frequency• Can be calibrated against an Network Time Protocol

standard within 30-40 parts per billion• On a 1 GHz Pentium machine

• Provides fine grain local clock to applications and performance evaluation framework


KURT-Linux: Event Scheduling• Explicitly schedules each event interrupt using

microsecond (x86) resolution PC timer chip• Simple and portable internal API: ARM, PPC• Resolution varies with the platform

• Explicit plan model is a simple way to ensure that execution of periodic real-time computations is controlled accurately• Within microseconds to tens of microseconds depending

on configuration options and CPU clock speed

• However 102 outlier events still occurred per 107

events• Interrupt blocking and lack of kernel preemption


KURT-Linux:Performance Evaluation• Basic principle of real-time and embedded system

development is that the accuracy of the performance evaluation methods must keep pace with the accuracy of the behavior control desired• KURT-Linux thus required performance evaluation

methods much more accurate than vanilla Linux provided

• Data Streams sub-system addresses this• Pseudo-device driver for kernel event recording• Utility library for application instrumentation

• Unified application/OS performance evaluation and analysis methods permits end-to-end computation view


Data Stream Overview


Data Streams• Basic data type is the event• Time stamp (TSC), unique ID, context information

• Instrumentation points in OS or application code produce an event when a thread of control cross them

• Aggregate data types have lower overhead• Histograms and counters

• Data added to aggregate by instrumentation points

• Administrative events can include snapshots of aggregate data in the output event stream during an experiment as well as at the end

• Instrumentation points can be individually enabled


Data Streams• Powerful post processing is key to analysis of often

huge data sets gathered from and experiment

• Post-processing filters are easy to use and to write• Python based implementation

• GUI tool support for filter configuration

• Constraint checking and scenario specific filters useful

• Visualization is also quite useful for understanding higher level patterns of computation behavior • Detection of unusual situations

• Data streaming across sockets especially useful for embedded systems


Pipeline Processing Visualization


LibeRTOS• Collaborative project with: Linutronix, several industrial

automation customers in Germany, and Washington U.

• Industrial-use version of KURT-Linux• KURT-Linux should remain research version and venue

• LibeRTOS will receive updates from research efforts

• LibeRTOS industrial driving problems enrich KURT-Linux research and development

• LibeRTOS motivated feature development• Unified Computation Control: Group Scheduling Model

• Configurability of many aspects of system semantics

• Tool support


LibeRTOS: Unified Computation Control

• Motivation: many application computations are implemented by groups of computational components

• Emphasizes flexible representation and control of entire computations, including internal Linux components

• Internal Linux computation types are “off-the-books”• Linux internal types: hard-IRQ, soft-IRQ, tasklets

• Execution manifests as noise in thread scheduling models

• Key Group Scheduling issues• Customizable scheduling of applications

• End-to-end scheduling of all computation components

• Precise fine-grain resource control and partitioning


Group Scheduling (GS) Model• Simple form of hierarchical scheduling permits composition of a

wide variety of scheduling semantics • Middleware version for portability, OS for complete control

• GS controls arbitrary groups of computational components• Threads: OS and Middleware versions

• Hard-IRQ, soft-IRQ, tasklet, bottom-halves only under OS

• Groups nest to form scheduling hierarchy

• Scheduling Decision Function (SDF) associated with each group selects among members when asked

• Application specific semantics can be created by implementing an SDF embodying those semantics


Group Scheduling Model• Generally we use decision

trees but could be DAGs• Scheduling Decision Tree

(SDT) is any graph subset• System Scheduling Decision

function (SSDF) is SDT controlling system as a whole

● This SSDF illustrates the “first refusal” semantics for GS under KURT-Linux/LibeRTOS

● Only desired computations are under GS others default to Linux vanilla scheduler control for convenience


Group Scheduling Model• Emphasizes modularity and configurability• Controls selected computations

• Does not require specification for all computations

• SSDF can clearly represent desired semantics

• Users can define SDFs • Functions in MW, Kernel modules in OS

• Application centric and state-aware SDF use was the driving problem for this paper• SSDF integrates several scheduling constraints


Group Scheduling Target Examples• Many computations have multiple components and it is

the computations developers really wish to schedule

• Ethernet Time Division Multiplexing on a LAN

• Data Streams instrumentation effect reduction

• Rodent behavioral experiment data collection & control

• Real-time graphics calculation and display • Coordinate execution of threads doing calculation with

execution of the X-server thread doing the display

• Application-aware scheduling • Balanced progress of multiple multi-threaded pipelines

for video processing, sensor fusion, or multi-track audio


Application Aware Driving Problem • Based on DARPA PCES driving problem• Critical and non-critical streams of data frames• Each stream is processed by a pipeline of threads• Processing time of each frame varies• By frame and pipeline stage

• Balanced progress by many streams is desired• Multiple stream video monitoring• Sensor fusion and control loops in more general terms

• Difficult to implement without an algorithm using application state information

• System where progress of a set of computations matters


Application Aware Driving Problem


Scheduling Scenario 11.Preference to Critical Streams2.Frames move through pipelines as fast as possible3.Receiving frames more important than processing4.Non-critical streams under default Linux scheduling


Scheduling Scenario 2

1.Preference to Critical Streams2.Frames move through pipelines as fast as possible3.Receiving frames more important than processing4.Equal execution opportunity for non-critical streams


Is Round Robin Enough?• As implemented in the RR SDF, each non-critical

stream received opportunity to use equal CPU share • We could have made it equal consumption

• Under either policy, the variable processing time of each pipeline stage for each frame means that frame progress is not balanced

• Such variable execution behavior is common to many groups of computations

• Need for behavior to meet specific and unchanging constraints despite variable behavior also common

• Application aware scheduling addresses this


Scheduling Scenario 31.Preference to Critical Streams2.Frames move through pipelines as fast as possible3.Receiving frames more important than processing4.Balanced Progress of non-critical Streams


Scenario 1 – Linux Scheduler


Linux Scheduler Conclusions• Linux scheduler is oblivious to non-critical stream • Existence

• Pipeline component relationships

• Frame progress over various streams thus varies widely

• Complete failure to produce desired application behavior


Scenario 2 – Round Robin SDF


Round Robin Scheduling Conclusions• Non-critical streams under GS control• Control computations as groups of threads

• Gives much more rational execution behavior

• RR gives streams opportunities to use roughly equal portions of the left over CPU

• RR is, however, unaware of progress by the application so progress is not balanced across streams

• Complete failure to produce desired application behavior


Scenario 3 – Frame Progress SDF


Frame Progress Scheduling Conclusions• Substitution of the FP SDF makes the SSDF aware of

the non-critical application state• Extremely tight balance between progress of all

streams since the SDF has access to published application state

• Application aware SDF is a simple and effective way to produce the desired application semantics


LibeRTOS: Configurability• Group scheduling provides flexible scheduling semantics

configurability for precise computation component control

• Interrupt enable/disable software semaphore control• Reduces interrupt response latency

• Permits configurability of IRQ handling semantics and integration under the group scheduling model

• Similar treatment of soft-IRQ and tasklet control permits configuration and integration under group scheduling

• Loadable module can completely replace default vanilla Linux semantics with the desired programming model

• Concurrency control in OS is a remaining issue


LibeRTOS: Tool Support• Data Streams

• Event name space GUI editor

• Post processing filter configuration GUI editor

• Used as EECS 448 Software Engineering projects (Sp04)

• Experiment data collection configuration GUI editor

• Group Scheduling• Hierarchy configuration language, GUI editor, and system software

interface to OS and MW group schedulers

• GUI is EECS 448 (SPR05) driving problem

• Eclipse Data Streams and Group Scheduling plugins• Under development


Project Support• Portions of the work described here have been

supported by a variety of sources and projects including:• Sprint

• NSF E&HS program

• DARPA PCES program

• DARPA ANTS program

• Linutronix (Eclipse plugin prototypes)

• NIH (Behavior Experiment support)


Conclusions• Linux is an increasingly popular platform for real-time

and embedded systems for a variety of strong reasons• KURT-Linux/LibeRTOS addresses many of the existing

constraints keeping Linux from addressing an even wider range of distributed real-time and embedded applications

• Configurability of the system makes it applicable to a variety of application areas beyond real-time• Real-time scheduling and behavior semantics is now

simply one configuration choice among many• Real-time embedded systems will remain a focus but will

not be the only one


Future Work• Unification of concurrency control interface in Linux • Will simplify code, making analysis and study easier

• Will provide clean structure within which to integrate OS concurrency control within LibeRTOS flexibly configurable framework

• Will permit much cleaner structure within which to integrate OS concurrency control and group scheduling

• Proxy execution • Simple accounting will track semaphore holders

• Integration with group scheduling will prevent deadlock when preemption of threads holding locks is permitted


Future Work• Consolidation of end-to-end scheduling for computation

components on real-time and embedded endsystems

• Extension of end-to-end scheduling to sets of distributed computation components crossing endsystem boundaries

• Integration of formal modeling of distributed real-time and embedded computation components to OS components• Extend current Washington University PhD thesis

• Explore use of LibeRTOS component oriented configurability for HW/SW co-design of OS and application support and integration with group scheduling


Future Work• Apply KURT-Linux/LibeRTOS to various target

applications areas

• Behavioral science experiment control and data collection• Dr. Fowler at KU and Dr. Snyder at Washington

University

• Video processing: Dr Gauch (KU) and Dr. Fowler

• Performance evaluation of GRID computations• CHARMM distributed molecular dynamics simulation

• Drs. Clark , Kuczera, Houdonougbo

• Apply group scheduling to improve performance

ittc iab 4/7/05 1 dr. douglas niehaus © 2005 information and telecommunication technology center...

Documents

embedded linux linux

linux services

linux internals

vanilla linux

overview linux

components of linux

time scheduling

linux system services