asqf fachgruppe automation/ automation day 1 realtime capable linux: requirements...

ASQF Fachgruppe Automation/ Automation Day 1

Realtime capable Linux: Requirements

(Standard)LinuxRealtime LinuxRealtime kernel

One (physical) address space with n-tasks

Application program 100% source compatibilitybetter binary compatibility

API

Response time: < 10 µs Response time: ~1ms

Response time: < 10 µssame

homogeneity

One process address spacewith n-threads

All tasks haveone common API implemented by

one common library

All threads of a process haveone common API implemented by

one common library


Realtime capable Linux: The important questions

• Is the POSIX-API suitable?

– Only a subset is relevant for realtime applications

– The mechanism for the notification of asynchronous events determine widespread usage

• Can it provide for competitive performance?

– Ideally it should be as good as specialized realtime kernels

– Ultimately cycle times of 31,25 µs should be possible

• What about additional changes to the Linux kernel ?

– Ideally none

– Only minimal changes allow for long time support


Thread management– Creation,Termination, Scheduling

Synchronization – Mutex – Spinlock

Communication – POSIX Message Queue– Semaphore– Shared Memory

Timer– types (periodic, one shot, absolute, relative)– resolution– clock tick– notification

The POSIX-API: Minimal set for realtime tasks

needs improvements


The POSIX Notification-API: The Timer-API

Timer sigeventobject

SIGEV_THREAD

CLOCK_REALTIME

SIGEV_SIGNAL

SIGEV_NONE

timer_create (clockid_t clockid, timer_t *timerid, struct sigevent *evp);

Problems:SIGEV_THREAD:

The implementation within the glibc is not realtime capable

another implementation fixes it (deterministic, performance improvement by a factor of 10)

SIGEV_SIGNAL:Sending a signal to the entire process is not deterministic. Potentially the entire thread list of a process must be scanned.

minimal overhead and deterministic behavior require sending to a thread (Linux can handle it)

no response based on signal handlers (signal handlers are for synchronous exceptions only)

having explicit wait only allows for extremely fast notification.


The POSIX Notification-API: Extensions(1)

t

timer fires

sigwait()Scenario a

sigwait()Scenario b

timeroverrun

sigwait()Scenario c

Can be queried This situation must get signalled

An additional call sigevent_set_notification() makes sending the signal to a thread available at the API level. This call maps the threadid of the API to the tid of the kernel allows via an additional encoding of the sigev_notify field for

supervision of timeliness implementation with best possible deterministic behavior

The existing sigevent structure needs no changes.

timer fires timer fires

POSIX definition (not very helpful)


The POSIX Notification-API: Extensions(2) A device must be a possible source of a timer tick (clock)

Example: A Profinet Realtime Ethernet Controller ERTEC provides for a 250 µsec tick, not necessarily synchronous with CLOCK_REALTIME additional call register_clock()

A firing timer is one special type of an event arbitrary asynchronous IO events should get signalled the same way additional call event_create()

Timer

sigeventobject

SIGEV_THREAD (realtime capable implementation of the library module)

CLOCK_REALTIME SIGEV_SIGNAL (inhibited)

direct IO-Event

create_event()

Timer

DeviceDriver

clock

register_clock()

sigevent_set_notification()

Signal to thread (optional with supervision)

Extensions to the POSIX API


Realtime capable Linux: Base concept

Linuxkernel

realtime kernel

threads controlled by realtime scheduler

threads controlledby Linux-Scheduler

system call entryexactly the same for both kernels

Ipipe (ADEOS)

Common Library

Linux ProcessPOSIX API

HW-Interrupts

Soft-Interrupts

static priorities(POSIX-counting)

99

0

99

0

realtimekernel Linux

Linux + Realtime

boundary implementation defined

Linux


Typical partitioning of a realtime thread setup phase time critical response path

Much functionality needed but neither

• realtime requirements nor

• high performance requirements

For all calls not implemented, the thread temporarily runs under the control of the Linux scheduler

Wait for event

event

Realtime capable Linux: Functionality of the realtime kernel

minimal functionality needed

minimal realtime kernel- timer functions and its notification

- communication and synchronization

- scheduling


Realtime capable Linux: Limits of homogeneity

Linux Kernel

Realtime Kernel

threads managed by therealtime scheduler

threads managedby theLinux scheduler

Ipipe

Linux process

HW-Interrupts

Soft-Interrupts

Signal source eg. timer

Signal source eg. IO event

A signal can only get sent to that domain which created the signal source This restriction is

important for best possible performance (allows for separation of implementation) acceptable and manageable (violations are reported)


Realtime capable Linux: Implementation Kernel

Two additional loadable modules:• A device driver

– functional extensions of the API (API-calls ioctl() system calls, additional internal interfaces for realtime capable device drivers)

– Interface for initialization/configuration of the realtime module eg. registration of a process as realtime process

• The realtime module (20K for x86)

– Functionality of the realtime kernel

– Communication between realtime and Linux kernel

Three modified kernel files:– futex.c (hook), posix_timers.c (clock registration), mqueue.c

Ipipe modifications:– Systemcall handling (additional hook at system call exit + optimization)

– Interrupt handling (optimization, support for truly preemptive realtime kernel)


Realtime capable Linux: Implementation glibc

Modified functions: • Initialization

– Registration of the realtime process

• Spinlock function also for static priorities• Realtime capable implementation for SIGEV_THREAD

Additional functions• Device driver functions via ioctls

– Registration of a timer clocks– Registration of an (IO-)event

• Thread specific sigevent notification (the kernel is able to do it) • fast message queues


Testscenario:

• Measurement with getuid(), since getpid() is handled inside glibc

• Measurement with time stamp counter (94 clocks resp. 12/5 clocks)

• glibc-2.3.6 generated with either int 80 or sysenter system entry (…/…)

Cel. 2,8 GHz1000 clocks = 357 ns

Athlon 2 GHz1000 clocks = 500 ns

343/147 Linux 2.6.15.4 original

ipipe original

1124/609

1397/914

Linu

x D

oma

inR

ealti

me

D

oma

in

Lin

ux +

Ipip

e pa

tch

+ AuD patch

+ AuD optimized

+ AuD patch

+ AuD optimized

1260/767

987/494 289/199

383/257

Realtime capable Linux: System call overhead in clocks

317/255

466/338

987/494

1155/641 /185


high priority thread

sem_post()/mq_send()

sem_wait()/mq_receive()

up

helper thread of lowest priority

down

Low priority thread must have

a chance to proceed to sem_wait() / mq_receive()

sem_wait()/mq_receive()

sem_post()/mq_send()

low priority thread

Realtime capable Linux: Performance inter-thread communication

Testscenario:Two threads communicating either

• via semaphores or

• through POSIX message queues

Note:Times for down are always longer than for up, since an additionalsem_wait() / mq_receive()-call is included.


RealtimeDomain

LinuxDomain

up down

2268

2615 4547

3980

3801

3361

3539

3780

4715

5377

15992

13304

744 1279

1071

1519 2075

4374

updown

down

down

up

up

Realtime capable Linux: Interthread communication in clocks Celeron 2,8 GHz

1000 clocks = 357 ns

Athlon 64 2,0 GHz 1000 clocks = 500 ns

up down up down

1963 2509

2530

3241 3912

6174

up down

Semaphore Message Queue Semaphore Message Queue

6721 8390

7266 19825

4715 7004

optimized

836 1417

1297 4943

1647 2215

2050 2771

2827 6666

3335 4197 7655 10364

7130 18575

5534 8106

System call entry: sysenter

Times for Realtime Linux • first switch to interrupted Linux thread• afterwards switch to woken up Linux thread (message queue)• semaphores (futex) need an additional proxy thread for futex_wake


ISR response

thread response

min max

525

1785

2688

284

3287

16149

17441

3276

min max

1024

1010

101

9446

Athlon64 2,0 GHz 1000 clocks = 500 ns

Celeron 2,8 GHz 1000 clocks = 357 ns

Realtime capable Linux: Interrupt response time in clocks Test scenario:• An application thread issues via a driver an

ipipe soft interrupt

• The ISR resumes via send_event another application thread (waiting with sigtimedwait())

• ISR frequency varied between 100 µsec and 1 sec

• System load generated via GUI (Eclipse-Start, file transfer, etc.)

The minimal figures are of interest in several respect:

• for high interrupt frequencies representative for the average and therefore the load

• typical for path length, and therefore useful for scalability projections

• What can be achieved with less powerful HW (e.g. SOC with mediocre memory interface)?

• for a dedicated processor of a multicore system (private TLB, L1-Cache, L2-Cache) the guaranteedworst case time comes closer to the minimal figure.

224 1382

1264 9240


Realtime capable Linux: Positioning to RT_PREEMPT

• The kernel implementation is transparent to application programs – An application binary for a dual domain system runs without any changes on a Linux

only system and therefore also on a RT_PREEMPT system

• Possible coexistence scenarios with RT_PREEMPT – RT_PREEMPT as an alternative if no extremely strong realtime requirements

– An additional realtime domain

• for extremely strong requirements for response time and thread communication,

• if demonstration is required (proof) tiny kernel, short paths large complex kernel, lengthy paths, nested PI mutexes, RCU locks

• Commonalities with RT_PREEMPT– The same modifications of the glibc are needed.

– The same kernel enhancements (registering a clock, realtime capable POSIX message queues) are needed


Realtime capable Linux: Conclusion

• The POSIX-Realtime-Extensions need some fine tuning

– Minimal extensions to the API definition improve usability and allow for best possible performance

• Realtime under Linux can be provided transparently

– The performance figures are comparable to specialized realtime kernels

• Ipipe is a sustainable base

– Small optimizations improve system call handling and interrupt response

asqf fachgruppe automation/ automation day 1 realtime capable linux: requirements...

Documents