using embedded linux for infrastructure systems
TRANSCRIPT
Using Embedded Linux for Infrastructure Systems
Embedded Linux Conference Europe 201414 Oct 2014
Yoshitake Kobayashi
2Embedded Linux Conference Europe 2014
Scope of this presentation
Make a Place for collaboration
NOTE
This project currently belongs to CE workgroup, Linux Foundation
Anyone interested in this activities, please contact us
3Embedded Linux Conference Europe 2014
Outline
Definition of Social Infrastructure Systems
Problem statement
Expected goal
Project scope
Activities
Current status
4Embedded Linux Conference Europe 2014
Definition of Social Infrastructure Systems (SI systems)
Embedded systems which uses for the following infrastructure systems Transport infrastructure
Energy infrastructure
Water management infrastructure
Communications infrastructure
Earth monitoring and measurement systems
The system includes Automation systems
Controllers
PLC
Microcontrollers
Signal controller
Multi-purpose controller
Mini-server based controller (includes PC based controller)
Sensor network systems
5Embedded Linux Conference Europe 2014
An Example of Infrastructure System
sensorController
(e.g. PLC)
6Embedded Linux Conference Europe 2014
IoT?
sensorController
(e.g. PLC)
THE INTERNET
OF THINGS
7Embedded Linux Conference Europe 2014
Infrastructure System becomes a part of IoT
sensor
THE INTERNET
OF THINGS
THE INTERNET
OF THINGSMoving Forward
Controller
(e.g. PLC)
8Embedded Linux Conference Europe 2014
Problem statement
The SI systems will be work with other systems such as:
Cloud computing
IoT / IoE related systems or devices
9Embedded Linux Conference Europe 2014
Problem statement
To connect with other systems, SI systems
Need to have same quality with new requirement
Need to have new technology to have new functionality
New requirements
Ensure a complicated performance requirement
Nondeterministic and deterministic programs runs on a same board
Collaborate with cloud platform
New functionality
Use fast non-volatile memory (e.g. MRAM, FeRAM)
10Embedded Linux Conference Europe 2014
Goal
Main goal
Develop reference implementations for SI system
Reference implementation may includes:
Linux kenel and filesystem for some target board
Target board: TBD
Open spec hardware is better for the implementation
Linux kernel related implementations should post to LKML for mainlining
Test-cases
11Embedded Linux Conference Europe 2014
Project scope
Security Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update e.g. Kpatch / kGraft
Real-time Hard real-time
Multi-purpose real-time system
Tests Reliability test
Long Term Support VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
12Embedded Linux Conference Europe 2014
Activities
This project will be separated into several different sub-project by its key technology
Initial activity
Survey to determine current common key issues for SI systems
Create a mail list
F2F meeting at Linux Foudation’s conference
Embedded Linux Conference / LinuxCon
Workgroup meeting will be arranged
Reviewed by members of the CE Workgroup Architecture Group
13Embedded Linux Conference Europe 2014
Estimated output
Publicly available pages on the elinux wiki which includes the follows:
A list of issues
Specification for reference implementations
A list of link to reference implementations
All kernel related implementation should post to LKML
Reference implementations for key issues will be available on GitHub or alternative service
14Embedded Linux Conference Europe 2014
Current status
Under survey to determine actual issues by discussion
Start to call for participation
15Embedded Linux Conference Europe 2014
Project scope
Security Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update e.g. kpatch / kGraft
Real-time Hard real-time
Multi-purpose real-time system
Tests Reliability test
Long Term Support VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
16Embedded Linux Conference Europe 2014
Project scope
Security Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update e.g. kpatch / kGraft
Real-time Hard real-time
Multi-purpose real-time system
Tests Reliability test
Long Term Support VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
17Embedded Linux Conference Europe 2014
Real-time
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
18Embedded Linux Conference Europe 2014
Issue on SCHED_DEADLINE
WakeupPeriod
Task executionTime
dl_timer
1.5ms
task_tick_dl1ms
Wakeup
dl_timer
1.5ms
1ms
Step1: Refill budget
Step2: Use budget
Period
2ms
Difficult to keep task’s budget if the budget has micro seconds granularity
Period: 1ms, Budget: 0.5ms
19Embedded Linux Conference Europe 2014
Support for micro seconds granularity
Period: 1ms, Budget: 0.5ms
Wakeup
Period
Task execution
Set HRTICK here
Step1: Refill budget
Step2: Use budget
Time
3.2ms
1ms 2ms 3ms
Wakeup
When a task’s budget is less than 1ms, set HRTICK for the rest of budget
Source code is available on GitHub ( https://github.com/ystk/sched-deadline )
20Embedded Linux Conference Europe 2014
Let’s play a movieModifiedOriginal
Play the movie about 3 times faster than normal speed Requirement for a frame decoding
Period: 10ms, budget: 3.6ms
When a frame decode is not finished in a period, the player shows “DEADLINE MISS” on screen and add 3ms penalty
“Big Buck Bunny” http://www.bigbuckbunny.org/
(c) copyright 2008, Blender Foundation
License: Creative Commons Attribution 3.0
21Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Reliability
Deta reliability evaluation for Linux filesystems
Functionality
Long-term testing by accelerated kernel
22Embedded Linux Conference Europe 2014
RT process and GP process runs on a board
CPU
Device 1 Device 2
Hard
ware
Ap
plicati
on
sO
S Linux
RT App
Core 1 Core 2
GP App
Goal
Run RT processes and GP processes on a hardware platform
CPU
Device 1 Device 2
RT App
RT core GP core
GP App
Linux
Isolate from GP application’s behavior
23Embedded Linux Conference Europe 2014
An example for isolation demo
Eth0 SD UART
Core1Core0
Web
server
RT
control
Pandaboard
Workload
(Web clients)
Puppy
Motor
Sensors
(Gyroscope,
etc.)
Motor torque
Sensor values
CAN
RS232C
Serial-CAN
converter
PointRT Control and UART
interrupts are isolated
from the Web server
and other interrupts.
IRQIRQ
24Embedded Linux Conference Europe 2014
DEMO
25Embedded Linux Conference Europe 2014
An issue while control the robot
Latency [μs]
Latency for RT app while running with GP appliation
Evaluation EnvironmentHardware・ Pandaboard- ARM Dual core 1.2GHz- Memory 1GB- SDcard
OS・ Linux 3.6.11.6-RT38
Application・ RT App- Periodic task
Period = 300 micro sec・ GP App- Random I/O on SDcard
0102030405060708090100
0 100 200 300 400 500 600 700 800 900
Num
ber
of sa
mple
s
Before(CPUset)
This latency cause a failure
26Embedded Linux Conference Europe 2014
Realtime performance improvement with Timer-Shielding
Timers Timers
Timermanagement
Hard
ware
Ap
plicati
on
sLin
ux
Kern
el
RT core GP core
RT App GP App
Timers Timers
Timer management
Hard
ware
Ap
plicati
on
sLin
ux
Kern
el
RT core GP core
RT App GP App
Migrator
Timer-Shield ONTimer-Shield OFF
Hardware Partition Management(Create / Modify / Delete)
Migrator
Prohibit to resister non RT related
timers
Migrate
Migrate non RT relate timers
API to manage hardware partitions
Regis
ter
IssueSome kernel task does not able to allocate to target core
27Embedded Linux Conference Europe 2014
An issue while control the robot
0102030405060708090
100
0 100 200 300 400
Nu
mb
er
of
sa
mp
les
Latency [μs]
After
Latency is less than 60us
Latency [μs]
Latency for RT app while running with GP appliation
Evaluation EnvironmentHardware・ Pandaboard- ARM Dual core 1.2GHz- Memory 1GB- SDcard
OS・ Linux 3.6.11.6-RT38
Application・ RT App- Periodic task
Period = 300 micro sec・ GP App- Random I/O on SDcard
0102030405060708090100
0 100 200 300 400 500 600 700 800 900
Num
ber
of sa
mple
s
Before(CPUset)
28Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
29Embedded Linux Conference Europe 2014
Data reliability test
Checker
Target
file
LOG
file
AAAAABBBBBCCCCCDDDDDEEEEE
OK
FFFFF
AAAAABBBBBCCCCCDDDDDEEEEE
OKAAAAABBBBBCCCCCDDDDDAAAAA
NGAAAAABBBBBCCCCCDDDDD
NG
?
size mismatchdata mismatch
Estimated file contents
Estimated file size
Power off while running writer processes
Verify all data
File size
Contents
Test case is available on GitHub ( https://github.com/ystk/fs-test )
30
Results of data reliability tests on Linux filesystems
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
* * * JFS XFS
Err
or
rate
[%
]
45.9%
Err
or
rate
[%
]
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
EXT4-
JOURNAL
EXT4-
ORDERED
EXT4-
WRITEBACK
* XFS BTRFS
82.4%84.7%43.4% 41.4%
43.2%
Err
or
rate
[%
]
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
EXT4-
JOURNAL
EXT4-
ORDERED
* JFS XFS
13.3%
kernel
2.6.18
kernel
2.6.31
kernel
2.6.33
File size mismatch rateData mismatch rate
Point 1:An filesystem hasdifferent characteristicson different kernel
Point 2:2.6.33 has high error rate on ordered and writeback mode
Point 3:Ext4-journal and Btrfshas good resluts
File system types
ke
rne
l ve
rsio
n
31Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
32Embedded Linux Conference Europe 2014
Long-term running test by accelerated kernel
START GOAL
Accelerate
Detect clock counterrelated errors (e.g. time_t)
Accelerate clock counter
jiffies = jiffies + (1 * ratio)
Issue
• Long-term testing takes really long time
→ We want results as fast as possible
Things that NOT able to be accelerated.
• CPU frequency
• HDD or SSD access speed
• Network link speed
• …
33Embedded Linux Conference Europe 2014
Evaluation with accelerated kernel
Xdaliclock works as a stopwatch
About 40 times faster
to get 100% CPU usage
returned incorrect value
after about 450 days
34Embedded Linux Conference Europe 2014
Enough?
Real-time latency is short enough
Data is safe enough
System works long enough
35Embedded Linux Conference Europe 2014
Summary There are many issues need to be solved for SI system
CE workgroup is considering to have sub project for it
Current status Under survey for actual issues
Start call for participation
Initial activity will be the follows: Clarifying issues to be solved
Create a mail list
Have F2F meetings at Linux Foundaton’s conference
Estimated output Publicly available pages on the elinux wiki or Linux Foundation’s web site
which includes the follows:
A list of issues
Specification for reference implementations
A list of link to reference implementations
All kernel related implementation should post to LKML
All source code for key issues will be opened
36Embedded Linux Conference Europe 2014
Discussion
Any comments and suggestions are welcome
Contact information
To get the latest information, please send an email to the following adderss:
Yoshitake Kobayashi [email protected]
Noriaki Fukuyasu [email protected]
37Embedded Linux Conference Europe 2014
40Embedded Linux Conference Europe 2014
Demo (Not Isolated)
41Embedded Linux Conference Europe 2014
Demo (Isolated)