![Page 1: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/1.jpg)
Sponsored in part by:
Single Core Equivalence Framework (SCE) For Certifiable Multicore Avionics
http://rtsl-edge.cs.illinois.edu/SCE/
a collaboration of:
Presenters: Lui Sha, Marco Caccamo, Heechul Yun, Renato Mancuso, Jung-Eun Kim With contributions from Rodolfo Pellizzoni and Man Ki Yoon Russell Kegley, Jonathan Preston and Dennis Perlman Greg Arundale and Richard Bradford
![Page 2: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/2.jpg)
Outline
1. Overview
2. CAST32 and SCE
3. DRAM, Bus and Cache management
4. IMA and I/O management
5. SCE Summary
SCE single-core equivalence
2
![Page 3: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/3.jpg)
Transition to Multi-Core
multi-core benefits: reduced space and weight
reduced power and cooling
increased computation
… and more
We have a large body of certified single-core hard real-time software
Existing software will be migrated en-masse, in addition to new software.
There are
serious risks
3
![Page 4: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/4.jpg)
• “Shared access to cache or other memory areas, operating systems / supervisors / hypervisors that can control and affect all the applications executing on all the cores, and ‘coherency fabrics / coherency modules / interconnects’ that control all the data transfers between the MCP cores, memory and the peripheral devices of the MCP via a shared bus.
• Many of these features … were not designed or verified for compliance with the current airborne software or hardware guidance material.
• It may therefore be difficult or even impossible to fully characterize and verify all the possible effects of these features, which may include unintended and unexpected behavior.”
- CAST 32
Inter-core Interferences
4
![Page 5: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/5.jpg)
• Lockheed Space Systems ported
some applications to a Freescale
P4080 testbed. Tests indicated that
• Recorded max delay (blue bar) of a task
increased 6x, when 7 cores were used, but
not when 8 cores were used. This makes
the determination of worst case
configuration difficult.
• Using SCE technology (red bar)
– Recorded max delay of a task increased
monotonically when more cores were used.
– WCET(8) > WCET(j), j = 1 to 7; and
increased less than 2x.
Critical Software can be Slowed
by up to 6X if Uncontrolled
Source: Lockheed Space Systems HWIL Testbed
5
• Currently SCE core isolation technology is software based. The isolation overhead increases when more cores are used.
• Hardware design support can greatly reduce the overhead but requires cooperation from chip makers.
Testbed Results
5
![Page 6: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/6.jpg)
• DO178 C was developed for single core chips under the assumption that a acceptably tight bound on a task T’s execution time (WCET) can be determined and reused for different task sets in a given platform.
• This constant WCET assumption makes schedulability analysis, timing tests and timing certification tractable. Without it, any change to any task mandates the recalculation of all other tasks’ WCETs.
• In a multicore chip, as is, physically concurrent sharing of globally available DRAM banks, memory bus, last level cache, and I/O channels invalidates the traditional constant WCET assumption.
• SCE generalizes the constant WCET assumption in the form of constant WCET(m) assumption, where m is the maximal number of cores that will be used in a multicore chip. SCE allows the reuse of DO178 C as is with WCET(1), a.k.a, WCET, replaced by WCET(m).
6
DO178C and the WCET Assumption
6
![Page 7: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/7.jpg)
Sponsored in part by:
Single Core Equivalence Framework (SCE) For Certifiable Multicore Avionics
SCE and CAST-32
a collaboration of:
Presenter: Marco Caccamo
![Page 8: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/8.jpg)
CAST-32 Position
DO-178B and DO-178C only address software on single-core processor
No existing material to adapt development and verification on MCPs
In MCPs, applications on separate cores may cause interference with each other
CAST-32/position g.i
1 MCP Interference Channels
2 Shared Memory and Cache
3 Planning and Verification of Resource Usage
4 Software Verification
position d
position e
position f
position h
8
![Page 9: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/9.jpg)
SCE Overview
a framework of OS-level techniques
implementable on commercial MCP platforms
for strict partitioning of shared resources
so that each core can be treated as a single-core chip
from a schedulability analysis
and certification perspective
SCE single-core equivalence
is
9
![Page 10: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/10.jpg)
CAST-32 Coverage
SCE single-core equivalence
and
MCP Interference Channels
CAST-32
“Applications running on different cores of a MCP do not execute independently from each other because the cores are sharing resources”
CAST-32/position d.i
1
Within SCE we have: • Identified and analyzed the main interference channels • Provided a mitigation strategy for each channel • Exported a set of equivalent, independent single-cores
“The applicant has conducted a functional interference analysis […] and has designed, implemented and verified a means of mitigation for each interference channel”
CAST-32/position d.ii
10
![Page 11: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/11.jpg)
CAST-32 Coverage
SCE single-core equivalence
and
Shared Memory and Cache
CAST-32
“WCET of the software applications hosted on one core can increase greatly due to repeated cache accesses by the processes hosted on the other core”
CAST-32/position e.i
2
SCE provides: • Per-process cache usage profiling mechanism • Deterministic shared cache allocation strategy • No inter- and intra-core interference on cache space
“The applicants have to describe their strategy for managing and verifying cache usage” and “to conduct analyses of worst-case effect of shared cache”
CAST-32/position e.ii
11
![Page 12: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/12.jpg)
CAST-32 Coverage
SCE single-core equivalence
and
Planning and Verification of Resource Usage
CAST-32
“If the overall available resources of the MCP are exceeded by the combined resource demand, the effects on the software may be unpredictable”
CAST-32/position f.i
3
SCE provides: • Per-core memory bandwidth regulation mechanism • Guarantee of operation below saturation point • Serialization of I/O transactions
“The applicants have to describe their plans to allocate, manage and measure the use of the interconnect used by applications and peripherals”
CAST-32/position f.ii
12
![Page 13: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/13.jpg)
CAST-32 Coverage
SCE single-core equivalence
and
Software Verification
CAST-32
“Existing guidance and standard industry practice for the integration and verification of hardware platforms, OSes and applications is the field of IMA systems”
CAST-32/position h.i
4
In summary, using SCE: • Perform per-core modular analysis and certification • Reuse consolidated software and engineering processes • Use an IMA approach on each equivalent single-core • Verification of SCE implementation is an open challenge
“A similar approach […] would be effective to the verification of software on an MCP” since it “would not impose any additional burden on the industry”
CAST-32/position h.i
13
![Page 14: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/14.jpg)
Sponsored in part by:
Single Core Equivalence Framework (SCE) For Certifiable Multicore Avionics
Tech. Overview and Cache Management
a collaboration of:
Presenter: Renato Mancuso
![Page 15: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/15.jpg)
Core 1 Core m
...
Interconnect
DRAM I/O
ch. 1 I/O
ch. n
...
Shared Cache
I/O Core
m Application Cores +
1 I/O Core
Shared Last Level Cache
Shared resources regulated by SCE Over-provisioned resources
Memory Controller
Shared Resources Regulated by SCE
Shared Interconnect
Shared I/O Peripherals
Shared DRAM memory
Shared Memory Controller
15
![Page 16: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/16.jpg)
SCE Tech. 1 – Colored Lockdown
m Application Cores +
1 I/O Core
Per-Core Assigned Cache
Shared Interconnect
Shared I/O Peripherals
Core 1
... I/O ch. 1
I/O ch. n
...
Assigned Cache
I/O Core
Core m
Assigned Cache
...
deconflict
Interconnect
DRAM
Memory Controller Shared DRAM
memory
Shared Memory Controller
Shared resources regulated by SCE Over-provisioned resources
16
![Page 17: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/17.jpg)
SCE Tech. 2 – MemGuard
m Application Cores +
1 I/O Core
Per-Core Assigned Cache
Core 1
... I/O ch. 1
I/O ch. n
...
Assigned Cache
I/O Core
Core m
Assigned Cache
...
Interconnect
DRAM
Shared resources regulated by SCE Over-provisioned resources
MC/1 MC/m
deconflict
...
Shared I/O Peripherals
Shared DRAM memory
Per-Core Assigned Mem.
Bandwidth
Shared Interconnect
17
![Page 18: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/18.jpg)
SCE Tech. 2 – Palloc
m Application Cores +
1 I/O Core
Per-Core Assigned Cache
Core 1
... I/O ch. 1
I/O ch. n
...
Assigned Cache
I/O Core
Core m
Assigned Cache
...
Interconnect
DRAM/1
Shared resources regulated by SCE Over-provisioned resources
MC/1 MC/m
deconflict
...
Shared I/O Peripherals
Per-Core DRAM banks
Per-Core Assigned Mem.
Bandwidth
Shared Interconnect
DRAM/m ...
18
![Page 19: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/19.jpg)
SCE Tech. 4 – I/O Scheduling
m Application Cores +
1 I/O Core
Per-Core Assigned Cache
Core 1
... I/O ch. 1
I/O ch. n
...
Assigned Cache
I/O Core
Core m
Assigned Cache
...
Interconnect
DRAM/1
Shared resources regulated by SCE Over-provisioned resources
MC/1 MC/m ...
Serialized I/O Transactions
Per-Core DRAM banks
Per-Core Assigned Mem.
Bandwidth
Shared Interconnect
DRAM/m ... deconflict
19
![Page 20: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/20.jpg)
SCE: Engineering Perspective
SCE dedicates 𝟏
𝒎 of shared resources to each core.
WCET of tasks directly depends on the number of active cores m.
To certify for up to m active cores, find WCET(m) for each task
WCET(m) can be derived from WCET calculated in isolation
WCET(𝑚) = WCET(1) + 𝜇 ⋅ 𝐿𝑠𝑖𝑧𝑒𝑚
𝐵𝑊𝑚𝑖𝑛−
1
𝐵𝑊𝑚𝑎𝑥
(*) Renato Mancuso, Rodolfo Pellizzoni, Marco Caccamo, Lui Sha, Heechul Yun, WCET(m) Estimation in Multi-Core Systems using Single Core Equivalence. In Proceedings of the 27th Euromicro Conference on Real-Time Systems (ECRTS 2015), Lund, Sweden. To appear2015
*
20
![Page 21: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/21.jpg)
Consider multi-core platform
Determine relevant
parameters
Profile workload and
define partitions
Collect experimental
measurements
Compute WCET(m)
Check per-partition
schedulability
Generate partition and I/O schedule
SCE: Engineering Perspective
SCE single-core equivalence
the
workflow
21
![Page 22: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/22.jpg)
Our LLC Management Model:
• Consider the LLC as a 2D array of lines
• Assign arbitrary sets of blocks to tasks
Sets
Ways
✔ Addresses all the sources of interference
✔ Converts the LLC cache in a deterministic object
at the granularity of a single memory page
✔ Allows the use of legacy code
✔ Provides flexibility in cache assignment
SCE: Colored Lockdown
(*) Renato Mancuso, Roman Dudko, Emiliano Betti, Marco Cesati, Marco Caccamo, Rodolfo Pellizzoni, Real-Time Cache Management Framework for Multi-Core Architectures. In Proceedings of the 19th IEEE International Conference on Real-Time and Embedded Technology and Applications Symposium (RTAS 2013), Philadelphia, PA, USA.
*
22
![Page 23: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/23.jpg)
• Aims at using the cache deterministically
• Has to deal with limited cache size
• Run task in sandbox, analyze memory accesses
• Find frequently accessed (hot) memory regions
1. Profiling
✔
✔
✔ ✔
✔
Sets
Ways
Colored Lockdown: Profiling
23
![Page 24: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/24.jpg)
• Leverages on the virtual → physical translation layer
• Used to move page mapping across sets (up/down)
• Transparent to the programmer
• Transparent to the application
2. Coloring
Sets
Ways
Colored Lockdown: Coloring
24
![Page 25: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/25.jpg)
• Uses architecture-specific lockdown features
• Used to allocate pages on selected ways (left/right)
• Can be implemented at OS-level
3. Lockdown
Sets
Ways
Colored Lockdown: Lockdown
25
![Page 26: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/26.jpg)
• DO 178C can be used for multicore, only if each core in a multicore chip is logically equivalent to a single core chip (SCE). SCE enables to modularly certify software one core at a time using DO 178C.
• Technologies to implement the SCE framework are open to innovation. However, violation of SCE objective means that we would allow the modification of applications in one core to “decertify” the applications in other cores.
• Challenges in SCE technology development and certification
– Currently SCE addresses the isolation challenges.
• Intercore communication support needs to be completed.
• How to use more than one core for big applications needs to be completed.
– SCE certification is architecture dependent. Validated hardware abstraction required.
– SCE integrates with low level RTOS operation and is harder to verify than application level software. But only needs to be done once for a platform.
Summary
26
![Page 27: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/27.jpg)
Sponsored in part by:
Single Core Equivalence Framework (SCE) For Certifiable Multicore Avionics
Memory Mnagement
a collaboration of:
Presenter: Heechul Yun
![Page 28: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/28.jpg)
This Talk
• Focus on DRAM and memory controller
• Present SW mechanisms for timing predictability
28
Core1 Core2 Core3 Core4
DRAM
Memory Controller
![Page 29: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/29.jpg)
Why Important?
• Memory is becoming a bottleneck
• Performance is very poor in the worst-case
29
Core1 Core2 Core3 Core4
DRAM
Memory Controller
![Page 30: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/30.jpg)
How Serious? • Synthetic worst-case experiments:
30
8.0
33.5
45.8
0
5
10
15
20
25
30
35
40
45
50
ARMCortex A15
IntelNahelem
IntelHaswell
solo
+ 1 co-runner
+ 2 co-runners
+ 3 co-runners
DRAM
Core1 Core2 Core3 Core4
bench N
orm
aliz
ed e
xecu
tio
n t
ime
co-runner(s)
Up to 45.8X slowdown
![Page 31: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/31.jpg)
Background: DRAM Organization
L3
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
• Have multiple banks – 8 ~ 16 banks per DIMM
• Different banks can be accessed in parallel
31
![Page 32: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/32.jpg)
Most-cases
L3
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
Mess
• Performance = ??
32
![Page 33: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/33.jpg)
Worst-case
• 1bank b/w – Less than peak b/w
– How much?
Slow
L3
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
33
![Page 34: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/34.jpg)
Outline
• Introduction
• DRAM Background
• Control mechanisms
– PALLOC: Space (bank) partitioning *
– MemGuard: Bandwidth partitioning **
• Conclusion
34
(*) Heechul Yun, Renato Mancuso, Zheng-Pei Wu, Rodolfo Pellizzoni. PALLOC: DRAM Bank-Aware Memory Allocator for Performance Isolation on Multicore Platforms. IEEE Intl. Conference on Real-Time and Embedded Technology and Applications Symposium (RTAS), IEEE, 2014
(**) Heechul Yun, Gang Yao, Rodolfo Pellizzoni, Marco Caccamo, and Lui Sha. MemGuard: Memory Bandwidth Reservation System for Efficient Performance Isolation in Multi-core Platforms. IEEE Intl. Conference on Real-Time and Embedded Technology and Applications Symposium (RTAS), IEEE, 2013.
![Page 35: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/35.jpg)
Problem
L3
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
• OS/hypervisor is unaware of DRAM banks
• Memory pages are spread all over multiple banks
???? Unpredictable Bank Conflict
OS/Hypervisor
35
![Page 36: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/36.jpg)
DRAM DIMM
PALLOC
CPC
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
• Aware of DRAM mapping
• Each page can be allocated to a desired DRAM bank
Flexible Allocation Policy
OS/Hypervisor
36
![Page 37: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/37.jpg)
PALLOC
L3
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1 Core2 Core3 Core4
• Private banking
– Allocate pages on certain exclusively assigned banks
Better Performance
Isolation
37
![Page 38: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/38.jpg)
Performance Slowdown
• PB: DRAM bank partitioning only; • PB+PC: DRAM bank and Cache partitioning • Bank (and cache) partitioning improves isolation, but far from ideal
– Due to Memory bus bandwidth contention (next technique)
38
Slo
wd
ow
n r
ati
o
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
buddy PB PB+PC
DRAM
Core1 Core2 Core3 Core4
bench co-runner(s)
![Page 39: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/39.jpg)
Problem
• Banks can be accessed in parallel
• But all banks share a memory bus
• Memory bandwidth << CPU demands
39
Memory bandwidth contention
Shared Cache
DRAM DIMM
Memory Controller (MC)
Bank 4
Bank 3
Bank 2
Bank 1
Core1
Core2
Core3
Core4
Shared memory bus
![Page 40: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/40.jpg)
MemGuard
40
• Goal: guarantee minimum memory b/w for each core • How: b/w reservation
Operating System
Core1
Core2
Core3
Core4
PMC PMC PMC PMC
DRAM DIMM
MemGuard
Multicore Processor Memory Controller
BW Regulator
BW Regulator
BW Regulator
BW Regulator
0.6GB/s 0.2GB/s 0.2GB/s 0.2GB/s
Reclaim Manager
![Page 41: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/41.jpg)
Reservation • Idea
– Reserve per-core memory bandwidth via the OS scheduler • Use h/w PMC to monitor memory request rate
41
1ms 2ms 0
Schedule a RT idle task
Suspend the RT idle task
Budget
Core
activity
2 1
computation memory fetch
as long as sum budgets <= guaranteed
memory bandwidth, queuing delay in the memory controller is small
![Page 42: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/42.jpg)
Impact of Reservation
42
LLC
mis
ses/
ms
Time (ms) Time (ms)
W/o MemGuard MemGuard (1GB/s)
LLC
mis
ses/
ms
![Page 43: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/43.jpg)
Conclusion
• Multicore certification is a huge challenge
• Main memory is an important interference channel – Bank (space) conflict
– Bandwidth contention
• Proposed control mechanisms – PALLOC: DRAM bank (space) control
– MemGuard: DRAM bandwidth (time) control
Improved performance isolation
43
![Page 44: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/44.jpg)
Sponsored in part by:
Single Core Equivalence Framework (SCE) For Certifiable Multicore Avionics
IMA & I/O Management
a collaboration of:
Presenter: Jung-Eun Kim
![Page 45: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/45.jpg)
The MCP IMA Challenge
• “Authorities are not currently aware of any MCP hardware and software implementations … in the way … currently ensured for the applications of an IMA on a single core
processor (SCP).“ in CAST 32
• “This paper may be extended in future to address MCPs with more than two active cores and MCP IMA implementations.” in CAST 32
![Page 46: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/46.jpg)
Migration: I/O Conflicts
– Zero-partition
: a special-purpose ‘I/O partition’
– Migrating multiple single-core IMAs to a multicore system.
• Multiple rate groups
• Shared I/O channel conflicts
• Synchronizing challenge
z
z
Z: zero-partition
core k
core k+1
z z
z
![Page 47: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/47.jpg)
All Things are Putting Together
Cache management
Memory management
IMA partition parameters determined
integrated
IMA partitions scheduling with conflict-free I/O
![Page 48: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/48.jpg)
(Cache locking) +
Processing Partition (memory bandwidth
regulated) +
(Cache Unlocking)
Generating IMA Partition Scheduling for Conflict-free I/O
Input
Output
. . .
Input
Output
Processing Partition
Input
Output
Processing Partition
. . .
. . .
. . .
. . .
I/O core Core 1 Core 2 Control (schedule) access to I/O devices
Processing partition
Cache locking +
Memory bandwidth regulation
+ Processing
+ Cache unlocking
One I/O at A time
![Page 49: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/49.jpg)
Idea – How to Solve
Bottleneck-first approach
Jung-Eun Kim, Man-Ki Yoon, Sungjin Im, Richard Bradford and Lui Sha, “Optimized Scheduling of Multi-IMA Partitions with Exclusive Region for Synchronized Real-Time Multi-Core System,” in Proceedings of the 16th ACM/IEEE Design, Automation, and Test in Europe (DATE 2013), pp. 970-975, Mar. 2013.
Jung-Eun Kim, Man-Ki Yoon, Richard Bradford and Lui Sha, “Integrated Modular Avionics (IMA) Partition Scheduling with Conflict-Free I/O for Multicore Avionics Systems,” in Proceedings of the 38th IEEE Computer Software and Applications Conference (COMPSAC 2014), Jul. 2014.
Allocate first Strictly periodic processing partitions
Search space reduced
Allocate Semi-periodic I/O partitions
![Page 50: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/50.jpg)
Result of a Practical Example 1 I/O Core + 2 Processing cores; Periods (core_1: 40,200,100,100,100,40; core_2: 60, 40, 100); LCM=600
(magnified)
I/O core (core 0)
core 1
core 2
D. Locke, L. Lucas and J. Goodenough, “Generic avionics software specification,” Software Engineering Institute, Pittsburgh, Pennsylvania,1990, CMU/SEI-90-
TR-008.
![Page 51: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/51.jpg)
SCE Summary • DO 178C can be used for multicore, only if each core in a multicore chip is
logically equivalent to a single core chip (SCE). That is, intercore interferences can be certifiably bounded and for all core workload configurations.
• SCE Technologies is open to innovation. However, violation of SCE objective means that we would allow the modification of applications in one core to “decertify” the applications in other cores.
• Challenges in SCE technology development and certification
– Currently SCE addresses the isolation challenges.
• Intercore communication support needs to be completed.
• How to use more than one core for big applications needs to be completed.
– SCE certification is chip architecture dependent, requires hardware primitives currently found in some Freescale chips.
– Validated hardware abstraction required.
– Verification and certification of SCE design & implementation are required.
![Page 52: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/52.jpg)
SCE: Engineering Perspective
52
SCE single-core equivalence
and
IMA
![Page 53: Single Core Equivalence Framework (SCE) For Certifiable ...rtsl-edge.cs.illinois.edu/CMAS/media/02-Lui.pdf · SCE Overview a framework of OS-level techniques implementable on commercial](https://reader034.vdocuments.us/reader034/viewer/2022051912/60037c02328ef41bc724dedd/html5/thumbnails/53.jpg)
SCE: Engineering Perspective
53
SCE single-core equivalence
and
IMA