outline

CHEP’04, 27th. Sep. – 1st. Oct., 2004

1

An Embedded Linux System An Embedded Linux System Based on PowerPCBased on PowerPC YE MeiYE [email protected]@ihep.ac.cnExperimental Physics Center Experimental Physics Center Institute of High Energy PhysicsInstitute of High Energy Physics

CHEP’04, 27th. Sep. – 1st. Oct., 2004

2

OutlineOutline

1.1. IntroductionIntroduction

2.2. The Readout SystemThe Readout System

3.3. Overview of the Overview of the TestbedTestbed

4.4. Design and ImplementationDesign and Implementation

5.5. ConclusionsConclusions

CHEP’04, 27th. Sep. – 1st. Oct., 2004

3

IntroductionIntroduction

•12 years since the commission of the BES at BEPC

• Both machine and detector have undergone the upgrade

•The peak luminosity of 1033cm-2sec-1

• The estimated event rate: about 4000Hz at J/ψpeak

• The new electronics will up to higher event rate and has much more noise environment.

The DAQ system will adopt embedded real time operation system(RTOS) and corresponding programming techniques to fit the readout requirement of the Front-End Electronics(FEE).

CHEP’04, 27th. Sep. – 1st. Oct., 2004

4

The Readout SystemThe Readout System

• Designed to adopt Motorola’s PowerPC

• Real-time operation system : Multi-level parallel data collecting and Real-time operation system : Multi-level parallel data collecting and processing schemesprocessing schemes

• Work with the readout electronics: performs initialization and Work with the readout electronics: performs initialization and calibrations of sub-systems; collect, pre-process and transfer datacalibrations of sub-systems; collect, pre-process and transfer data

• Several readout crates are connected to a readout PC through fast Ethernet, thus constituting a readout branch

• All the readout branches are connected to the online computer farm through gigabit switch

• The events are recorded in persistent media.

CHEP’04, 27th. Sep. – 1st. Oct., 2004

5


• The main tasks of the readout system are:The main tasks of the readout system are:

• collecting event data from the front-end electronics after collecting event data from the front-end electronics after

Level 1 triggerLevel 1 trigger

• pre-processing and assemblingpre-processing and assembling

• transferring data fragments from the VME readout crate transferring data fragments from the VME readout crate

to the readout PC to the readout PC

CHEP’04, 27th. Sep. – 1st. Oct., 2004

6

Benefit from Embedded LinuxBenefit from Embedded Linux

• High price of the commercial RTOS: VxWorks, LynxOS

• The possibility of using the embedded real-time Linux as the operating system of the readout system.

In comparison with VxWorks (Single Seat) : 1:3In comparison with VxWorks (Single Seat) : 1:3

CHEP’04, 27th. Sep. – 1st. Oct., 2004

7

Overview of the Development EnvironmentOverview of the Development Environment

TimeSys Analysis ToolsWindows

Online Software , Linux

Gigabit Ethernet

100M Ethernet

Readout PCRH 7.3 ~ 9.0

TimeSys Linux Target

FEEs

PowerPC:MVME5500

Debug

Host X86

TimeSys SDK & IDE Env.RH 7.3 ~ 9.0

Host X86

CHEP’04, 27th. Sep. – 1st. Oct., 2004

8


To accomplish above tasks, the testbed system must be designed to have the following functions:

• To collect data from the VME readout modules (ADC and TDC) at To collect data from the VME readout modules (ADC and TDC) at high speed and full utilize the VME bus bandwidth;high speed and full utilize the VME bus bandwidth;

• To set up the multi-level buffering techniques that solve the To set up the multi-level buffering techniques that solve the “bottleneck” problem caused by the VME data fragments assembling “bottleneck” problem caused by the VME data fragments assembling and the network transmission; and the network transmission;

• To use the necessary task synchronization strategy and software To use the necessary task synchronization strategy and software protocols that ensure the correct parallel processing;protocols that ensure the correct parallel processing;

• To build the effective channel between the kernel module and the To build the effective channel between the kernel module and the user module;user module;

• To simulate the Front-End Electronics modules to generate the To simulate the Front-End Electronics modules to generate the VMEbus trigger and Monte Carlo data;VMEbus trigger and Monte Carlo data;

• To establish the APIs between the readout code and the online To establish the APIs between the readout code and the online software.software.

CHEP’04, 27th. Sep. – 1st. Oct., 2004

9

Battery

NVRAMRTC

FLASH32 MB

(soldered)8MB

(sockets)

66 MHz PCI Local Bus -- A32/D64-bitA32/D64-bit 100 MHz PCI Bridge & Memory Controller

Discovery

Universe II

133MHz SDRAM

512MBPowerPC ISA

MPC7455@ 1 GHz

L3 Cache200MHz

2MB

SDRAMMezzanine

512MB

66 MHz PCI Local Bus

PMC FP I/O

PMCSlot 2

P1

IPMCSlot 1

P2

PCIBridge

33 MHz PCI Local Bus -- A32/D64-bit

10/100Ethernet

PHY

10/100 BaseTX F/P I/O

10/100/1000BaseTXF/P I/O

PMCspanConnector

GigabitEthernet

Controller

PMC FP I/OJumperSelectable

Port 1

Port 2

A32/D64-bit

UART

Header

EIA232

FP I/OCom 1 Com 2

PowerPC Intro PowerPC Intro —— MVME5500 MVME5500

Features:• MPC7455 microprocessor at 1 GHz• 2MB of L3 cache • 100 MHz front-side bus • 256/512MB of on-board ECC SDRAM —

expandable up to 1GB with optional

memory expansion module• 40MB Flash memory • Dual IEEE P1386.1 compatible 32/64-bit

33/66 MHz PMC expansion slots • 64-bit PCI expansion mezzanine

connector allowing up to four more

PMCs • Dual serial ports • 10/100 and Gb Ethernet interfaces• Single VME slot when configured with

PMC modules and memory mezzanine• MVME51xx I/O compatibility• MOTLod firmware

CHEP’04, 27th. Sep. – 1st. Oct., 2004

10

PowerPC Intro PowerPC Intro —— MVME2431 MVME2431

• MotorolaMotorola 2400 2400 Series:MVME2431Series:MVME2431

• CPU:MPC750 PowerPC(350MHZ) CPU:MPC750 PowerPC(350MHZ)

• Up to 1MB of backside L2 cache Up to 1MB of backside L2 cache

• 32MB SDRAM 32MB SDRAM

• 8MB of on-board Flash(bankB), 8MB of on-board Flash(bankB),

1MB of socketed Flash(bankA),1MB of socketed Flash(bankA),

• 10/100BaseT Ethernet10/100BaseT Ethernet

• One serial portOne serial port

• PPCBug:evaluation and debugging PPCBug:evaluation and debugging

tooltool

CHEP’04, 27th. Sep. – 1st. Oct., 2004

11

Fully Preemptible Linux GPL Kernel

Standard Linux/POSIX APIs

TimeSys Linux RTOS OverviewTimeSys Linux RTOS Overview

PrioritizableInterruptHandler

Constant Time Scheduler

RTOS Modules• POSIX Real-Time• Priority Inversion

Control KLM• High Resolution

Timer KLM

Reservations Add-on

PrioritizableSoft-IRQ(aka Bottom

Half)

Linux App Linux App JTime Add-on…

CHEP’04, 27th. Sep. – 1st. Oct., 2004

12

TimeSys Linux RTOS:TimeSys Linux RTOS:Software Development Kit (SDK)Software Development Kit (SDK)

TimeSys Linux RTOS SDKs

2.4.21 Linux Kernel• GPL Performance Improvements• RTOS Kernel Loadable Modules• Certified drivers for hardware• CGL and CE Linux features

Root File System• CGL & CE Linux Packages• 100+ Packages & 196 Tools

Reservations Add-on• Loadable kernel module• CPU and Network

resource management

JTime Add-on• RTSJ compliant Java

Virtual Machine

GUI-Based Tools• TimeStorm Linux Development

Suite • TimeStorm IDE• TimeTrace

Certified Toolchains• gcc/g++ 3.2, gdb 5.3, glibc 2.2.5,

binutils 2.13, KGDB 5.2.1• Windows or Linux hosting

CHEP’04, 27th. Sep. – 1st. Oct., 2004

13

Professional & Designer Editions: Integrated Professional & Designer Editions: Integrated with TimeStorm IDEwith TimeStorm IDE

• Full-featured IDE for any Linux

• Kernel, driver and application development, debug, and deployment

• Linux & Windows based IDE

• Utilizes standard tools

• GCC, GDB, Make, ……

• Foundation for all TimeStorm Tools

• Built on Eclipse framework

• Extensibility: 350+ open-source and commercial plug-ins available

• Eclipse v2.1 and CDT v1.0.1 Easily create, download and

debug C/C++/Java code

CHEP’04, 27th. Sep. – 1st. Oct., 2004

14

Professional & Designer Editions: Integrated Professional & Designer Editions: Integrated with TimeTracewith TimeTrace

• Graphically represent exactly what is happening within a system.

• Critical events unfold within an application

• Context switches

• Mutex locks/unlocks

• Scheduling events

• Timer events

• Many others including user defined

• Dynamically instrumented on “production” code!

CHEP’04, 27th. Sep. – 1st. Oct., 2004

15

Measurement TechniqueMeasurement Technique

• Test the performance of the system such as:Test the performance of the system such as:

• The time spend on VME read and write operationsThe time spend on VME read and write operations

• The speed of DMA read operationThe speed of DMA read operation

• System latency when receive interruptSystem latency when receive interrupt

• Network throughput and the CPU rate on network Network throughput and the CPU rate on network

transferstransfers

• Integrated Test: Block/SignalIntegrated Test: Block/Signal

• Response JitterResponse Jitter

CHEP’04, 27th. Sep. – 1st. Oct., 2004

16

System LatencySystem Latency

Ti meSys VxWorksItem MVME5500->2431 MVME5100->2431

One VME Read 1942(ns) 1330(ns)

One VME Write 407(ns) 440(ns) DMA(A32/D32/BLT) 18. 6 MBps 18. 9 MBpssi ze=4096(Bytes)

DMA(A32/D64/MBLT) 35. 9 MBps 37. 9 MBpssi ze=4096(Bytes)

DMA Overhead(K) 7. 1/ 17. 1(μ s) 8. 6(μ s)Interrupt Latency（ H.S.） 9. 9/ 14.4(μ s) 10. 2(μ s)Network Speed(1024) 94. 26(Mbps) 90. 4(Mbps)Network CPU Rate 15% 52%Network Idle 85% 48%

CHEP’04, 27th. Sep. – 1st. Oct., 2004

17

Readout ModuleReadout Module

Front End(MVME2431)

Front End(MVME2431)

Online Software

MVME5500

DMADMA PACKPACK NETNET

DataSize

RngBuf RngBuf

DataSize

DA

TA

VM

EB

us

Shake Hand

DATA

DATA

Trig.

Shake Hand

Handler

Signal

CHEP’04, 27th. Sep. – 1st. Oct., 2004

18

Result with Block ModuleResult with Block Module

• Intr(blk) +DMA +Pack+NetIntr(blk) +DMA +Pack+Net• msg_q(2)+rngBuf(2)msg_q(2)+rngBuf(2)

• 1 Trig.1 Trig.

Event Rate : 10300Event Rate : 10300 CPU = 72.8%CPU = 72.8%

• 16 Trig.16 Trig.


• VxWorks Ref.VxWorks Ref. • 1 Trig.1 Trig. Event Rate : 11600Event Rate : 11600 CPU = 88%CPU = 88%

• 16 Trig. 16 Trig. Event Rate : 16000Event Rate : 16000 CPU = 84%CPU = 84%

CHEP’04, 27th. Sep. – 1st. Oct., 2004

19

The Result with Signal ModuleThe Result with Signal Module

• Intr(sig)+Pack+DMA+NetIntr(sig)+Pack+DMA+Net• msg_q(2)+rngBuf(2)msg_q(2)+rngBuf(2)

• 1 Trig.1 Trig.




CHEP’04, 27th. Sep. – 1st. Oct., 2004

20

The Result with IPC Message QueueThe Result with IPC Message Queue

• Intr(sig)+Pack+DMA+NetIntr(sig)+Pack+DMA+Net• Msg_IPC(2)+rngBuf(2)Msg_IPC(2)+rngBuf(2)

• 1 Trig.1 Trig.




CHEP’04, 27th. Sep. – 1st. Oct., 2004

21

Jitter Tests of Real-Time OS PerformanceJitter Tests of Real-Time OS Performance

(Under Heavy Idle Task and Network Load) (Under Heavy Idle Task and Network Load)• Intr(sig)+DMA+Net+rngBuf(1)Intr(sig)+DMA+Net+rngBuf(1)• Intr(blk)+DMA+Net+rngBuf(1)Intr(blk)+DMA+Net+rngBuf(1)

Exit When the rngBuf fullExit When the rngBuf full

11

1 1 1 1 1

0

0. 2

0. 4

0. 6

0. 8

1

1. 2

256/247267(TB)512/159466(TB)1024/153629(TB)2048/74679(TB)4096/63183(TB)8192/52724(TB)16384/52049(TB)

CHEP’04, 27th. Sep. – 1st. Oct., 2004

22

TimeSys Linux/Real-Time

Embedded Linux

Real-Time Unix

Desktop OSCommercial Unix

Widely-Used RTOS

Benchmark from TimeSys/Linux RTOSBenchmark from TimeSys/Linux RTOS

CHEP’04, 27th. Sep. – 1st. Oct., 2004

23

Long Time Test RunLong Time Test Run

80008000Hz,Hz,2020HrsHrs

1000010000Hz,Hz,1717HrsHrs 1800018000Hz,Hz,6262HrsHrs

Net: 12MB/sNet: 12MB/s

CHEP’04, 27th. Sep. – 1st. Oct., 2004

24

ConclusionsConclusions

• As experiment discussed above the strategy using the As experiment discussed above the strategy using the embedded embedded real-time Linuxreal-time Linux as the operating system of the as the operating system of the PowerPC PowerPC is able to fitis able to fit the essential requirement of the the essential requirement of the readout system. readout system.

• In addition to the task of readout and net transmission In addition to the task of readout and net transmission there is still there is still enough CPU idleenough CPU idle to ensure the stabilization of to ensure the stabilization of the system. the system.

• Considering the nice price and performance of the Considering the nice price and performance of the TimeSys Linux/real-time the TimeSys Linux/real-time the DAQ could take into accountDAQ could take into account the possibility of adopting the the possibility of adopting the real-time Linuxreal-time Linux..

CHEP’04, 27th. Sep. – 1st. Oct., 2004

25

ThanksThanks

27th Sep. 200427th Sep. 2004Congress CentreCongress CentreInterlaken, SwitzerlandInterlaken, Switzerland

outline

Documents

linux hostingchep04

mb sdram

pci local busp1p2pcibridge33

mb of socketed flashbanka

professional designer

mb of backside l2 cache

application development

timestorm toolsbuilt