Characterizing the Performance of SpaceWire

on a LEON3FT Ken Koontz, Andrew Harris,

David Edell

Introduction   SpaceWire is emerging as standard high-performance data interface   Recent NASA missions include LRO/Mini-RF, SDO, JWST   Future NASA missions may include SPP, ILN/RLL, JEO

  SpaceWire interfaces are becoming standard on new flight HW   BAE and Aeroflex have processors with SpaceWire

  SpaceWire advertises 2 to 400 Mbps, but…

  Are these speeds really achievable, and under what conditions?

  As part of an IR&D project, we characterized the performance of SpaceWire on a LEON3FT.

  What we found was interesting…

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 2

SpaceWire Background   ESA standard ECSS-E-ST-50-12C (31 July 2008), plus several

“protocol” extensions (-51C, -52C, -53C, …)   SpaceWire packets are composed of a sequence of data characters,

followed by an End-Of-Packet (EOP) character   No explicit length field, which simplifies routing and router design   Data characters are 10 bits: 8-bits data, 2-bits control

  Maximum uni-directional transfer rate is 80% signaling rate (e.g., 10 MHz signaling rate == 8 Mbps data rate).

  SpaceWire packets include a destination address and protocol ID   Destination address used for routing (path or logical addressing)   Protocol ID used at each end-node to identify protocol stack

  For our experiment, we used:   Logical addressing (single byte), point-to-point (no router)   CCSDS Packet Transfer Protocol (CPTP), one SpaceWire packet

contained one CCSDS telemetry packet, single byte protocol ID Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 3

LEON3FT Background

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 4

  SPARC V8 system-on-chip, developed by Gaisler Research   Aeroflex UT699 is a flight-qualified, rad-hard LEON3FT realization   SPARC V8 with FPU and MMU   4 on-chip SpaceWire ports (end-nodes only, no router functions)   2 CAN, 1 PCI, 1 Ethernet, 1 memory controller w/EDAC

The Experiment   Measure uni-directional performance in “flight-like conditions”:

  Point-to-point SpaceWire link (no router, no blocking, no topology issues)   Vary link speed over a predetermined range (2, 5, 10, 25, and 50 MHz)   Vary packet size over a predetermined range (2^N bytes, N=4..16, 16 to 64 Kbytes)   Set transmit and receive speeds the same (adjusted at outermost loop)   Send 50,000 packets at each configuration in innermost loop

  Measure start, end, elapsed time at receiver   Measure CPU load at receiver using flight-like background monitor

  Send CCSDS packets over SpaceWire using CPTP, with standard processing on transmitter and receiver (buffer pools, checksums, sequence counters)

  Hardware   2 Aeroflex LEON3FT commercially available development boards   SYS_CLK = 66 MHz, SPW_CLK = 100 MHz

  System software   VxWorks 6.5   Aeroflex/GR BSP version 1.1.2   GNU compiler, -O2 optimization level

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 5

Packets Per Second vs. SpaceWire Packet Size

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 6

Link Utilization (%) vs. SpaceWire Packet Size

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 7

CPU Load (%) vs. SpaceWire Packet Size

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 8

Effective Data Throughput (Mbps) vs. SpaceWire Packet Size

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 9

Observations

  With small packets, we observed an upper limit of ~4200 packets/sec (or 238 microsecs/packet).

  As packet size increased, processing load remained high until the effect of the DMA units kicked in. Where this occurred varied by link speed.

  We were able to achieve very low CPU load (< 10%) and very high link utilization (> 90%) at 2, 5, and 10 MHz.

  We pegged the CPU load at 100% at 25 & 50 MHz link speeds, until the DMA units kicked in (2K bytes@25 MHz, 8 Kbytes @50 MHz)

  We were only able to achieve 60% link utilization at 50 MHz, even with very large 64 Kbyte packets. This also cost 30% CPU load.

  We did not test 100 MHz (50 MHz LEON3FT max rate). Performance expected to be worse than 50 MHz, even for large packets.

  We were surprised that 1 Kbyte packets at 10 MHz required 50% CPU load and only 80% link utilization.

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 10

Limitations and Optimizations   If you need to use checksums, and implement them in software…   …then expect to pay a price!

  Avoid copying packets as much as possible!   In the stock driver, packets are DMA’d into a driver buffer, and then

copied to an application buffer.   This can kill performance with large packets!

  The driver and application should use a common buffer pool   Pass pointers to buffers between software elements   Copy data only on TX and RX, and with the aide of DMA   This is common practice with slower interfaces such as 1553 (1 Mbps)   SpaceWire drivers and applications need to play catch-up

  Use interrupts only when they are necessary in the application   The GRSPW interfaces can support DMA wo/interrupts   The VxWorks driver only supports DMA w/interrupts   If speed is the utmost concern, remove VxWorks and code in Bare C!

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 11

Questions?

Characterizing the Performance of SpaceWire on a LEON3FT, FSW-10 Workshop 12