Lossless packet compression: a technique to deal with bandwidth limitations

David Evans

The problem with housekeeping..

A mix of different field lengths and data types

Different packets are then written into the same packet store

This mixed up data is hard to compress.

Initial steps..

ROSETTA

7zip14%

winzip27%

7zip9%

winzip16%

3) Group packet types before compressing

1) Recreate a real on-board packet store on the ground

2) Compress it with commercial products to determine level of information redundancy RICE

100%

A bit of lateral thinking..

....#...##...#..###.#.#.#.##............##.#..##....#.##.#..#..###.#.#######.......#...##...#..###.#.#.#.##...#........##.#..##....#.##.#..#..###.##......#.......#...##...#..###.#.#.#.##..#.........##.#..##....#.##.#..#..###.##.....##.......#...##...#..###.#.#.#.##.#..........##.#..##....#.##.#..#..###.##....#.#.......#...##...#..###.#.#.#.##.#.#........##.#..##....#.##.#..#..###.##....###.......#...##...#..###.#.#.#.##.##.........##.#..##....#.##.#..#..###.##...#..#.......#...##...#..###.#.#.#.##.###........##.#..##....#.##.#..#..###.##...#.##.......#...##...#..###.#.#.#.###...........##.#..##....#.##.#..#..###.##...##.#.......#...##...#..###.#.#.#.###..#........##.#..##....#.##.#..#..###.##...####.......#...##...#..###.#.#.#.###.#.........##.#..##....#.##.#..#..###.##..#...#.......#...##...#..###.#.#.#.###.##........##.#..##....#.##.#..#..###.##..#..##.......#...##...#..###.#.#.#.####..........##.#..##....#.##.#..#..###.##..#.#.#.......#...##...#..###.#.#.#.####.#........##.#..##....#.##.#..#..###.##..#.###.......#...##...#..###.#.#.#.#####.........##.#..##....#.##.#..#..###.##..##..#.......#...##...#..###.#.#.#.######........##.#..##....#.##.#..#..###.##..##.##...

traditional feed

transposed feed

7zip

winzip

9%

10%

....#...##...#..###.#.#.#.##............##.#..##....#.##.#..#..###.#.#######.......#...##...#..###.#.#.#.##...#........##.#..##....#.##.#..#..###.##......#.......#...##...#..###.#.#.#.##..#.........##.#..##....#.##.#..#..###.##.....##.......#...##...#..###.#.#.#.##.#..........##.#..##....#.##.#..#..###.##....#.#.......#...##...#..###.#.#.#.##.#.#........##.#..##....#.##.#..#..###.##....###.......#...##...#..###.#.#.#.##.##.........##.#..##....#.##.#..#..###.##...#..#.......#...##...#..###.#.#.#.##.###........##.#..##....#.##.#..#..###.##...#.##.......#...##...#..###.#.#.#.###...........##.#..##....#.##.#..#..###.##...##.#.......#...##...#..###.#.#.#.###..#........##.#..##....#.##.#..#..###.##...####.......#...##...#..###.#.#.#.###.#.........##.#..##....#.##.#..#..###.##..#...#.......#...##...#..###.#.#.#.###.##........##.#..##....#.##.#..#..###.##..#..##.......#...##...#..###.#.#.#.####..........##.#..##....#.##.#..#..###.##..#.#.#.......#...##...#..###.#.#.#.####.#........##.#..##....#.##.#..#..###.##..#.###.......#...##...#..###.#.#.#.#####.........##.#..##....#.##.#..#..###.##..##..#.......#...##...#..###.#.#.#.######........##.#..##....#.##.#..#..###.##..##.##...

traditional feed

transposed feed

Let’s try something really simple…

7zip

winzip

9%

10%

....#...##...#..###.#.#.#.##............##.#..##....#.##.#..#..###.#.#######.......#...##...#..###.#.#.#.##...#........##.#..##....#.##.#..#..###.##......#.......#...##...#..###.#.#.#.##..#.........##.#..##....#.##.#..#..###.##.....##.......#...##...#..###.#.#.#.##.#..........##.#..##....#.##.#..#..###.##....#.#.......#...##...#..###.#.#.#.##.#.#........##.#..##....#.##.#..#..###.##....###.......#...##...#..###.#.#.#.##.##.........##.#..##....#.##.#..#..###.##...#..#.......#...##...#..###.#.#.#.##.###........##.#..##....#.##.#..#..###.##...#.##.......#...##...#..###.#.#.#.###...........##.#..##....#.##.#..#..###.##...##.#.......#...##...#..###.#.#.#.###..#........##.#..##....#.##.#..#..###.##...####.......#...##...#..###.#.#.#.###.#.........##.#..##....#.##.#..#..###.##..#...#.......#...##...#..###.#.#.#.###.##........##.#..##....#.##.#..#..###.##..#..##.......#...##...#..###.#.#.#.####..........##.#..##....#.##.#..#..###.##..#.#.#.......#...##...#..###.#.#.#.####.#........##.#..##....#.##.#..#..###.##..#.###.......#...##...#..###.#.#.#.#####.........##.#..##....#.##.#..#..###.##..##..#.......#...##...#..###.#.#.#.######........##.#..##....#.##.#..#..###.##..##.##...

traditional feed

transposed feed

RLE14%

Our simple preprocessing and fax machine algorithm (less than a page of C++ in total) was compressing ROSETTA housekeeping almost as well as the best algorithm on the market

Does this work on all missions?

38.20Earth ObservationGoce28.00AstronomyHerschel25.93Technology DemoProba-118.23InterplanetaryVenus Express14.06InterplanetaryRosetta5.75Human SpaceflightColumbus

COMPRESSION (% of original)

MISSION TYPEMISSION NAME

Parameter and bit level compressibility

0

20

40

60

80

100

120

140

160

1 107 213 319 425 531 637 743 849 955 1061 1167 1273 1379 1485 1591 1697 1803 1909 2015 2121 2227

Figure 3. Compressibility (% of original size) of each bit column for a packet that compresses well [CDMU Herschel Periodic P1 HK Parameter Report]

Parameter and bit level compressibility

0

20

40

60

80

100

120

1 161 321 481 641 801 961 1121 1281 1441 1601 1761 1921 2081 2241 2401 2561 2721 2881 3041 3201 3361 3521 3681 3841 4001 4161

Figure 5. Compressibility (% of original size) of each bit column for a packet that compresses badly [Herschel SCM Mode TM]

Major influencing factor is AOCS design, mission environment is less important

Resistance from an unexpected quarter..

Impractical

Risky

Not worth it

How much data do we need to store before compressing it?

0

10

20

30

40

50

60

70

80

90

100

10 50 100 250 500 1000 5000

# Packets

% O

rigin

al s

ize

48

Will it run on real hardware?

The bit transposition and RLE algorithms were recoded to respect space coding standards and loaded to a real LEON2 processor in ESA-ESTEC.

The algorithm compressed a 42.4 kilobytes file (from VEX) to 6.55 kilobytes (i.e. 15% of the original) in 0.5 seconds.

Implemented in one day, worked first time.

DANGERTEST IN PROGRESS

Now let’s address the risk argument..

Impractical

Risky

Not worth it

Too Risky

The error correction on the space link works at frame level, i.e. either all the data in a frame is received or the full frame is lost.

Compression in itself does not increase the risk of information loss

ONLY IF The information in each frame can be interpreted independently of the others.

The simple beauty of RLE

“If two identical bytes are followed by a third, insert a one byte counter of how many more repeats follow”

“FF FF FF FF FF FF FF FF 1C” becomes “FF FF 06 1C”

Incredibly simple to compress and expand

Self contained – so no risk increase!

The effect of bit transposition

....#...##...#..###.#.#.#.##............##.#..##....#.##.#..#..###.#.#######.......#...##...#..###.#.#.#.##...#........##.#..##....#.##.#..#..###.##......#.......#...##...#..###.#.#.#.##..#.........##.#..##....#.##.#..#..###.##.....##.......#...##...#..###.#.#.#.##.#..........##.#..##....#.##.#..#..###.##....#.#.......#...##...#..###.#.#.#.##.#.#........##.#..##....#.##.#..#..###.##....###.......#...##...#..###.#.#.#.##.##.........##.#..##....#.##.#..#..###.##...#..#.......#...##...#..###.#.#.#.##.###........##.#..##....#.##.#..#..###.##...#.##.......#...##...#..###.#.#.#.###...........##.#..##....#.##.#..#..###.##...##.#.......#...##...#..###.#.#.#.###..#........##.#..##....#.##.#..#..###.##...####.......#...##...#..###.#.#.#.###.#.........##.#..##....#.##.#..#..###.##..#...#.......#...##...#..###.#.#.#.###.##........##.#..##....#.##.#..#..###.##..#..##.......#...##...#..###.#.#.#.####..........##.#..##....#.##.#..#..###.##..#.#.#.......#...##...#..###.#.#.#.####.#........##.#..##....#.##.#..#..###.##..#.###.......#...##...#..###.#.#.#.#####.........##.#..##....#.##.#..#..###.##..##..#.......#...##...#..###.#.#.#.######........##.#..##....#.##.#..#..###.##..##.##...

traditional feed

transposed feedInstead of losing rows, you now lose columns…..a small set of parameters over a longer time as opposed to all parameters over a short time.

Is this a problem? If so, we can already state our maximum exposure….

48 packets.

But what about if I can’t afford to lose anything?

Simply use the time and bandwidth released to send the frames twice or implement a closed loop..

Original Repeat

This compression technique does not increase mission risk and the resources it releases can be used to reduce risk far below that normally accepted.

Those missions that are concerned about the risk of information loss must compress their housekeeping telemetry. Especially in contingency cases.

One more…

Impractical

Risky

Not worth it

Time is money

The time saved by HKTM compression will reduce how long an operation takes and open up choices for when it can be performed.

This will cut reaction times, an important operational metric.

The time saved will enable operational feedback loops not possible before.

Operations costs are dominated by manpower.

Manpower costs are dominated by time.

Ease the pain of packet design

Preparation tasks can account for a significant proportion of the total operations cost. Part of that work is to carefully select parameters and their respective sampling rate in the packets.

Do we really need this parameter in this

mode?

What sample rate is needed?

A common solution is to design asynchronous events to flag significant changes. This sounds easy!

How does compressing telemetry in this way help?

It effectively removes all “compressible” parameters (vast majority) from the packet design trades.

No need to guess if these parameter histories might be needed one day

No need to design and test asynchronous events for these parameters

No need to predict spacecraft behavior - to decide on trigger metrics

One can simply select as many of these parameters to be in the housekeeping as

one wants

Improving observability

Can sample “compressible” parameters at high rates with a low impact on bandwidth usage.

Less need to guess or extrapolate when trying to reconstruct events

Less chance of missing important information due to low sampling rates(short abnormal behavior like spikes, transients, high frequency switching)

More chance of discovering correlations between parameter behavior and important events or trends

Richer, finer information on the ground to analyze

Is it compatible with the present infrastructure?

Packet Store

Compressed Packet Store

Packet extractionBit transpositionRLE encodingStore in special “frame length” packets

Mission Control Sys tem

Router

Transfer to mission control centre using normal PUS services, CCSDS frames, coding, modulation etc

Pre-processor

Recovered packets passed to miss ion control system for normal processing

Compressed packets routed to pre-processor for decompression

The future

It is fair to say that it is becoming a standard solution for future studies section at ESA/ESOC for Phase 0/A work.

Baseline for PROBA-3

Identified as an enabling technology for the Mars Atmospheric Sample Return Mission

ESA patent filed in the USA

Conclusion

The Bit-Transposition and RLE algorithm

• Compresses housekeeping data very effectively

• Is extremely simple

• demonstrated good performance on target hardware

• Requires little change to the present CCSDS/PUS based infrastructure

• Requires just 48 stored packets as input

• Does not increase the risk of information loss

It will

• Reduce ground station usage and increase mission return

• reduce ground reaction times

• enable more efficient use of operations manpower

• reduce engineering costs in the preparation phase

• increase spacecraft observability and telemetry flexibility

• decrease mission risk

Conclusion

Thank you

Backup: Comparison with RICE