an exploitation of tufte’s small multiples

An Exploitation ofTufte’s Small Multiples

Carl T. Russell

Colorado Springs, Colorado

[email protected]

Operational Overview

• Theater Missile Defense System Exerciser (TMDSE)– Geographically-distributed hardware-in-the-loop (HWIL) simulation

– Used to investigate terminal-phase theater Family of Systems (FoS) interoperability

• TMDSE stimulates Tactical Segments – Provides simulated threat environment via DIS PDUs

– Provides communications connectivity• Emulates Joint Data Network (JDN)• Tactical Segments exchange TADIL-J messages over an

emulated JTIDS communications network consisting primarily of gateways and T1 lines

Operational Question Addressed in This Paper:What are the time delays (latencies) associated with the

emulated tactical communications network?

Analytical Overview

• Large sample sizes

• Widely varied sample sizes

• ANOVA dumb, tables unwieldy– Everything significant in ANOVA

– Detailed tables of latencies have several hundred pages

Analytical Question Addressed in This Paper:How well can quantitative graphics be used to

assess the communications latencies?

TMD Operational Architecture

AFC

TMDSE Tactical Segments

TDDSJNIC Test

Exercise

Controller

Azusa, CA

Kirtland AFB, NM

Schriever AFB, CO Syracuse, NY

Dahlgren, VA

Huntsville, AL

Redstone Arsenal, AL

TIBS

AFC (JNIC)AFC (JNIC)

JTAGSJTAGS

MTSMTSPATRIOTPATRIOT

THAADTHAAD

AEGIS AAW & AEGIS Linebacker

AEGIS AAW & AEGIS Linebacker

USMC ADCPUSMC ADCP

TMDSE Test Architecture

T1 Link

* Each Router provides a connection (not shown) to the JDN Gateway

Joint Data Net (Link 16) Emulation

T1 Link T1 Link = DIS + TADIL J + Voice Coord

LocalLink

TMDSE ControlSegment (TCS)

Tactical Communications

Environment Segment (TCES)

Tactical Hardware-In-the

LoopTHAADX1

X.25

T1 Link

USMC ADCP& TPS 59

X1

1553 RS232

TIBS / TDDS

AEGISLINEBACKER

X1

1553

*

X2 PATRIOT

RemoteGateway

X2

FMS/DX2

X.25

TIBS

TDDS

T1 Link

JTAGSX1

AFCX1

X1

TIBS/TDDS

CRC(MTS only)

X1

X1

AEGISAAW

ACSIS

X1

1553

T1 Link

Adjunct Processor

MSIM & DIS Gateways

ACSIS

Test Exercise Controller (TEC)Joint National Integration Center (JNIC), Colorado

TEC Sub-Systems

TTCX1

*Router

CentralGateway

X1

LANX.25

RES

RES WorkStation

X1 Tactical System Drivers

Segment (TSDS)

Router

Tactical Driver

RemoteEnvironment

RemoteEnvironment

RemoteEnvironment

RemoteEnvironment

RemoteEnvironment

RemoteEnvironment

Adjunct Processor

Router Router Router Router Router Router

RemoteGateway

X1

RemoteGateway

X1

RemoteGateway

X1

RemoteGateway

X1

RemoteGateway

X1

RemoteGateway

X3

CentralGateway

(TCS)

RemoteEnvironment

RES Tactical Driver

X1

TIBS

Communications Latency in TMDSE

• TADIL-J messages recorded and time-stamped– At transmitting node

– At transmitting gateway

– At receiving gateways

– At receiving nodes

• TADIL-J messages matched end-to-end for analysis– Latencies calculated point-to-point and end-to-end

– Flawed process, 95%+ successful

TransmittingNode

Transmitting Gateway

ReceivingGateways

Receiving Nodes

point-to-point

latency

end-to-endlatencies

point-to-point

latencies

point-to-point

latencies

Analysis Challenges (1 of 3)—Large Sample Sizes—

Number of Messages for Latency CalculationLocation ofReceived Message

MsgType Test Run I Test Run II Test Run III

a 2126 2033 1774b 1072 973 976c 478 528 507d 124 95 166

TransmittingGateway

e 1210 1324 1831a 17668 16640 15884b 8584 7664 8014c 3802 4220 4174d 981 745 1300

ReceivingGateway

e 9619 10524 14562a 12755 12229 12352b 6474 5817 6341c 3802 3062 3181d 981 529 974

ReceivingNode

e 5964 6141 8803

Analysis Challenges (2 of 3)—Widely Varied Sample Sizes—

Factors and levels (1143 Factor Combinations)– Test run (3 levels)– Message type (5 levels)– Transmitting segment (8 levels nested in message type)– Receiving segment (7 levels nested in message type)

Total Number of Factor Combinations=1143

Min=1

10%=11

25%=40

50%=103

75%=217

90%=450

max=1099

* *

50

100

0 400 9004 16

36

64

100

144 196

256

324

484

576

676

784

1024Number of Messages per Factor

Combination(Square Root Scale)

Num

ber

of

Fact

or

Com

bin

ati

ons

0

Analysis Challenges (3 of 3)—ANOVA Dumb, Tables Unwieldy—

• Everything is significant in ANOVA

• Tables of latencies have several hundred pages

Source DF F Ratio Prob>F

Test Run 2 529.2 <0.0001

Message Type 4 63.8 <0.0001

Test Run x MsgType 8 8.5 <0.0001

Transmitter 7 1897.4 0.0000

Receiver 6 19.8 <0.0001

Test Run x Transmitter 14 196.9 0.0000

Test Run x Receiver 12 5.2 <0.0001

Graphical Approach Works—Initial Display—

A B C D E GF

a

TADIL-J Message Type and Receiving Segment

Tim

e D

elay

0

A B C D E GF

b

A B C D E GF

c

A B C D E GF

d

A B C D E GF

e

Time Delays for Key Messages Transmitted by Segment “B” on Test Run I,by Message Type and Receiving Segment

– 80% boxplots used– No latencies for receipts by “B” (“B” was transmitting)– No latencies for message type “e” (“B” did not transmit any messages of type “e”)– No latencies for receiving node “H” (message matching unreliable)

Tufte’s notion of “small multiples”—“graphics can be shrunk way down”—will be used on subsequent slides to exploit this initial display

Transmit to Receive Time Delays by Test Run, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

TestRun

I

II

ALL

III

A DCB E F HG

Transmitting Segment

ALL

Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.

Delays corresponding to Segment H node receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.

a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

Time Delays on Run I by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

Nodeto Gateway

Gatewayto Gateway

Transmitter to Receiver

Gatewayto Node

Type DelayA DCB E F HG


ALL



a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

Time Delays on Run II by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

A DCB E F HG


ALL


Delays corresponding to Segment H node receipts are not displayed , and the sending node for each cell has no receiving latency distribution except node to gateway.

a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

Nodeto Gateway

Gatewayto Gateway


Gatewayto Node

Type Delay

Time Delays on Run III by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

A DCB E F HG


ALL



a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

Nodeto Gateway

Gatewayto Gateway


Gatewayto Node

Type Delay

Transmit to Receive Time Delays by Test Run, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Not Adjusted for Apparent Node Clock Errors.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

TestRun

I

II

ALL

III

A DCB E F HG


ALL



a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

Gateway to Node Time Delays by Test Run, Type, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots).Times Not Adjusted for Apparent Node Clock Errors.

}

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

TestRun

I

II

III

A DCB E F HG


Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line.

Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.Delays corresponding to Segment H receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.

a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

}}

Segment E biased up( fast node clock)

Segment F biased up( fast node clock)

Segment F biased up( fast node clock)


Segment F biased down( slow node clock)}

Segment G biased down( slow node clock)

}

}}


Node to Gateway Time Delays by Test Run, Type, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots).Times Not Adjusted for Apparent Node Clock Errors.

} }}

TestRun

I

II

III

A DCB E F HG


Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line.

0

Tim

e D

elay

0

Tim

e D

elay

0

Tim

e D

elay

a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e a

Message Type

b c d e

}}

}}

} }

} } }} } }

} } }} } }}}}

}}}}}} } }

}

Segment E biased down( fast node clock)

Segment G biased up( slow node clock)

Segment H biased up( slow node clock)

Segment F biased down( fast node clock)

Segment F biased up( slow node clock)

Segment H biased up( slow node clock)



Segment F biased down( fast node clock)

Summary

• Simple statistical graphics can produce rich, easily understood analyses via tables of miniature displays– Motivated by Tufte and others, “Data Density and Small Multiples”

“Shrink Principle”—“Graphics can be shrunk way down”

– Works very well on 1200 dpi printer with tabloid paper

– Still marginally acceptable on letter paper (or transparencies)

• Such graphics still a hard sell– DoD likes three bullets per slide, three words per bullet

– Tufte says decision makers can be led through dense graphics but must be kept focused—then credibility of analysis increased

Moral:Push for denser graphics despite resistance

an exploitation of tufte’s small multiples

Documents

segment b

message typereceiving

messages of type eno

time delays latencies

communications latencies

message type e b

test run

levelsmessage type