an exploitation of tufte’s small multiples
DESCRIPTION
An Exploitation of Tufte’s Small Multiples. Carl T. Russell Colorado Springs, Colorado [email protected]. Operational Overview. Theater Missile Defense System Exerciser (TMDSE) Geographically-distributed hardware-in-the-loop (HWIL) simulation - PowerPoint PPT PresentationTRANSCRIPT
An Exploitation ofTufte’s Small Multiples
Carl T. Russell
Colorado Springs, Colorado
Operational Overview
• Theater Missile Defense System Exerciser (TMDSE)– Geographically-distributed hardware-in-the-loop (HWIL) simulation
– Used to investigate terminal-phase theater Family of Systems (FoS) interoperability
• TMDSE stimulates Tactical Segments – Provides simulated threat environment via DIS PDUs
– Provides communications connectivity• Emulates Joint Data Network (JDN)• Tactical Segments exchange TADIL-J messages over an
emulated JTIDS communications network consisting primarily of gateways and T1 lines
Operational Question Addressed in This Paper:What are the time delays (latencies) associated with the
emulated tactical communications network?
Analytical Overview
• Large sample sizes
• Widely varied sample sizes
• ANOVA dumb, tables unwieldy– Everything significant in ANOVA
– Detailed tables of latencies have several hundred pages
Analytical Question Addressed in This Paper:How well can quantitative graphics be used to
assess the communications latencies?
TMD Operational Architecture
AFC
TMDSE Tactical Segments
TDDSJNIC Test
Exercise
Controller
Azusa, CA
Kirtland AFB, NM
Schriever AFB, CO Syracuse, NY
Dahlgren, VA
Huntsville, AL
Redstone Arsenal, AL
TIBS
AFC (JNIC)AFC (JNIC)
JTAGSJTAGS
MTSMTSPATRIOTPATRIOT
THAADTHAAD
AEGIS AAW & AEGIS Linebacker
AEGIS AAW & AEGIS Linebacker
USMC ADCPUSMC ADCP
TMDSE Test Architecture
T1 Link
* Each Router provides a connection (not shown) to the JDN Gateway
Joint Data Net (Link 16) Emulation
T1 Link T1 Link = DIS + TADIL J + Voice Coord
LocalLink
TMDSE ControlSegment (TCS)
Tactical Communications
Environment Segment (TCES)
Tactical Hardware-In-the
LoopTHAADX1
X.25
T1 Link
USMC ADCP& TPS 59
X1
1553 RS232
TIBS / TDDS
AEGISLINEBACKER
X1
1553
*
X2 PATRIOT
RemoteGateway
X2
FMS/DX2
X.25
TIBS
TDDS
T1 Link
JTAGSX1
AFCX1
X1
TIBS/TDDS
CRC(MTS only)
X1
X1
AEGISAAW
ACSIS
X1
1553
T1 Link
Adjunct Processor
MSIM & DIS Gateways
ACSIS
Test Exercise Controller (TEC)Joint National Integration Center (JNIC), Colorado
TEC Sub-Systems
TTCX1
*Router
CentralGateway
X1
LANX.25
RES
RES WorkStation
X1 Tactical System Drivers
Segment (TSDS)
Router
Tactical Driver
RemoteEnvironment
RemoteEnvironment
RemoteEnvironment
RemoteEnvironment
RemoteEnvironment
RemoteEnvironment
Adjunct Processor
Router Router Router Router Router Router
RemoteGateway
X1
RemoteGateway
X1
RemoteGateway
X1
RemoteGateway
X1
RemoteGateway
X1
RemoteGateway
X3
CentralGateway
(TCS)
RemoteEnvironment
RES Tactical Driver
X1
TIBS
Communications Latency in TMDSE
• TADIL-J messages recorded and time-stamped– At transmitting node
– At transmitting gateway
– At receiving gateways
– At receiving nodes
• TADIL-J messages matched end-to-end for analysis– Latencies calculated point-to-point and end-to-end
– Flawed process, 95%+ successful
TransmittingNode
Transmitting Gateway
ReceivingGateways
Receiving Nodes
point-to-point
latency
end-to-endlatencies
point-to-point
latencies
point-to-point
latencies
Analysis Challenges (1 of 3)—Large Sample Sizes—
Number of Messages for Latency CalculationLocation ofReceived Message
MsgType Test Run I Test Run II Test Run III
a 2126 2033 1774b 1072 973 976c 478 528 507d 124 95 166
TransmittingGateway
e 1210 1324 1831a 17668 16640 15884b 8584 7664 8014c 3802 4220 4174d 981 745 1300
ReceivingGateway
e 9619 10524 14562a 12755 12229 12352b 6474 5817 6341c 3802 3062 3181d 981 529 974
ReceivingNode
e 5964 6141 8803
Analysis Challenges (2 of 3)—Widely Varied Sample Sizes—
Factors and levels (1143 Factor Combinations)– Test run (3 levels)– Message type (5 levels)– Transmitting segment (8 levels nested in message type)– Receiving segment (7 levels nested in message type)
Total Number of Factor Combinations=1143
Min=1
10%=11
25%=40
50%=103
75%=217
90%=450
max=1099
* *
50
100
0 400 9004 16
36
64
100
144 196
256
324
484
576
676
784
1024Number of Messages per Factor
Combination(Square Root Scale)
Num
ber
of
Fact
or
Com
bin
ati
ons
0
Analysis Challenges (3 of 3)—ANOVA Dumb, Tables Unwieldy—
• Everything is significant in ANOVA
• Tables of latencies have several hundred pages
Source DF F Ratio Prob>F
Test Run 2 529.2 <0.0001
Message Type 4 63.8 <0.0001
Test Run x MsgType 8 8.5 <0.0001
Transmitter 7 1897.4 0.0000
Receiver 6 19.8 <0.0001
Test Run x Transmitter 14 196.9 0.0000
Test Run x Receiver 12 5.2 <0.0001
Graphical Approach Works—Initial Display—
A B C D E GF
a
TADIL-J Message Type and Receiving Segment
Tim
e D
elay
0
A B C D E GF
b
A B C D E GF
c
A B C D E GF
d
A B C D E GF
e
Time Delays for Key Messages Transmitted by Segment “B” on Test Run I,by Message Type and Receiving Segment
– 80% boxplots used– No latencies for receipts by “B” (“B” was transmitting)– No latencies for message type “e” (“B” did not transmit any messages of type “e”)– No latencies for receiving node “H” (message matching unreliable)
Tufte’s notion of “small multiples”—“graphics can be shrunk way down”—will be used on subsequent slides to exploit this initial display
Transmit to Receive Time Delays by Test Run, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
TestRun
I
II
ALL
III
A DCB E F HG
Transmitting Segment
ALL
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.
Delays corresponding to Segment H node receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
Time Delays on Run I by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
Nodeto Gateway
Gatewayto Gateway
Transmitter to Receiver
Gatewayto Node
Type DelayA DCB E F HG
Transmitting Segment
ALL
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.
Delays corresponding to Segment H node receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
Time Delays on Run II by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
A DCB E F HG
Transmitting Segment
ALL
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.
Delays corresponding to Segment H node receipts are not displayed , and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
Nodeto Gateway
Gatewayto Gateway
Transmitter to Receiver
Gatewayto Node
Type Delay
Time Delays on Run III by Type Delay, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Adjusted for Apparent Node Clock Errors.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
A DCB E F HG
Transmitting Segment
ALL
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.
Delays corresponding to Segment H node receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
Nodeto Gateway
Gatewayto Gateway
Transmitter to Receiver
Gatewayto Node
Type Delay
Transmit to Receive Time Delays by Test Run, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots). Times Not Adjusted for Apparent Node Clock Errors.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
TestRun
I
II
ALL
III
A DCB E F HG
Transmitting Segment
ALL
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line. Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.
Delays corresponding to Segment H node receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
Gateway to Node Time Delays by Test Run, Type, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots).Times Not Adjusted for Apparent Node Clock Errors.
}
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
TestRun
I
II
III
A DCB E F HG
Transmitting Segment
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line.
Within message type, left to right receiving segment order is: A, B, C, D, E, F, G.Delays corresponding to Segment H receipts are not displayed, and the sending node for each cell has no receiving latency distribution except node to gateway.
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
}}
Segment E biased up( fast node clock)
Segment F biased up( fast node clock)
Segment F biased up( fast node clock)
Segment E biased up( fast node clock)
Segment F biased down( slow node clock)}
Segment G biased down( slow node clock)
}
}}
Segment E biased up( fast node clock)
Node to Gateway Time Delays by Test Run, Type, Transmitting Segment, Message Type and Receiving Segment (80% Boxplots).Times Not Adjusted for Apparent Node Clock Errors.
} }}
TestRun
I
II
III
A DCB E F HG
Transmitting Segment
Within message type, the “whiskers” on the boxplots extend from the 10th to the 25th percentile and the 75th to the 90th percentile of the latency distribution for each receiving segment; the 50th percentile (median) is marked with a horizontal line.
0
Tim
e D
elay
0
Tim
e D
elay
0
Tim
e D
elay
a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e a
Message Type
b c d e
}}
}}
} }
} } }} } }
} } }} } }}}}
}}}}}} } }
}
Segment E biased down( fast node clock)
Segment G biased up( slow node clock)
Segment H biased up( slow node clock)
Segment F biased down( fast node clock)
Segment F biased up( slow node clock)
Segment H biased up( slow node clock)
Segment E biased down( fast node clock)
Segment E biased down( fast node clock)
Segment F biased down( fast node clock)
Summary
• Simple statistical graphics can produce rich, easily understood analyses via tables of miniature displays– Motivated by Tufte and others, “Data Density and Small Multiples”
“Shrink Principle”—“Graphics can be shrunk way down”
– Works very well on 1200 dpi printer with tabloid paper
– Still marginally acceptable on letter paper (or transparencies)
• Such graphics still a hard sell– DoD likes three bullets per slide, three words per bullet
– Tufte says decision makers can be led through dense graphics but must be kept focused—then credibility of analysis increased
Moral:Push for denser graphics despite resistance