testing communications in the connected...

12 May 2015

Testing Communications in the Connected Car

Chris Wellens

InterWorking Labs

iwl.com

+1.831.818.7963

chrisw@iwl.com

InterWorking Labs

Saving the World from

Network Failure!

12 May 2015

Testing Communications in the Connected Car

● InterWorking Labs helps engineers:

➢ Understand how well a device or app will perform under adverse network conditions....

➢ With network emulators and protocol testers …

➢ To optimize device or app performance.

12 May 2015

Testing Communications in the Connected Car

Tonight's Topic:

How do connected car

[manufacturers, suppliers, aftermarket players]

test, verify, and validate the proper

[implementation, interoperability, robust operation]

In the automotive network?

12 May 2015

Testing Communications in the Connected Car

Agenda

1. What components require testing?

2. What industry standards are used?

3. How are those standards implemented?

4. How do you test the communications?

– What types of testing should be applied?

5. ECU and related modules testing

6. Infotainment and Driver Assistance testing

12 May 2015

Testing Communications in the Connected Car

1. What vehicle components require communications testing?

Any component that exchanges data with:

another component another vehicle other external communication nodes (vehicle-to-

infrastructure communication).

Depending on the industry standard communication protocols

12 May 2015

Testing Communications in the Connected Car

2. What Industry Standards Are Used?

AUTOSAR: The AUTomotive Open System Architecture

See www.autosar.org

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

➔ But AUTOSAR only applies to ECUs!

➔ Infotainment needs more bandwidth!

➔ TODAY FUTURE

AUTOSAR “Classic” Platform

“Adaptive” AUTOSAR Platform

For ECUs For ECUs, Infotainment, Driver Assistance,And more

12 May 2015

Testing Communications in the Connected Car

3. How are those industry standard solutions implemented?

● You can always read the standards specifications and do the implementation yourself.

● Or, you can license an AUTOSAR implementation from an embedded system supplier:

– Our host: Mentor Graphics

– Others: Wind River, QNX, Electrobit, Vector, etc.

● Or, you can look for an open source solution.

12 May 2015

Testing Communications in the Connected Car

4. Now we address communications testing.

12 May 2015

Testing Communications in the Connected Car

What are the different types or kinds of testing that should be applied?

Functional Tests include:

‣ Negative Testing

‣ Inopportune Testing

‣ Conformance/Compliance Testing

‣ Interoperability Testing

‣ Deep-path Testing

And ...

12 May 2015

Testing Communications in the Connected Car

Stress and Reliability Tests include:

‣ Load Testing

‣ Stress Testing

‣ Performance Testing

‣ Line Speed Testing

‣ Robustness (Security) Testing

‣ Endurance Testing

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

AUTOSAR requires the TCP/IP family.

TCP is 40 years old –

Isn't it thoroughly debugged at this point?

So glad you asked that question ...

12 May 2015

Testing Communications in the Connected Car

➢ InterWorking Labs used its Maxwell PRO TCP/IP Test Suite to test Windows 7 with all updates applied.

➢ Windows 7 passed the IPv6 Ready Logo program

➢ Windows 7 passed at the highest level.

AND here's what we found ...

12 May 2015

Testing Communications in the Connected Car

● Out of 396 TCP/IP Protocol Tests:

✗ 327 tests passed✗ 69 tests failed✗ 10 IPv6 protocol-specific tests failed✗ 18 tests of other protocols using IPv6 failed

(that passed over IPv4)● For the IPv6 failures, the most serious ones

involved incorrect checksums and IPv6 fragments delivered out of order

12 May 2015

Testing Communications in the Connected Car

Of the non-IPv6 protocol tests that failed, the ones that could cause problems to customers are:

✔ incorrect checksums

✔ incorrect issuance of ICMPv4 redirects

✔ lack of support for overlapping IPv4 fragments

✔ dropping of some IPv4 datagrams containing unknown options

✔ poor handling of out-of-order TCP segments

✔ poor handling of lost TCP SYN connections.

12 May 2015

Testing Communications in the Connected Car

CONCLUSION:

The TCP/IP protocol family requires testing on all products.

12 May 2015

● Or

12 May 2015

Testing Communications in the Connected Car

The TCP family is used in AUTOSAR.

How do we test for robustness, security, interoperability and conformance/compliance?

✔ We will use a test suite program to send pathological TCP segments to the device under test.

✔ The device will have to respond appropriately.

✔ It should not hang or crash.

✔ It should exhibit the proper behavior as defined in the standards document.

So how does that work conceptually?

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

Conformance/Compliance Fuzz Tests

Sends pathological segments Sends pathological segments

Reports all errors Monitor to check for DUT crash

Grades each test: Pass/Fail/etc DUT crashed or did not crash

Reports cause of the error Human tester must analyze and find the code causing the crash

Junior engineer Senior engineer required

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

Here's an example of a negative test:

Negative testing verifies that the device under test (DUT) responds correctly to error conditions or unacceptable input conditions.

SUMMARY: Append unknown option with a length of 41 to all segments.

DESCRIPTION: This test adds an unknown option to all TCP packets unless there is already an option in the packet or the packet is so large that adding an option would cause fragmentation.

The option to be added here consists of 6 octets. However the length octet in the option has a value of 41, which is too large to fit into the TCP header of any TCP packet (and doesn't match the actual number of octets actually in the option).

The option type is 31 which has not been allocated by IANA.

12 May 2015

Testing Communications in the Connected Car

Negative Test Example continued.

...

ASSUMPTIONS: None.

EXPECTATIONS: The affected segments must be discarded and the connection may be reset.

REQUIREMENT LEVEL: Must

REFERENCES: RFC 793: section 3.1 RFC 1122: section 4.2.2.5 {Cope with illegal option length}

NOTE: Linux fails this test! Linux accepts the packet, and sends it up the protocol stack. The wrong layer responds.

12 May 2015

Testing Communications in the Connected Car

Here's an example of a robustness / security test:

Robustness testing is the process of subjecting a device under test to particular input streams in an attempt to cause the device to fail. The input streams may be one of three types:

‣ Random input streams

‣ Valid input streams, and

‣ Invalid input streams.

Our aim is a precise, controlled, repeatable input stream.

12 May 2015

Testing Communications in the Connected Car

Robustness/ Security Test (A variation on the “Ping of Death” test) …

Enough fragments are sent to create 65536 bytes of data in any destination buffer (one more than is allowed.) The MTU is 512 + header length. A final fragment of 8 bytes occurs at the end. Fragments are transmitted in order of increasing segment offset.

Expected Outcome:

The fragments must eventually be discarded before anything is delivered to upper protocol layers. The device under test must not crash!

This test is also an example of a Conformance/Compliance test, because the requirements are defined in the following specifications:

RFC 791, pages 8, 13, 24, 25

RFC 1122, section 3.3.2.

12 May 2015

Testing Communications in the Connected Car

Let's look at how this works in the lab ...

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

6. Infotainment and Driver Assistance Testing

For Infotainment and Driver Assistance

how can we assure that

performance will be acceptable

under all conditions?

12 May 2015

Testing Communications in the Connected Car

Defining the performance metrics of mobile networks:

Mobile networks perform more poorly than their wired counterparts on the following four performance metrics:

1) Throughput

2) Round Trip Time (RTT)

3) Jitter

4) Packet Loss

Evaluate the performance of our app or device under worst case conditions ...

12 May 2015

Testing Communications in the Connected Car

See References on last page

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

✔ Why use a network emulator?

✔ I can do ad hoc testing with some 3G/4G phones!

Goal is precise, deterministic test results for all boundary conditions.

Accuracy. Repeatability.

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

Consider 4G/LTE worst case conditions...

● You must tune your app or device to perform at these limits

● What if you cannot accommodate the limits?

12 May 2015

Testing Communications in the Connected Car

Consider Apple's Siri...

● “I'm really sorry, Chris, but I cannot answer that right now.”

12 May 2015

Testing Communications in the Connected Car

Infotainment and driver assistance applications and a geosynchronous satellite

12 May 2015

Testing Communications in the Connected Car

The signal from the ground station to the geosync satellite travels about 22,000 miles up and then 22,000 miles back down.

Geosync satellite communication automatically introduces a network delay of about 0.24 seconds or 240 milliseconds.

Other variables include the link rate, min and max payload size, header overhead, queue length, and so on.

How will your app perform in your moving car when data is coming from the satellite?

12 May 2015

Testing Communications in the Connected Car

12 May 2015

Testing Communications in the Connected Car

Summary ...

1. Components that require testing

2. Industry standards used and implemented:

AUTOSAR, TCP/IP family, 3G, 4G/LTE

3. Types of testing: stress, reliability, functionality ...

4. Testing of ECU related modules

6. Performance testing of Infotainment and Driver Assistance

12 May 2015

Testing Communications in the Connected Car

References 1 of 2:A Close Examination of Performance and Power Characteristics of 4G LTE Networks, 2012,

http://www.cs.columbia.edu/~lierranli/coms6998-7Spring2014/papers/rrclte_mobisys2012.pdf

One standard deviation added to median from the document Characterizing 4G and 3G Networks: Supporting

Mobility with Multi-Path TCP, UMass Amherst Technical Report UM-CS-2012-022,

http://people.cs.umass.edu/~yungchih/publication/12_mtcp_4g_tech_report.pdf

http://iwl.com/white-papers/emulating-3g-4g-networks – “Emulating 3G and 4G Networks”

Explains how data from the above academic research is incorporated into emulations

http://iwl.com/white-papers/network-protocol-testing – “Network Protocol Testing Overview”

Categories of network testing

12 May 2015

Testing Communications in the Connected Car

References 2 of 2:

http://iwl.com/white-papers/network-impairments/causes-and-correlation – “Causes and Correlation of Network Impairments”Explains the cause of phenomenon like excessive fragmentation and its impact on the network

http://www.eetimes.com/document.asp?doc_id=1280616&page_number=1 – “Automotive Ethernet: Evolution in the Fast Lane”Overview of where the standards and technologies are going, with emphasis on EthernetAny questions or comments, send a note to info@iwl.com

NOTE: © Copyright InterWorking Labs, Inc. ALL RIGHTS RESERVED.Distribute freely among colleagues; contact us to reuse any material in other published works.

12 May 2015

Testing Communications in the Connected Car

References continued:

http://iwl.com/white-papers/network-impairments/causes-and-correlation – “Causes and Correlation of Network Impairments”Explains the cause of phenomenon like excessive fragmentation and its impact on the network http://www.eetimes.com/document.asp?doc_id=1280616&page_number=1 – “Automotive Ethernet: Evolution in the Fast Lane”Overview of where the standards and technologies are going, with emphasis on EthernetAny questions or comments, send a note to info@iwl.com

NOTE: © Copyright InterWorking Labs, Inc. ALL RIGHTS RESERVED.Distribute freely among colleagues; contact us to reuse any material in other published works.