presentation 6
TRANSCRIPT
Integration of Mobile Apps into the Vehicle HMI
Terminal Mode Deep Dive
Presenter:Matthias Benesch, Daimler AG
Jörg Brakensiek, Nokia Corporation
Overview
� Introduction
� VNC Protocol
� Demonstration – Part I
� Device Attestation
� Audio
� Service Negotiation
� Demonstration – Part II
� Summary
Jörg Brakensiek, Nokia Research Center Palo Alto | June 2, 20102
Introduction – Guiding Principles
Technology properties
• Existing
• Non-proprietary
• Well known
• Stable• Stable
• Do extend only if necessary
Be Application & OS Agnostic
• Support legacy application (no change needed)
• Allow for optimization to automotive
• Can be implemented on existing platforms
Introduction – Terminal Mode Stack
Terminal Mode
RTP & BT(Audio)
VNC(Display & Control)
UPnP(Service Discovery & Initiation)
DeviceAttestation
WLAN USB 2.0 …
Internet Protocol Abstraction Layer
(Service Discovery & Initiation)
BT
VNC Protocol – Overview
Display ControlRFB Protocol
VNC Server
VNC Client
CE DisplayCE Display
HU Display
User Input
Copy CE display to Head-unit display
Receive user input events from the Head-Unit
Consumer Electronics
Device
Automotive Head Unit
Display Control
VNC Protocol – Framebuffer: Concept
Memory
FB
Memory
FB
Sca
lingRFB Protocol
Add
H
eade
r
Rem
ove
Hea
der
VNC Protocol – Framebuffer: Protocol
R G BRGB 5655 6 5
Framebuffer (FB) Update Header
Position of update in source FB• X / Y position
• Width / Height
• Encoding
A R G BARGB 888
8 8 8 8
VNC Protocol – Framebuffer: Messages
Set FB Size, CE Pixel Format
Set Head-Unit Pixel FormatSet
up
Pha
se
Request FB Update
Provide FB Update
Ope
ratio
n P
hase
Full Update
VNC Protocol – Framebuffer: Updates
Incremental Update
Phone
Model
MTU
Size
FB Size
[kByte]
Home Screen Camera Viewer
Full Update Full Update Change Inc. Update
N97 1500 900 4 FPS 3 FPS 22 % 12 FPS
N97 8260 900 7 FPS 5 FPS 24 % 16 FPS
Pro: Reduce fragmentation and data transfer size
Con: Additional Processing for incremental analysis
VNC Protocol – Framebuffer: Optimization
8 8 8 8 A R G B
5 6 5
R G B
Color Conversion
5 6 5
Phone
Model
Color
Format
FB Size
[kByte]
Home Screen Camera Viewer
Full Update Full Update Change Inc. Update
N97 RGB888 900 7 FPS 5 FPS 24 % 16 FPS
N97 RGB565 450 12 FPS 10 FPS 22 % 20 FPS
Pro: Reduce Data transfer size
Con: Color conversion processing at phone and head-unit
FB Request
FB Update
FB Request
FB Update
VNC Protocol – Framebuffer: Optimization
FB Changed
Wait for Change
Pro: Reduced CPU load
Con: External trigger from Display Driver / UI framework
Phone
Model
Color
Format
No Wait Wait for FB Update
Incremental CPU Load Change Incremental CPU Load
N97 RGB888 70 FPS 100 % 18 % 7 FPS 17 %
N97 RGB565 70 FPS 100 % 15 % 7 FPS 12 %
Application: Performance Monitor
VNC Protocol – Framebuffer: Optimization
A R G B A R G B A R G B A R G B A R G B
Run-Length Encoding
3 R G B 0 R G B
Phone
Model
Update RAW Encoding RLE Encoding
Update CPU Load Average RL Update CPU Load
N97 Full 7 FPS 70 % 6 pixel/run 19 FPS 85 %
N97 Increment 7 FPS 17 % 6 pixel/run 7 FPS 10 %
Application: Performance Monitor
Pro: Reduced CPU load
Con: Run-Length Encoding processing at phone and head-unit
Cro
p
VNC Protocol – Framebuffer: Extension
• Scaling support
• Display resolution & sizeClient
Display Config.
Memory
FB
Sca
le
• Distance to driver
Virtual FB
Back
Option
+
-
Config.
VNC Protocol – Framebuffer: Extension
Application
Device Status
Key lock Voice input
Device lock Driver Distraction
Dim display Framebuffer rotation
Night mode Framebuffer orientation
VNC Protocol – Input Events: Protocol
Key Events
• Unique key identifier
• Key press & Key release
Pointer Events
• Event position ( x / y )
• Button mask (left, middle, right)• Button mask (left, middle, right)
Phone does not distinguish between local and remote events
Keyboard events defined in /usr/include/keysymdef.h
VNC Protocol – Input Events: Identifier
Specific Device Events Key Identifier
ITU Key 0 (‘0’, ' ‘) 0x3000 0100
ITU Key 1 (‘1’, '.', ‘,') 0x3000 0101
ITU Key 2 (‘2’, ‘A’, ‘B’, ‘C’) 0x3000 0102
ITU Key 3 (‘3’, ‘D’, ‘E’, ‘F’) 0x3000 0103
Phone call start 0x3000 0200
Phone call end 0x3000 0201
Specific Head-Unit Events Key Identifier
Shift Right 0x3000 0001
Shift Left 0x3000 0002
Shift Up 0x3000 0004
Shift Up-Right 0x3000 0005
Shift Up-Left 0x3000 0006
Shift Down 0x3000 0008
Soft key – Left 0x3000 0202
Soft key – Middle 0x3000 0203
Soft key – Right 0x3000 0204
Application key 0x3000 0205
Multimedia – Play 0x3000 0400
Multimedia – Pause 0x3000 0401
Multimedia – Stop 0x3000 0402
Multimedia – Forward 0x3000 0403
Shift Down-Right 0x3000 0009
Shift Down-Left 0x3000 000A
Push 0x3000 0020
Rotate clock wise 0x3000 0050
Rotate anti-clock wise 0x3000 0060
Head-Unit Key 0 0x3000 0300
Head-Unit Key 1 0x3000 0301
Head-Unit Key 255 0x3000 03FF
VNC Protocol – Input Events: ExtensionTouch Head-Unit
Application Dev
ice
Eve
nts
Device
Request Event Mapping
Non-Touch Head-Unit + Knob
Kno
b E
vent
s
Event Mapping
Device Events
VNC Protocol – Input Events: Extension
City:City:
Remove
City:
Show Keyboard
City: Hel Event List
White List:
{ e, u, o, a, s, f, v, m }
Remove Keyboard
VNC Protocol – Driver Distraction
Ok to show, while driving
Not ok to show, while driving
VNC Protocol – Driver Distraction
Problem
• Not all content is save to show on the head-unit display while driving.
• Head-unit cannot reason about the content on its
Context Information Structure
X-position of rectangle
Y-position of rectangle
Width of rectangle
Height of rectangle
Context Information Encodingabout the content on its display from the pixel data.
Solution
• Provide Context Information together with the pixel data.
Context Information Encoding
Unique application identifier
Trust Level for application category
Trust Level for content category
Application category
Content category
Driver Distraction Rules followed
VNC Protocol – Driver Distraction
Trust Level
No Trust
User Configuration
Self-Registered Application
Registered Application
Application Category
Unknown Application
UI Framework
Phone call application
Media applications
Content Category
Text
Video
Image
Vector Graphics
Application certificateMessaging applications
Navigation
Browser
Productivity
UI less applications
System
3D Graphics
User Interface
Car Mode
Media Audio
Voice Audio
Misc. content
Driver Distraction Rules
Minimum font size required.
No video is shown.
No automatic scrolling text
Maximum feedback time allowed
VNC Protocol – Driver Distraction
Ok to show, while driving
Not ok to show, while driving
Application not allowed!
Go back to Home Screen
Audio Input and Output – Overview
Bluetooth
A2DP & HFP
RTP over
UDP/IP
BT Audio Support• BT A2DP for application audio
• BT HFP for phone audio
RTP Audio Streaming• For application and phone audio (bi-directional)
• Audio can be as well voice commands or text-to-speech
Device Attestation – Threats
Fake TM SW
Navigation Application not allowed!
Context
Inform
ation
Filter Device
SMS Navigation
Fake Device
Navigation
Device Attestation – Trust model
Head-unit Manufacturer (HM)
SKHM/PKHM
TrustsCertifies
Device Manufacturer (DM)
CertifiesSKDM/PKDMCertDM = Sig(SKHM, PKDM)
Head-unit (H)
Trust root PKHM
Trusts
Device (D)
CertD = Sig(SKDM, PKD)SKD/PKD
Certifies
DeviceAttestationProtocol
Device Attestation – Attestation Protocol
Random Nonce, Attested software id
Measure requested software componentMeasure requested software componentExtend with IP address and Port numberCreate attestation signature (Quote)
CertDM, CertD, Quote, IP, Port
Verify CertDM with PKHM (� PKDM)Verify CertD with PKDM (� PKD)Verify Quote with PKDSave IP and port
Summary
Terminal Mode consists of a set of basic protocols, enabling user-interface replications
• VNC for display synchronization and event input
• RTP and BT for audio transport
• UPnP for service negotiation
Driver distraction is taken care of in various elements
• Context Information
• Individual Launch and Termination
• Device Attestation
Full specification for download at www.nokia.com/terminalmode
Jörg Brakensiek, Nokia Research Center Palo Alto | June 2, 201027
What does Terminal Mode not Address?
� Providing IP connectivity to the Head-Unit (IP tethering)
� Video and high-end Gaming Use cases for rear-seat entertainment systems
� Provide the CE device access to car data
� Provide the head-unit access to CE device data
� Certification process for automotive applications