how to handle international keyboards for virtual desktops ... · how to handle international...
TRANSCRIPT
How to Handle International Keyboards for Virtual Desktops in Cloud
Zhenjun Zhuo([email protected]), Qiang Wan([email protected])
VMware, Product Globalization
2
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
Virtual Desktops in Cloud
Desktop VMs
3
International Keyboards
4
International Keyboard Issues
Windows 17%
Linux 5%
Mac 20%
iOS 21%
Android 13%
ChromeOS 8%
Web 16%
INTERNATIONAL KEYBOARD ISSUES ACROSS DESKTOPS/DEVICES
German 20%
French 20%
Japanese 18%
Simplified Chinese
8%
Traditional Chinese
8%
Korean 8%
Spanish 5%
Portuguese 2%
General 6%
Others 5%
INTERNATIONAL KEYBOARD ISSUES ACROSS LANGUAGES
• Above international keyboard issues are filtered from VMware bug tracking system as of July 2016 • General: general international keyboard issues are applying to all languages • Others: others includes 15 other input languages 5
Keyboard Handling Architecture Agent
Client
Keyboard
SendInput
Network
Keyboard Event Filtering
Keyboard Input Steam
Key Packets
Key Packets
6
7
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
Windows Keyboard Input
8
* https://msdn.microsoft.com/en-us/library/windows/desktop/ms646267(v=vs.85).aspx
Simulating Input: SendInput
UINT WINAPI SendInput( _In_ UINT nInputs, _In_ LPINPUT pInputs, _In_ int cbSize );
Scan code
Unicode
SendInput OR Keyboard Packets
9
* https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310(v=vs.85).aspx
Language Synchronization: InstallLayoutOrTip
BOOL InstallLayoutOrTip( _In_ LPCWSTR psz, _In_ DWORD dwFlags, );
Layout Id
Lang Id
Keyboard State
Packet AND InstallLayoutO
rTip
Layout Id Lang Id
“0409” (United States) “040C” (French)
10
Remapping Keys - From US 101 to JA 106
US 101 JA 106
@ "
^ &
* (
( )
] [
\ ]
Some mismatches:
11
Remapping Keys - From US 101 to JA 106
• Add 20 key remaps in registry: ID US 101 JA 106
0 tilde Shift + @
1 Shift + tilde Shift + ^
2 Shift + 2 @
3 Shift + 6 ^
4 Shift + 7 Shift + 6
5 Shift + 8 Shift + :
6 Shift + 9 Shift + 8
7 Shift + 0 Shift + 9
8 Shift + - Shift + \
9 = Shift + -
ID US 101 JA 106
10 Shift + = Shift + ;
11 [ [
12 Shift + [ Shift + [
13 ] ]
14 Shift + ] Shift + ]
15 Shift + ; :
16 ' Shift + 7
17 Shift + ' Shift + 2
18 \ Yen/Vertical bar
19 Shift + \ Shift + Yen/Vertical bar 12
13
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
Windows Client: General Process /* WM_KEYDOWN, WM_KEYUP */ bool type = get_event_type(e); /* Windows Scan Code */ uint scancode = get_scancode(e); /* Filtering and special handling */ if(is_special(scancode, type) { /* do something for special events*/ } /* Generate and send keyboard package */ if(scancode) { r = send_keyboard_event(scancode, type); }
Following Slides: 1. Handling special events 2. Language synchronization
14
Handling Special Events: Issue
* Cannot catch KEYDOWN messages for VSCAN_HANGUL_EN and VSCAN_HANJA_EN
15
Handling Special Events: Solution
Bool sendExtraDownToGuest = FALSE; if (up) { if (vscancode == VSCAN_HANGUL_EN || vscancode == VSCAN_HANJA_EN) { sendExtraDownToGuest = TRUE; } }
16
Language Synchronization: Client Process
EnumEnabledLayoutOrTip
• langId • layoutId
• langId • layoutId
• langId • layoutId
Send to Agent
* Windows API
17
* https://msdn.microsoft.com/en-us/library/windows/desktop/bb847907(v=vs.85).aspx
18
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
Mac Client: General Process /* NSKeyDown, NSKeyUp, NSFlagsChanged */ bool type = get_event_type(e); /* Mac ADB(Apple Desktop Bus) code */ uint adb_code = get_ADBCode(e); /* Translate Mac ADB code to scan code*/ scancode = Mapping_ADBtoScan(adb_code); /* Generate and send keyboard package */ if(scancode) { r = send_keyboard_event(scancode, type); }
19
Map Mac-specific keys
20
21
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
Web Client: General Process /* keydown, keyup, keypress */ bool type = get_event_type(e); /* keyCode is used to represent a unique key * Need to handle browser difference here */ var keyCode = get_keyCode(e); /* Translate keyCode to scan code or Unicode*/ var result = Mapping_KeytoSU(keyCode); /* Generate and send keyboard package */ if(result.value) { r = send_keyboard_event(result.value, result.isunicode, type); }
Following Slides: 1. Get keyCode from different browsers 2. Example of special events
22
Keyboard Event Properties
* https://dvcs.w3.org/hg/d4e/raw-file/tip/key-event-test.html
23
Browser Difference: “a” of EN
Keydown Keyup Keypress
keyCode which keyIdentifier keyCode which keyIdentifier keyCode which keyIdentifier
Edge/20 65 65 65 65 97 97
Firefox/47.0 65 65 65 65 0 97
Chrome/51 65 65 U+0041 65 65 U+0041 97 97 U+0041
Safari/9.1.1 65 65 U+0041 65 65 U+0041 97 97
24
Browser Difference: Get keyCode var keyCode = 0, isUnicode = false, property = ‘keyCode’; if (e.keyCode) { keyCode = e.keyCode; } else if (e.which) { keyCode = e.which; property = ‘which’ ; } else if (e.keyIdentifier) { keyCode = e.keyIdentifier; property = ‘keyIdentifier’; isUnicode = true; } else { /* Special events handling */ }
25
Browser Difference: German Characters Demo
26
Browser Difference: Issue(“ü” of DE)
Keydown Keyup Keypress
keyCode which keyIdentifier keyCode which keyIdentifier keyCode which keyIdentifier
Edge/20 186 186 186 186 252 252
Firefox/47.0 0 0 0 0 0 252
Chrome/51 186 186 U+00BA 186 186 U+00BA 252 252 U+00FC
Safari/9.1.1 219 219 U+00FC 219 219 U+00FC 252 252
27
Browser Difference: Solution
if (this.keyboardLayoutId == "de-DE" &&e.keyCode == 0 && WMKS.BROWSER.isFirefox()){ keyCode = Mapping_keytokeyCode(e.key); }
• Check other properties and map to keyCode, for example: ‘key’
e.key keyCode
ü/Ü 252
ö/Ö 246
ä/Ä 228
• Mapping table for German characters:
28
29
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
iOS Client and Android Client
30
Onscreen vs. Physical Keyboards
• No modifier keys and functional keys
31
Onscreen vs. Physical Keyboards
• Long tap and popup alternate keys
32
Onscreen vs. Physical Keyboards
• Emoji characters
33
Onscreen vs. Physical Keyboards
• 10-key input for CJK phones
34
Keyboard Controller Workflow
35
Direct Input Mode
Local Input Mode
Accessory Keys
Btn onClick Send Scan
code
Key-Scancode Mapping
Send Unicode
Send Scan code
On Enter or Tap
Send Btn
On Text Change
Send Unicode
Yes No
Use Case 1: Typing Korean Characters • Typical blocks for Korean characters
• For example • Typing “한글” (Hangul)
• ㅎㅏㄴ(한)ㄱㅡㄹ(글) • Typing “가” (ga)
• ㄱㅏ(가) • Typing backspaces for “간” (gan)
• History issues • Jamos typed failed to form a Korean character • Typing backspace key, the last Korean character failed to decompose into jamos
I M I M
I M
M I M
F * From wikipedia
간 가 ㄱ
36
Use Case 1: Typing Korean
37
Use Case 2: Typing French
38
39
Introduction
Windows Client Mac Client
Web Client
iOS Client and Android Client
International Keyboard Testing
Agent
International Keyboard Testing Matrix
For Platform P = Language L = , and browser B=
Matrix M = P * L + P*L* B
Windows Linux Mac iOS Android Chrome OS
German
French
Japanese
Simplified Chinese
Traditional Chinese
Korean
IE - - - - -
Chrome Chrome Chrome - Chrome
Chrome
Firefox Firefox Firefox - - -
- - Safari Safari
- -
40
International Keyboard Test Cases
41
Automate repetitive tasks that testers slowly become less focused
2915 Overall international keyboard test cases
2889 Automated
99% Automation rate
Automation for Intl Keyboard E2E Testing
Tools Sample Function
Windows C# keybd_event
Mac Apple Script tell application "System Events" to key code {1}
Linux Python Virtkey
iOS Instruments UIAButton tap
Android UI Automator device.click(int x, int y)
42
Takeaways
• International keyboard handling process • Use cases and demos • International keyboard testing
43
Questions & Answers
44