audio engineering society 143rd convention · 2019. 3. 17. · aes143 iot smart speaker testing...
TRANSCRIPT
-
AES143 IoT Smart Speaker Testing
Audio Engineering Society143rd Convention
Challenges of IoT Smart Speaker Testing
Glenn Hess and Daniel Knighten
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
1
-
AES143 IoT Smart Speaker Testing
“Look Ma! No inputs or outputs.”
➢ How do we test a speaker where the only input or output is the cloud?
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
2
-
AES143 IoT Smart Speaker Testing
Test Setup Diagram
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
3
-
AES143 IoT Smart Speaker Testing
Physical Setup
IEEE 1329-2010 Test Setup
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
4
-
AES143 IoT Smart Speaker Testing
Microphone Measurement Steps
1. Combine recorded activation word with frequency sweep equalized at the MRP of the mouth simulator
– Sample stimulus
2. Playback stimulus, then download from the cloud service
– Virtual Audio Cable was used to capture the playback stream.
3. Analyze captured response recording
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
5
-
AES143 IoT Smart Speaker Testing
Speaker Measurement Steps1. Create activation phrase
2. Upload test stimulus to cloud service– The example cloud music service would not accept a linear .wav file.
Conversion to 320 kbps MP3 was used.
– The transparency of the MP3 was tested by encoding and decoding the file and comparing them using Transfer function analysis.
3. Play activation word via mouth simulator and capture response with reference microphone
4. Analyze captured response signal
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
6
-
AES143 IoT Smart Speaker Testing
Correcting for Sampling Error
✓ Smart speaker testing is an asynchronous process
✓ The analysis system and the device under test do not share a sample clock
✓ Clock skew or sampling rate mismatch between the systems can lead to measurement error
✓ All measurements were corrected for sampling rate mismatch by precisely measuring the actual playback frequencies and re-sampling the response waveforms to match the stimulus waveforms.
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
7
-
AES143 IoT Smart Speaker Testing
Sample Microphone Measurement Results
Freq. (Hz)
100 500 1k 5k 10k
Sens.(dB re 1 FS/Pa)
-15.2 -10.1 -9.6 -11.0 -164
Freq. (Hz)
100 500 1k 4k
THD(%)
3.7 0.2 0.3 0.01
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
8
-
AES143 IoT Smart Speaker Testing
Sample Speaker Measurement Results
Freq.(Hz) 100 500 1k 5k 10k
Sens.(dB re 20u Pa/FS)
44.4 88.0 88.6 85.3 84.6
Freq. (Hz)
100 500 1k 5k
THD(%)
5.9 1.2 1.4 0.3
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
9
-
AES143 IoT Smart Speaker Testing
Conclusions
• As long as there is a path to inject a test signal and extract a response signal, we can measure it!
• Sampling rate mismatch must be addressed between device and analysis system.
• Once the mystery surrounding recording and playback is removed, the actual behavior of the systems is relatively conventional.
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
10
-
AES143 IoT Smart Speaker Testing
Future Work
✓ Objective testing of voice recognition accuracy – Word error rate testing
✓ Activation behavior in the presence of interfering noise and talkers
✓ Effects on objective measurements with non-stationary test signals, e.g. noise, speech, etc…
✓ Directivity response
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
11
-
AES143 IoT Smart Speaker Testing
References▪ IEC 61094-4, “Measurement Microphones Part 4:
Specifications for Working Standard Microphones”, November 1995
▪ IEEE 1329-2010, “Standard Method for Measuring Transmission Performance of Speakerphones”, October 2010
▪ ITU-T Recommendation P.51, “Artificial Mouth”, August 1996
▪ Karlheinz Brandenburg, “MP3 and AAC Explained”, presented at AES17th International Conference on High-Quality Audio Coding, August 1999
▪ Steve Temme et al., “The Challenges of MP3 Player Testing”, presented at the AES 122nd Convention, May 2007
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
12
-
AES143 IoT Smart Speaker Testing
Questions?
Property of Indy Acoustic Research LLC and Listen Inc. All Rights Reserved.
13