drive testing. aqua. pvqa

Drive Testing. AQuA. PVQA.

Valeri Sitnikov

Sevana Oü

Tallinn, Estonia

[email protected]

+3726170331

mailto:[email protected]

End-to-End Drive Test

GSM,

WCDMA,V

oLTE,

VoWiFi,

RCS

Codec

(NB, WB)

Drive Tester

Voice Core, PackeCore, Evolved Packet Core

AQuA

PVQA

Drive Test Analysis

AQuA Metrics

MOS score according to P.800 – reliable objective MOS scores from 1 to 5

Percentage of sound quality similarity between reference and test audio

Duration distortion

Delay or advancing of audio signal activity

Sound signal activity mistiming

Signal spectrum damages

Signal spectrum vibration

Unidirectional signal spectrum analysis: amplification or attenuation

Most affected signal spectrum: low, medium, high bands

Average pitch

Pitch deviation

Pitch frequencies distribution distortion

Quantization steps distribution distortion

Amplitude clipping level

RMS

Sound signal: Minimum Energy, Maximum Energy, Average Energy

Sound signal: Min. Sample, Max. Sample, Average Sample

Drive Tester

Android ported NDK library, developed for Samsung phones

Linux, Windows compatible application/library, outperforming existing algorithms in performance

Any Linux-like operating system with standard C++ compiler

Android Drive Tester

PESQ. POLQA. AQuA. Feature Comparison Table.

Feature PESQ POLQA AQuA

Measurements in wideband audio

NO YES YES

Accurate measurements in CDMA networks

NO YES YES

VAD detection in AMR codec

NO YES YES

GSM/WCDMA accurate measurements

NO YES YES

Variable delay up to 1 sec in VoIP

NO YES YES



Strong linear distortions NO YES YES

Limitations with speech enhancement devices (noise

reduction, gain control)

YES NO NO

Works in NB (up to 8 kHz Sample Rate) bandwidths

NO YES YES

Works in WB (up to 16 kHz Sample Rate) bandwidths

NO/YES YES YES

Works in SWB (above 16 kHz Sample Rate) bandwidths

NO YES YES



Works with audio sampling rate higher than 48kHz

NO NO YES

Works with all standard and non-standard codecs audio (AMR-WB, EVRC-WB, iLBC, AMB+, AAC, SLIK,

G.711, G.729)

NO YES YES

Works with all technologies (GSM, WCDMA, LTE, VoIP, VoIP

over IMS)

NO YES YES

Can be used in testing terminals and hand-free applications

YES YES YES

Considers codec degradations (tandem codecs, packet loss,

frame errors, bit errors)

NO YES YES



Customizable perceptual model and algorithm

NO NO YES

Considers network specific degradations

NO YES YES

Copes with P.800 MOS NO YES YES

Talking quality YES NO NO/YES

MOS score for long sequences of speech (more than 30

seconds)

NO NO YES



Noisy listening environments NO NO YES

Music clips NO NO YES

Noise suppressors NO NO YES

Non-telephony applications YES NO YES

Language independent YES NO YES



Multi-dimensional audio(Works with Stereo)

NO NO YES

Unified perceptual model for all types of audio

(speech, music)

NO NO YES

Able of working with long audio(1 hour and longer)

NO NO YES

AQuA has some key features vital for sound quality evaluation during deployment and operation of VoLTE, VoWi-Fi, RCS:

Language independentLong test calls support

Ability to work with Stereo sound

PVQA Metrics

MOS score according to P.800 – reliable objective MOS scores from 1 to 5

Impairments detection (reasons for QoE drop down):

Noise

DeadAir

Amplitude clipping

VAD Clipping

Clicking

Crackling

Fuzzy voice

Echo

DTMF

By just waveform analysis PVQA is able to discover such network issues as

Packet loss (DeadAir + Crackling)

Hardware issues (Amplitude clipping, VAD clipping)

PVQA Analysis

• Non-intrusively calculated MOS score with standard deviation of 0.6 MOS to intrusive AQuA test score

• CSV reports on impairments that caused sound quality degradation

Time; Average; SNR-00; DeadAir-01; DeadAir-00; AMPC-00; VADC-00; Click-00;

0.000 : 0.799; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

0.799 : 1.598; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

1.598 : 2.397; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

2.397 : 3.196; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

3.196 : 3.995; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

3.995 : 4.794; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

4.794 : 5.593; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

5.593 : 6.392; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

6.392 : 7.191; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

7.191 : 7.990; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

7.990 : 8.789; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

8.789 : 9.588; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

9.588 : 10.387; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

10.387 : 11.186; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

11.186 : 11.985; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

11.985 : 12.784; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

12.784 : 13.583; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

13.583 : 14.382; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

14.382 : 14.882; 0.013333; 0.000000; 0.080000; 0.000000; 0.000000; 0.000000; 0.000000; Ok


0.000 : 0.799; 0.014583; 0.000000; 0.087500; 0.000000; 0.000000; 0.000000; 0.000000; Ok

0.799 : 1.598; 0.018750; 0.000000; 0.112500; 0.000000; 0.000000; 0.000000; 0.000000; Poor

1.598 : 2.397; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

2.397 : 3.196; 0.085340; 0.000000; 0.425000; 0.000000; 0.000000; 0.050000; 0.037037; Poor

3.196 : 3.995; 0.039583; 0.000000; 0.225000; 0.000000; 0.000000; 0.012500; 0.000000; Poor

3.995 : 4.794; 0.018750; 0.000000; 0.100000; 0.000000; 0.000000; 0.012500; 0.000000; Poor

4.794 : 5.593; 0.034395; 0.000000; 0.162500; 0.000000; 0.000000; 0.025000; 0.018868; Poor

5.593 : 6.392; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

6.392 : 7.191; 0.055228; 0.000000; 0.300000; 0.000000; 0.000000; 0.012500; 0.018868; Poor

7.191 : 7.990; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

7.990 : 8.789; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

8.789 : 9.588; 0.043750; 0.000000; 0.225000; 0.000000; 0.000000; 0.037500; 0.000000; Poor

9.588 : 10.387; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

10.387 : 11.186; 0.114623; 0.000000; 0.612500; 0.000000; 0.000000; 0.037500; 0.037736; Poor

11.186 : 11.985; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

11.985 : 12.784; 0.014583; 0.000000; 0.075000; 0.000000; 0.000000; 0.012500; 0.000000; Ok

12.784 : 13.583; 0.004167; 0.000000; 0.025000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

13.583 : 14.382; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; 0.000000; Ok

14.382 : 14.832; 0.014815; 0.000000; 0.066667; 0.000000; 0.000000; 0.022222; 0.000000; Ok


0.000 : 0.799; 0.014583; 0.000000; 0.087500; 0.000000; 0.000000; 0.000000; 0.000000; Ok

0.799 : 1.598; 0.018750; 0.000000; 0.112500; 0.000000; 0.000000; 0.000000; 0.000000; Poor

1.598 : 2.397; 0.087500; 0.000000; 0.500000; 0.000000; 0.000000; 0.025000; 0.000000; Poor

2.397 : 3.196; 0.148920; 0.000000; 0.812500; 0.000000; 0.000000; 0.062500; 0.018519; Poor

3.196 : 3.995; 0.093750; 0.000000; 0.525000; 0.000000; 0.000000; 0.037500; 0.000000; Poor

3.995 : 4.794; 0.035417; 0.000000; 0.187500; 0.000000; 0.000000; 0.025000; 0.000000; Poor

4.794 : 5.593; 0.133333; 0.000000; 0.712500; 0.000000; 0.000000; 0.087500; 0.000000; Poor

5.593 : 6.392; 0.077083; 0.000000; 0.400000; 0.000000; 0.000000; 0.062500; 0.000000; Poor

6.392 : 7.191; 0.117728; 0.000000; 0.637500; 0.000000; 0.000000; 0.050000; 0.018868; Poor

7.191 : 7.990; 0.036900; 0.000000; 0.177215; 0.000000; 0.000000; 0.025316; 0.018868; Poor

7.990 : 8.789; 0.070833; 0.000000; 0.375000; 0.000000; 0.000000; 0.050000; 0.000000; Poor

8.789 : 9.588; 0.096836; 0.000000; 0.500000; 0.000000; 0.000000; 0.062500; 0.018519; Poor

9.588 : 10.387; 0.052083; 0.000000; 0.300000; 0.000000; 0.000000; 0.012500; 0.000000; Poor

10.387 : 11.186; 0.125000; 0.000000; 0.712500; 0.000000; 0.000000; 0.037500; 0.000000; Poor

11.186 : 11.985; 0.139506; 0.000000; 0.750000; 0.000000; 0.000000; 0.050000; 0.037037; Poor

11.985 : 12.784; 0.132311; 0.000000; 0.750000; 0.000000; 0.000000; 0.025000; 0.018868; Poor

12.784 : 13.583; 0.128145; 0.000000; 0.700000; 0.000000; 0.000000; 0.050000; 0.018868; Poor

13.583 : 14.382; 0.086478; 0.000000; 0.475000; 0.000000; 0.000000; 0.025000; 0.018868; Poor

14.382 : 14.832; 0.107407; 0.000000; 0.600000; 0.000000; 0.000000; 0.044444; 0.000000; Poor

PVQA Analysis

MOS = 2.894737

PVQA Analysis

PVQA Root Cause Analysis

VAD clipping related to shortening of voice snippets due to failures in voice activity detection, the so-called squelch

Amplitude clipping with noise - form of waveform distortion that occurs when an amplifier is overdriven and attempts to deliver an output voltage or current beyond

its maximum capability.

PVQA Pattern Discovery

There are two types of patterns PVQA allows to register and disover:

• Network specific impairments (prevailing of clipping, noise, or signals with

crackling)

• Combination of impairments specific for certain events, e.g. packet loss

Packet Loss pattern

Pitch

o AQuA offers reliable end-to-end sound quality assessment for both NB and WB audio providing reasonable MOS scores according to P.800.

o AQuA is working on laptops, servers and Android devices providing high performance in MOS calculation and analysis.

o AQuA can be ported to other systems that have standard C++ compiler (Linux-type operating systems, iOS)

o PVQA provides reliable approach for non-intrusive MOS score calculation

o PVQA provides additional information on impairments that cause voice quality degradation:

o Root Cause Analysis

o Impairments Pattern Discovery

o Packet loss detection by waveform analysis

THANK YOU!

drive testing. aqua. pvqa

Technology