chuhan gao, kassem fawaz, sanjib sur, and suman banerjee ... · sanjib sur:...

Proceedings on Privacy Enhancing Technologies ; 2019 (2):146–165

Chuhan Gao, Kassem Fawaz, Sanjib Sur, and Suman BanerjeePrivacy Protection for Audio Sensing AgainstMulti-Microphone AdversariesAbstract: Audio-based sensing enables fine-grained hu-man activity detection, such as sensing hand gesturesand contact-free estimation of the breathing rate. Apassive adversary, equipped with microphones, canleverage the ongoing sensing to infer private informa-tion about individuals. Further, with multiple micro-phones, a beamforming-capable adversary can defeatthe previously-proposed privacy protection obfuscationtechniques. Such an adversary can isolate the obfusca-tion signal and cancel it, even when situated behinda wall. AudioSentry is the first to address the privacyproblem in audio sensing by protecting the users againsta multi-microphone adversary. It utilizes the commod-ity and audio-capable devices, already available in theuser’s environment, to form a distributed obfuscatorarray. AudioSentry packs a novel technique to carefullygenerate obfuscation beams in different directions, pre-venting the multi-microphone adversary from cancelingthe obfuscation signal. AudioSentry follows by a dynamicchannel estimation scheme to preserve authorized sens-ing under obfuscation. AudioSentry offers the advantagesof being practical to deploy and effective against an ad-versary with a large number of microphones. Our ex-tensive evaluations with commodity devices show thatAudioSentry protects the user’s privacy against a 16-microphone adversary with only four commodity obfus-cators, regardless of the adversary’s position. AudioSen-try provides its privacy-preserving features with littleoverhead on the authorized sensor.

Keywords: keywords, keywords

DOI 10.2478/popets-2019-0024Received 2018-08-31; revised 2018-12-15; accepted 2018-12-16.

Chuhan Gao: University of Wisconsin-Madison, E-mail:[email protected] Fawaz: University of Wisconsin-Madison, E-mail:[email protected] Sur: University of South Carolina, E-mail:[email protected] Banerjee: University of Wisconsin-Madison, E-mail:[email protected]

1 IntroductionThe ubiquity of commodity devices with microphonesand speakers have made audio-based sensing of humanactivity and health attributes feasible and attractive.Contact-free and audio-based sensing systems can sensefine-grained human gestures [12, 24, 42, 54, 60], move-ments [28, 30], behavior [10, 21, 22, 29], and health at-tributes [33, 37, 53, 55]. Akin to sonars, these audio-based sensing systems transmit near-ultrasound signalsand analyze their reflections off the human body. Asthe mechanical motion of the individuals and their bodyparts shapes the reflected signal, the sensing system canidentify a range of human attributes. For example, mov-ing hands in a certain direction results in a distinct re-flected signal which allows for identifying hand gestures.Also, fine-grained finger movements result in detectablechanges in the reflected signal, enabling the develop-ment of new user interfaces (UIs) for devices. Chestmovement due to breathing impacts the reflected sig-nal which allows for estimating health situation of anindividual.

Unfortunately, a stealthy adversary can passively ex-ploit audio-based sensing at a low cost. While situatedoutside the user’s environment, such an adversary canemploy a set of microphones, or gain access to the vul-nerable user-owned audio devices, to capture and ana-lyze the audio sensing signals that reflect off the user.Capturing these reflections can lead to serious privacyand security threats [35]. These threats extend beyondidentifying users’ attributes to impacting their physicalenvironment:

– Smart home systems and personal devices are in-creasingly employing audio sensing to accept humangestures as inputs [12, 24, 54]. In February 2018, El-liptic Labs announced a new SDK that adds gesturerecognition capability to any programmable devicewith a speaker and microphone [6]. The adversarycan stealthily record the audio sensing signals re-flected by user’s gestures and replay them to con-trol the smart environment (e.g., light, doors andsecurity camera). As audio reflections leak more in-formation than gestures, an adversary can benefitfrom the ongoing sensing to discretely monitor user

Privacy Protection for Audio Sensing Against Multi-Microphone Adversaries 147

behavior, such as their room occupancy and per-formed activities.

– Audio-based sensing enables contact-free monitor-ing of the health conditions of individuals. For ex-ample, researchers have demonstrated the efficacyof audio sensing in the detection of sleep apnea(this technology has already been licensed by a sleephealth-care company) [33] and the monitoring ofbreathing patterns [37, 53, 55]. A stealthy adver-sary can leverage the ongoing sensing to remotelycollect sensitive health information about the user.

Preventing privacy leaks from contact-free sensinghas not been addressed previously, except for Phy-Clock [38], which thwarts the sensing capability of asingle-antenna adversary in the context of wireless radiosignal sensing. PhyClock employs a single obfuscator toprevent an adversary from correctly decoding the sens-ing signal and its reflections. It is well-known, however,that an adversary with multiple antennas can performbeamforming by focusing its receiver at a particular di-rection, thereby isolating and removing the effect of anyobfuscation signal [49]. As microphone arrays are verycheap and accessible [1], a single obfuscator cannot pre-vent privacy leaks from audio-based sensing.

In this work, we first investigate how a passive andmulti-microphone adversary can circumvent a single-obfuscator defense, even when separated by a wall.Then, we propose a mechanism that prevents a multi-microphone adversary’s from monitoring the user’s be-havior, activities and health attributes from a distance.We consider two representative scenarios of audio sens-ing: hand-gesture recognition and breathing rate estima-tion. We find that an adversary, behind a wall and 6.5meters far from the audio-sensing system, can correctlyidentify user’s gestures and breathing rate. An adversarywith sensitive microphones can eavesdrop from even amore extended range.

Naively adding more obfuscators does little to re-duce privacy leaks from the multi-microphone adversarybecause of three challenges. First, there is the issue ofpracticality; the users are unlikely to purchase custom-built obfuscator hardware that are dedicated to protecttheir privacy, especially when privacy is a secondary con-cern for individuals [57]. Second, even if the first issuewere to be overcome, determining how many obfuscatorsto deploy is not straightforward. The attacker’s abilityto resolve signal directions is directly proportional to thenumber of microphones it uses; trying to scale the num-ber of deployed obfuscators with the adversary’s micro-phones is a losing battle. Third, the authorized sensing

of the user has to be preserved even when obfuscationis running, without requiring any hardware changes.

We propose AudioSentry, a system that prevents anunauthorized adversary from stealthily identifying usergestures and health attributes while preserving this ca-pability for an authorized sensor. In particular, it pro-tects the user against a strong passive adversary de-ploying an external set of microphones. AudioSentry ad-dresses the aforementioned challenges by employing thewidely available commodity devices such as smart TVs,laptops, and smartphones as a distributed obfuscatorarray. Instead of purchasing dedicated devices, the usercan install a program/app on their existing device toenable AudioSentry. From a high-level perspective, Au-

dioSentry has two components: Multi-directional beam-forming and obfuscation signal cancellation.

Multi-directional beamforming: AudioSentry

uses distributed beamforming (from the user’s commod-ity devices) to transmit independent obfuscation signalsacross all different directions, some of which will coverthe user. The reflected signal off the user contains acomponent from the known sensing signal and anotherfrom a randomized obfuscation signal. Since the signalsreceived along other directions are independent of theobfuscation signal hitting the user, the adversary can-not isolate the reflections from the sensing signal. Thisdesign decision offers a significant advantage in the racebetween AudioSentry and the adversary; with a limitednumber of commodity devices, AudioSentry can thwartsensing capability of an attacker with a larger numberof microphones.

Obfuscation Signal Cancellation: In the pres-ence of multiple obfuscators, it is essential for AudioSen-

try to cancel their interference as to preserve autho-rized sensing. Interference cancellation requires estimat-ing the audio channel between each obfuscator and thesensor which is challenging because of the user’s move-ment. To address this problem, we propose a novel in-verse channel estimation scheme that runs concurrentlywith sensing. In this scheme, each of obfuscators esti-mates the channel with the authorized sensor and com-municates the estimated channel with the obfuscationsignal over a secure out-of-band channel, such as WiFi.With access to such information, the authorized sensor,but not the adversary, can cancel the interference causedby obfuscation.

We have implemented AudioSentry using commodityspeakers and evaluated it extensively against differentadversaries. Our evaluation shows that, with only fourcommodity speakers, AudioSentry limits the informationinferred about the user even when the adversary in-


creases its array 16 microphones. With 16 microphones,the adversary’s gesture recognition accuracy is close to21% (11% for a random guess) and its average error forbreath rate estimation is more than seven breaths perminute (the average human breathing rate is between 12and 20 bpm). AudioSentry achieves this privacy protec-tion across various adversary placements/configurationsand with little overhead on the authorized sensing.

In summary, this paper has the following contribu-tions:

– demonstrating that a single obfuscator is ineffectiveagainst a modest and far multi-microphone adver-sary, even when a wall separates them (Sec. 3);

– a novel mechanism to employ a limited number ofcommodity devices to thwart the sensing of an at-tacker with a larger number of microphones (Sec. 5);

– a novel inverse channel estimation approach to pre-serving authorized sensing even with the presenceof multiple obfuscators (Sec. 6); and

– the implementation and extensive evaluation of Au-dioSentry which highlights its privacy preserving ca-pabilities and low overhead (Sec. 7).

2 BackgroundIn this section, we provide a primer on audio-based sens-ing and signal obfuscation.

2.1 Audio-based Sensing

In an audio sensing scenario, there are two entities:the sensor and the target. The sensor has two com-ponents, a transmitter (speaker) and a receiver (micro-phone). While the microphone and speaker are usuallyco-located (e.g., the smartphone as a sensor), they donot have to be as such. The speaker emits a known sens-ing signal, S(t) towards the target (e.g., an individual).S(t) which can be modeled as a baseband signal s(t)modulated at a carrier/sensing frequency fc, where fcis in the inaudible range, typically between 17 kHz and20 kHz. The signal S(t) travels through a physical chan-nel to reach the microphone of the receiver. Within thisphysical channel, when S(t) hits the target, the signalundergoes three phenomena: (1) the user absorbs partof it, (2) another part reflects back to the environmentin different directions, and (3) the reflected part under-

goes a Doppler shift (shift in the frequency) due to themechanical movement of the target.

As a result, the sensor’s microphone receives rφ(t)along an incident angle φ, which can be expressed as:

rφ(t) = α× s(t)× cos(2π(fc + fD)(t−∆t)), (1)where α is the amplitude change, ∆t is the delay andfD is the Doppler shift. While these three parametersare each function of φ, we slightly abuse the notationby omitting φ from α, ∆t and fD in Eq. (1). A singlemicrophone receives omni-directionally where there isno notion of φ. On the other hand, an array of micro-phones is able to separate received signal across differentdirections.

The parameters α, ∆t and fD, characterize the au-dio channel between the target and the sensor. In the au-dio channel, both free space propagation and reflectionattenuate the signal amplitude, resulting in a smaller α.The propagation time (function of the distance), as wellas the target movement, determine the delay ∆t. Finally,the target movement determines the value of Dopplershift fD. Audio sensing applications utilize these threeparameters to characterize the target reflecting the sens-ing signal. For example, the Doppler shift created byhand movement can be used to identify different handgestures.

2.2 Sensing Obfuscation

An obfuscator aims to hide these three channel parame-ters, at every signal path, to prevent the adversary fromobtaining useful information. Only the authorized sen-sor should be able to sense the user and retrieve informa-tion such as gesture or biometrics. Consider the case forobfuscating the adversary’s received audio signal rφ(t).The obfuscator emits a signal oφ(t) along direction φ sothat the microphone receives: ro(t) = oφ(t) + rφ(t). Theobfuscator designs oφ(t) in such a way that the receivedsignal ro(t) leaks little information about the parame-ters of rφ(t): α, ∆t, and fD.Amplitude gain α: Obfuscating the gain is straightfor-ward as the obfuscator randomly modifies the amplitudeof the obfuscation signal through changing its normal-ized volume between 0 and 1.Delay ∆t: Directly delaying the obfuscation signal inthe digital domain introduces discrete delay (incrementsof 22 µs for Fs = 44.1 kHz audio sampling rate). Instead,an obfuscator can add an additional initial phase to thesensing signal to manipulate the delay with higher res-olution. With an initial phase θ, the obfuscation signal


is:oφ(t) = cos(2π(fc + fD)t+ θ) (2)

which can be organized as

oφ(t) = cos(

2π(fc + fD)(t+ θ

2π(fc + fD)

))The introduced equivalent delay is −θ

2π(fc+fD) , whichis a continuous function of θ. The obfuscator can thenmodify θ to obtain fine-grained control over the equiva-lent delay, without actually delaying the signal.Doppler shift fD: To emulate Doppler shift effect,the obfuscator transmits signals with different frequen-cies to cover the possible Doppler shift range of theoriginal sensing signal. Specifically, the obfuscator si-multaneously transmits on the frequencies {fl, · · · , fl +n∆f, · · · , fh} to cover the reflected signal (rφ(t)). TheDoppler shift can be estimated as fD = ∆v

c fc, where ∆vis the relative speed between reflector and receiver, c isthe speed of sound, and fc is the sensing frequency.Obfuscation signal: In order to hide all three channelparameters, the obfuscation signal along a certain signalpath, φ, can be given as:

oφ(t) =fl+n∆f≤fh∑

n

αn × cos(2π(fl + n∆f)t+ θn) (3)

3 System and Threat Models

3.1 System Model

Our audio sensing scenario involves a sensor and theuser. The sensor employs a commodity device, with aspeaker and a microphone, to sense the user’s behav-ior and attributes as explained earlier in Sec. 2. In thesame environment, AudioSentry leverages the existenceof other commodity devices, each with omni-directionalspeakers and microphones, to provide its privacy pre-serving features. AudioSentry requires those devices tobe programmable and equipped with WiFi, such assmart TVs (or those with Chromecast), laptops, smarthome assistants (e.g., Google Home mini), tablets, andsmartphones. AudioSentry runs a software componenton those commodity devices to transmit a specifically-coordinated obfuscation signal that prevents the unau-thorized entities from sensing the user while preservingthis capability for the authorized sensor.

3.2 Threat Model

An adversary aims to stealthily exploit the ongoing au-dio sensing to identify the user’s behavior and healthattributes. To that end, the adversary can be one of thefollowing two types: (i) It deploys an array of micro-phones at a distance (e.g., at a neighboring apartment).The adversary performs its attack with a dedicatedmicrophone array, and does not have access/controlover the user-owned commodity devices. (ii) Alterna-tively, it could potentially compromise user-owned andmicrophone-enabled devices to achieve stealthy audio-based sensing. Such compromises can happen on mo-bile/browser platforms by abusing microphone permis-sions [13, 16, 17, 43]. Note than attacker can performthis attack with a single microphone, but in this paper,we consider a stronger threat model than can thwartsingle-obfuscator defenses. We assume that the adver-sary is capable of synchronizing its microphone array toachieve receiving beamforming, to identify the differentsignal paths (more on that in Sec. 3.3). The adversarycan steer its beam direction electronically during offlineprocessing, without needing to adjust the array config-uration when eavesdropping.

Further, we assume a strong passive adversary thatcan recover the precise locations of the target user, theauthorized sensor, and the obfuscators through blindsource separation. Conventionally, blind source separa-tion is a signal processing technique that helps the re-ceiver to recover independent signal streams from un-known sources. In our model, the knowledge of all sig-nal source locations allows the adversary to accuratelyperform beam steering without the need for source sep-aration, yielding the strongest performance. Note thatthe precise user location is not absolutely necessary forblind source separation, but can significantly improvethe performance of a strong adversary with such infor-mation.

The adversary aims to extract the physical chan-nel information of the sensing signal reflected off thetarget user, which contains physical information aboutthe user. By eavesdropping and analyzing the physicalchannel information, the adversary can obtain privatepersonal information about the target users in their pri-vate spaces, such as health condition inferred from res-piration rate information. Besides, the adversary canreplay the recorded reflected signal to impersonate theuser’s gestures and behavior. This possible attack raisesalarming security risks when audio sensing serves as auser-input mechanism for devices [54].


Fig. 1. Experiment setupof a 8-microphone adversaryand a single obfuscator. Oxshort for obfuscator.

Fig. 2. Adversary micro-phone array, separated withthe user by 6 meters andwith a wall in between.

Finally, AudioSentry utilizes WiFi as a secure out-of-band (OOB) channel to synchronize the authorizedsensor with the other commodity devices. We assumethat all devices (authorized sensor and obfuscators) areconnected to the same WiFi network that is properly se-cured, such that the adversary cannot intercept/modifythe WiFi traffic. AudioSentry has no option but to trustthose commodity devices; otherwise, its operation willbe compromised and will fail to deliver the promisedprivacy provisions.

3.3 Privacy Threats from Passive Sensing

The state-of-art in protecting the privacy of users froma passive adversary focuses on employing a single omni-directional obfuscator, as explained in Sec. 2.2. In thefollowing, we show that a single omni-directional obfus-cator does little to prevent a multi-microphone adver-sary from correctly identifying the user’s fine-grainedgestures and accurately estimating the breathing rate.In addition, we show that the adversary can effectivelycompromise user’s privacy even when a wall separatesboth of them.

3.3.1 Experiment Setup

Our experimental setup (Fig. 1) consists of four entities:the sensor, the user, an obfuscator, and an adversary be-hind a concrete wall with around 12 to 17 cm in width.The sensor has a speaker that emits the sensing signaland the obfuscator is another speaker emitting an ob-fuscation signal. Fig. 2 shows the adversary microphonearray consisting of eight microphones, each with -30dBsensitivity. Each pair of adjacent microphones are sepa-rated by 7 cm. The cost of the microphone array is lessthan $120 (including the synchronization board and themicrophones). The array is placed right next to the con-

crete wall, while the legit sensor, obfuscator device, anduser are in another room.

Close to the sensor, a user moves his hand back andforth. The adversary aims to utilize its multiple micro-phones to cancel the effect of the obfuscation signal andestimate user’s hand movement as well as the breath-ing rate. Behind-the-wall adversary demonstrates thefeasibility of a practical attack, where the adversary’smicrophones are deployed outside user’s room behind awall or door. The distance between the adversary’s ar-ray and the target is 6 m, with the user standing around5.5 m away from the wall. Beyond this range, it becomesdifficult for our low-cost microphone array to succeed inattack, but With more sensitive microphones, the adver-sary can considerably increase its sniffing range.

In the first scenario, the adversary tries to identifythe user’s gesture by observing the Doppler shift pattern.The sensor transmits a single tone at 16 kHz as S(t), andthe obfuscator transmits o(t) such that:

o(t) =fl+n∆f≤fh∑

n

αn × cos(2π(fl + n∆f)t+ θn) (4)

where fl = 15.85 kHz, fh = 16.15 kHz, n is integer,∆f = 5 Hz, and θn changes randomly between 0 and 2πevery 100 ms.

In the second scenario, the adversary attemptsto identify the user’s breathing rate. We implementthe breathing rate estimation system, ApneaApp [33],where the sensor plays Frequency Modulated Contin-uous Wave (FMCW) signals between 16 kHz and 18kHz. Each chirp is 512 samples long, and the receivergroups ten chirps to improve FFT resolution similar ToApneaApp. The obfuscator plays the same signal as inEq. (4), with fl and fh set to 16 kHz and 18 kHz, respec-tively (to cover the frequency range of the transmittedsignal).

3.3.2 Signal Recovery

The objective of the adversary is to obtain hu,a whichis the acoustic channel between the user and adversary.To that end, the adversary obtains each of S(t), o(t)and ra(t) = (α · S(t) + β · o(t)) ∗ hu,t(t) by steering itsreceiving beam off-line towards the sensor, obfuscatorand the user, respectively. The adversary utilizes its mi-crophone array to achieve directional beamforming [48]by delaying the signal received by each microphone by adifferent amount, which coherently combines the signalfrom a certain direction.


Fig. 3. Doppler shift pattern extracted by the adversary be-hind a wall 6 meters away from the user.

0 10 20 30 40 50 60

Chest m

ovem

ent

Time (s)(a)

Detected breath rate: 12.1 bpm

0

10

20

30

40

0 0.5 1 1.5 2 2.5 3 3.5 4

Am

plit

ude (

dB

)

Frequency (Hz)(b)

Fig. 4. Chest motion signal recovered by adversary behinda wall 6 meters away from the user, in (a) time domain (b)frequency domain

To recover the channel, hu,a, the adversary removesthe effect of (α · S(t) + β · o(t)) from the received signalra(t). In our experiments we use a special preamble tocalibrate α and β. A real adversary could easily utilizepast information to achieve the calibration. Nonetheless,we find that even if we do not consider these two param-eters, the Doppler shift pattern can still be recovered, al-beit with slight quality degradation. hu,a(t) is obtainedas:

hu,a(t) = F−1 F{ra(t)}F{αS(t) + βo(t)} (5)

where F represents the Fourier Transformation.For both applications, we experimented with a num-

ber of state-of-the-art approaches [12, 15, 24, 33, 54, 55]to decide on the best performing techniques. We settledon the approach of Das et al. [15] for the gesture recog-nition application and that of Nadakumar et al. [33] forthe breathing rate estimation application.

Extracting hand gestures. In the first scenario,a user moves his hand forward, backward and then for-ward. Fig. 3 shows the Doppler shift, extracted fromhu,a(t) over 1.75 seconds. It is clear from the figure howthe Doppler shift pattern is consistent with the usermovement, although an obfuscator was running.

Extracted breathing rate. In the second sce-nario, we estimate the channel hu,a(t) for 60 seconds,during which the user is breathing normally. Using theestimated channel, we extract the chest movement as ev-ident from Fig. 4. The figure shows the extracted chestmovement pattern in the time and frequency domains.The frequency spectrum of the chest movement revealsa peak at 0.201 Hz, which corresponds to 12.1 breathsper minute (bpm); the ground truth was 11 bpm.

Fig. 5. High-level overview of AudioSentry. The adversaryreceives the combined reflections resulting from s(t) (bluebeam) and o1(t) (purple beam) hitting the user. Since the Oxarray sends decorrelated beams in all directions, AudioSentryprevents the adversary from receiving o1(t) and estimating thechannel.

We also conducted the same attack experimentswith the adversary mic array placed in the same roomwith the user, with the same 6 m distance between them.Without the wall, the adversary receives much strongeraudio signals from the user, legit sensor and obfuscator,leading to even better estimation accuracy.

4 AudioSentry OverviewAudioSentry addresses the above privacy threats by al-lowing only authorized clients to sense the user. Froma high-level perspective, AudioSentry employs signal ob-fuscation to hide the sensing signal’s reflections off theuser from a multi-microphone adversary. Such adversarycan focus its receiving beam into different signal pathsto isolate the obfuscation signal and then remove its ef-fect from the user reflections. AudioSentry counters thisadversary by applying obfuscation along different sig-nal paths, which necessitates using multiple speakers. Itpreserves user’s privacy while by addressing three chal-lenges: the number of speakers needed, the practicalityof deployment, and preserving the authorized sensing.

Multi-beam Beamforming.AudioSentry could attempt to obfuscate as many signalpaths as possible, which locks it in an arms-race betweenits number of speakers and the adversary’s number ofmicrophones. Alternatively, AudioSentry only hides thesignal paths corresponding to the user reflections whilepreventing the leak of the obfuscation signal (otherwisethe adversary can cancel its effect). To hide those signalpaths, AudioSentry employs a novel multi-beam beam-forming mechanism. It generates decorrelated and ran-domized signals along the 360 degrees, as evident inFig. 5.


As such, the reflections of the user will have twooverlapping components: those resulting from the orig-inal (and known) sensing signal and those from theobfuscation signal. Because the obfuscation signal isdecorrelated in different directions, the adversary can-not employ beamforming to isolate the obfuscation sig-nal. Without access to the obfuscation signal hittingthe user, the adversary cannot properly discern the re-flections from the sensing signal.

Distributed Obfuscation with Commodity Devices.Generating decorrelated obfuscation beam requires mul-tiple speakers. It is highly impractical to require the userinstall a set of dedicated obfuscator devices in their envi-ronment, given that privacy is usually a secondary con-cern [57]. AudioSentry meets this challenge through lever-aging the commodity devices, such as smart TVs, homeassistants, and PCs, in the user’s environment to form adistributed array of audio obfuscators. Our evaluationsof AudioSentry shows that it needs only four speakersto defend against an adversary with 16 microphones.Commodity devices that are already available in homeenvironments can easily meet this requirement. Smart-phones, tablets and laptops generally pack two speak-ers [2, 4]. Smart speakers1 contain multiple speakers; forexample, Apple HomePod has 7 speakers [3]. Also, con-nected TVs2 usually have two speakers. These commod-ity devices only need to perform AudioSentry obfuscationwhen the audio sensing application is active, which pre-serves Ox device battery life. In addition, devices suchas TV and smart speakers are typically plugged in, with-out battery life concern.

There are three problems associated with beamform-ing on a set of distributed and heterogeneous devices:device synchronization, beamforming weight computa-tion and audio hardware heterogeneity. We address thefirst issue by retrofitting an existing mobile device syn-chronization solution that forms a distributed micro-phone array on a set of smartphones [48]. For the sec-ond problem, we first perform audio-based localizationto determine the relative locations among obfuscators.Then, we estimate the phase offset of each transmitter tosteer multiple beams to different signal paths. AudioSen-try overcomes the third issue by performing a one-timeinitial calibration to compensate for audio hardware di-versity at run time as will be evident in Sec. 6.

1 16% of Americans own a smart speaker [7].2 58% of TV households in the US have a connected TV [5].

Fig. 6. Linear array directional beamforming example.Preserving Authorized Sensing.As a result of obfuscation, the authorized sensor experi-ences interference from the obfuscation signals. We de-vise an interference cancellation mechanism that pre-serves the functionality of the authorized sensor. Theinterference, to be canceled, is a function of each of theobfuscation signals and the channel between each obfus-cator and the sensor. The authorized sensor has to knoweach of the obfuscation signals and each of the channelsas to cancel them. Achieving the first task is straight-forward; AudioSentry utilizes WiFi as an OOB channelto communicate the obfuscation signal information fromeach obfuscator to the sensor.

For the second task, we propose a dynamic channelestimation scheme; it has to be dynamic because theuser movement continuously changes the channel. Oneapproach is to have each obfuscator transmit a knownsignal sequence so that the sensor can perform the chan-nel estimation. This process introduces additional inter-ference to the channel and requires more coordinationbetween the obfuscators. Instead, AudioSentry leveragesthe channel reciprocity to have the sensor broadcast atraining signal on a close, but different, frequency rangefrom that of the sensing. After receiving the trainingsignal, each obfuscator estimates the channel betweenitself and the sensor. Then, it sends the estimated chan-nel parameters over the OOB channel to the sensor.

5 Multi-Beam BeamformingIn this section, we explain how AudioSentry employs mul-tiple commodity devices to create a distributed beam-forming array. As described earlier, this array gener-ates decorrelated obfuscated signals along different di-rections.

5.1 Directional Audio Transmission

Audio beamforming employs multiple speakers (micro-phones) to transmit (receive) directional audio signals— i.e., boost the transmitted (received) signal in specificdirections and decrease it towards others.Consider the N -speaker array, in linear arrangement,with distance d separating the speakers in Fig. 6. To


create a directional signal towards an Angle of Depar-ture (AoD) ψ, the signal from each speaker is delayedby a specific time delay to compensate for the differentpath length of each of the signals. For example, Fig. 6shows the path length difference of the transmitted sig-nal from 1st speaker relative to the ith speaker’s alongψ as (i−1)d cosψ. To constructively combine the signalsalong ψ, a classical beamforming algorithm delays thetransmitted signal from the ith speaker by (i− 1)d cosψ

c

(c is the speed of sound in air).We adapt this classical beamforming algorithm to

fit AudioSentry’s context. AudioSentry employs a set ofheterogeneous and arbitrarily placed speaker elementsto steer an obfuscation beam in different directions.In order to steer the beam towards a particular direc-tion, AudioSentry picks a reference speaker and computesthe desired delay τ between the reference and each ofthe other speakers based on their positions (estimatingthe position will be described later). As long as eachspeaker’s signal is constructively combined with the ref-erence speaker’s signal along ψ, the overall signal wouldbe boosted along this direction. Suppose the referencespeaker and another speaker, i, have coordinates (x0, y0)and (xi, yi), then their path length difference along ψ(same angular coordination system as Fig. 6) is:

pdi =√

(y0 − yi)2 + (x0 − xi)2 cos(ψ − arctan

(y0 − yi

x0 − xi

))(6)

Then, the obfuscation signal to be transmitted fromthe speaker i has to be delayed by τi = di

c . We follow thesame approach of Sec. 2.2 to control the delays of theobfuscation signals by manipulating their initial phases.

5.2 Steering Multiple Beams

AudioSentry transmits obfuscation beams with differentchannel parameters (delay, magnitude, and frequencyshift) along different AoDs. Suppose we aim to createmultiple obfuscation signals o1(t), · · · , om(t) with AoDas ψ1, · · · , ψm, respectively. We create each directionalom(t) following the method described in Sec. 5.1. Forsimplicity, we first assume each om(t) is a narrow bandsignal cos(2πfmt) and extend to wideband signal inSec. 5.3. The signal transmitted by the ith speaker tocreate om(t) along ψm is:

oim(t) = αm cos(2πfm(t− τ im) + θi) (7)where τ im is the delay created by the ith speaker relativeto the reference speaker in the way described in Sec. 5.1.

The signal received at a far away position along directionψ can be expressed as (propagation loss omitted):

om(t, ψ) =∑i

αm cos(2πfm(t−τ im)+θm+ pdiλm

2π), (8)

where pdm is the path difference as defined by Eq. (6)and λm = c

fm. Next, we have each speaker i play∑

m oim(t). With N speakers, we are able to steer N − 1independent beams. Since the parameters of each om(t)are independent, AudioSentry creates different obfusca-tion signals along different directions.

5.3 Obfuscation Signal Design

The obfuscator samples αm and θm from two uniformdistributions in the range of (0, 1) and (0, 2π), respec-tively. It continuously varies both parameters to createtime-varying channel parameters that are uncorrelatedwith the actual channel.

To cover the Doppler shift of the sensing signal, Au-dioSentry performs wideband obfuscation, which can beobtained by summing the expression of Eq. (8) over thedifferent tones fm covering the desired frequency range:{fl, · · · , fl + n∆f, · · · , fh}. Since the sensor has a fixedlocation, the obfuscator sets the values of fl and fh ac-cording to the maximum velocity of the target, whichdepends on the sensing application. ∆f – a function ofthe signal collection time τ – is chosen such that theobfuscation signal contains as much frequency compo-nents as possible. Given τ and the sampling rate Fs,the receiver collects Fs · τ samples in the time domain.The frequency resolution, ∆f , is then the width of eachFFT bin: FS

FS ·τ = 1τ .

Even though the obfuscation signals parameterschange continuously (which affects the obfuscationbeam pattern), the obfuscation signal still consists ofde-correlated beams in all the directions.

5.4 Enabling a Distributed Speaker Array

AudioSentry relies on commodity devices to form a dis-tributed obfuscator array. Its beamforming techniquehas two requirements: (i) the speakers in the array needto be synchronized to audio sample level and (ii) therelative positions of the speakers need to be known. For-tunately, both of these problems have been solved forcommodity devices [36, 48]; AudioSentry adopts both so-lutions to enable its operation as follows.


AudioSentry employs Dia [48] to achieve sample-levelsynchronization between the audio I/O clocks commod-ity devices. It first synchronizes the CPU clocks ofthe different devices by timestamping the WiFi bea-cons overheard from the access point. Then, it synchro-nizes the audio I/O clock of each device with its CPUclock. The average audio synchronization error of Diafor Nexus smartphones is between 1 to 3 audio samplesat a sampling rate equal to 44100 Hz. AudioSentry per-forms the synchronization over WiFi, which we assumeis secure. Aside from jamming, the adversary cannotmanipulate the synchronization process to hinder Au-

dioSentry’s protection.AudioSentry addresses the localization issue through

BeepBeep [36], which can achieve an accuracy of 0.8 cmin ranging commodity smartphones. AudioSentry breaksthe localization into a series of stages, by ranging eachpair of its devices at each stage – taking n(n−1)/2 stagesfor n devices in the speaker array. AudioSentry needs toperform the relative localization of its devices only onceunless some device moves (where it has to repeat forthe moving device relative to two other devices witha known location). At each stage (involving a pair ofdevices), a device (initiator) emits a specially-designedsound signal (beep), while simultaneously recording itsown and the peer’s (responder) beep. By measuring thetime between these two beeps, each device can computethe round trip time of the beep, consequently the dis-tance between the two devices. With more than threedevices in the array, AudioSentry can easily map the pair-wise distances to locations relative to the reference de-vice (every three devices form a triangle which can bedefined using the lengths of its sides). Apart from Beep-Beep, a recently proposed audio localization schemeSonoloc [18] can also be applied in AudioSentry. Sonoloclocalizes up to 100 commodity devices in a fully auto-matic fashion without user intervention, and only re-quires software changes to the commodity audio devices,similar to BeepBeep. Both these two solutions requiresminimum user effort.

The original BeepBeep and Sonoloc are potentiallyvulnerable to the adversary playing forged beeps at ahigh power. The forged beeps can lead the initiator tomiscalculate peer locations, significantly impairing thebeamforming performance. AudioSentry addresses thisproblem by having the initiator prepend a preamblewaveform to the beep. The initiator randomly decideson the preamble and communicates it to the responderover secure WiFi connection. The responder can verifywhether the beep originated from the initiator or not.Note that the adversary recording the preamble and

playing it back does not compromise the localizationprocedure, as the forged response only deceives the ini-tiator if it arrives earlier than the true response. Thelatter can happen only if the adversary is closer to theinitiator than the responder.

5.5 Privacy Properties

AudioSentry achieves its privacy protection through atwo-step process. First, it hides the reflections off-the-user from the sensing signal by covering them with thereflections from the obfuscation signal. Second, it usesbeamforming to prevent leaking the obfuscation signalto the adversary.

Single-path Obfuscation. Since the distributedobfuscator array covers the 2π range, one obfuscatorpath is guaranteed to hit the target user. With the usernot being a perfect mirror, the obfuscating signal hittingthe user deflects over a wide angle range (almost π as ourexperiments show). In practice, the obfuscators are dis-tributed in the environment around the user, and eachof their signals deflects around the π range, togethercovering the whole 2π range almost for certain. As a re-sult, the adversary receives on the path between it andthe target user: r(t) = (α · S(t) + β · oi(t)) ∗ hu,a(t). Thereceived signal contains two components, the reflectionsfrom the sensing signal and those from the obfuscationsignal. Since oi(t), by design, is independent from S(t),the adversary cannot extract hu,a(t) if it does not haveaccess to oi(t).

Obfuscation Signal Leak. The second task of Au-dioSentry is then to prevent the adversary from receivingoi(t). The adversary is only likely to succeed if its loca-tion allows it to be in the signal path of the obfuscationsignal hitting the user; the obfuscation signal is not azero-width beam. In the most extreme case, with a sin-gle obfuscator (Sec. 3.3), the two copies of obfuscationsignal received by the adversary are highly correlated.The adversary is successful at sensing the user with ac-curacy.

Nevertheless, AudioSentry utilizes the multi-directional beamforming technique to create uncor-related and non-overlapping beams towards differentdirections. As AudioSentry utilizes a distributed array,it can leverage more transmitters to create more non-overlapping and narrower obfuscation beams [32, 52].

Fig. 7 shows simulated beam patterns created bya linear obfuscator as well as a distributed array. Botharrays consist six obfuscators and generate five indepen-dent beams. The transmitters of the linear array are


-70

-50

-30

-10

10

30

0 0.5 1 1.5 2 2.5 3

Am

plit

ude (

dB

)

AoD (rad)

Overall Beam

(a)-25

-15

-5

5

15

0 0.5 1 1.5 2 2.5 3

Am

plit

ude (

dB

)

AoD (rad)(b)

Fig. 7. Obfuscator beam pattern created by (a) linear arraywith half-wavelength separation (b) distributed array. Eachcolored line represents a different beam.

each separated by half-wavelength. On the other hand,the obfuscators of the distributed array are randomly lo-cated within a 4 m by 8 m area. In the textbook case ofthe linear array [50, 51] (which we show for comparison),it is evident that each beam is made up of one singlewide main lobe and a series of very weak side lobes. Inthe distributed array (the real-world case of AudioSen-

try), the individual beams do not exhibit a well-definedshape. Instead, they contain a series of thin lobes withsimilar strengths pointing at different directions.

In the case of AudioSentry, which employs a dis-tributed obfuscator array, it is difficult for the adversaryto position itself in the same lobe as the target user toperform effective eavesdropping. The reason is that eachof the obfuscation beams is a series of very thin lobes, in-stead of a single wide main lobe in the case of the lineararray. We also confirm this observation in our evalua-tion findings, displayed in Fig. 9. There, we show thatthe adversary’s performance is consistently low when wemove its position around the target user. In addition,the adversary is unlikely to recover the original obfusca-tion signal even if it collects signals at various locations.Since the Ox signal directions are randomly changed, itis almost impossible for the adversary to keep movingto the correct angle to receive the correct Ox signal thatit needs to remove from the received signal.

6 Preserving Authorized SensingThe key to preserving authorized sensing is to cancel theobfuscation signal at the authorized sensor – essentiallyan interference cancellation problem. The interference(obfuscation signals) received by the authorized sensorcan be expressed as:

I(t) =∑i

oi(t) ∗ hOi,S (9)

where hOi,S represents the channel from Oi to the au-thorized sensor. The obfuscation signal transmitted byOi, denoted by oi(t), can be known via the secure WiFi

link between Oi and authorized sensor. The sensor hasto know each hOi,S to perform its sensing.

A common approach to estimate the audio chan-nel hOi,S is the Swept Sine technique [19], where theobfuscator’s speaker plays an exponential time-growingfrequency sweep signal as a channel sounding sequence.The sensor’s microphone deconvolves the received signalwith the original one (sounding sequence) to obtain thechannel impulse response.

Reverse Channel Estimation.Allowing each obfuscator to transmit a channel sound-ing sequence is not scalable. It introduces coordinationproblems between the obfuscators in the time and fre-quency domains, as each obfuscator has to perform thechannel sounding on a separate frequency or time slot.Instead, AudioSentry leverages the audio channel reci-procity [19] to estimate the channel between each obfus-cator and the authorized sensor. The authorized sensor’sspeaker plays a single channel sounding sequence to bereceived by all obfuscators’ microphones. The obfusca-tors need not play multiple channel sounding sequencessimultaneously. This operation is feasible as the obfus-cators, which are the mobile devices, TV, laptops orsmart speakers at home, are already equipped with mi-crophones, so our solution requires no additional hard-ware.

AudioSentry places the channel sounding sequenceat a different frequency range from the sensing signalso that the obfuscators can transmit oi(t) and receivethe swept sine signal at the same time. Once the sound-ing sequence is received, the obfuscator estimates thechannel and sends it to the authorized sensor via WiFi.Given the obfuscation signal described in Equation 3,AudioSentry uses a sweep sine signal in the frequencyrange of fl − 1000 to fl − 500 Hz. It is played every 100ms by the authorized sensor, which provides ten chan-nel measurements per second. The 500 Hz gap betweenthe channel sounding signal and the obfuscation signalserves to prevent interference.

Obfuscation Signals Synchronization at the Sensor.Although the obfuscators are synchronized for transmis-sion beamforming, different distances between the ob-fuscators and the authorized sensor result in differentarrival time of the signals. Hence, Eq. (9) should be re-written as:

I(t) =∑i

oi(t− τi) ∗ hOi,S (10)


where each oi(t) experiences a different delay τi. Au-

dioSentry compensate for this delay by adding a pream-ble, a sweep sine signal spanning 200 Hz, to each obfus-cator’s transmitted signal oi(t) for the authorized sensorto synchronize them by performing correlation. The au-thorized sensor distinguishes each preamble by placingthem on different frequencies. Since the obfuscators andthe authorized sensor are connected via WiFi, the autho-rized sensor acts as a central controller to allocate all theobfuscators preambles to non-overlapping frequencies.

As commodity devices have limited audio samplingrate (mostly up to 44.1 kHz), which supports at most22 kHz sound playback, this approach limits the max-imum number of co-existing obfuscators. Nonetheless,each obfuscator’s preamble only takes up 200 Hz band-width; a 2 kHz unused frequency range in the ultrasoundrange would make enough room for up to 10 obfuscators.Consider the ApneaApp [33] example where the systemtransmits a chirp between 18 kHz to 20 kHz as the sens-ing signal. The obfuscation signal and channel soundingpreamble occupy additional 300 Hz and 1000 Hz, respec-tively. Thus, there is still 2.7 kHz within the inaudiblerange (16 kHz to 22 kHz) – enough to fit around tenobfuscators.

Removing the Obfuscation Signal.Eventually, the sensor obtains the estimated channelhOi,S and the obfuscation signal oi(t) from each of theobfuscators. The authorized sensor then removes the es-timated interference factor, I(t) =

∑i oi(t)∗ hOi,S , from

its received signal. At this stage, the sensor can performits sensing logic over the cleaned signal (without inter-ference).

Commodity Speaker and Microphone FrequencyResponses.Although the audio channel between an obfuscator andthe authorized sensor is reciprocal, their speakers andmicrophones are very likely to exhibit different fre-quency responses. Let the channel sounding sequencebe x(t), obfuscator’s speaker and microphone frequencyresponses as DOTx and DORx, authorized sensor’s speakerand microphone frequency response as DSTx and DSRx.Then, the signal received in AudioSentry’s reverse chan-nel estimation is DSTx ∗ x(t) ∗ DORx. In comparison, thesignal received in the regular channel sounding processis DOTx ∗ x(t) ∗DSRx.

AudioSentry requires a one-time initial calibration tocompensate for the frequency responses of speakers and

microphones. In the calibration process, the user placesthe authorized sensor next to each obfuscator, where Au-

dioSentry has each device transmit a channel soundingsequence once at a time. By deconvolving the receivedsignal with the sounding sequence, AudioSentry obtainsDOTx∗DSRx when using the obfuscator as the transmitter,and DSTx∗DORx the other way around. The user does notneed to repeat this process as the device’s audio hard-ware do not experience significant changes over time.

When AudioSentry intends to estimate hOi,S , the sen-sor plays the channel sounding sequence x(t) and theobfuscator Oi receives yi(t). The sensor estimates thechannel hOi,S as:hOi,S = yi(t)∗−1x(t)∗−1(DSTx∗DORx)∗(DOTx∗DSRx) (11)

Computational Overhead.This interference cancellation scheme poses little over-head on the real-time performance of the authorizedsensing. The additional computations of the authorizedsensor involve convolving the estimated channel withthe obfuscation signal for each obfuscator, and then sub-tracting them from the received signal. Thus the com-plexity is O(Q log(Q)R), where Q is the length of obfus-cation signal, and R is the number of obfuscators. In atypical deployment, these values are 4410 and 6, respec-tively.

We evaluate AudioSentry’s computational overheadby measuring the total sensing time for an authorizedsensor with and without enabling AudioSentry. We use aGoogle Pixel smartphone running Android 7.1.1 as theauthorized sensor. With six obfuscators, the average ad-ditional delay introduced by AudioSentry is around 360ms for both gesture detection and breath rate estima-tion. This delay is unlikely to hinder user experience asit is insignificant compared to the time the user needs toperform the gesture (1 to 2 seconds) and the breathingmeasurement time (30 to 60 seconds).

7 Implementation and EvaluationWe now present a prototype of AudioSentry along withits evaluation.


Fig. 8. AudioSentry’s operation in sensing applications.

7.1 AudioSentry’s Operation

Fig. 8 shows the basic operations of our AudioSentry’sprototype during the sensing which follows this proce-dure.

Initiation process. The user controls AudioSentry

through an end device, such as a smartphone. First, theuser installs AudioSentry on the sensor device and desig-nates it as the authorized sensor device. The user selectsa group of commodity devices in the environment andconnects them to the same WiFi access point. Next, sheinstalls the AudioSentry app on those devices and desig-nates them as the obfuscators. At this point, the userneeds to perform the one-time bootstrapping process,described in Sec. 6, between each obfuscator device andthe authorized sensor. The information about the fre-quency response of each device is stored at the autho-rized sensor to be used for obfuscation signal cancella-tion.

Obtaining obfuscator positions. AudioSentry

asks the user to choose a fixed device as a referencepoint for the relative localization procedure. As de-scribed in Sec. 5.2, AudioSentry requires the relative po-sitions of obfuscators to perform directional beamform-ing. When the obfuscators are initially deployed, Au-

dioSentry utilizes the audio ranging approach describedin Sec. 5.4 to perform obfuscator localization. It commu-nicates the relative location information to the obfusca-tors via WiFi. It only repeats this process for a devicethat has been moved.

Application specific parameter setting. De-pending on the particular sensing applications, AudioSen-try sets the parameters for obfuscation signal, such as fl,fh and ∆f , which we describe in more detail below.

Start the sensing application. The user can startthe sensing application with her end device, which in-structs obfuscators to start the obfuscation simultane-ously. During the sensing, AudioSentry keeps performingthe channel estimation and obfuscation signal cancel-lation at the authorized sensor as long as the sensingapplication is active. During this process, the obfusca-tor distributed synchronization process (Sec. 5.4) runsonce every one second.

7.2 Experiment Setup

We use six commodity speakers as the obfuscator ar-ray; we distribute them randomly in a 4m × 8m livingroom, with random orientations to emulate the position-ing of commodity devices in a practical home environ-ment. For the adversary microphone array, we use thebottom microphones of multiple Pixel phones in a lineararray formation, with every two phones separated by 7cm. We use eight microphones to form the adversary’sarray unless otherwise specified. We synchronize the ad-versary microphones offline so that we can fully analyzethe adversary’s performance. Using the phones for theadversary array instead of a synchronized board waspurely a logistical decision; it does not affect the fidelityof the reported results. Our authorized sensing systemconsists of two co-located Pixel phones, one acting asthe transmitter and the other as the receiver.

We implement two audio sensing scenarios: gesturerecognition and breathing rate estimation. For the for-mer, we implement a recognition algorithm that identi-fies nine different hand-based gestures. This algorithm,akin to those proposed in the literature, extracts theDoppler shift pattern to recognize the gesture with amatching pattern. For breath rate estimation, we de-termine the breath rate as the highest peak in the fre-quency spectrum of chest motion within the normalbreath rate range (below 20 bpm). In our evaluation,a user (one of the authors) performs a set of gestureswhile both the sensing system and AudioSentry are run-ning.

The obfuscation signal design follows the descrip-tion of Section 5.3. We set the values of ∆f , fl, andfh so that the Doppler shift from the obfuscated sig-nal covers that from the sensing signal’s reflections. Inour scenarios, the user’s gesture and breathing induce arelative movement up to 2 m/s. A sensing signal at 20kHz creates a Doppler shift equal to 118 Hz. Therefore,in our evaluation, we set fl and fh to fc − 150 Hz andfc + 150 Hz, respectively (fc is the sensing frequency –depending on the application).

Following suit of earlier audio sensing applications,we set ∆f = 5 Hz to ensure that the obfuscated Dopplershift pattern cannot be distinguished from that of theauthorized sensor.

Although it’s more practical for the adversary toplace its microphones behind a wall, we only evaluateAudioSentry against the adversary in the same room withthe user for the similar reason we explained in Sec. 3.3.2:It’s much easier to recover the sensing signals with-out the wall’s blockage. If AudioSentry is able to defend


this ’stronger’ adversary, it could also defend against a’weaker’ adversary behind a wall.

7.3 Evaluation Results

Evaluation Metrics.We evaluate our system using three metrics: mutualinformation, gesture recognition accuracy, and breath-ing rate estimation error. The latter two metrics areapplication-dependent; gesture accuracy measures theportion of correctly identified gestures at the adversary.The estimation error metric measures the absolute dif-ference between the ground truth and the adversary’sestimate of the breathing rate.

We also report the mutual information betweenthe channel extracted by the adversary and the autho-rized sensor, each viewed as a random variable. Foreach measurement scenario, we frequently sample thesensor’s and adversary channel estimates to obtain anempirically-derived distribution of both. The mutual in-formation between two random variables characterizesthe amount of information one variable reveals aboutanother.

Definition 7.1. Mutual Information: Let the randomvariable representing the adversary’s estimated channelbe H, and the random variable representing the au-thorized sensor’s estimated channel be H. Then themutual information can be computed as I(H;H) =∑h

∑h p(h, h) log( p(h,h)

p(h)p(h)).

The mutual information metric quantifies how much ofthe sensing signal’s reflections is embedded in the adver-sary’s recovered channel. The higher the metric is, themore descriptive is the adversary’s estimated channel ofthe user’s sensing. This metric offers the advantage ofbeing application independent. It quantifies the informa-tion leakage in the adversary’s estimated channel, irrel-evant to what signal processing algorithm the adversaryemploys after obtaining the channel.

For each experiment, with a different location andnumber of adversary microphones, we record the chan-nel between the adversary and the user along with allthe beams the adversary’s array can discern. Then, weapply Eq. (5) to obtain the adversary’s channel using theknown sensing signal and each of the recorded beams. Inthe following, we report the results for the beam thatresults in a channel estimate that maximizes the mutualinformation with authorized sensor’s extracted channel.

0.5

1

1.5

2

2.5

3

3.5

-3 -2 -1 0 1 2 3

Mu

tua

l in

form

atio

n (

bit)

Attacker angle (rad)

Linear Ox arrayDistributed Ox array

wo Ox

Fig. 9. Adversary mutual information when eavesdroppingfrom different directions.

Adversary Position. We first investigate the im-pact of the adversary’s position. In this experiment, weuse six obfuscators and uniformly move the position ofthe adversary array around the target user, who ran-domly performs one of the nine gestures (similar to theone shown in Fig. 3). The target user is in the centerof the living room, and the separation between the userand the authorized sensor is 2.5 m. We use two typesof obfuscator array configuration, linear and distributed.The speakers have half-wavelength separation in the lin-ear array, and the array is 1.5 m away from the user.Also, the array aligns with the user and adversary whenthe adversary is at the angle of 1.3 rad in Fig. 9. Thedistributed array’s speaker positions are randomly cho-sen inside the room. Fig. 9 shows the measured mutualinformation when the adversary array is placed at dif-ferent angles relative to the user. The authorized sensoris at the angle of 0 rad.

It is evident that both obfuscator arrays provide sig-nificant privacy protection, with the mutual informationdropping in half when employing the obfuscators (from3 bits to 1.5 on average). With around 3 bits of mu-tual information, without an obfuscator, the adversaryachieves accurate detection results for both the gestureand breath rate. On the other hand, with around 1.5 bitsmutual information, the adversary’s estimated channelis completely corrupted (see examples in Fig. 10 (a) andFig. 11), which causes the detection to fail.

Interestingly, we find that the linear obfuscator ar-ray fails to provide decent protection when the adver-sary, the user, and the obfuscator array are aligned. Inthis case (at 1.3 rad), the same obfuscation beam hitsboth the adversary and the user, because of the lineararray’s wide main lobe. This allows the adversary to re-move the obfuscation signal from the user’s reflection,thus revealing more information about the channel. Onthe other hand, the distributed array does not have thisdrawback, where we find that the mutual informationsuffers some fluctuation, but is consistently low regard-less of the adversary’s position. This result confirms thearguments explained in Sec. 5.5.


-100

-50

0

50

100

0 0.25 0.5 0.75 1 1.25

Fre

qu

en

cy (

Hz)

Time (s)

0

30

60

90

120

150

Am

plit

ud

e

(a) (b)

Fig. 10. Doppler shift pattern extracted by (a) adversary (b)authorized sensor. The adversary is located at angle 0 rad.

0 10 20 30 40 50 60

Chest m

ovem

ent

Time (s)(a) 0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5 3 3.5 4

Am

plit

ude (

dB

)

Frequency (Hz)(b)

Fig. 11. Adversary extracted chest motion signal in (a) timedomain (b) frequency domain. The adversary is located atangle 0 rad.

0 10 20 30 40 50 60

Chest m

ovem

ent

Time (s)(a) 0

10

20

30

40

50

60

0 0.5 1 1.5 2 2.5 3 3.5 4

Am

plit

ude (

dB

)

Frequency (Hz)

Detected breath rate: 13.0 bpm

(b)

Fig. 12. Authorized sensor extracted chest motion signal in(a) time domain (b) frequency domain.

Next, we examine the application level performanceof the adversary with distributed obfuscator array.Fig. 10 presents examples of the Doppler shift pat-terns extracted by the adversary and authorized sen-sor. Across all adversary positions, the average gesturerecognition accuracy of the adversary and the autho-rized sensor are 12% and 94%, respectively. Note thatfor nine total gestures, an accuracy of 12% is equivalentto a random guess.

Fig. 11 shows an example of the chest motion signalextracted by the adversary, which contains no observ-able peak corresponding to the correct breathing rate.On the other hand, the chest motion signal extracted bythe authorized sensor, plotted in Fig. 12, clearly showsa peak at 0.21 Hz (13.0 bpm), while the true breathingrate is 14 bpm. For all adversary positions, the averageestimation error for the adversary and authorized sensorare 8.8 and 0.6 bpm, respectively.

Adversary array size. Then, we investigate theimpact of the adversary’s array size. We change the num-ber of microphones on the array from 6 to 16 and mea-sure the adversary’s mutual information as well as theapplication level performance. For each array size, wealso change the number of obfuscators. The authorizedsensor is placed at the angle of -1.5 rad, 2.5 m from theuser. The adversary array is placed at the angle whichmaximizes the mutual information, 1.5m from the user.

0

0.5

1

1.5

2

2.5

3

3.5

6 8 10 12 14 16Mu

tua

l in

form

atio

n (

bit)

Adversary array size

4 Ox 6 Ox

Fig. 13. Adversary mutualinformation with different ar-ray size. Error bars indicatestandard deviation.

0

0.5

1

1.5

2

2.5

3

3.5

4 5 6Mu

tua

l in

form

atio

n (

bit)

Number of Ox

Fig. 14. Adversary mutualinformation with differentnumber of Ox. Error barsindicate standard deviation.

0

0.1

0.2

0.3

0.4

0.5

6 8 10 12 14 16

Recognitio

n a

ccura

cy


4 Ox 6 Ox

(a) 0

2

4

6

8

10

12

14

6 8 10 12 14 16Estim

ation e

rror

(bpm

)


4 Ox 6 Ox

(b)Fig. 15. Adversary’s performance in (a) gesture recognition(b) breathing rate estimation with different array size

The user performs a gesture as the mutual informationis being measured.

Fig. 13 shows that the mutual information slowlyincreases with the adversary’s array size. However, thistrend is far from being linear; increasing the array sizeprovides limited benefits to the adversary. Althoughhaving a larger array offers more fine-grained receivingdirectionality, AudioSentry’s multi-directional beamform-ing technique ensures that the adversary is unlikely toobtain the true obfuscation signal in the user body’sreflection. As a result, the channel estimated by the ad-versary does not improve significantly with the arraysize. The same observation holds true for both 4 and6 obfuscators; using more obfuscators provides slightlybetter privacy protection.

Fig. 15 shows the gesture recognition accuracy andbreath rate estimation error. Not surprisingly, the de-tection performance improves slightly as the adversaryarray expands, albeit with a marginal benefit. Even with16 obfuscators, the gesture recognition accuracy is 21%and 16% for 4 and 6 obfuscators, and the breath rateestimation error is higher than 7 bpm.

These results highlight an important aspect if Au-

dioSentry. It provides privacy protection with a limitednumber of obfuscator against a large adversary array.

Number of obfuscators. We assess the effect ofthe number of obfuscators in the environment on theperformance of the obfuscation and authorized sensing.We only change the number of obfuscators, while keep-ing the experiment setup as before (adversary with eightmicrophones, 6.5 m from the adversary and at a locationthat maximizes mutual information). Fig. 14 presents


0

0.2

0.4

0.6

0.8

1

4 5 6

Recognitio

n a

ccura

cy

Number of Ox

Adversary Authorized

(a) 0

2

4

6

8

10

12

14

4 5 6Estim

ation e

rror

(bpm

)

Number of Ox


(b)Fig. 16. Adversary and authorized sensor’s performance in (a)gesture recognition (b) breathing rate estimation with differentnumber of Ox.

the adversary mutual information, where we observea decrease as more obfuscators are used. Fig. 16 alsoshows similar observations: where the adversary’s per-formance in these two applications is worse on averagewhen we use more obfuscators. On the other hand, al-though increasing the number of obfuscators introducesmore interference to the authorized sensor, our interfer-ence cancellation technique proves to be resilient to thenumber of obfuscators. As a result, we see in Fig. 16that the authorized sensing performance is consistentlyhigh. With six obfuscators, the gesture detection accu-racy is around 95%, and the breath rate estimation erroris within one bpm.

Synchronization error. We further investigatehow the synchronization error among distributed ob-fuscators could harm the obfuscation and authorizedsensing performance. We still use six obfuscators andintroduce additional random delays on each speaker toemulate possible synchronization errors that could hap-pen in a practical commodity device array. For example,we create an average synchronization error of 3 samplesby varying the delay randomly between 0 and 6 samples.For each obfuscator, we vary the delay randomly every100 ms.

Fig. 17 shows the adversary’s mutual informationunder different levels of synchronization error. The mu-tual information increases with the error because thebeamforming is affected by synchronization accuracy.Under perfect synchronization, each beam covers a dis-tinct and non-overlapping range of the AoD. Withhigher error, the direction (AoD) of each beam may de-viate from the intended direction, which causes differentbeams to overlap largely. As a result, The same beamswill cover a broader range of AoD, which causes theoverall obfuscation signal to be correlated along a widerrange of angles. The adversary will enjoy an increasedchance to cancel the obfuscation signal. Fortunately, aswe explain in Section 5.4, the measured average audiosynchronization error is fewer than two samples, whichis more than sufficient to achieve sufficient obfuscationfor AudioSentry.

0

0.5

1

1.5

2

2.5

3

3.5

0 2 4 6 8 10 12 14Mutu

al in

form

ation (

bit)

Average sync error (sample)

Fig. 17. Adversary mutual information under different syn-chronization error. Error bars indicate standard deviation.

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10 12 14

Recognitio

n a

ccura

cy



(a) 0

2

4

6

8

10

12

14

0 2 4 6 8 10 12 14Estim

ation e

rror

(bpm

)



(b)Fig. 18. Adversary and authorized sensor’s performance in(a) gesture recognition (b) breathing rate estimation underdifferent synchronization error.

In Fig. 18, we show the gesture recognition andbreathing rate estimation accuracy as a function of thesynchronization error. We find that although the obfus-cation effectiveness degrades as the synchronization er-ror increases in general, the impact if not significantwhen the average synchronization error is no greaterthan 4 audio samples. Under 4 samples of average er-ror, adversary’s average accuracy in gesture recognitionis around 13%, and the average error in breathing rateestimation is around 7.2 bpm. As the synchronizationerror increases to 14 samples, these two metrics become32% and 3.6 bpm, which provides a decent estimationof the user’s activity and physical status.

We notice that the synchronization error hardly hasany impact on the authorized sensing accuracy. This re-sult highlights the efficacy of our synchronization ap-proach introduced in Sec. 6 as part of AudioSentry’s in-terference cancellation. The synchronization preambleof each obfuscator allows the authorized sensor to es-timate the relative timing of each copy of obfuscationsignal and perform interference cancellation.

8 DiscussionPrivacy-aware Sensing. AudioSentry’s obfuscationmechanism can also be built into a privacy-aware sens-ing system, instead of utilizing commodity devices asobfuscators. For example, the sensing system could addadditional speakers as built-in obfuscators, akin to Au-

dioSentry’s distributed obfuscators. In such a case, someof AudioSentry’s operation could be simpler; there is needfor extra procedures for synchronization or localization


as the speakers will be co-located on the same board.Also, there will be no need to trust the commodity de-vices in the same environment, where an adversary cancompromise multi-microphone devices such as AmazonEcho. The built-in obfuscator, however, presents someissues. Since it still needs to perform reverse channel esti-mation (to cancel the effect of obfuscation), each built-inobfuscator needs a co-located microphone, which signifi-cantly increases the hardware and space requirement. Aprivacy-aware sensor with six obfuscators, for example,requires seven microphone-speaker pairs. In contrast,AudioSentry achieves its privacy protection without re-quiring any dedicated hardware.

Active attacker. We design AudioSentry to thwarta strong passive attacker with multiple microphones andfull knowledge of the environment. AudioSentry is equallyeffective against an active attacker operating within thesame frequency range as AudioSentry’s devices. Still, theactive attacker could employ costlier ultrasound speak-ers and microphones to overcome AudioSentry’s protec-tion. Moreover, an active attacker could also affect thechannel estimation procedure by jamming the obfusca-tors to prevent the authorized sensor from operatingcorrectly. This attack, however, is not specific to Au-

dioSentry; a regular active adversary can jam the sensorand prevent it from sensing the environment. AudioSen-try could detect such an attacker, by having the sensorcommunicate the training sequence over the OOB chan-nel to detect if there has been any interference in thechannel.

Channel hopping. Channel hopping has beenused in some wireless communication systems for se-curity purposes. However, applying this technique inaudio sensing is impractical for two reasons. First, anadversary can listen to the entire spectrum and moni-tor the sensing frequency as it hops – a problem that islikely to for electromagnetic-based sensing as well. Sec-ond, there is not sufficient audio bandwidth on mostcommodity devices. Such devices’ operating frequencyis below 24 kHz and audio sensing applications oper-ate in between 18 kHz and 22 kHz (inaudible range).Most audio sensing applications require high bandwidthfor higher accuracy. For example, ApneaApp requiresat least 2 kHz bandwidth to accurately estimate chestmovement, which makes frequency hopping difficult inpractice.

Ultrasound Frequencies. One potential concernis the effect of AudioSentry’s obfuscation signals on petsand wildlife, some of which are capable of hearing ul-trasound signals. However, AudioSentry only operates inthe same frequency range as the audio sensing applica-

tion it protects; it does not create ultrasound “noise" inadditional frequency ranges.

Eavesdropping private speech. An adversarythat can sniff audio sensing signal is also likely to be ableto eavesdrop private user conversations, a severe privacythreat. However, addressing both problems (eavesdrop-ping on sensing vs. conversations) requires different ap-proaches. The information in human speech is the audiowaveform itself, while for sensing, the information is em-bedded in the audio channel, and the raw audio signal isnot directly relevant. There are ongoing research effortsin protecting the privacy of human speech [40, 41], butit’s out of the scope of AudioSentry.

9 Related WorkAudio sensing applications. The low propagationspeed of audio signal has enabled a wide range of au-dio sensing applications, many on commodity devices.One of the most popular applications is to detect ges-ture and body movements [10, 12, 15, 21, 22, 24, 29,34, 42, 54, 56]. SoundWave detects hand gestures byplaying an inaudible tone, measuring, and analyzingthe Doppler shift pattern [24]. FingerIO proposes anOFDM-based audio ranging design to track finger posi-tion with millimeter-level accuracy [34]. A more recentwork, LLAP, can detect more fine-grained hand move-ments using a continuous wave radar, where it extractsthe phase change of the audio signal to track finger po-sition [54]. Shake and Walk demonstrates the feasibil-ity of localizing and determining the direction of devicemovement using audio signals [26]. The similar principlealso enables tracking and localization, such as trackingsmartphones to emulate a mouse or for interactive ap-plications [8, 12, 36, 46, 47, 59]. The ability to detectand distinguish finger strokes at different positions alsoenables an extended human-computer interaction inter-face [14, 25].

Audio sensing has been utilized in health monitor-ing, especially for breathing rate sensing, thanks to itsability to detect centimeter-level movements. For exam-ple, ApneaAPP is a breath monitoring application run-ning on a smartphone [33]. As an FMCW radar, it playsa sweep frequency chirp in the inaudible human range(18kHz → 20kHz), and perform FFT on the reflectedsignal to detect chest motion. This information can beused to detect diseases such as sleep apnea remotelyin a non-intrusive way. Similarly, SonarBeat [55] usesa smartphone as a continuous wave radar (18kHz to


22kHz) to measure the phase of the reflected audio sig-nal to estimate chest motion, thus being able to mea-sure breath rate. All of those techniques have not beendesigned with privacy in mind. They allow a passive ad-versary to infer a set of privacy-infringing informationabout the users, including their movements, behavior,interactions, and health attributes. AudioSentry targetsthese privacy threats.

Sensing and communication obfuscation. Phy-Cloak addresses the sensing privacy problem in the RFdomain through an obfuscation technique [38]. It uses asingle specialized full-duplex radio as both the obfusca-tor and authorized sensor, where the device forwards thesignal for sensing and distorts the channel parametersto mislead the adversary. The single-obfuscator designmakes PhyCloak ineffective against a multi-antenna at-tacker with directional beamforming capabilities. Also,extending PhyCloak’s design to multiple obfuscators ishardly feasible, as it’s impractical for regular users todeploy several dedicated full-duplex radios (commodityRF devices are not full-duplex).

Other works employ friendly jamming to protectthe content of the communication between wireless de-vices. Implantable medical device (IMD) Shield pro-tects the communication between an IMD and its au-thorized reader [23]. It jams the RF signal transmit-ted by the IMD to prevent eavesdropping and unau-thorized connections using a full-duplex radio. Simi-larly, BLE-Guardian is an active obfuscator that jamsthe advertisement messages from Bluetooth Low En-ergy (BLE) devices to prevent privacy leakage [20]. Dueto their single jammer nature, both and similar tech-niques [9, 11, 27, 31, 39, 44, 45, 58, 61] are ineffec-tive against a multi-antenna receiver, which could sepa-rate the obfuscation signal and the communication sig-nals [49].

Sound injection to prevent audio recording.Recently, researchers have established that it is possibleto play specially crafted sound in the ultrasonic rangeto inject an audio signal in lower frequencies into a mi-crophone [40, 62]. This technique can thwart an eaves-dropper attempting to record conversations. It, however,faces the same drawbacks of the other approaches whenrelying on a single obfuscator. Extending to a multiple-obfuscator system has the same challenges which we ad-dress in this work.

10 ConclusionIn this paper, we show that a multi-microphone adver-sary can overcome a single obfuscator from a distanceand with a wall separates them. Taking advantage ofcommodity devices in the environment, AudioSentry per-forms multi-beam beamforming to de-correlate the ob-fuscation signals towards different directions. While ef-fective with all types of audio sensing applications inprinciple, we extensively evaluate AudioSentry for thetwo most popular applications: gesture and breath ratesensing. Our evaluations show that AudioSentry can de-fend against an adversary with a large microphone arraywith only a small number of obfuscators, while still be-ing able to preserve authorized sensing.

The design principle of AudioSentry can be extendedto other domains as well, such as RF sensing andspeech recording, to protect user’s privacy against multi-receiver adversaries. For WiFi sensing, one problemworth investigating in future work is how to utilizecommodity devices without full duplex capabilities toachieve obfuscation as well as preserving authorizedsensing. For speech recording, we can adapt AudioSen-

try to leverage a mechanism such as backdoor [40].

ACKNOWLEDGEMENTSWe appreciate the anonymous reviewers for their in-sightful comments. This research was supported in partby Wisconsin Alumni Research Foundation, and theUS National Science Foundation through CNS-1719336,CNS-1647152, CNS-1629833, CNS-1343363, and CNS-1838733.

References[1] [n. d.]. Amazon Alexa Premium Far-Field Voice Development

Kit. https://developer.amazon.com/alexa-voice-service/dev-kits/amazon-premium-voice. ([n. d.]).

[2] [n. d.]. Apple Ipad Tech Specs. https://www.apple.com/ipad-9.7/specs/. ([n. d.]).

[3] [n. d.]. HomePod reinvents music in the home. https://www.apple.com/newsroom/2017/06/homepod-reinvents-music-in-the-home/. ([n. d.]).

[4] [n. d.]. Pixel 2 Tech Specs. https://store.google.com/us/product/pixel_2_specs?hl=en-US. ([n. d.]).

[5] 2017. The Nielsen Total Audience Report: Q2 2017.(November 2017). Retrieved May 8, 2018 from http://www.nielsen.com/us/en/insights/reports/2017/the-

https://developer.amazon.com/alexa-voice-service/dev-kits/amazon-premium-voice

https://developer.amazon.com/alexa-voice-service/dev-kits/amazon-premium-voice

https://www.apple.com/ipad-9.7/specs/

https://www.apple.com/ipad-9.7/specs/

https://www.apple.com/newsroom/2017/06/homepod-reinvents-music-in-the-home/



https://store.google.com/us/product/pixel_2_specs?hl=en-US

https://store.google.com/us/product/pixel_2_specs?hl=en-US

http://www.nielsen.com/us/en/insights/reports/2017/the-nielsen-total-audience-q2-2017.html



nielsen-total-audience-q2-2017.html[6] 2018. Elliptic Labs Makes Smart Speakers More Intelli-

gent with New INNER REFLECTION™ Ultrasound Vir-tual Sensor. (February 2018). Retrieved May 8, 2018from http://www.ellipticlabs.com/2018/02/26/elliptic-labs-touch-free-ultrasound-gesture-technology-leveraging\-the-qualcomm-snapdragon-neural-processing-engine-2/

[7] 2018. The Smart Audio Report, Fall/Winter 2017 from NPRand Edison Research. (January 2018). Retrieved May 8, 2018from https://www.nationalpublicmedia.com/smart-audio-report/

[8] Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, EricLarson, and Shwetak Patel. 2013. DopLink: using the dopplereffect for multi-device interaction. In Proceedings of the2013 ACM international joint conference on Pervasive andubiquitous computing. ACM, 583–586.

[9] Daniel S. Berger, Francesco Gringoli, Nicolò Facchi, IvanMartinovic, and Jens Schmitt. 2014. Gaining Insight onFriendly Jamming in a Real-world IEEE 802.11 Network. InProceedings of the 2014 ACM Conference on Security andPrivacy in Wireless & Mobile Networks (WiSec ’14).105–116. https://doi.org/10.1145/2627393.2627403

[10] Andreas Braun, Stefan Krepp, and Arjan Kuijper. 2015.Acoustic tracking of hand activities on surfaces. In Proceed-ings of the 2nd international Workshop on Sensor-basedActivity Recognition and Interaction. ACM, 9.

[11] James Brown, Ibrahim Ethem Bagci, Alex King, and UtzRoedig. 2013. Defend Your Home!: Jamming UnsolicitedMessages in the Smart Home. In Proceedings of the 2NdACM Workshop on Hot Topics on Wireless Network Securityand Privacy (HotWiSec ’13). 1–6. https://doi.org/10.1145/2463183.2463185

[12] Ke-Yu Chen, Daniel Ashbrook, Mayank Goel, Sung-HyuckLee, and Shwetak Patel. 2014. AirLink: sharing files betweenmultiple devices using in-air gestures. In Proceedings of the2014 ACM International Joint Conference on Pervasive andUbiquitous Computing. ACM, 565–569.

[13] Qi Alfred Chen, Zhiyun Qian, and Zhuoqing Morley Mao.2014. Peeking into Your App without Actually Seeing It:UI State Inference and Novel Android Attacks.. In USENIXSecurity Symposium. 1037–1052.

[14] Seungeun Chung and Injong Rhee. 2016. vTrack: virtualtrackpad interface using mm-level sound source localizationfor mobile interaction. In Proceedings of the 2016 ACMInternational Joint Conference on Pervasive and UbiquitousComputing: Adjunct. ACM, 41–44.

[15] Amit Das, Ivan Tashev, and Shoaib Mohammed. 2017.Ultrasound based gesture recognition. In Acoustics, Speechand Signal Processing (ICASSP), 2017 IEEE InternationalConference on. IEEE, 406–410.

[16] Lucas Davi, Alexandra Dmitrienko, Ahmad-Reza Sadeghi,and Marcel Winandy. 2010. Privilege escalation attacks onandroid. In international conference on Information security.Springer, 346–360.

[17] William Enck, Peter Gilbert, Seungyeop Han, Vasant Ten-dulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung,Patrick McDaniel, and Anmol N Sheth. 2014. TaintDroid:an information-flow tracking system for realtime privacymonitoring on smartphones. ACM Transactions on ComputerSystems (TOCS) 32, 2 (2014), 5.

[18] Viktor Erdélyi, Trung-Kien Le, Bobby Bhattacharjee, PeterDruschel, and Nobutaka Ono. 2018. Sonoloc: Scalablepositioning of commodity mobile devices. (2018).

[19] Angelo Farina. 2007. Advancements in impulse responsemeasurements by sine sweeps. In Audio Engineering SocietyConvention 122. Audio Engineering Society.

[20] Kassem Fawaz, Kyu-Han Kim, and Kang G Shin. 2016. Pro-tecting Privacy of BLE Device Users.. In USENIX SecuritySymposium. 1205–1221.

[21] Biying Fu, Dinesh Vaithyalingam Gangatharan, Arjan Kui-jper, Florian Kirchbuchner, and Andreas Braun. 2017. Ex-ercise Monitoring On Consumer Smart Phones Using Ul-trasonic Sensing. In Proceedings of the 4th internationalWorkshop on Sensor-based Activity Recognition and Interac-tion. ACM, 9.

[22] Biying Fu, Jakob Karolus, Tobias Grosse-Puppendahl,Jonathan Hermann, and Arjan Kuijper. 2015. Opportunitiesfor activity recognition using ultrasound doppler sensing onunmodified mobile phones. In Proceedings of the 2nd inter-national Workshop on Sensor-based Activity Recognition andInteraction. ACM, 8.

[23] Shyamnath Gollakota, Haitham Hassanieh, Benjamin Rans-ford, Dina Katabi, and Kevin Fu. 2011. They can hear yourheartbeats: non-invasive security for implantable medical de-vices. In ACM SIGCOMM Computer Communication Review,Vol. 41. ACM, 2–13.

[24] Sidhant Gupta, Daniel Morris, Shwetak Patel, and DesneyTan. 2012. Soundwave: using the doppler effect to sense ges-tures. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems. ACM, 1911–1914.

[25] Chris Harrison, Julia Schwarz, and Scott E Hudson. 2011.TapSense: enhancing finger interaction on touch surfaces. InProceedings of the 24th annual ACM symposium on Userinterface software and technology. ACM, 627–636.

[26] Wenchao Huang, Yan Xiong, Xiang-Yang Li, Hao Lin, XufeiMao, Panlong Yang, and Yunhao Liu. 2014. Shake and walk:Acoustic direction finding and fine-grained indoor localiza-tion using smartphones. In INFOCOM, 2014 ProceedingsIEEE. IEEE, 370–378.

[27] Yu Seung Kim and Patrick Tague. 2014. Proximity-basedWireless Access Control Through Considerate Jamming. InProceedings of the ACM MobiCom Workshop on Securityand Privacy in Mobile Environments (SPME ’14). 19–24.https://doi.org/10.1145/2646584.2646588

[28] Tao Li, Yimin Chen, Jingchao Sun, Xiaocong Jin, and Yan-chao Zhang. 2016. iLock: Immediate and Automatic Lockingof Mobile Devices Against Data Theft. In Proceedings of the2016 ACM SIGSAC Conference on Computer and Commu-nications Security (CCS ’16). ACM, New York, NY, USA,933–944. https://doi.org/10.1145/2976749.2978294

[29] Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang,and Marco Gruteser. 2015. Snooping keystrokes with mm-level audio ranging on a single phone. In Proceedings of the21st Annual International Conference on Mobile Computingand Networking. ACM, 142–154.

[30] Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the22nd Annual International Conference on Mobile Computingand Networking. ACM, 69–81.


http://www.ellipticlabs.com/2018/02/26/elliptic-labs-touch-free-ultrasound-gesture-technology-leveraging\-the-qualcomm-snapdragon-neural-processing-engine-2/



https://www.nationalpublicmedia.com/smart-audio-report/

https://www.nationalpublicmedia.com/smart-audio-report/

https://doi.org/10.1145/2627393.2627403

https://doi.org/10.1145/2463183.2463185

https://doi.org/10.1145/2463183.2463185

https://doi.org/10.1145/2646584.2646588

https://doi.org/10.1145/2976749.2978294


[31] Ivan Martinovic, Paul Pichota, and Jens B. Schmitt. 2009.Jamming for Good: A Fresh Approach to Authentic Com-munication in WSNs. In Proceedings of the Second ACMConference on Wireless Network Security (WiSec ’09). 161–168. https://doi.org/10.1145/1514274.1514298

[32] Raghuraman Mudumbai, D Richard Brown Iii, UpamanyuMadhow, and H Vincent Poor. 2009. Distributed trans-mit beamforming: challenges and recent progress. IEEECommunications Magazine 47, 2 (2009), 102–110.

[33] Rajalakshmi Nandakumar, Shyamnath Gollakota, andNathaniel Watson. 2015. Contactless sleep apnea detec-tion on smartphones. In Proceedings of the 13th AnnualInternational Conference on Mobile Systems, Applications,and Services. ACM, 45–57.

[34] Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, andShyamnath Gollakota. 2016. Fingerio: Using active sonar forfine-grained finger tracking. In Proceedings of the 2016 CHIConference on Human Factors in Computing Systems. ACM,1515–1525.

[35] Rajalakshmi Nandakumar, Alex Takakuwa, Tadayoshi Kohno,and Shyamnath Gollakota. 2017. CovertBand: ActivityInformation Leakage Using Music. Proc. ACM Interact. Mob.Wearable Ubiquitous Technol. 1, 3, Article 87 (Sept. 2017),24 pages. https://doi.org/10.1145/3131897

[36] Chunyi Peng, Guobin Shen, Yongguang Zhang, Yanlin Li,and Kun Tan. 2007. Beepbeep: a high accuracy acousticranging system using cots mobile devices. In Proceedings ofthe 5th international conference on Embedded networkedsensor systems. ACM, 1–14.

[37] Kun Qian, Chenshu Wu, Fu Xiao, Yue Zheng, Yi Zhang,Zheng Yang, and Yunhao Liu. 2018. Acousticcardiogram:Monitoring Heartbeats using Acoustic Signals on Smart De-vices. In Proceedings of the IEEE International Conferenceon Computer Communications (Infocom ’18).

[38] Yue Qiao, Ouyang Zhang, Wenjie Zhou, Kannan Srinivasan,and Anish Arora. 2016. PhyCloak: Obfuscating Sensingfrom Communication Signals.. In USENIX Annual TechnicalConference.

[39] Hanif Rahbari and Marwan Krunz. 2014. Friendly CryptoJam:A Mechanism for Securing Physical-layer Attributes. InProceedings of the 2014 ACM Conference on Security andPrivacy in Wireless & Mobile Networks (WiSec ’14).129–140. https://doi.org/10.1145/2627393.2627415

[40] Nirupam Roy, Haitham Hassanieh, and Romit Roy Choud-hury. 2017. Backdoor: Making microphones hear inaudiblesounds. In Proceedings of the 15th Annual InternationalConference on Mobile Systems, Applications, and Services.ACM, 2–14.

[41] Nirupam Roy, Sheng Shen, Haitham Hassanieh, andRomit Roy Choudhury. 2018. Inaudible Voice Commands:The Long-Range Attack and Defense. In 15th {USENIX}Symposium on Networked Systems Design and Implementa-tion ({NSDI} 18). 547–560.

[42] Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, PeipeiXu, and Longfei Shangguan. 2016. AudioGest: enablingfine-grained hand gesture detection by decoding echo sig-nal. In Proceedings of the 2016 ACM International JointConference on Pervasive and Ubiquitous Computing. ACM,474–485.

[43] Roman Schlegel, Kehuan Zhang, Xiao-yong Zhou, MehoolIntwala, Apu Kapadia, and XiaoFeng Wang. 2011. Sound-comber: A Stealthy and Context-Aware Sound Trojan forSmartphones.. In NDSS, Vol. 11. 17–33.

[44] Matthias Schulz, Francesco Gringoli, Daniel Steinmetzer,Michael Koch, and Matthias Hollick. 2017. Massive ReactiveSmartphone-based Jamming Using Arbitrary Waveforms andAdaptive Power Control. In Proceedings of the 10th ACMConference on Security and Privacy in Wireless and MobileNetworks (WiSec ’17). 111–121. https://doi.org/10.1145/3098243.3098253

[45] W. Shen, P. Ning, X. He, and H. Dai. 2013. Ally FriendlyJamming: How to Jam Your Enemy and Maintain YourOwn Wireless Connectivity at the Same Time. In 2013IEEE Symposium on Security and Privacy. 174–188. https://doi.org/10.1109/SP.2013.22

[46] Adam Smith, Hari Balakrishnan, Michel Goraczko, and Nis-sanka Priyantha. 2004. Tracking moving devices with thecricket location system. In Proceedings of the 2nd inter-national conference on Mobile systems, applications, andservices. ACM, 190–202.

[47] Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. 2013.Spartacus: spatially-aware interaction for mobile devicesthrough energy-efficient audio sensing. In Proceeding of the11th annual international conference on Mobile systems,applications, and services. ACM, 263–276.

[48] Sanjib Sur, Teng Wei, and Xinyu Zhang. 2014. Autodirectiveaudio capturing through a synchronized smartphone array. InProceedings of the 12th annual international conference onMobile systems, applications, and services. ACM, 28–41.

[49] Nils Ole Tippenhauer, Luka Malisa, Aanjhan Ranganathan,and Srdjan Capkun. 2013. On limitations of friendly jammingfor confidentiality. In Security and Privacy (SP), 2013 IEEESymposium on. IEEE, 160–173.

[50] Barry D Van Veen and Kevin M Buckley. 1988. Beamforming:A versatile approach to spatial filtering. IEEE assp magazine5, 2 (1988), 4–24.

[51] Junyi Wang, Zhou Lan, Chin-Sean Sum, Chang-Woo Pyo,Jing Gao, Tuncer Baykas, Azizur Rahman, Ryuhei Funada,Fumihide Kojima, Ismail Lakkis, et al. 2009. Beamformingcodebook design and performance evaluation for 60GHzwideband WPANs. In Vehicular Technology Conference Fall(VTC 2009-Fall), 2009 IEEE 70th. IEEE, 1–6.

[52] Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. InACM SIGCOMM Computer Communication Review, Vol. 44.ACM, 235–246.

[53] Tianben Wang, Daqing Zhang, Yuanqing Zheng, Tao Gu,Xingshe Zhou, and Bernadette Dorizzi. 2018. C-FMCWBased Contactless Respiration Detection Using AcousticSignal. Proc. ACM Interact. Mob. Wearable UbiquitousTechnol. 1, 4, Article 170 (Jan. 2018), 20 pages. https://doi.org/10.1145/3161188

[54] Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-freegesture tracking using acoustic signals. In Proceedings of the22nd Annual International Conference on Mobile Computingand Networking. ACM, 82–94.

[55] Xuyu Wang, Runze Huang, and Shiwen Mao. 2017. Sonar-Beat: Sonar phase for breathing beat monitoring with smart-phones. In Computer Communication and Networks (IC-

https://doi.org/10.1145/1514274.1514298

https://doi.org/10.1145/3131897

https://doi.org/10.1145/2627393.2627415

https://doi.org/10.1145/3098243.3098253

https://doi.org/10.1145/3098243.3098253

https://doi.org/10.1109/SP.2013.22

https://doi.org/10.1109/SP.2013.22

https://doi.org/10.1145/3161188

https://doi.org/10.1145/3161188


CCN), 2017 26th International Conference on. IEEE, 1–8.[56] Hiroki Watanabe, Tsutomu Terada, and Masahiko

Tsukamoto. 2013. Ultrasound-based movement sensing,gesture-, and context-recognition. In Proceedings of the2013 International Symposium on Wearable Computers.ACM, 57–64.

[57] Ryan West. 2008. The psychology of security. Commun.ACM 51, 4 (2008), 34–40.

[58] Matthias Wilhelm, Ivan Martinovic, Jens B. Schmitt, andVincent Lenders. 2011. WiFire: A Firewall for WirelessNetworks. In Proceedings of the ACM SIGCOMM 2011Conference (SIGCOMM ’11). 456–457. https://doi.org/10.1145/2018436.2018518

[59] Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning amobile device into a mouse in the air. In Proceedings of the13th Annual International Conference on Mobile Systems,Applications, and Services. ACM, 15–29.

[60] Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, andWenguang Mao. 2017. Strata: Fine-Grained Acoustic-basedDevice-Free Tracking. In Proceedings of the 15th AnnualInternational Conference on Mobile Systems, Applications,and Services (MobiSys ’17). 15–28. https://doi.org/10.1145/3081333.3081356

[61] B. Zhang, Q. Zhan, S. Chen, M. Li, K. Ren, C. Wang, andD. Ma. 2014. ssrPriWhisper : Enabling Keyless SecureAcoustic Communication for Smartphones. IEEE Internet ofThings Journal 1, 1 (Feb 2014), 33–45. https://doi.org/10.1109/JIOT.2014.2297998

[62] Guoming Zhang, Chen Yan, Xiaoyu Ji, Tianchen Zhang,Taimin Zhang, and Wenyuan Xu. 2017. DolphinAttack:Inaudible voice commands. In Proceedings of the 2017 ACMSIGSAC Conference on Computer and CommunicationsSecurity. ACM, 103–117.

https://doi.org/10.1145/2018436.2018518

https://doi.org/10.1145/2018436.2018518

https://doi.org/10.1145/3081333.3081356

https://doi.org/10.1145/3081333.3081356

https://doi.org/10.1109/JIOT.2014.2297998

https://doi.org/10.1109/JIOT.2014.2297998

chuhan gao, kassem fawaz, sanjib sur, and suman banerjee ... · sanjib sur:...

Documents