DO YOU HEAR WHAT I HEAR? FINGERPRINTING SMART DEVICES THROUGH EMBEDDED ACOUSTIC

COMPONENTSCSS2014 -Anupam Das, Nikita Borisov , Matthew Caesar

Presenter: Harshitha Chidananda

Introduction• Mobile devices, including smartphones, PDAs, and tablets, are quickly

becoming widespread in modern society

• 1 in every 5 people in the world owns a smartphone• 1 in every 17 own a tablet

2012 2013 2014 20150

250,000

500,000

750,000

1,000,000

1,250,000

1,500,000

1,750,000

2,000,000

2,250,000PC(Desktop & Notebook) Tablet Mobile Phone

Year

Thou

sand

s of

Uni

ts

Introduction• 75% of the mobile phones are smartphones and highly-featured

phones

• Motion detection• Gesture detection• Audio Genre detection

• Location detection• Interaction with nearby devices

• CompassSensors could be potential source for unique fingerprints.

Main problem• Fingerprinting smart devices can

jeopardize privacy• Software strength is not enough• Hardware level idiosyncrasies are used to

fingerprint physical devices:• Microphones• Speakers

Why Fingerprint Smartphones?• Smartphones can be fingerprinted for:

• Targeted Advertisement• Secondary Authentication factor

Related Work: Fingerprint techniquesSoftware based approaches• Browser based

• Cookies• Analyze Bing and Hotmail• JavaScript and Flash• Device drivers

Hardware based approaches• Radio Transmitter

• Accelerometer

• Microphones tested for fingerprinting(no variations)

Main points and contributions Study the feasibility of using microphones and speakers

embedded in smartphones. Exploit fabrication imperfections to fingerprint smartphones

through playback and recording of audio samples. Explore different acoustic features and analyze their ability to

successfully fingerprint smartphones. Examines the effect of background noise and other variables on

fingerprinting accuracy

Source of fingerprinting• Microphone• Speaker

1 Storage : modify/delete USB storage and SD card contents2 Network communication : full Internet access3 Network communication : view network state4 System tools : prevent device from sleeping5 Phone calls : read phone state and identity6 Hardware controls : control vibrator7 System tools : automatically start at boot8 Network communication : view Wi−Fi state9 Your location : fine (GPS) location

10 Your location : coarse (network−based) location11 System tools : retrieve running applications12 Your personal information : read contact data13 Your messages : read SMS or MMS14 Your messages : receive SMS15 Hardware controls : take pictures and videos16 Hardware controls : record audio17 System tools : modify global system settings

Top 17 permissions out of 173 total permissions

• Scenario 2: Stealthy App

App

1.Play audio

2. Audio signal

3.Record

audio4. Extract Fingerprint

Need access to only:1. Microphone2. Internet

connection

• Scenario 1: External attacker locally present1. Attacker records audio signal from distance2. Create a fingerprint of the recorded audio and link the fingerprint to a unique smartphone

Requires: microphones

Real World Apps

CHALLENGES• Determining acoustic features and audio analysis techniques are

effective in identifying unique signatures of devices.• Audio properties that could be used includes:

• Spectral entropy• Zero crossing

• Analysis algorithms that could be used are:• Principle component analysis• Linear discriminant analysis• Feature selection

OverviewKey observation:• Imperfections in smart device hardware induce unique signatures

on received and transmitted audio streams• These unique signatures are used to fingerprint device.

3 finger printing scenarios

Speaker Microphone Speaker + Microphone

Microphones• Micro Electro Mechanical System(MEMS)• Noise and echo canceling capabilities

Sound Wave

Mechanical Energy

Capacitive Change

Voltage Change

Microphone• The sensitivity of the microphone depends on:

• How well the diaphragm deflects to acoustic pressure• Gap between back-plate and flexible diaphragm

• Imperfections can arise due to: • Slight variations in the chemical composition of components from one

batch to the next • Wear and tear in the manufacturing machines • Changes in temperature and humidity

Speaker• Sound waves are produced whenever electrical current flows

through the voice coil.• Each speaker component can introduce into the generated sound.

Electrical current

Magnetic field around voice coil Mechanical

EnergySound Wave

Fingerprinting Microphones

•Convince the user to install an app on their phone•Observer inputs from the device’s microphone

Fingerprinting Speakers

This scenario is suited for the case where the attacker has deployed nearby microphones to capture audio signal (e.g., ringtone) from user device.

Fingerprinting both speakers and Microphones

The attacker could potentially play a theme song at the start of the game and at the same time make a recording of the audio clip

ApproachListener module• Responsible for receiving and recording

audioAnalyzer module• Identify device signatures

Fingerprint

Mechanism:1)Extract auditory fingerprints2)Efficiently search for matching fingerprints

Acoustic Features

Classification Algorithms• Training step

• Collect a no. of observations from a set of devices• Data Point

• Each observation corresponds to a set of featuresApproaches• K-nearest neighbors

• Associates an incoming data point with the device corresponding to the nearest “learned” data points

• Gaussian mixture models• Computes a probability distribution for each device and determines the

maximal likely association

Feature Selection• Are there any dominant features?• Identifying the dominant feature set benefits us in two ways:

• Less computation • Improve accuracy as there might be conflicting features

Feature Reduction (Dimensionality reduction)

Feature Extraction[new features=function(old

features)]e.g., PCA, LDA

Feature Selection[subset of old features]

EVALUATION

Setup• 266 square foot (14’x19’) office room• Nine-foot dropped ceilings• The room also receives a minimal amount

of ambient noise from • Air conditioning• Desktop computers• florescent lighting.

• Laptop(ACER Aspire 5745 ) is used to emulate the attacker

Experimental SetupInvestigate:• Different maker-model• Same maker-model• Combination Duration of the audio clips varies from 3 to 10 seconds

Evaluation Metrics• 2 classification algorithms:• K-NN(k nearest neighbors)• GMM(Gaussian Mixture Model)

TP: True PositiveFP: False PositiveFN: False Negative

Evaluation Algorithms

Dataset• 50% Training, 50% testing• Varying classification

algorithms from 1 to 5

DIFFERENT MAKER AND MODEL

Feature Exploration

MFCCs

• Mel-Frequency Cepstral Coefficients of the same audio sample take from:• 3 different handsets • Same vendor

• Coefficients vary significantly• Exploit this feature.

Fingerprinting using Microphone

• F1-score of 97%• Smartphones can be

fingerprinted using microphones

Fingerprinting using Microphone and Speaker

• Android App only• F1-score of 100%• Malicious app having access to

microphone and speaker can successfully fingerprint smartphones.

Fingerprinting using Speaker

• F1-score of 100%• Smartphones can be

fingerprinted using Speaker

Instrument Human Speech Song0123456789

10

0 0 00

2.59999999999999

00 0 0

Speaker Microphone

Genre

1 - F

1_sc

ore

(%)

So we can accurately distinguish smartphones of different make and model.

Different Make & Model SetsMaker Model #Apple iPhone 5 1

HTC Nexus One

14

Samsung

Nexus S 8Galaxy

S33

Galaxy S4

10

Motorola

Droid A855

15

Sony Ericsson

W518 1

Total 52

SAME MAKER AND MODEL

Motorola Droid A855 handsets (15)

Feature Exploration

MFCCs are the dominant features for all categories of audio excerpt.

Fingerprinting using Microphone

• F1-score of 95%• Feasible option

Fingerprinting using Microphone and Speaker

• Android App only• F1-score of 100%• App developed by authors - Best

Fingerprinting using Speaker

• F1-score of 94%• Viable Option

Instrument Human Speech Song0123456789

10

1.7 1.7

5.54.7

0

3.90000000000001

0 0 0

Speaker Microphone

Genre

1 - F

1_sc

ore

(%)

Fingerprinting both the speaker and microphone seems to provide better results.

Same Make & Model SetsMaker Model #Apple iPhone 5 1

HTC Nexus One

14

Samsung

Nexus S 8Galaxy

S33

Galaxy S4

10

Motorola

Droid A855

15

Sony Ericsson

W518 1

Total 52

COMBINATIONAL: MAKER AND MODEL

50 Android Phones

Feature Exploration

MFCCs are the dominant features for all categories of audio excerpt.

Fingerprinting using Microphone and Speaker

• Android App only• F1-score of 98%• Malicious app having access

to microphone and speaker can successfully fingerprint smartphones

Instrument Human Speech Song0123456789

10

1.7 0.700000000000003

0

Genre

1- F

1_sc

ore

(%)

Maker Model #Apple iPhone 5 1

HTC Nexus One

14

Samsung

Nexus S 8Galaxy

S33

Galaxy S4

10

Motorola

Droid A855

15

Sony Ericsson

W518 1

Total 52

Heterogeneous Smartphones

SENSITIVITY ANALYSIS

Sensitivity AnalysisImpact of Sampling Rate• Record Ringtone• 3 Different frequencies• As Sampling frequency

decreases, precision/recall also decreases.

• Higher sampling frequency includes more fine-tuned information

Sensitivity AnalysisImpact of Varying Training Size• Expected results• Performance increases as

training size increases• 3 samples per class

• F1-score of 90%• Too many training samples are

not required.

Sensitivity AnalysisImpact of Varying Distance between Speaker and Recorder• As distance increases, F1-score

decreases.• Works only up to a certain

distance0.1 1 2 3 4 50

102030405060708090

100

k-NN GMM

Distance (in meters)

F1 S

core

Sensitivity AnalysisImpact of Ambient Background Noise• Crowded environment• F1-score of 91%

STRENGTHS AND WEAKNESSESSTRENGTHS• Requires small amount of data• Very Good evaluation• Good results• Diverse

WEAKNESS• Repetitive• Unstructured

OPEN ISSUES• Privacy concerns in smartphones• Consequences of fingerprinting smart devices through acoustic

channels.• More experiments on Apple• Find out if this is applicable to laptops as well

MY THOUGHTS AND CONCLUSIONMy thoughts• Easy to understand• Extensive evaluation• Good illustrations• Good graphical representation• Good results• The application developed by the

authors results in good performance

ConclusionIt is feasible to fingerprint smart-devices through on-board acoustic components like

• microphones • speakers

Able to accurately attribute ~98% of all recorded audio excerpts from 50 different Android devices.

THANK YOUQuestions?