sensor-level privacy for thermal cameras - focus...

Sensor-level Privacy for Thermal Cameras

Francesco Pittaluga

Aleksandar Zivkovic

Sanjeev J. Koppal

University of Florida

Imaging and Tracking People

Surveillance Military

Gaming IoT Mobile

2Intro Digitization Noise Exposure Conclusion

Balancing Privacy and Utility

Hospitals Schools

Retirement homes Workplaces

3Intro Digitization Noise Exposure Conclusion

Conventional Privacy Processing

Scene

Camera

ComputerStorageSecure

Data

4

Post-capture processing Lock data with cryptography

Edit images computationally

Boyle 2000, Sweeney 2002, Johnson et al. (IETF) 2003,

Gross et al. 2009, Agrawal and Narayanan 2011, …

Intro Digitization Noise Exposure Conclusion

Conventional Privacy Processing

Scene

Camera

ComputerStorageSecure

Data

Post-capture strategy has an inherent vulnerability

5Intro Digitization Noise Exposure Conclusion

Privacy Preserving Computational Cameras

Scene

Only capture light-field samples that are needed

Computational Camera

Secure

Data

6Intro Digitization Noise Exposure Conclusion

Computational Camera: Three Approaches

SceneComputational Camera

Secure

DataComputerStorage

X Pattern recognition

in every frameX Special

OpticsX New sensor

7

Chattopadhyay and Boult 2007, Winkler and Renner 2010, Narayanan and Mrityunjay 2011

Nelson et al. 2005,Winkler et al. 2014, Fernandez-Berni et al. 2014

Three Problems Limited Adoption

Zhang et al, 2014, Pittaluga and Koppal 2015

Intro Digitization Noise Exposure Conclusion

Our Idea: Sensor-level Privacy for Thermal Cameras

Scene

Our idea: use a thermal sensor

(Wavelengths > 3µm)

Computational Camera

Secure

DataComputerStorage

8

Reliable Physics

Based Processing Off the Shelf

Optics

Thermal

sensor

Intro Digitization Noise Exposure Conclusion

Thermal Cameras are Coming Soon!

Thermal cameras are not exotic anymore

Thermal face recognition works

FLIR One 80x60

~$250

Melaxis 16x4

~$75FLIR A6751SC

9Intro Digitization Noise Exposure Conclusion

Three Sensor Level Approaches

SceneComputational Camera

Secure

Data

Thermal Sensor

10Intro Digitization Noise Exposure Conclusion

Three Sensor Level Approaches

Thermal Sensor

Digitization Noise Exposure

11Intro Digitization Noise Exposure Conclusion

Three Sensor Level Approaches

Thermal Sensor

Digitization Noise Exposure

12Intro Digitization Noise Exposure Conclusion

Humans are broadband

13

Spectral Power Response vs Wavelength

8

11

7 14

Human Spectral Response

r(λ)

(μm))

𝑊

𝑚2 𝑠𝑟 𝜇𝑚

Intro Digitization Noise Exposure Conclusion

Facial skin temperature reaches an equilibrium

37 Degrees

(Celsius)

Outside temperature known

= T Celsius

Facial skin temperature F (T, 37)

14Intro Digitization Noise Exposure Conclusion

This mapping is known

Olesen and Parsons 2002

15

Skin

Tem

pera

ture

°C

Ambient Temperature °C

Intro Digitization Noise Exposure Conclusion

Removing pixels in this range during digitization

ASIC Modification

16

Pixel

Readout

Output

Voltage

A-to-D

Masking measurements

based on temp. range

16-bit

decoder

Lower

Voltage

Upper

VoltageUpper Bound

Comparator

Lower Bound

Comparator

AND

AND

NOT

Seq. Counter

Random-Access

Memory

Address (18.0)

Data(15.0)

Intro Digitization Noise Exposure Conclusion

Digitization Result

17Intro Digitization Noise Exposure Conclusion

Digitization Result

18Intro Digitization Noise Exposure Conclusion

Three Sensor Level Approaches

Thermal Sensor

Digitization Noise Exposure

19Intro Digitization Noise Exposure Conclusion

Adding noise to the bolometer

20

GFID

GSK

VCC

Blind Bolometer

Active Bolometer

Vbus

Vout

Cint

Tunable Bias

Voltages

Intro Digitization Noise Exposure Conclusion

The effect of bias voltages

21

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

Exposure 5 Exposure 15

Exposure 25 Exposure 75

Intro Digitization Noise Exposure Conclusion

Calibrating for noise and privacy

22

GSK

GFID

5 V

5 V

0 V

0 V

9000

0

0 120Grayscale values

Histogram of values for the highest standard deviation

Occurrences

2𝜎 = 14 graylevels

Bias voltages with exposure set to 5

For a flat lambertian plane

Intro Digitization Noise Exposure Conclusion

Noise Result: Head Tracking

23Intro Digitization Noise Exposure Conclusion

Three Sensor Level Approaches

Thermal Sensor

Digitization Noise Exposure

24Intro Digitization Noise Exposure Conclusion

Temperature and radiant power

25

Spectral Power Response vs Wavelength

8

11

7 14

s(λ)

Camera sensitivity

(μm)

𝑊

𝑚2 𝑠𝑟 𝜇𝑚

Pixel Number vs Radiant Power

Φ𝑐𝑜𝑙𝑑 Φℎ𝑜𝑡

𝐼max~

𝐼min~

𝐼max

Φ = λ𝑡

λℎ

𝑠 λ 𝑟 λ 𝑑λ

Human Spectral Response

r(λ)

Φℎ𝑢𝑚𝑎𝑛

r(λ)r(λ)

Intro Digitization Noise Exposure Conclusion

No capture region

26

Spectral Power Response vs Wavelength

8

11

7 14

Φ𝑚𝑖𝑛 = λ𝑡

λℎ

𝑠 λ [𝑟 λ −∆ 𝜆

2] 𝑑λ

Φ𝑚𝑎𝑥 = λ𝑡

λℎ

𝑠 λ [𝑟 λ +∆ 𝜆

2] 𝑑λ

(μm)

𝑊

𝑚2 𝑠𝑟 𝜇𝑚

Pixel Number vs Radiant Power

Φ𝑚𝑖𝑛 Φ𝑚𝑎𝑥

𝐼max

“No

Ca

ptu

re”

s(λ)

Camera sensitivity

Δ(𝜆)Human Spectral Response

r(λ)

Intro Digitization Noise Exposure Conclusion

Exposures that remove no capture region

27

Spectral Power Response vs Wavelength

8

11

7 14

s(λ)

Camera sensitivity

(μm)

𝑊

𝑚2 𝑠𝑟 𝜇𝑚

Underexposure

Overexposure

Pixel Number vs Radiant Power

Φ𝑚𝑖𝑛 Φ𝑚𝑎𝑥

𝐼max~

𝐼min~

𝐼max

“No

Ca

ptu

re”

𝑡 ≥𝑔 𝐼𝑚𝑎𝑥(𝑔)

Φ𝑚𝑖𝑛

𝑡 ≤𝑔 𝐼𝑚𝑖𝑛(𝑔)

Φ𝑚𝑖𝑛

Human Overexposed Human Underexposed

Human Spectral Response

r(λ)

Intro Digitization Noise Exposure Conclusion

Optimal algorithm to obtain exposures

28

Grossberg and Nayar 2003

𝝽(𝑛, 𝑇) = Γ𝑚𝑖𝑛

Φ𝑚𝑖𝑛

ℎ𝑑𝑒𝑠′ − ℎ′ 𝑝𝜔𝑑Φ +

Φ𝑚𝑎𝑥

Γ𝑚𝑎𝑥

ℎ𝑑𝑒𝑠′ − ℎ′ 𝑝𝜔𝑑Φ 𝜔 =

0 ℎ𝑑𝑒𝑠′ (Φ) < ℎ′(Φ)

1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒,

…

Error Function Binary Weights

1. 𝑇𝑖 > 0

2. ∀ 𝑇𝑖 𝑇𝑖 ≥ 𝐼𝑚𝑎𝑥

Φ𝑚𝑖𝑛⊕ 𝑇𝑖 ≤

𝐼𝑚𝑖𝑛

Φ𝑚𝑎𝑥

𝑎𝑟𝑔𝑚𝑖𝑛 𝑛,𝑇 𝝽(𝑛, 𝑇) 𝑠. 𝑡.

Intro Digitization Noise Exposure Conclusion

Optimal algorithm to obtain exposures

Algorithm assumes a single no capture region

Brute force search is therefore tractable

29

Objective function score

Optimal solution for

112 exposures

Grid search index

HDR Image

Intro Digitization Noise Exposure Conclusion

HDR Results

Over Under Fusion

30Intro Digitization Noise Exposure Conclusion

HDR Results

Over Under Fusion

31Intro Digitization Noise Exposure Conclusion

HDR Results

Over Under Fusion

32Intro Digitization Noise Exposure Conclusion

HDR Results

33Intro Digitization Noise Exposure Conclusion

Method Comparison

34Intro Digitization Noise Exposure Conclusion

Comparison

Digitization Noise Exposure

• Low noise

• Good image

quality

• Real-time

• Hardware and

firmware upgrades

• Low noise

• No hardware

modification

• Good image quality

• Multiple Images and

more capture time

• Real-time

• No hardware

modification

• Low image quality

• Noisy

35Intro Digitization Noise Exposure Conclusion

Future Work

Pilot deployment program of private sensors at

UF Health Shands Hospital.

Generate database of private face images to for

privacy challenge.

Generate database of private videos for activity

recognition in a hospital setting.

36Intro Digitization Noise Exposure Conclusion

DHS

Sanjeev Koppal Andreas Enqvist

This material is based upon work supported by theU.S. Department of Homeland Security under Grant Award

Number, 2014-DN-077-ARI083-01. The views and conclusionscontained in this document are those of the authors

and should not be interpreted as necessarily representing theofficial policies, either expressed or implied, of the U.S. Department

of Homeland Security.

Acknowledgements

Intro Digitization Noise Exposure Conclusion

Summary: Three Sensor Level Approaches

Digitization Noise Exposure

38Intro Digitization Noise Exposure Conclusion