video technology for security: problems and solutions dr. dmitry gorodnichy video recognition...

Video Technology for security:Problems and Solutions

Dr. Dmitry Gorodnichy

Video Recognition Systems*Project Leader

Rail and Urban Transit Security

Workshop

MontrealNovember 2007

2. Video Recognition Systems (Dmitry Gorodnichy)

Outline

1. Video Technology (VT) – History of VT– Who we are: NRC/IIT Video Recognition Systems (VRS) project– VT within GoC (VT4NS’07 meeting)– Important VT facts: What vendors don’t tell

2. On Intelligent Surveillance– Need-to-know facts: status quo, real challenges, real solutions

• On “motion-detection” • Next-generation surveillance: object-detection based

– Example: ACE Surveillance (Annotated Critical Evidence) – Future trends

3. On Video-based Face Recognition– Not as Photograph-based: New constraints, applications & approaches– Future trends


History of VT

Video recognition when became possible to process video frames fast (>12 fps)

Pattern Recognition

Video

Recognition

ComputerVision

Analog

XX century

XXI century

First videowhen became possible to display video frames fast (>12fps)

Digital

Wireless, IP


Next-generation VT = Video Recognition

• Aka - Intelligent Video, Smart Video, - Video Analysis & Content Extraction (VACE) - Perceptual Vision

• IS NOT about capturing data (better lenses, grabbers, coders, transmitters), - but about understanding captured data (better theory)

• IS very young area and IS NOT an extension BUT a mixture of:• Pattern Recognition (plates, character, passport rec.)

• Computer Vision (video capture and processing)

• Biological Vision (how brain does it)


Who we are:Video Recognition Systems

• Started 2001 within NRC/IIT– Formerly, as Perceptual Vision project

• Do both research / services and development / licensing – Worked on Canadarm2 (2001-2)

– Known for Nouse® (Nose as mouse) tool for disabled (2003-7)

• Emphasis on Security & Surveillance since 2004– Intelligent surveillance

– Face recognition from Video

• Work with Industry, Academics & OGDs: – Esp. CBSA, RCMP, DRDC.

• Partner of USA DTO / VACE program (Disruptive Technology Office / Video Analysis and Content Extraction)


VRS key technologies

• Object detection and tracking– Automated Teleoperator– ACE Surveillance™

• Faces in Video – Face detection, tracking– Face recognition from Video

• Other– Image Search (Roth)– Marker-based tracking (Fiala)

17:00-24:00 monitoring 2 mins of

summarized video


• No national / regional VT program yet. • Decisions influenced by vendors / short-term solutions

No national standards for capturing /saving video data.

• E.g. over 30 different video systems deployed in Ottawa No policy to handle evidence:

• E.g. is data original, not altered

• Many local initiatives, not coordinated– City of Calgary (traffic abnormalities detection with CCTV cams)– Cornwall Canada US border* (DVR). Pilot project #1 “port-runner”– Ottawa/Montreal Airports* (CCTV, PTZ DVR), …

•This is about to be changed (2007)•Follow the USA DTO/VACE model

GofC Workshop VT4NS’07 Report


Facts to know (what VT vendors may not tell)

1. Video capture is no longer expensive or bad– Composite video/RCA (CCTV analog)– USB2 cams and digitizers– Firewire cams – Wireless & IP cameras– IR (night) cams– Multi-channel framegrabers for CCTV

2. Beware of “high resolution” cameras1. It’s unlikely the real resolution

2. It doesn’t help making video more “intelligent”

3. It’s Intelligence that’s missing

http://www.axis.com/products/cam_207w/index.htm


Problems

1. Environment/Setup – light/weather, field of view …

2. Objects/Activities – non-collaborative actions

3. Misconceptions (in interest of vendors) 1. The more, the better – NO

2. A human can see, so the system will (one day) – NO

3. “Baggage of the past”: using old tools for NEW problems

4. Real-time constraint – for “alarm” systems

5. Resolution1. Video image is small: 720 x 480 (HD) or 360 x 240 (tape)

2. Objects occupy small part: <1/8 of image But is resolution really a problem ?


Recorded from TV(320 x 240 video)

Despite “bad” resolution + orientation, expression, occlusion

(Faces are 30x30 pixels!)

You don’t have problem recognizing people & activities

But computers can’t do it! (yet) Even on a studio made video-clip with perfect FOV and lighting!


Intelligent Surveillance: problems & solutions


Two Big problems

1. Storage space consumption• Typical assignment:

2-16 cameras, 7 or 30 days of recording, 2-10 Mb per min.

1.5 GB per day per camera / 20 - 700 GB total !

2. Data management and retrieval• London bombing video backtracking experience:

“Manual browsing of millions of hours of digitized video from thousands of cameras proved impossible within time-sensed period”[by the Scotland Yard trying to back-track the suspects]


Misconception about“Motion-based” capture

• Term “Motion-based” is coined to make people believe that video recognition is happening, which is not!

• It’s actually illumination-change-based, as it uses simple point brightness comparison:

– Which often happens not because of motion!• Changing light / weather (esp. in 24/7 monitoring)• Against sun/light, out of focus, blurred, thru glass• Reflections, diffraction, optical interferences• Image transmission, compression losses


Solution:- Do as much as possible Video Recognition in real-time

BEFORE saving video !- Object-based surveillance (not change-based) !

Example: A.C.E. Surveillance(Annotated Critical Evidence)

- Based on recent advances in object detection / tracking.- Replaces video clips with annotated JPG images

– Compresses 1 Gb of video into 2 Mb of easy to browse still images – Annotations: size, velocity, colour of detected objects.

• Enables efficient new Zoom-on-Evidence browsing

Next-generation surveillance


Motion-based capture

•Many captured snapshots are useless: either noise or redundant

•Without visual annotation, motion information is lost.

•Hourly distribution of snapshots is not very useful


ACE Capture

•Each captured snapshot is useful.

•Object location and velocity shown using graphical annotation

•Hourly distribution of snapshots is indicative of what happened in each hour, provides good summarization of activities over long period of time.


Example: Monitoring in XXI-st century

• In real-time mode: watch closely if alarm sounds.

• If away from desk: Last captured CES shows whether anything happened. Then play-back all CES-es.

• In archival mode: “zoom on evidence” – zoom on a day, on hour, then on event - point and click (for high res as needed)


Zoom-on-Evidence Browsing

Delivery EntryBack Door Entry

On

wee

k-da

yO

n w

eek-

end


Future trends

• In software (video recognition algorithms):– Better object detection & tracking

• For complex motions• For moving cameras

– Better annotation: activity recognition • In hardware:

– Smart PTZ cams: PTZ on objects– Smart IP cams: send only when/what is needed– Video + hi/res photocamera / other sensors – Synchronized cameras

• In mentality / logistics:– More inter-department VT initiatives– More constrained/proper setups and tasks


Video-based Face Recognition:

problems & solutions


Intentional misconception?

Over last 5 years $XXX.XXX.XXX already spent on applying face recognition to video data…

And what?

Still Face Recognition Vendor Test (www.frvt.org) is seen: “in making the video

data of better quality” (2006)

Instead of developing approaches which can deal with low-resolution data

0

20

40

60

80

100

In

photos

In

video

By humans

By computers


Important

Photographic facial data and video-acquired facial data are two different image-based modalities

– different nature of data – different biometrics– different approaches– different testing benchmarks

Face recognition in video requires video-based framework


Photos vs Video

Photos:- High spatial resolution- No temporal knowledge

E.g. faces:1. in controlled environment

(similar to fingerprint registration)

2. “nicely” forced-positioned 3. 60 pixels IOD

(IOD = intra-ocular distance)

Video:- Low spatial resolution- High temporal resolution

( Individual frames of poor quality)

1. in unconstrained environment (in a “hidden” camera setup)

2. don’t look into camera, don’t even face camera

3. 10-20 pixels IOD

Yet, for humans, video (even of this “low” quality) is often even more informative than a photograph !


Canonical Face Modelused in passports

Adopted by ICAO’02 for passport-type documents(used in Canada, USA, EU)

• One picture per person• IOD=60 (Width=120 pixels)

Used - To store faces in databases- In recognition algorithms

• But can it be used for video?• Should it be used ?


Proper approaches

1. Work on low-res images

2. Accumulate facial data over time

Good video-based recognition implies accumulation of data over time!

Anything based on a single frame won’t be good.

Types of multi-frame facial data fusion• Super-resolution

• Neuro-biological (synaptic adaptation)

• 3D face models


VRS Example: person recognition in TV

programs

• People: recognize faces starting from IOD > 10 pixels • Good news (2002) – computers can detect such faces

• Limitation: may not be suitable for ceiling cameras


Discussion

• IOD < 10 Body/gait biometrics should be used

• IOD > 11 Some face “recognition” from video can be performed

• IOD > 40 Person identification may be possible (under some conditions: many shapshots under good angles).

Future trends:- Monitoring of limited-access premises- Multiple-camera person tracking and backtracking - Verification (e.g. with access card)- Forced face registration (as in check-ins / “hidden” eye-level cameras)

Questions: • Email: [email protected] ([email protected]) • VRS project site:

http://vrs.iit.nrc.ca http://vrs.iitc.ca (after Nov. 1, 2007)www.videorecognition.com, www.perceptual-vision.com

video technology for security: problems and solutions dr. dmitry gorodnichy video recognition...

Documents