Automated video surveillance: challenges and solutions.

ACE Surveillance (Annotated Critical Evidence) case study.

Dmitry Gorodnichy and Tony Mungham

Laboratory & Scientific Services DirectorateCanada Border Services Agency

www.videorecognition.com/ACE

2

Outline

Problems with status-quo Video Surveillance Real-time and archival problems Operational considerations

Next generation solution - Video Analytics based “Motion detection” myth and problem “Object detection” as example of real intelligence

ACE Surveillance – first fully-functional object- detection-based prototype Year long tests with different levels of complexity What that means for future of Video Surveillance

Conclusions

3

Role of Video Technology (VT)

In the context of enhancing security, Video Technology (VT) is one of the most demanded technologies of the 21st century It is publicly acceptable It provides rich in content data

Multi-million funding in Canada and worldwide: CBSA Port Runner project invested 10s of Millions in

CCTV upgrades Transport Canada opens $35M of funding towards

procurement of CCTV

4

VT at CBSA

CBSA is a major user of CCTV systems at POEs Most major CCTV installations start to leverage VT Current task: to lead applied R&D to push VT to help

CBSA apply S&T innovative approaches to border management: Event detection and notification to provide effective response

to events Traffic trends analysis to assist with border management Video storage management to manage the cost of storage

and meet obligations under the privacy act Data integration/fusion of contextualised video information

5

Problem with status quo use of CCTV surveillance

Modes of operation:

1. Active - personnel watch video at all times

2. Passive - in conjunction with other duties

3. Archival - for post-event analysis

Current systems and protocols are not efficient in either mode!

Problem in real-time modes: an event may easily pass unnoticed . due to false or simultaneous alarms, lack of time needed to rewind and analyse all video

streams.

6

Problems in Archival mode:

Due to temporal nature of data:

1. Storage space consumption problem• Typical assignment:

2-16 cameras, 7 or 30 days of recording, 2-10 Mb / min.1.5 GB per day per camera / 20 - 700 GB total !

2. Data management and retrieval problem• London bombing video backtracking experience:

“Manual browsing of millions of hours of digitized video from thousands of cameras proved impossible within time-sensed period”

[by the Scotland Yard trying to back-track the suspects]

7

Operational considerations

Lots of CCTV infrastructure: Many local initiatives, not coordinated

Most video technology decisions are influenced by vendors - short-term solutions

Over 30 different video systems within the same dept. (at RCMP)

A national program with proper benchmark-based planning and evaluation of VT is required Leveraging advances recently made in S&T Technical standards for capturing /saving video data. Policies in when, where and how VT should be used.

8

Video Technology today

Video Analytics (Video Recognition)

Analog

20th century

21st century

First video recording

Digital

Wireless, Network Connected (IP)

9

Next generation Video Technology

Is Video Analytics basedalso identified as:

Video Recognition, Intelligent Video,Smart Video / Smart CameraVideo Analysis & Content ExtractionPerceptual Vision

is not much about capturing better data (better lenses, grabbers, coders, transmitters)

but about understanding captured data (better theory)

10

Status-quo “video intelligence”

Transport Canada CCTV Reference Manual for Security Application .

Australian Government National code of practice for CCTV applications in urban transport

USA Government :recommended security Guidelines for Airport Planning, Design and Construction.

…. refer to “Motion-based” capture as Intelligent Surveillance Technology, and make their recommendations based on thereon.

11

“Motion-detection” is not intelligent!

Term “Motion-based” is coined to make people believe that video recognition is happening, which is not!

It’s actually illumination-change-based, as it uses simple point brightness comparison:

Which often happens not because of motion!Changing light / weather (esp. in 24/7 monitoring)Against sun/light, out of focus, blurred, thru glassReflections, diffraction, optical interferences Image transmission, compression losses

12

“Object-detection” is intelligent …

… but few can do it, since necessary advances in video recognition theory became possible only recently (>2002).

In 2002 National Research Council of Canada (NRC) starts developing Video Recognition Systems to leverage its scientific Video Recognition expertise for the industry.

In 2005, it develops ACE Surveillance:an object-detection-based Automated surveillanCE prototype capable of automatically extracting Annotated Critical Evidence from live video.

NRC becomes also the organizer of the first Canadian academic workshops dedicated to Video Processing for Security (since 2004)

13

What is ACE Surveillance?

A Windows software that performs real-time video analytics by integrating best object detection and tracking algorithms.

Replaces video clips with annotated still images: Compresses 1 Gb of video into 2 Mb of easy to browse and

analyze still images ACE Surveillance output:

A 7-hour activity from day to night (17:00 - 24:00) is summarized into 2 minutes (600Kb) of Annotated Critical Evidence snapshots.

Note illumination changes! - Watch tree shadows and sun light.

14

ACE Surveillance architecture

Works with ordinary USB cameras or CCTV cameras with USB video converters.

Adds on top of existing infrastructure using an ordinary desktop computer.

Video clips (Tb)...

Archival mode of operation

Real-time mode of operation

Last captured CES

ALARM!

C.E.S. (Gb)

ACE Capture ACE Browser

15

Adds on top of existing infrastructure

16

Status quo “Motion-based” capture(Courtesy: NRC-IIT Video Recognition Systems project)

1. Many captured snapshots are useless: either noise or redundant

2. Without visual annotation, motion information is lost.

3. Hourly distribution of snapshots is not useful

17

ACE Surveillance “Object-based” capture(Courtesy: NRC Video Recognition Systems project)

1. Each captured shot is useful.

2. Object location and velocity shown augmentent.

3. Hourly distribution of shots is indicative of what happened in each hour, provides good summarization of activities over long period of time.

18

ACE Surveillance testing benchmarks

Tested in different levels of complexity: lighting conditions, object motion patterns, camera location environmental constraints.

most difficult - outdoors in unconstrained environments with little or no object motion consistency (as around a private house in a regular neighbourhood).

most easy - in controlled indoor environment where minimal direct sunlight is present and where all objects are of approximately the same size and exhibit similar motion pattern (as at access gate inside the business building).

VT within CBSA 19

Outdoor, wireless, eye-level Outdoor, webcam, overview Indoor with sunlight, CCTV Indoor w/o sunlight, CCTV

20

Enables efficient detection of abnormal activities Delivery EntryBack Door Entry

On

wee

k-da

yO

n w

eek-

end

More than usual

21

ACE Surveillance results

In real-time mode: alarm sounds & last captured evidence (time-stamped) is shown.

In archival mode: “Zoom on the evidence” browsing of captured evidences – zoom on a day, on hour, then on event - point and click (for high res as needed)

Made Commissioners much more aware of activities.

22

Conclusions

Affordable automated (intelligent) video surveillance (AVS) is possible! To replace traditional DVR OR to supplement them: DVR for 1 month + AVS for 1 year

However: Requires extra training from security officers. Requires new protocols to handle automatically extracted

evidence. - From forensic prospective, data that are not original and have

been processed by a computer can not be considered as evidence. Requires new privacy policies.

- Surveillance data are normally not kept for a long period of time (<1 month), due to their size. AVS allows to store on local machine many months (even years) of evidence data.

23

ACE surveillance case study outcome

ACE Surveillance (which is developed by a research lab) provides a reference standard against which can be measured solutions coming from industry.

It deals with common misconceptions related to deploying intelligent video surveillance systems (IVS): “motion detection” myth vs real object detection and tracking. The “one-fit-all” myth. - Extra video analytics expertise is required to

set and operate IVS. better video data (better resolution or compression) do not imply

better video intelligence. - ACE Surveillance is shown to work with regular TV quality data (320 by 240 pixels).

However better quality of video image is needed for forensic purposes as evidence

Due to closing of the project by NRC, CBSA takes lead on it.