crime early warning: automated data mining of cad and rms
DESCRIPTION
The genesis of HunchLab was the idea to mine law enforcement agencies' CAD and RMS databases to detect unusual levels of activity in particular areas and then send alerts to the appropriate police staff. While crime analysis tools often are aiming to display what has happened, the concept of a geographic early warning system, such as within HunchLab, tries to answer the question: "what is unusual that is happening?" http://www.azavea.com/products/hunchlab/features/early-warning/TRANSCRIPT
![Page 1: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/1.jpg)
340 N 12th St, Suite 402Philadelphia, PA 19107
www.azavea.com/hunchlab
Crime Early Warning Systems
Automated Data Mining of CAD and RMS Databases
![Page 2: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/2.jpg)
About Us
Robert CheethamPresident & [email protected]
Jeremy HeffnerHunchLab Product [email protected]
![Page 3: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/3.jpg)
Agenda
• Company Background• The Backstory• HunchLab
– Concept of Early Warning / Data Mining– Demonstration of Hunches– Underlying Statistics
• Q&A
![Page 4: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/4.jpg)
About Azavea
• Founded in 2000
• 27 people
• Based in Philadelphia
– Also Boston & Minneapolis
• Geospatial + web + mobile
– Software development
– Spatial analysis services
![Page 5: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/5.jpg)
Clients & Industries
• Public Safety• Municipal Services• Public Health• Human Services• Culture • Elections & Politics• Land Conservation• Economic Development
![Page 6: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/6.jpg)
Azavea & Governments
![Page 7: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/7.jpg)
The Backstory
![Page 8: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/8.jpg)
How Phila PD uses GIS
Customized Map Products
Weekly CompStat Meetings
Web Crime Analysis
![Page 9: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/9.jpg)
Complainant
CAD
Verizon911
911 Operator
RadioDispatcher
Police Officer
District48 Desk
INCT
Daily download& Geocoding Routines
Incident ReportCompleted by Officer District X
District Y
District Z
Maps distributedThrough Intranet,
Printing, CompStat
INCT & PARS – main database sources
over 5,000 incidents daily, over 2 million annually
PARS
![Page 10: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/10.jpg)
The Context
1,500,000 people
7,000 police officers
1,000 civilian employees
2,000,000 new incidents / year
3 crime analysts
![Page 11: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/11.jpg)
What we did
• Weekly Compstat• Lots of maps• Automation of map creation• Web-based systems
![Page 12: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/12.jpg)
… but what if we could…
Accelerate the cycle Proactively notify Automate the process
![Page 13: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/13.jpg)
Prototype
ArcViewVB & MapObjects
MS SQL Server
Crime Incidents Database
Shapefiles
and
GRIDs
Process Documentation
.ini file
![Page 14: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/14.jpg)
![Page 15: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/15.jpg)
… but there was a problem …
![Page 16: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/16.jpg)
It was crap … sort of.
![Page 17: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/17.jpg)
We needed ….
1. Better Statistics
2. Notification
3. Very Straightforward
![Page 18: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/18.jpg)
![Page 19: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/19.jpg)
web-based crime analysis, early warning, and risk forecasting
![Page 20: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/20.jpg)
Crime Analysis
– Mapping (spatial / temporal densities)
– Trending
– Intelligence Dashboard
Early Warning
– Statistical & Threshold-based Hunches (data mining)
– Alerting
Risk Forecasting
– Near Repeat Pattern
– Load Forecasting
![Page 21: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/21.jpg)
Crime Analysis – What has happened?
– Mapping (spatial / temporal densities)
– Trending
– Intelligence Dashboard
Early Warning – What is out of the ordinary?
– Statistical & Threshold-based Hunches (data mining)
– Alerting
Risk Forecasting – What is likely to happen?
– Near Repeat Pattern
– Load Forecasting
![Page 22: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/22.jpg)
Early Warning
![Page 23: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/23.jpg)
Early Warning
• Geographic Early Warning System– A system to alert staff of an unusual situation in a
particular location– Ingests data sets to automatically “cook on” and only
involves staff when a statistically unusual situation is found
HunchLab Database
Operational Database Alerting
System
Geostatistical Engine
Operational DatabaseOperational Databases
![Page 24: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/24.jpg)
Data Mining
• What do we mean by data mining?– The process of “cooking on” the data to reveal
something new (unusual)• Benefits
– Automated discovery process– Can examine large data sets without additional staff
time• Major crime incidents• Minor crime incidents
– Near real-time alerts• Limitations
– Can’t determine why something unusual is happening, only that it is happening
![Page 25: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/25.jpg)
Early Warning
bit.ly/crimespikedetector
![Page 26: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/26.jpg)
Demo
![Page 27: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/27.jpg)
What is a Hunch?
• A proposed hypothesis, saved into the system, and continually tested for validity
• Incident Attribute Requirements– Location (x, y)– Time (timestamp)– Classification
• Hunch Attributes– Location (area)– Time (recent / historic periods)– Classification
• Analyses– Statistical Hunch– Threshold Hunch
![Page 28: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/28.jpg)
Hunch Parameters: Location
• Address & Radius• Precinct/County/Country• Custom Drawn Area• Mass Hunch
![Page 29: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/29.jpg)
Hunch Parameters: Time
• Statistical Hunch– Recent Past– Historic Past
![Page 30: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/30.jpg)
Hunch Parameters: Classification
• Category• Time of Day• Narrative
![Page 31: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/31.jpg)
Hunch Helper
![Page 32: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/32.jpg)
Email Alert
![Page 33: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/33.jpg)
Hunch Details
![Page 34: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/34.jpg)
The Statistics
![Page 35: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/35.jpg)
What do we know?
• Hunch– Geographic region (that we care about)– Recent time frame (to alert on) – Historic time frame (to compare against)– Classification (that we are interested in)
![Page 36: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/36.jpg)
What do we know?
• Hunch– Geographic region (that we care about)– Recent time frame (to alert on) – Historic time frame (to compare against)– Classification (that we are interested in)
Within Hunch Outside of Hunch
Recent past ? ?
Historic past ? ?
![Page 37: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/37.jpg)
Hypergeometric Distribution
• Arises when selecting items at random from a heterogenous pool without replacement– Example
• A bag contains 45 black marbles and 5 white marbles• What is the chance of picking 4 white marbles when we
draw 10 marbles?
Tony SmithUniversity of Pennsylvania
Drawn Not Drawn
White Marbles
4 1
Black Marbles
6 39
![Page 38: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/38.jpg)
Hypergeometric Distribution
Drawn Not Drawn Total
White Marbles
4 = k 1 = m – k 5 = m
Black Marbles
6 = n-k 39 = N + k – n - m
45 = N – m
Total 10 = n 40 = N - n 50 = N
en.wikipedia.org/wiki/Hypergeometric_distribution
![Page 39: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/39.jpg)
What do we know?
• Hunch– Geographic region (that we care about)– Recent time frame (to alert on) – Historic time frame (to compare against)– Classification (that we are interested in)
Within Hunch Outside of Hunch
Recent past ? ?
Historic past ? ?
![Page 40: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/40.jpg)
What do we know?
• Valid Hunch– The current condition (and all worse conditions) is
unlikely to simply be due to chance
![Page 41: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/41.jpg)
Demo
![Page 42: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/42.jpg)
Research Topics
![Page 43: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/43.jpg)
Research Topics
• Mobile Interfaces• Analysis
– Real-time Functionality• Consume real-time data streams & conduct ongoing
analysis
![Page 44: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/44.jpg)
Research Topics
• Risk Forecasting– Load forecasting enhancements
• Machine learning-based model selection• Weather and special events
– Combining short and long term risk forecasts– Risk Terrain Modeling
![Page 45: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/45.jpg)
Q&A
![Page 46: Crime Early Warning: Automated Data Mining of CAD and RMS](https://reader034.vdocuments.us/reader034/viewer/2022051210/54b3a88c4a7959de248b456b/html5/thumbnails/46.jpg)
Contact Us
Robert CheethamPresident & [email protected]
Jeremy HeffnerHunchLab Product [email protected]