municipal government meets nosql

Municipal Governmentmeets NoSQL

Ryan BrionesIT Director - Software Architecture and DesignDepartment of Innovation and TechnologyCity of Chicago

Tuesday, November 13, 12

This is my bossBrett GoldsteinChief Data OfficerCommissioner, DoIT (CIO)


This is my bossBrett GoldsteinChief Data OfficerCommissioner, DoIT (CIO)Data Junkie


My Job

Crazy Ideas

Reality


2010Tuesday, November 13, 12

2011How do we tell the story about a place?


How do we tell the story about a place?

911 CPS CTA CFD 311


2012How can we use realtime data to make better decisions?


Bring all the data all in one place

Make it available in realtime

Find relationships in the data


Spatial / Temporal


Grow elastically

Loose schema

Works well with geospatial data


Grow elastically

Loose schema



{pos: [41.884267623865384,

-87.62977480888367]}/* ... */db.windygrid.ensureIndex({pos: “2d”})/* ... */db.windygrid.find({ $within: { $box:

[ [41.88227, -87.63208] [41.88499, -87.62838] ] }})


Grow elastically

Loose schema



Router

Primary Shard 1 Primary Shard 2 Primary Shard 3

Secondary Shard 1 Secondary Shard 2 Secondary Shard 3

Secondary Shard 1 Secondary Shard 2 Secondary Shard 3

C

C

C

69GB data200K objs


Grow elastically

Loose schema



{type: “311”,shardkey: “2a08c28b3ca3bb65bd...”where: { location: [-180,180], ... },when: { shardtime: 0, ... },what: { ... },who: { ... }

}


Challenges

$within/$box

safe scaling


The Future

Predictive AnalyticsNatural-language ProcessingMachine LearningDistributed ComputingVisualizations


Ryan BrionesIT Director - Software Architecture and Design

Department of Innovation and TechnologyCity of Chicago

github.com/Chicagogithub.com/ryanbriones

@ChicagoCDO@ryanbriones


municipal government meets nosql

Documents

geospatial data

realtime data

c secondary shard

loose schematuesday

realtimefind relationships

spatial temporaltuesday

doit ciotuesday

comchicago github