ai for smart city innovations with open data (tutorial)

160
AI FOR SMART CITY INNOVATIONS WITH OPEN DATA DR. BIPLAV SRIVASTAVA ACM DISTINGUISHED SCIENTIST, ACM DISTINGUISHED SPEAKER SENIOR RESEARCHER AND MASTER INVENTOR, IBM RESEARCH – INDIA 1 Tutorial on 27 July 2015 @ IJCAI 2015

Upload: biplav-srivastava

Post on 16-Aug-2015

363 views

Category:

Technology


4 download

TRANSCRIPT

Page 1: AI for Smart City Innovations with Open Data (tutorial)

AI FOR SMART CITY INNOVATIONS WITH OPEN DATA DR. BIPLAV SRIVASTAVA A C M D I S T I N G U I S H E D S C I E N T I S T , A C M D I S T I N G U I S H E D S P E A K E R S E N I O R R E S E A R C H E R A N D M A S T E R I N V E N T O R , I B M R E S E A R C H – I N D I A

1 Tutorial on 27 July 2015 @ IJCAI 2015

Page 2: AI for Smart City Innovations with Open Data (tutorial)

Why This Tutorial?

Tutorial on 27 July 2015 @ IJCAI 2015

�  Sustainability is a key imperative of modern societies �  AI techniques have high potential to impact the

world �  But they need data which is not always available �  Open data is often the most promising source to start

making quick impact �  Eventual aim should be to scale innovations with

other data sources

2

Page 3: AI for Smart City Innovations with Open Data (tutorial)

What to Expect: Tutorial Objectives

Tutorial on 27 July 2015 @ IJCAI 2015

�  The aim of the tutorial is to ¡  Make early and experienced researchers aware, and equip them to create, societal innovations

with AI techniques like semantics, knowledge representation, data integration, machine learning, planning, scheduling, logic, trust and agents, and open data, that is increasingly, readily available, globally from government and other sources.

�  Relation to other tutorials 1.  Tutorial on Composing Web APIs – State of the art and mobile implications, in conjunction with 1st International Conference

on Mobile Software Engineering and Systems (MobiSOFT), held with 39th International Conference on Software Engineering (ICSE), by Biplav Srivastava; Hyderabad, India, June 2, 2014.

2.  Tutorial on AI-Driven Analytics In Traffic Management, in conjunction with International Joint Conference on Artificial Intelligence (IJCAI-13), by Biplav Srivastava, Akshat Kumar, at Beijing, China, Aug 3-5, 2013.

3.  Tutorial describing the traffic space and relevance of AI techniques was held at 26th Conference on Artificial Intelligence (AAAI-12), at Toronto, Canada. Formally called “Tutorial Traffic Management and AI”, by Biplav Srivastava and Anand Ranganathan, its details are at: http://www.aaai.org/Conferences/AAAI/2012/aaai12tutorials.php

4.  Tutorial highlighting planning and scheduling techniques for traffic management was held at ICAPS 2010. Formally called “Planning and Scheduling for Traffic Control” by Scott Sanner. Its details are available at: http://users.cecs.anu.edu.au/~ssanner/Papers/traffic_tutorial.pdf

5.  Tutorial on Open Data in Practice, in conjunction with the World Wide Web (WWW 2012), by Hadley Beeman, in Lyon, France on the 16th of April, 2012. Slides at: http://www.w3.org/2012/Talks/0417-LD-Tutorial/

6.  Tutorial on How to Publish Linked Data on the Web, in conjunction with International Semantic Web Conference (ISWC 2008), by Tom Heath, Michael Hausenblas, Christian Bizer, Richard Cyganiak, Olaf Hartig, Karlsruhe. Slides and video at: http://videolectures.net/iswc08_heath_hpldw/

�  Disclaimer: we are only providing a sample of Smart City space intended to whet audience interest in the available time.

3

Page 4: AI for Smart City Innovations with Open Data (tutorial)

Acknowledgements All my collaborators over last 5 years, and especially those in: �  Government agencies around the world

¡  City: Boston, USA; New York/ New Jersey area, USA; Silicon Valley, USA; Dubuque, IA; Dublin, Ireland, Stockholm, Sweden; Ho Chi Minh City, Vietnam; New Delhi, India; Bengaluru, India; Nairobi, Kenya; Tokyo, Japan

¡  Country: India, Singapore

�  Academia ¡  India: IIT Delhi, IISc CiSTUP, IIIT Delhi, IIT BHU ¡  USA: Boston University, Wright State University, University of Southern California,

Arizona State University ¡  Vietnam: Ho Chi Minh University

�  IBM: Akshat Kumar, Anand Ranganathan, Raj Gupta, Ullas Nambiar, Srikanth Tamilselvam, L V Subramaniam, Chai Wah Wu, Anand Paul, Milind Naphade, Jurij Paraszczak, Wei Sun, Laura Wynter, Olivier Verscheure, Eric Bouillet, Francesco Calabrese, Tsuyoshi Ide, Xuan Liu, Arun Hampapur, Nithya Rajamani, Vivek Tyagi, Rauam Krishnapuram, Shivkumar Kalyanraman, Manish Gupta, Nitendra Rajput, Krishna Kummamuru, Raymond Rudy, Brent Miller, Jane Xu, Steven Wysmuller, Alberto Giacomel, Vinod A Bijlani, Pankaj D Lunia, Tran Viet Huan, Wei Xiong Shang, Chen WC Wang, Bob Schloss, Rosario Usceda-Sosa, Anton Riabov, Magda Mourad, Alexey Ershov, Eitan Israeli, Evgenia Gyana R Parija, Ian Simpson, Jen-Yao Chung, Kohichi Kajitani, Larry L Light, Lisa Amini, Marco Laumanns, Mary E Helander, Milind Naphade, Sebastien Blandin, Takayuki Osogami, Tony R Heritage, Ulysses Mello, Wei CR Ding, Wei CR Sun, Xiang XF Fei, Yu Yuan, Bipin Joshi, Vishalaksh Agarwal, Pallan Madhavan, Ravindranath Kokku, Mukundan Madhavan, Rashmi Mittal, Sandeep Sandha, Sukanya Randhawa, Karthik Vishweshvariah, Guruduth Banavar

For discussions, ideas and contributions. Apologies to anyone unintentionally missed. Material gratefully taken from multiple sources. Apologies if any citation is unintentionally missed. Tutorial on 27 July 2015 @ IJCAI 2015 4

Page 5: AI for Smart City Innovations with Open Data (tutorial)

Outline

�  Motivating Examples �  Basics

¡  AI: Analytics to process data, derive insights and enable action ¡  Smart City

÷  Challenges ÷  Innovation needs – value desired ÷  Critical considerations different from other applications

¡  Open Data ÷  Introduction and issues ÷  Giving semantics for evolution

¡  Access via APIs �  Applications

¡  Open data as disruptor technology: ÷  patents, corruption, citizen engagement

¡  Health ¡  Environment Pollution ¡  Transportation ¡  Tourism

�  Discussion

5 Tutorial on 27 July 2015 @ IJCAI 2015

Page 6: AI for Smart City Innovations with Open Data (tutorial)

Examples

Tutorial on 27 July 2015 @ IJCAI 2015 6

Page 7: AI for Smart City Innovations with Open Data (tutorial)

Tutorial on 27 July 2015 @ IJCAI 2015 7

[India] Ganga – Local Ground Situation @ Varanasi (Assi/ Tulsi Ghats) + Patna

Photos of/ at Assi/ Tulsi Ghat, Varanasi on 25 March 2015 during 1700-1800 Hrs

Assi Ghat post recent cleanup Bathing on Tulsi Ghat

A nullah draining into Ganga A manual powered boat

Photos at Gandhi Ghat, Patna on 18 March 2015 during 1700-1800 Hrs

Page 8: AI for Smart City Innovations with Open Data (tutorial)

Example –River Water Pollution

�  Value – To individuals, businesses, government institutions ¡  Example – Can I take a bath? Will it cause me dysentery? ¡  Example – How should govt spend money on sewage treatment for maximum

disease reduction? �  Data – Quantitative as well as qualitative

¡  Dissolved oxygen, ¡  pH, ¡  … 30+ measurable quantities of interest

�  Access – ¡  Today, little, and that too in water technical jargon ¡  In pdf documents, website

Key Idea: Can we make insights available when needed and help people make better decisions?

8 Tutorial on 27 July 2015 @ IJCAI 2015

Page 9: AI for Smart City Innovations with Open Data (tutorial)

We All See Traffic Daily. An Illustration from Across the Globe

Source: Google map for New York City and New Delhi; Search done on Aug 20, 2010

Characteristics New York City, USA

New Delhi, India

Beijing, China Moscow, Russia Ho Chi Minh City, Vietnam

Sao Paolo, Brazil

1 How is traffic pre-dominantly managed

Automated control, manual control

Manual control

Automated control, manual control

Automated, manual control

Manual control Automated, manual control, Rotation system (# plate based)

2 How is data collected Inductive loops, cops, video, GPS

Traffic surveys, cops

Video, GPS, cops GPS, some video, cops

Traffic surveys, cops Video, GPS, cops

3 How can citizens manage their resources

GPS devices, alerts on radio, web, road signs (variable)

Alerts on radio

alerts on radio, road signs (variable), mobile alerts

GPS, radio, road signs, mobile alerts

Alerts on radio GPS devices, alerts on radio, web

4 Traffic heterogeneity by vehicle types(Low: <10; Medium 10-25; High: >25)

Low High Low Low Medium Low

5 Driving habit maturity (Low: <10 yrs; Medium: 10-20; High: > 20)

High Low Low Low Low Medium

6 Traffic movement Lane driving Chaotic Lane driving Lane driving Chaotic Lane Driving 9

Page 10: AI for Smart City Innovations with Open Data (tutorial)

Example –Traffic Management

�  Value – To individuals, businesses, government institutions ¡  Example – Can I reach office on time? Where to park if I take my car? ¡  Example – How much overt-time does the city need to give today? Where

should I deploy my traffic cops today? ¡  Example – When to service city’s buses?

�  Data – Quantitative as well as qualitative ¡  Volume – traffic count ¡  Speed on road ¡  City events

�  Access – ¡  Today, little and on city websites ¡  Facebook sites

Key Idea: Can we make insights available when needed and help people make better decisions?

10 Tutorial on 27 July 2015 @ IJCAI 2015

Page 11: AI for Smart City Innovations with Open Data (tutorial)

Basics: AI

Tutorial on 27 July 2015 @ IJCAI 2015 11

Page 12: AI for Smart City Innovations with Open Data (tutorial)

Advanced AI Techniques (Analytics) like Planning & Machine Learning make use of data and models to provide insight to guide decisions

Models

Analytics

Data

Insight

Data sources: Business automation

Instrumentation Sensors

Web 2.0 Expert knowledge

“real world physics”

Model: a mathematical or

algorithmic representation of

reality intended to explain or predict some aspect of it

Decision executed automatically or

by people

12 Tutorial on 27 July 2015 @ IJCAI 2015

Page 13: AI for Smart City Innovations with Open Data (tutorial)

Example: Tutorials

�  Are they useful? (Descriptive) ¡  Answering needs an assessment about the event

�  If it happens next time, how many will attend? (Predictive) ¡  Above + Answering needs an assessment about unknowns

(e.g., future) �  Should you attend? (Prescriptive)

¡  Above + Answering needs understanding the goals and current status of the individual

13 Tutorial on 27 July 2015 @ IJCAI 2015

Page 14: AI for Smart City Innovations with Open Data (tutorial)

Analytics Landscape

Degree of Complexity

Com

petit

ive

Adv

anta

ge

Standard Reporting

Ad hoc reporting

Query/drill down

Alerts

Simulation

Forecasting

Predictive modeling

Optimization

What exactly is the problem?

What will happen next if ?

What if these trends continue?

What could happen…. ?

What actions are needed?

How many, how often, where?

What happened?

Stochastic Optimization

Based on: Competing on Analytics, Davenport and Harris, 2007

Descriptive

Prescriptive

Predictive

How can we achieve the best outcome?

How can we achieve the best outcome including the effects of variability?

14 Tutorial on 27 July 2015 @ IJCAI 2015

Page 15: AI for Smart City Innovations with Open Data (tutorial)

Real-World Applications of ICT Follow a Pattern

n Value (from Action, Decisions) – Providing benefits that matter, to people most in need of, in a timely and cost-efficient manner. Going beyond technology to process and people aspects.

n Data + Insights – Available, Consumable with Semantics, Visualization / Analysis

n Access - Apps (Applications), Usability - Human Computer Interface, Application Programming Interfaces (APIs)

Tutorial on 27 July 2015 @ IJCAI 2015 15

Page 16: AI for Smart City Innovations with Open Data (tutorial)

ML Reference

�  WEKA ¡  Website: http://www.cs.waikato.ac.nz/~ml/weka/index.html ¡  WEKA Tutorial:

÷  Machine Learning with WEKA: A presentation demonstrating all graphical user interfaces (GUI) in Weka.

÷  A presentation which explains how to use Weka for exploratory data mining. ¡  WEKA Data Mining Book:

÷  Ian H. Witten and Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques (Second Edition)

÷  http://www.cs.waikato.ac.nz/ml/weka/book.html ¡  WEKA Wiki: http://weka.sourceforge.net/wiki/index.php/Main_Page

�  Jiawei Han and Micheline Kamber, Data Mining: Concepts and Techniques, 2nd ed. �  http://www.kdnuggets.com/2015/03/machine-learning-table-elements.html

Tutorial on 27 July 2015 @ IJCAI 2015 16

Page 17: AI for Smart City Innovations with Open Data (tutorial)

Basics: Smart City

Tutorial on 27 July 2015 @ IJCAI 2015 17

Page 18: AI for Smart City Innovations with Open Data (tutorial)

What is a Smart City?

Tutorial on 27 July 2015 @ IJCAI 2015

Smart city can mean one or more of the following: �  As a resource optimization objective, it is to know and manage a

city's resources using data.

�  As a caring objective, it is about improving standard of life of citizens with health, safety, etc indices and programs.

�  As a vitality objective, it is about generating employment and doing sustainable growth.

A city leadership can choose among these or define their own objective(s) and manage with measurements to pro-actively achieve it

18

See other FAQs at: https://sites.google.com/site/biplavsrivastava/research-1/intelligent-systems/scfaqs

Page 19: AI for Smart City Innovations with Open Data (tutorial)

Tutorial on 27 July 2015 @ IJCAI 2015 19

Cities are traditionally built and governed by independent departments operating as domains of functions

C i t y

I n f r a s t r u c t u r e

D a t a

Water Energy Transport Security Planning Food . . . Science Health ICT

City

Responsibility

Department

Responsibility

Project

Responsibility

Task

Responsibility

Typically lacking holistic view

Ope

rati

onal

Sys

tem

s Before

Page 20: AI for Smart City Innovations with Open Data (tutorial)

Tutorial on 27 July 2015 @ IJCAI 2015 20

D o

IT

An integrated Smarter City Framework – a comprehensive management system across all core systems, will anchor the vision to executable steps

I n f r a s t r u c t u r e

D a t a

City

Responsibility

Department

Responsibility

Project

Responsibility

Task

Responsibility

Ope

rati

onal

Sys

tem

s

C i t y M a n a g e m e n t Analytics, Insight, Visualization, Control Center, etc.

Water Energy Transport Security Planning Food . . . Science Health . . .

D o

W

D o

E

D o

T

D o

S

D o

P

D o

F

D o

. . .

D o

S

D o

H

. . .

B u s i n e s s P r o c e s s e s a n d A p p l i c a t I o n s

Your City

After

Page 21: AI for Smart City Innovations with Open Data (tutorial)

Tutorial on 27 July 2015 @ IJCAI 2015 21

Smarter Cities solution paths leverage a similar approach

Uni

que

valu

e re

aliz

ed

Use of Smarter Cities capabilities

ManageData 1

AnalyzePatterns 2

Optimize Outcomes 3

Integrate service information to improve department operations

Develop integrated view to improve outcomes and compliance

Leverage end-to-end case management to optimize service delivery

Ç Improve service levels È Reduce fraud and abuse

Ç Focus on the citizen Ç Savings from overpayment Ç Assistance with compliance

Ç Integrated case management Ç Automation of citizen support È Reduce operating costs

Page 22: AI for Smart City Innovations with Open Data (tutorial)

Basics: Open Data

Tutorial on 27 July 2015 @ IJCAI 2015 22

Page 23: AI for Smart City Innovations with Open Data (tutorial)

Open Data

�  Open data is the notion that data should not be hidden, but made available to everyone. The idea is not new.

�  Scientific publications follow this: “standing on the shoulders of giants” ¡  Science stands for repeatability of results and

hence, sharing ¡  The scientific community asserts that open

data leads to increased pace of discovery. (See: Ray P. Norris, How to Make the Dream Come True: The Astronomers' Data Manifesto, At http://www.jstage.jst.go.jp/article/dsj/6/0/6_S116/_article, Accessed 2 Apr, 2012)

�  Governments are the new source for open data ¡  Data.gov efforts world-wide; 400+

governmental bodies, including 20+ national agencies, including India, have opened data

¡  In India, additional movement is “Right to Information Act”

Tutorial on 27 July 2015 @ IJCAI 2015 23

Page 24: AI for Smart City Innovations with Open Data (tutorial)

Not to Be Confused With Orthogonal Trend – Big Data

�  Volume �  Variety �  Velocity �  Veracity �  …

Cartoon critical of big data application, by T. Gregorius. http://upload.wikimedia.org/wikipedia/commons/thumb/b/b3/Big_data_cartoon_t_gregorius.jpg/220px-Big_data_cartoon_t_gregorius.jpg

Tutorial on 27 July 2015 @ IJCAI 2015 24

Page 25: AI for Smart City Innovations with Open Data (tutorial)

400+Data Catalogs of Public Data

As on 21 July 2015

Tutorial on 27 July 2015 @ IJCAI 2015 25

Page 26: AI for Smart City Innovations with Open Data (tutorial)

Data.gov (USA)

As on 16 June 2015 Tutorial on 27 July 2015 @ IJCAI 2015

26

Page 27: AI for Smart City Innovations with Open Data (tutorial)

City Level – Chicago, USA

Tutorial on 27 July 2015 @ IJCAI 2015 27 As on 16 June 2015

Page 28: AI for Smart City Innovations with Open Data (tutorial)

Data.gov.in (India)

As on 16 June 2015 Tutorial on 27 July 2015 @ IJCAI 2015

28

Page 29: AI for Smart City Innovations with Open Data (tutorial)

City Level – Buenos Aires, AR

Tutorial on 27 July 2015 @ IJCAI 2015 29 As on 21 July 2015

Page 30: AI for Smart City Innovations with Open Data (tutorial)

Peek into the Future - Amsterdam

http://citydashboard.waag.org/ Tutorial on 27 July 2015 @ IJCAI 2015 30

Page 31: AI for Smart City Innovations with Open Data (tutorial)

Illustration of Levels

Source: http://5stardata.info/

Does Opening Data Make It Reusable? No

1

2

3

4

5

Tutorial on 27 July 2015 @ IJCAI 2015 31

Page 32: AI for Smart City Innovations with Open Data (tutorial)

Linking of Open Data for Reusability

32

Source: http://5stardata.info/

Source: http://lab.linkeddata.deri.ie/2010/star-scheme-by-example/

Tutorial on 27 July 2015 @ IJCAI 2015

Page 33: AI for Smart City Innovations with Open Data (tutorial)

India: Right to Information Act

�  Any citizen “may request information from a "public authority" (a body of Government or "instrumentality of State") which is required to reply expeditiously or within thirty days.” ¡  Passed by Parliament on 15 June 2005 and came fully into force on 13

October 2005. Citation Act No. 22 of 2005 �  Lauded and reviled

¡  Brought transparency ¡  Also,

÷  Increased bureaucracy ÷  Shortcomings in preventing corruption

�  More information ¡  http://en.wikipedia.org/wiki/Right_to_Information_Act ¡  http://rti.gov.in

Tutorial on 27 July 2015 @ IJCAI 2015 33

Page 34: AI for Smart City Innovations with Open Data (tutorial)

Data Quality in Public Data in India

� Right to Information ¡  Not even 1* ¡  Information available to requester, but no one else

� Data.gov.in ¡  2-3* ¡  Available in CSV, etc but not uniquely referenceable

� Open data movements are moving to linked data form for semantics

34 Tutorial on 27 July 2015 @ IJCAI 2015

Page 35: AI for Smart City Innovations with Open Data (tutorial)

Semantics for Published Data

35

Classify data in public domain. Use schema.org as illustration.

¡  Select an area (e.g., food, news events, crime, customs, diseases, …) ¡  Build + disseminate the catalog tags via a website ¡  Encourage publishers to use meta-data tags and enable search

Catalog/ ID

General Logical

constraints

Terms/ glossary

Thesauri “narrower

term” relation

Formal is-a

Frames (properties)

Informal is-a

Formal instance

Value Restrs. Disjointness, Inverse, part-of…

Tutorial on 27 July 2015 @ IJCAI 2015 Credits: Ontologies Come of Age McGuinness, 2001 From AAAI Panel 99 – McGuinness, Welty, Uschold, Gruninger, Lehmann Plus basis of Ontologies Come of Age – McGuinness, 2003

Page 36: AI for Smart City Innovations with Open Data (tutorial)

�  Abstract:  This  document  describes  a  core  ontology  for  organiza7onal  structures,  aimed  at  suppor7ng  linked-­‐data  publishing  of  organiza7onal  informa7on  across  a  number  of  domains.  It  is  designed  to  allow  domain-­‐specific  extensions  to  add  classifica7on  of  organiza7ons  and  roles,  as  well  as  extensions  to  support  neighbouring  informa7on  such  as  organiza7onal  ac7vi7es.  

1.  Introduc7on  2.  Conformance  3.  Namespaces  4.  Overview  of  ontology  5.  Design  notes  6.  Notes  on  style  7.  Organiza7onal  structure  

7.1  Class:  Organiza7on  7.1.1  Property:  subOrganiza7onOf  7.1.2  Property:  transi7veSubOrganiza7onOf  7.1.3  Property:  hasSubOrganiza7on  7.1.4  Property:  purpose  7.1.5  Property:  hasUnit  7.1.6  Property:  unitOf  7.1.7  Property:  classifica7on  7.1.8  Property:  iden7fier  7.1.9  Property:  linkedTo  

7.2  Class:  FormalOrganiza7on  7.3  Class:  Organiza7onalUnit  7.4  Notes  on  formal  organiza7ons  7.5  Notes  on  organiza7onal  hierarchy  7.6  Notes  on  organiza7onal  classifica7on  

8.  Repor7ng  rela7onships  and  roles  8.1  Class:  Membership  

8.1.1  Property:  member  8.1.2  Property:  organiza7on  8.1.3  Property:  role  8.1.4  Property:  hasMembership  8.1.5  Property:  memberDuring  8.1.6  Property:  remunera7on  

8.2  Class:  Role  8.2.1  Property:  roleProperty  

8.3  Property:  hasMember  8.4  Property:  reportsTo  8.5  Property:  headOf  8.6  Discussion  

9.  Loca7on  9.1  Class:  Site  

9.1.1  Property:  siteAddress  9.1.2  Property:  hasSite  9.1.3  Property:  siteOf  9.1.4  Property:  hasPrimarySite  9.1.5  Property:  hasRegisteredSite  9.1.6  Property:  basedAt  

9.2  Property:  loca7on  10.  Projects  and  other  ac7vi7es  

10.1  Class:  Organiza7onalCollabora7on  11.  Historical  informa7on  

11.1  Class:  ChangeEvent  11.1.1  Property:  originalOrganiza7on  11.1.2  Property:  changedBy  11.1.3  Property:  resultedFrom  11.1.4  Property:  resul7ngOrganiza7on  

A.  Change  history  B.  Acknowledgments  C.  References  

C.1  Norma7ve  references  C.2  Informa7ve  references   http://www.w3.org/TR/vocab-org/ Tutorial on 27 July 2015 @ IJCAI 2015

Illustration: W3C Organization

36

Page 37: AI for Smart City Innovations with Open Data (tutorial)

Usage of W3C’s Org Ontology – Community Directory

@prefix skos: <http://www.w3.org/2004/02/skos/core#> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . @prefix vcard: <http://www.w3.org/2006/vcard/ns#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix dir: <http://dir.w3.org/directory/schema#> . @prefix directory: <http://dir.w3.org/directory/orgtypes/> . @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix gr: <http://purl.org/goodrelations/v1#> . @prefix org: <http://www.w3.org/ns/org#> . <> foaf:primaryTopic <#org> . <#org> a org:Organization, dir:Organization, gr:BusinessEntity, vcard:Organization ; rdfs:label "International Business Machines" ; gr:legalName "International Business Machines" ; vcard:organization-name "International Business Machines" ; skos:prefLabel "International Business Machines" ; dir:isOrganizationType directory:commercial ; vcard:url <http://www.ibm.com> ; vcard:logo <http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png> ; rdfs:comment """International Business Machines Corporation (NYSE: IBM), or IBM, is an American multinational technology and consulting corporation, with headquarters in Armonk, New York, United States. IBM manufactures and markets computer hardware and software, and offers infrastructure, hosting and consulting services in areas ranging from mainframe computers to nanotechnology.""" . <#org> org:siteAddress <#address-1NewOrchardRoad+Armonk+UnitedStates> . <#address-1NewOrchardRoad+Armonk+UnitedStates> a vcard:VCard, vcard:Address ; vcard:street-address "1 New Orchard Road " ; vcard:locality "Armonk " ; vcard:country-name "United States" ; vcard:region "New York" ; vcard:postal-code "10504-1722" .

37 Tutorial on 27 July 2015 @ IJCAI 2015

Page 38: AI for Smart City Innovations with Open Data (tutorial)

Still Confused on Semantics? Start with Linked Data Glossary

Tutorial on 27 July 2015 @ IJCAI 2015 38

Page 39: AI for Smart City Innovations with Open Data (tutorial)

Open Data References

�  Concept ¡  Open Data, At http://en.wikipedia.org/wiki/Open_data, ¡  Open 311, At http://open311.org/ ¡  Catalog of Open Data, At http://datacatalogs.org/dataset ¡  Data City Exchange: http://www.imperial.ac.uk/digital-city-exchange

�  India specific ¡  Open data report in India, At http://cis-india.org/openness/publications/ogd-report

�  Standards ¡  W3C, At http://www.w3.org/2011/gld/ ¡  5 Star Linked Data ratings, At http://www.w3.org/DesignIssues/LinkedData.html

�  Applications and ecoystems ¡  Introduction to Corruption, Youth for Governance, Distance Learning Program, Module 3, World Bank

Publication. Accessed on June 15th 2011, At http://info.worldbank.org/etools/docs/library/35970/mod03.pdf

¡  Dublinked, At http://dulbinked.ie

39 Tutorial on 27 July 2015 @ IJCAI 2015

Page 40: AI for Smart City Innovations with Open Data (tutorial)

Basic: Access via APIs

Tutorial on 27 July 2015 @ IJCAI 2015 40

Page 41: AI for Smart City Innovations with Open Data (tutorial)

Business  

Source: Bessemer Venture Partners 2012

Business Capabilities as Services are being via APIs and delivered as-a-service, allowing Businesses to engage with Clients and Partners with speed at Scale

41 Tutorial on 27 July 2015 @ IJCAI 2015 41

Page 42: AI for Smart City Innovations with Open Data (tutorial)

Example: API Registry

42

As on 16 July 2015

Page 43: AI for Smart City Innovations with Open Data (tutorial)

API Example http://www.programmableweb.com/api/sabre-instaflights-search

43 Tutorial on 27 July 2015 @ IJCAI 2015

Page 44: AI for Smart City Innovations with Open Data (tutorial)

A Composition (Mashup) Example

44 Tutorial on 27 July 2015 @ IJCAI 2015

Page 45: AI for Smart City Innovations with Open Data (tutorial)

REST v/s Web Services?

45

REST •  support limited integration styles, and

involves fewer decisions on architectural alternatives

•  This simplifies client-side integration steps (at the cost of lessening automation in system evolution); more focus on do-it-yourself

Source: Pautasso et al, RESTful Web Services vs. “Big” Web Services: Making the Right Architectural Decision, WWW 2008 45

Page 46: AI for Smart City Innovations with Open Data (tutorial)

Example: Open 311 (http://open311.org/)

Tutorial on 27 July 2015 @ IJCAI 2015

�  Refers to non-emergency events like graffiti, garbage, down trees, abandoned car, … ¡  Not human life threatening ¡  60+ cities support it world-wide

46

Page 47: AI for Smart City Innovations with Open Data (tutorial)

Scaling with Open 311

Tutorial on 27 July 2015 @ IJCAI 2015 47

Page 48: AI for Smart City Innovations with Open Data (tutorial)

Discovering Open 311 of a City

Tutorial on 27 July 2015 @ IJCAI 2015

�  http://311api.cityofchicago.org/open311/discovery.json �  Result

{"changeset":"2012-09-14T08:00:00-05:00”, "contact":"Contact [email protected] for assistance", "key_service":"Visit http://test311api.cityofchicago.org/open311 to request an API Key", "endpoints": [{"specification":"http://wiki.open311.org/GeoReport_v2", "url":"http://311api.cityofchicago.org/open311/v2", "changeset":"2012-09-14T08:00:00-05:00”, "type":"production","formats":["text/xml","application/json"]}, {"specification":"http://wiki.open311.org/GeoReport_v2", "url":"http://test311api.cityofchicago.org/open311/v2", "changeset":"2012-09-14T08:00:00-05:00” , ”type”:"test","formats":["text/xml","application/json"]}]}

48

Page 49: AI for Smart City Innovations with Open Data (tutorial)

Demonstration: Open 311

Tutorial on 27 July 2015 @ IJCAI 2015

�  List of services ¡  http://311api.cityofchicago.org/open311/v2/services.json ¡  Result

[{"service_code":"4ffa4c69601827691b000018","service_name":"Abandoned Vehicle","description":"Abandoned vehicles are taken to auto pound 3S or 3N where they are -- if not redeemed by the owners -- sold for scrap.","metadata":true,"type":"batch","keywords":"code:SKA","group":"Streets & Sanitation"}, {"service_code":"4ffa9cad6018277d4000007b","service_name":"Alley Light Out","description":"One or more alley lights out, on a wooden pole in the alley itself, are reported under this service request type. Important information needed when reporting alley lights out includes: the exact address that the light/lights are behind, how many lights are out, and if the light(s) are completely out or if they blink on and off intermittently. Alley light repairs are done during the day when the lights are not on, so this information is essential to expedite the repair work.","metadata":true,"type":"batch","keywords":"code:SFA","group":"Transportation"},

…]

�  Details of a service ¡  http://311api.cityofchicago.org/open311/v2/services/4ffa4c69601827691b000018.json ¡  Result

{"service_code":"4ffa4c69601827691b000018", "attributes": [{"variable":true,"code":"FQSKA1", "datatype":"singlevaluelist","required":false,"order":1, "description":"Vehicle Make/Model", "values": [{"key":"ASVEAV","name":"(Assembled From Parts,Homemade)"}, {"key":"HOMDCYL","name":"(Homemade Motorcycle, Moped.Etc.)"}, {"key":"HMDETL","name":"(Homemade Trailer)"}, …] ...]}

49

Page 50: AI for Smart City Innovations with Open Data (tutorial)

Demonstration: Open 311

Tutorial on 27 July 2015 @ IJCAI 2015

�  http://311api.cityofchicago.org/open311/v2/services/4ffa9cad6018277d4000007b.json �  Result

{"service_code":"4ffa9cad6018277d4000007b", "attributes": [{"variable":true,"code":"ISTHELI2", "datatype":"singlevaluelist","required":true,"order":1, "description":"Is the light located in your alley or the street?", "values":[{"key":"ALLEY","name":"Alley"}, {"key":"STREET","name":"Street"}]},

{"variable":true,"code":"POLEWORM", "datatype":"singlevaluelist","required":true,"order":2, "description":"Is the pole wooden or metal?", "values":[{"key":"METAL","name":"Metal"}, {"key":"WOODEN","name":"Wooden"}]}, {"variable":true,"code":"ISTHELI3", "datatype":"singlevaluelist","required":true,"order":3, "description":"Is the light directly behind this address?", "values":[{"key":"NO","name":"No - Light Not Directly Behind Address"} ,{"key":"YES","name":"Yes - Light Directly Behind Address"}]}, {"variable":true,"code":"A511OPTN", "datatype":"string","required":false, "datatype_description":"Enter number as 999-999-9999","order":4, "description":"Input mobile # to opt-in for text updates. If already opted-in, add mobile # to contact info."}]}

50

Page 51: AI for Smart City Innovations with Open Data (tutorial)

Chicago: Service Tracking

Tutorial on 27 July 2015 @ IJCAI 2015 51

Page 52: AI for Smart City Innovations with Open Data (tutorial)

Example: Application over Open Data (Chicago)

Tutorial on 27 July 2015 @ IJCAI 2015 52

Page 53: AI for Smart City Innovations with Open Data (tutorial)

Example of API Design– APIs for Temperature at Conference Location

�  API examples ¡  Get temperature (input: current, last, input instant) ¡  Get temperature interval (input: day) ¡  Get average temperature (input: time range)

�  REST or web-service �  Semantic annotation on input and output

Tutorial on 27 July 2015 @ IJCAI 2015 53

Page 54: AI for Smart City Innovations with Open Data (tutorial)

Every citizen is a potential city event sensor •  Citizen notices 311 event worth reporting •  Reports event using mobile

•  Launches mobile application •  Browses recent already-reported events •  Creates new event report

•  [Is pre-enabled or gets any needed credentials to report event] •  Identifies service type for new event •  Shares location using mobile device (coordinates) •  Can add location annotations (road, district, city) and description

•  Get confirmation of submission •  Get updates on service request

Extreme Personalization

=

Location Intelligence

Empowered Citizen

+

SocialAnalytics

+ +

ALLGOV SCENARIO: CROWDSOURCING 311* EVENT REPORTING

Tutorial on 27 July 2015 @ IJCAI 2015 54

Page 55: AI for Smart City Innovations with Open Data (tutorial)

Browsing Services in One’s City: Mary M. can look at the 311 services her city provides On selecting the icon, •  She sees a small set of categories

(health, building, traffic, cityimage, others) around which all the city’s services are grouped.

•  She can look at a list of services and check out the agencies involved

•  If there has been a change in agency responsible or new services added for an agency, she can note that directly

Browsing Services in Other Cities: Her colleagues from another city are visiting. She may want to bring a window (instantiate an app with browse city pattern) to look at what that city offers to their citizens [Alternatively, if she is travelling to another city, she may be interested to know how that city does compared to her’s, by which agency, etc.] On selecting the icon, •  See sees a small set of familiar categories (health, building, traffic,

cityimage, others) regardless of what the city calls its services •  She can look at a list of services and check out the agencies

involved

If her city does something different, she can show that to her colleagues in her or other cities.

Tutorial on 27 July 2015 @ IJCAI 2015 55

Page 56: AI for Smart City Innovations with Open Data (tutorial)

A Demonstration of AllGov Pattern with Open 311

Tutorial on 27 July 2015 @ IJCAI 2015 56

Page 57: AI for Smart City Innovations with Open Data (tutorial)

Applica7on  Pa\ern  

¡ What  is  it?:  A  pa\ern  is  any  applica7on  using  APIs,  with  some  informa7on  generalized  (i.e.,  removed  and  parameterized)  

¡ Business  Value:  A  pa\ern    ÷ standardizes  the  usage  experience  by  promo7ng  similar  behavior  (for  users)    ÷ simplifies  applica7on  development  by  templa7zing  API  interac7ons  (for  developers)  

÷ serves  as  the  organiza7on’s  memory  of  the  best-­‐prac7ces  in  developing  a  class-­‐of-­‐applica7ons  even  when  the  specific  APIs  may  not  be  relevant  (for  business)  

¡ Key  Technical  Issue  ÷ What  pa\erns  should  one  build  ?  Theore7cally,  there  exists  a  trivial  method  to  blindly  generate  a  pa\ern  from  any  applica7on.  Any  pa\ern  development  process  has  to  do  be\er  than  this  baseline.  

÷ How  should  the  pa\erns  be  used  in  prac7ce?  ÷ Building  a  tool-­‐enabled  process  around  Pa\ern-­‐based  programming  

Tutorial on 27 July 2015 @ IJCAI 2015 57

Page 58: AI for Smart City Innovations with Open Data (tutorial)

Applica7on  Pa\ern  

¡ Approach  followed  in  AllGov  ÷  Common  steps  taken  by  a  role  player  is  a  candidate  pa\ern  ÷  Common  steps  that  can  be  executed  in  the  same  infrastructure  is  a  candidate  pa\ern  

¡  Pa\ern  1:  Browse  city  services  pa\ern  [User  Role:  Govt.  Dept  Admin;  Environment:  PRODUCTION  system]  ÷  find  a  city's  services  ÷  find  a  service's  defini7on  ÷  find  services  of  a  par7cular  high-­‐level  category  (example:  building,  graffi7,  ...)  

¡  Pa\ern  2:  Create  service  request  pa\ern  [User  Role:  Developer;  Environment:  TEST  system]  ÷  Browse  city  services  ÷  Browse  raised  city  service  requests  ÷  Create  a  new  service  request  

¡  Pa\ern  3:  Create  service  request  pa\ern  [User  Role:  General  ci7zen  of  a  par,cular  City;  Environment:  PRODUCTION  system]  

÷  Browse  city  services  ÷  Browse  raised  city  service  requests  ÷  Create  a  new  service  request  

Tutorial on 27 July 2015 @ IJCAI 2015 58

Page 59: AI for Smart City Innovations with Open Data (tutorial)

AllGov Scenario Deconstruction (flows)

Customer Mobile

AllGov City Services

1

2

External IBM Client

browse events get recent events

Request confirmation

get service types create request

Post location coordinates

Post details on Event, location

3 Notify service completed

P1, P1+

P2, P3

Tutorial on 27 July 2015 @ IJCAI 2015 59

Page 60: AI for Smart City Innovations with Open Data (tutorial)

Video Demonstration - AllGov

Tutorial on 27 July 2015 @ IJCAI 2015 60

Page 61: AI for Smart City Innovations with Open Data (tutorial)

Applications with Open Data

Tutorial on 27 July 2015 @ IJCAI 2015 61

Page 62: AI for Smart City Innovations with Open Data (tutorial)

Open Data as Disruptor Technology

Tutorial on 27 July 2015 @ IJCAI 2015

Is happening in areas where information can disrupt status-quo �  Granting and Defending Patents �  Detecting Corruption �  Citizen Engagement

62

Page 63: AI for Smart City Innovations with Open Data (tutorial)

Patents

Tutorial on 27 July 2015 @ IJCAI 2015

�  Are for novel, useful and non-obvious ideas �  Which have not been known read (read: published

and available in public domain)

63

Page 64: AI for Smart City Innovations with Open Data (tutorial)

IP (Patent) Grants and Defense

Tutorial on 27 July 2015 @ IJCAI 2015 64

Page 65: AI for Smart City Innovations with Open Data (tutorial)

Example: Nutmeg

Tutorial on 27 July 2015 @ IJCAI 2015 65

http://www.tkdl.res.in/

Page 66: AI for Smart City Innovations with Open Data (tutorial)

Corruption

Tutorial on 27 July 2015 @ IJCAI 2015 66

Page 67: AI for Smart City Innovations with Open Data (tutorial)

Corruption - “the misuse of public office for personal gains”

* Source: http://cpi.transparency.org/cpi2012/results/

Corruption afflicts both public and corporate services world wide. It is known that it has a significant negative impact on the growth of economies and hence, is universally considered undesirable.

Corruption : “Monopoly + Discretion – Accountability” (Klitgaard, Robert E. Controlling corruption. Berkeley: U. of California Press, 1988)

Tutorial on 27 July 2015 @ IJCAI 2015 67

Page 68: AI for Smart City Innovations with Open Data (tutorial)

A Nation’s Competitiveness and Corruption Perception

Don’t Go Hand-in-Hand

For Promoting Growth, Corruption Perception has

to be Removed

Page 69: AI for Smart City Innovations with Open Data (tutorial)

Latin America’s Competitiveness

Tutorial on 27 July 2015 @ IJCAI 2015 69

Source: http://americasmi.com/en/expertise/articles-trends/page/the-cost-of-corruption-to-latin-americas-competitiveness

Page 70: AI for Smart City Innovations with Open Data (tutorial)

Some Key Questions Related to Corruption

�  Exchange of money: can a service for which the customer does not pay a fee (free service) be termed corrupt? Or conversely, can a corrupt practice only happen if the customer pays for a service?

�  Human agents: can a service be corrupt if the agent delivering the service is not a human but an automated agent?

�  Contention for resources: can corruption happen if delivering it requires no contention of resources? Alternatively, if resources are scarce, will an objective way of allocating them help remove corruption?

Tutorial on 27 July 2015 @ IJCAI 2015 70

Page 71: AI for Smart City Innovations with Open Data (tutorial)

Metamodel – Expressing Key Concepts for Corruption

Provider  

Ac7vity  Process  

Task   Decision  Inputs   Outputs  Escala7on  

Requestor  

0..1   *  

1   +  

Person  

Organiza7on  

1  

1  

1  

1  

1   1   1  

*  Process  Instance   *  

Ac7vity  Instance  

1  

+  

Execu7on  Time  

Execu7on  Cost  

1  

1  1  

1  

1  

Tutorial on 27 July 2015 @ IJCAI 2015 71

A Computational Model for Corruption Assessment, Nidhi Rajshree, Nirmit V. Desai and Biplav Srivastava, IJCAI 2013 Workshop on Semantic Cities, Beijing, 2013

Page 72: AI for Smart City Innovations with Open Data (tutorial)

Framework Evaluation, by Example National Registration - Kenya

1. Submit supporting documents

2. Validate

docs

4. Handover serialized App Form

11. App signed and stamped by

Chief Asst. Officer

12. Submit documents to

NRB

13. Verify identity of the

applicant

14. Process ID Card

17. Collect

ID Card

- Proof of birth - Proof of citizenship - Proof of residence

5. Fill and submit application form

- Form 101 - Form 136 A - Form 136 C

6. Take finger prints

7. Click photograph for

ID card

8. Handover the waiting card

10. Submit documents

to Chief

3. Vetting

15. Send ID card to the

Registration Office - Additional proof of

residence

Ancestral home town is a border district or age >> 18

Insufficient documents

Sufficient documents

9. Receive waiting card and wait for processing

16. Receive ID Card from

NRB

Citiz

en

Reg

istr

atio

n O

ffice

r

Satisfied

Not satisfied

Vett

ing

Com

mitt

ee

Ch. A

sst.

Offi

cer

NR

B O

ffice

r

Page 73: AI for Smart City Innovations with Open Data (tutorial)

National Registration

Kenya India (Aadhar) USA (Social Security)

•  The decision node, 3 - vetting, and the activity, 13 - verify identity, are discretionary with no clear mechanism on how to accomplish them.

•  In contrast, the checks for documents having been submitted are objective.

•  There is no Service Level Agreement (SLA) for the process.

•  The ID process is monopolistic since only a single authority

•  (registration office) can process it. •  The process has little reviewability

and low visibility since there is no escalation mechanism.

•  18 Proofs of Identity (PoI) and 33 Proofs of Address (PoA) documents are permitted for making the request.

•  The process also allows discretion by allowing at- tested documents from high-level officials.

•  The cost and time limits for the service are prescribed.

•  The process, however, can only be handled by a single agency creating a monopoly.

•  In SS, a clear list of documents proving US citizenship (or legal residence), age and identity is listed.

•  There is little room for discretion because no category allows a signed attestation by a high-level official to be acceptable

•  The cost and time limits for the service are prescribed.

•  The process, however, can only be handled by a single agency creating a monopoly.

Tutorial on 27 July 2015 @ IJCAI 2015 73

Page 74: AI for Smart City Innovations with Open Data (tutorial)

Framework Evaluation, by Example

International Driving Permit (IDP)

1. Submit supporting documents

2. Validate

docs

5. Handover Appl Form

10. Stamp and sign

the IDP

13. Collect IDP

- Driver’s license - Passport - Air tickets - VISA

5. Fill and submit application form

- Form CMV1

+ 4. DL Address change process

8. Verify

applicants driving skills

DL address not under RTO jurisdiction

Insufficient documents

DL address under RTO jurisdiction

Citiz

en

Fron

t Des

k O

ffice

r

Satisfied

Not satisfied

Insp

ecto

r

Reg

iona

l Tr

ansp

ort

Offi

cer

3. Validate address

7. Send applicant for DL Test

6. Verify DL issuance

date

9. Send application to Regional Transport

Officer

11. Send IDP to front

desk officer

12. Receive IDP from Regional

Transport Officer

Address has not changed

DL issued within 3 months

Address has changed

DL issued within more than 3 months

Tutorial on 27 July 2015 @ IJCAI 2015 74

Page 75: AI for Smart City Innovations with Open Data (tutorial)

International Driving License

India (IDP) USA (AAA) •  Service execution cost is specified

(of Rs 500) but not service execution time given.

•  There is no escalation mechanism •  The check whether all documents

have been sub- mitted is objective. •  The IDP is monopolistic since only

a single authority (RTO) can process it.

•  The process has little reviewability and low visibility since there is no escalation mechanism.

Procedure involves filling a form online, visiting the office of an authorized agency with a valid state-issued driver’s license, photos and fees, and getting the permit. Here, there are multiple agencies to process the request and the prerequisite driver license can be verified objectively (e.g., with social security databases). •  No monopoly •  Objective criteria

Tutorial on 27 July 2015 @ IJCAI 2015 75

Page 76: AI for Smart City Innovations with Open Data (tutorial)

Tackling Corruption

Tackling corruption pro-actively: �  Open Government Data

¡  Increases transparency hence increasing the risk of being caught (i.e., increasing accountability) in the act of corruption

¡  Makes benchmarking by Service Level Agreements (SLAs) possible

�  Process Redesign ¡  Ensures a robust process design reducing corruption hotspots ¡  Formalizes adequate data needs, reduces monopoly & discretion

�  Automation ¡  Automation needs outcomes and inputs to be formally defined ¡  Reduces discretion, forces data formalization (input, output, outcome)

Corruption : “Monopoly + Discretion – Accountability” (Klitgaard, Robert E. Controlling corruption. Berkeley: U. of California Press, 1988)

Tutorial on 27 July 2015 @ IJCAI 2015 76

Page 77: AI for Smart City Innovations with Open Data (tutorial)

Corruption – It’s All Around

Tutorial on 27 July 2015 @ IJCAI 2015 77

Page 78: AI for Smart City Innovations with Open Data (tutorial)

Citizen Engagement

Tutorial on 27 July 2015 @ IJCAI 2015

�  Reporting problems �  Finding help �  Generally: People-as-sensors

78

Page 79: AI for Smart City Innovations with Open Data (tutorial)

Chicago: Food Poisoning

Tutorial on 27 July 2015 @ IJCAI 2015 79

http://www.foodbornechicago.org/

Page 80: AI for Smart City Innovations with Open Data (tutorial)

Hottest Trend in Public Health

Tutorial on 27 July 2015 @ IJCAI 2015 80

Page 81: AI for Smart City Innovations with Open Data (tutorial)

Health

Details: Africa (2014-), India (2013-)

Tutorial on 27 July 2015 @ IJCAI 2015 81

Page 82: AI for Smart City Innovations with Open Data (tutorial)

Two Tales from (Public) Health

Cutting-edge Technical Progress •  Enormous improvement in our

understanding of diseases. E.g., Computational epidemiology

•  Enormous advances in treating diseases are being made ÷  We are living longer - A baby girl born

in 2012 can expect to live an average of 72.7 years, and a baby boy to 68.1 years. This is 6 years longer than the average global life expectancy for a child born in 1990. (Source: WHO 2014 Health Statistics)

•  Data on disease outbreaks is more available than ever before thanks to open data movement (E.g., data.gov, data.gov.in)

Stone-age Ground Reality �  Half of the top 20 causes of deaths

in the world are infectious diseases, and maternal, neonatal and nutritional causes, while the other half are due to noncommunicable diseases (NCDs) or injuries. (Source: WHO 2014 Health Statistics)

�  Worse – Indifference, mismanagement in response to communicable diseases - late response to known diseases, in known period of the year ¡  E.g.: Japanese Encephalitis (JE) has been

prevalent for ~3 decades in some parts of India killing 600+ every year

¡  District level health experience is not reused over time and in similar regions

Tutorial on 27 July 2015 @ IJCAI 2015 82

Page 83: AI for Smart City Innovations with Open Data (tutorial)

Ebola Data

Crowd sourced

Online

National Government

International Bodies

Tutorial on 27 July 2015 @ IJCAI 2015 83

Page 84: AI for Smart City Innovations with Open Data (tutorial)

Case Study: Dengue (Mosquito-borne) �  Overall cost of a Dengue case is US$ 828 (Sabchareon et al 2012). �  From 9 countries in 1960s, it has spread to more than 110 countries now

�  Prevention methods COMMUNITY 1.   Mosquito Coils & Candles: The use of mosquito coils, candles & vapor mats indoors and outdoors of homes to combat

mosquitoes. 2.   Window screens & Bed Nets: The use of window screens in homes and bed nets in bedrooms to keep mosquitos out. 3.   Insecticide Application: Application of insecticide to kill mosquitos that invade homes and surrounding areas. 4.   Larviciding at Home: Application of larvicide in homes to kill larvae that live in stagnant water breeding sites like small

ponds, gutters, cisterns, barrels, jars, and urns. 5.   Household/Community Cleanup: Organize cleanups within communities in the surrounding housing areas and

individual homes to recycle potential breeding sites like discarded plastic bottles, cans, old tyres, and any trash that can hold water for mosquitoes to breed in.

GOVERNMENT 6.   Surveillance For Mosquitoes: Conduct periodical surveillance in hotspot areas and other communities to look for signs of

mosquitoes. 7.   Medical Reporting: To collate and compile reports of dengue cases and statistics to prioritize and focus dengue and vector

mosquito control efforts and actions for best results. 8.   Effective Publicity & Campaigns: To foster and champion effective campaigns amongst communities and create adequate

public awareness of combating dengue. 9.   Enforcement: Support and enforce the public and communities to practice effective dengue vector elimination under

existing laws and implement new laws as appropriate for public health. 10.  Insecticide Fogging: Conduct fogging in areas that have mosquitoes and dengue outbreak hotspots to kill adult mosquitoes. 11.   Public Education:  Foster, promote, and participate in public education in schools and  all possible public meeting places to

inform communities how to eliminate dengue vector mosquitoes, recognize early symptoms of the disease, and proper medical care and reporting.

CORPORATE 12.  Education: To undertake community service initiatives and campaigns through marketing expertise and the media of TV,

radio, and newspapers. 13.  PR/CSR: To use public relations and customer service relations to reach communities on the fight against dengue. 14.  Adult Mosquito Traps: To provide adult mosquito traps and other measures within the work areas to protect employees

and workers from mosquitoes bites that transmit dengue. 15.   Mosquito Repellants: Provide mosquito repellants to employees and workers within the work areas for further protection. 16.   Mosquito Control Materials, Methods, and Agents:  To provide the tools to the public and government that are

necessary for dengue mosquito vector control like pesticides, biocontrol agents,  mosquito traps, repellants, and other means  to prevent dengue by eliminating the mosquito vectors.

WHO, 2013, Dengue Control. At http://www.who.int/Denguecontrol/research/en/, Accessed 21 June 2013. Entogenex, 2013, Integrated Mosquito Management. At http://www.entogenex.com/what-is-integrated-mosquito- management.html, Accessed 21 June 2013. Tutorial on 27 July 2015 @ IJCAI 2015 84

Page 85: AI for Smart City Innovations with Open Data (tutorial)

So, Do We Control Dengue

Effectively? NO

Source: http://nvbdcp.gov.in/den-cd.html

Data for India •  Increasing

number of states every year

•  No consistent reduction of cases

1"

10"

100"

1000"

10000"

100000"

C" C" C" C" C" C"

2008" 2009" 2010" 2011" 2012" 2013*"

Andhra"Pradesh"

Arunachal"Pradesh"

Assam"

Bihar"

Cha9sgarh"

Goa"

Gujarat"

Haryana"

Himachal"Pd."

J"&"K"

Jharkhand"

Karnataka"

Kerala"

Madhya"Pd."

Meghalaya"

Maharashtra"

Manipur"

Mizoram"

Nagaland"

Orissa"

Punjab"

Rajasthan"

Sikkim"

Tamil"Nadu"

Tripura"

UPar"Pradesh"

UPrakhand"

West"Bengal"

A&"N"Island"

Chandigarh"

Tutorial on 27 July 2015 @ IJCAI 2015 85

Page 86: AI for Smart City Innovations with Open Data (tutorial)

(ROI) Metrics

�  Expense for disease control ¡  $/person spent: How much money (in $) is spent for a given method divided by the population

of the region. Lower is better.

�  Impact of a disease control method ¡  Reduction: What is the magnitude of reduction in disease cases due to a method, expressed as

a percentage, in a time period (e.g., year, disease season)? Higher is better. ¡  Cases/ person: How many reported cases of a disease occurred in a time period divided by the

population of the region when a method was adopted? Lower is better.

�  Cost-effectiveness: ¡  Cases / $: how many cases were reported for a disease per dollar spent on controlling it in a

given time period? Lower is better.

86 Tutorial on 27 July 2015 @ IJCAI 2015 86

Page 87: AI for Smart City Innovations with Open Data (tutorial)

Major Methods to Tackle Dengue

�  M1: Public awareness campaigns: to prevent conditions conducive to disease propagation, to improve reporting

�  M2: Chemical Control: Aerosol space spray �  M3: Biological Control: Use of biocides �  M4: Distributing equipments: bednets, insecticide-

treated curtains �  M5: Vaccination against the disease

87 Tutorial on 27 July 2015 @ IJCAI 2015 87

Page 88: AI for Smart City Innovations with Open Data (tutorial)

Dengue Control Case Studies from Literature

88

•  An approach may use 1 or more method(s)

•  They incur different costs per person

•  Their efficacy is subject to various factors

Still, can we reuse these results in new areas?

Tutorial on 27 July 2015 @ IJCAI 2015 88 Details:

Vandana Srivastava and Biplav Srivastava, Towards Timely Public Health Decisions to Tackle Seasonal Diseases With Open Government Data , International Workshop on the World Wide Web and Public Health Intelligence (W3PHI-2014), AAAI 2014

Page 89: AI for Smart City Innovations with Open Data (tutorial)

Challenge: Prescribe Methods to Use for a Hypothetical, Illustrative Area - Sundarpur

�  City is Sundarpur ¡  Made up of 10 districts ¡  10,000 people in each district.

�  Disease control ¡  Each district allocates $10,000 per annum to prevent disease. ¡  The city has a district-level health administrator per district and then an

overall citywide public health administrator.

�  What approach/ method should the district health officer use? What should the city health officer recommend? ¡  a mix of control methods to produce the maximum reduction feasible. ¡  Default option is to do nothing. This is unfortunately followed a lot!

89 Tutorial on 27 July 2015 @ IJCAI 2015 89

Page 90: AI for Smart City Innovations with Open Data (tutorial)

Cost-benefits for Different Approaches

90

* represents assumption made to compensate for missing data.

Tutorial on 27 July 2015 @ IJCAI 2015 90

Page 91: AI for Smart City Innovations with Open Data (tutorial)

Prescription for Sundarpur

�  Best tactical option for administrators at Sundarpur (at district and the whole city level) ¡  is O1_A1 since it brings the maximum reduction. ¡  If the administrators are interested to cover the maximum number of people in the given

budget, the best method is still O1_A1. ¡  If the administrators are interested to show maximum reduction in cases for a pocket of the

city (sub- district level which may be more prone to the disease), they may choose O4_A4 but it costs maximum and thus can be perceived as taking resources away from the not- directed areas.

�  Strategic option ¡  Select top-2 (O1_A1 and O2_A2), and try them in 5 districts each in one year. It hedges risk of

variability between Sundarpur and old location of previous studies. ¡  Based on efficacy, decide the single best option for Sundarpur in subsequent year. ¡  She may also use the vaccine option only when the disease outbreak is above certain

threshold.

91 Tutorial on 27 July 2015 @ IJCAI 2015 91 Details:

Vandana Srivastava and Biplav Srivastava, Towards Timely Public Health Decisions to Tackle Seasonal Diseases With Open Government Data , International Workshop on the World Wide Web and Public Health Intelligence (W3PHI-2014), AAAI 2014

Page 92: AI for Smart City Innovations with Open Data (tutorial)

New Data Practices

�  Find correlation among methods (positive or negative) ¡  We assumed independence ¡  Needs: Historic Data, Experiment Design

�  Learn rate of return for approaches and methods (new combinations not tried in health literature) ¡  Need: Collect data on efficacy of method individually

�  Find similarity among regions ¡  Data Need: Spatio-temporal modeling/ STEM

�  Multi-objective optimization ¡  Examples: Effectiveness of approach, Reduction of case, people coverage ¡  Needs: Data about approaches tried historically

92 Tutorial on 27 July 2015 @ IJCAI 2015 92

Page 93: AI for Smart City Innovations with Open Data (tutorial)

Request to Medical Community on Data

�  Report both cost and effectiveness of approaches and methods ¡  Overlooking one hampers reuse of results

�  Interact with AI community to learn and try mixed approaches that reduce cost and improve overall effectiveness ¡  All combinations cannot be tried on the ground due to practical

constraints ¡  Get more effective approaches rolled out faster targeted to new

regions

93 Tutorial on 27 July 2015 @ IJCAI 2015 93

Page 94: AI for Smart City Innovations with Open Data (tutorial)

Environment Pollution

Details: Singapore (2012-2013), Varanasi (2015-)

94 Tutorial on 27 July 2015 @ IJCAI 2015

Page 95: AI for Smart City Innovations with Open Data (tutorial)

Water Cycle (aka Hydrological Cycle)

Source: Economist, May 20, 2010 95 Tutorial on 27 July 2015 @ IJCAI 2015

Page 96: AI for Smart City Innovations with Open Data (tutorial)

Fresh Water: Supply and Demand

Source: Economist, May 20, 2010

Supply Demand

96 Tutorial on 27 July 2015 @ IJCAI 2015

Page 97: AI for Smart City Innovations with Open Data (tutorial)

Water Challenges

�  Increasing demand due to ¡  Population ¡  Changing water-intensive lifestyle ¡  Industrial growth

�  Shrinking supplies ¡  Erratic rains due to climate change ¡  Sewage / effluent increase

�  Poor management ¡  Below cost, unsustainable, pricing ¡  Delayed or neglected maintenance

Water is the next flash point for wars

97 Tutorial on 27 July 2015 @ IJCAI 2015

Page 98: AI for Smart City Innovations with Open Data (tutorial)

[India] Ganga – Local Ground Situation @ Varanasi (Assi/ Tulsi Ghats) + Patna

Photos of/ at Assi/ Tulsi Ghat, Varanasi on 25 March 2015 during 1700-1800 Hrs

Assi Ghat post recent cleanup Bathing on Tulsi Ghat

A nullah draining into Ganga A manual powered boat

Photos at Gandhi Ghat, Patna on 18 March 2015 during 1700-1800 Hrs

98 Tutorial on 27 July 2015 @ IJCAI 2015

Page 99: AI for Smart City Innovations with Open Data (tutorial)

Example –River Water Pollution

�  Value – To individuals, businesses, government institutions ¡  Example – Can I take a bath? Will it cause me dysentery? ¡  Example – How should govt spend money on sewage treatment for maximum

disease reduction? �  Data – Quantitative as well as qualitative

¡  Dissolved oxygen, ¡  pH, ¡  … 30+ measurable quantities of interest

�  Access – ¡  Today, little, and that too in water technical jargon ¡  In pdf documents, website

Key Idea: Can we make insights available when needed and help people make better decisions?

99 Tutorial on 27 July 2015 @ IJCAI 2015

Page 100: AI for Smart City Innovations with Open Data (tutorial)

Value of Water Pollution Data

�  Government for business decisions ¡  Source attribution ¡  Sewage treatment ¡  Public Health

�  Individuals for personal decisions ¡  Bathing (Religious, Lifestyle) ¡  Recreation ¡  Community practices

100 Tutorial on 27 July 2015 @ IJCAI 2015

Page 101: AI for Smart City Innovations with Open Data (tutorial)

Use-case: Individual

101

�  Name: which bathing site should one use? ¡  Based on distance (cost of travel), risk of

disease, exposure to pollutants, suitability to occasion

�  Total sites in Varanasi (ghats): 87 ¡  Popular: 5 ¡  #1 religious rites (puja):

Dashashwamedh Ghat ¡  Cremation (non-bathing) ghats: 2;

Manikarnika and Harishchandra Ghat ¡  Bathing ghats: All – cremation = 85

41.  Lali Ghat 42.  Lalita Ghat 43.  Mahanirvani Ghat 44.  Mana Mandira Ghat 45.  Manasarovara Ghat 46.  Mangala Gauri Ghat 47.  Manikarnika Ghat 48.  Mehta Ghat 49.  Meer Ghat 50.  Munshi Ghat 51.  Nandesavara Ghat 52.  Narada Ghat 53.  Naya Ghat 54.  Nepali Ghat 55.  Niranjani Ghat 56.  Nishad Ghat 57.  Old Hanumanana Ghat 58.  Pancaganga Ghat 59.  Panchkota 60.  Pandey Ghat 61.  Phuta Ghat 62.  Prabhu Ghat 63.  Prahalada Ghat 64.  Prayaga Ghat 65.  Raj Ghat built by Peshwa Amrutrao 66.  Raja Ghat / Lord Duffrin bridge /

Malaviya Bridge 67.  Raja Gwalior Ghat 68.  Rajendra Prasad Ghat 69.  Ram Ghat 70.  Rana Mahala Ghat 71.  Rewan Ghat 72.  Sakka Ghat 73.  Sankatha Ghat 74.  Sarvesvara Ghat 75.  Scindia Ghat 76.  Shivala Ghat 77.  Shitala Ghat 78.  Sitala Ghat 79.  Somesvara Ghat 80.  Telianala Ghat 81.  Trilochana Ghat 82.  Tripura Bhairavi Ghat 83.  Tulsi Ghat 84.  Vaccharaja Ghat 85.  Venimadhava Ghat 86.  Vijayanagaram Ghat 87.  Samne Ghat

1.  Mata Anandamai Ghat 2.  Assi Ghat 3.  Ahilya Ghat 4.  Adi Keshava Ghat 5.  Ahilyabai Ghat 6.  Badri Nayarana Ghat 7.  Bajirao Ghat 8.  Bauli /Umaraogiri / Amroha Ghat 9.  Bhadaini Ghat 10.  Bhonsale Ghat 11.  Brahma Ghat 12.  Bundi Parakota Ghat 13.  Chaowki Ghat 14.  Chausatthi Ghat 15.  Cheta Singh Ghat 16.  Dandi Ghat 17.  Darabhanga Ghat 18.  Dashashwamedh Ghat 19.  Digpatia Ghat 20.  Durga Ghat 21.  Ganga Mahal Ghat (I) 22.  Ganga Mahal Ghat (II) 23.  Gaay Ghat 24.  Gauri Shankar Ghat 25.  Genesha Ghat 26.  Gola Ghat 27.  Gularia Ghat 28.  Hanuman Ghat 29.  Hanumanagardhi Ghat 30.  Harish Chandra Ghat 31.  Jain Ghat 32.  Jalasayi Ghat 33.  Janaki Ghat 34.  Jatara Ghat 35.  Karnataka State Ghat 36.  Kedar Ghat 37.  Khirkia Ghat 38.  Shri Guru Ravidass Ghat[5] 39.  Khori Ghat 40.  Lala Ghat

Source: http://en.wikipedia.org/wiki/Ghats_in_Varanasi

Note: ghats are specialities of most cities along Ganga – Haridwar, Allahabad, Patna

Tutorial on 27 July 2015 @ IJCAI 2015 101

Page 102: AI for Smart City Innovations with Open Data (tutorial)

Pollu7on  Example:  Leather  Tanneries  in  Kanpur,  India  

•  > 700 tanneries in Kanpur –  Employing > 100,000 people –  Bringing > USD 1B revenue

•  Discharge water after leather processing to river or Sewage treatment plants (STPs) –  Requirement

•  Must have their own treatment facility •  Or, have at least chrome recovery unit

–  But don’t due to costs which is a burden to main operations •  Installation •  Operations : electricity, manpower, technology upgrade, …

–  State pollution board is supposed to do inspections but doesn’t do effectively •  Government’s STPs do not process chrome, the main pollutant •  98 tanneries banned in Feb 2015 by National Green Tribunal; more

threatened

102 Tutorial on 27 July 2015 @ IJCAI 2015

Page 103: AI for Smart City Innovations with Open Data (tutorial)

India/Ganga – Very Little Data Data.gov.in https://data.gov.in/catalog/water-quality-data-river-ganga

Sr.  No.   Sta,on-­‐Loca,on   Distance  in  Kms.  

Dissolved  Oxygen  during  1986  (mg/l)  

Biological  Oxygen  Demand  in  1986  (mg/l)  

Dissolved  Oxygen  during  2011  (mg/l)  

Biological  Oxygen  demand  during  2011  (mg/l)  

1   Rishikesh   0   8.1   1.7   7.6   1.4  

2   Hardwar  D/s   30   8.1   1.8   7.4   1.6  

3   Garhmukteshwar   175   7.8   2.2   7.5   1.7  

4   Kannauj  U/S   430   7.2   5.5   7.9   1.7  6   Kanpur  U/S   530   7.2   7.2   7.7   3.3  7   Kanpur  D/S   548   6.7   8.6   7.6   3.8  

8   Allahabad  U/S   733   6.4   11.4   7.8   5.3  

9   Allahabad  D/S   743   6.6   15.5   7.8   5.1  

10   Varanasi  U/S   908   5.6   10.1   8   2.9  

11   Varanasi  D/S   916   5.9   10.6   8   4.3  12   Patna  U/S   1188   8.4   2   7   1.8  13   Patna  D/S   1198   8.1   2.2   7.1   2.5  

103 Tutorial on 27 July 2015 @ IJCAI 2015

Page 104: AI for Smart City Innovations with Open Data (tutorial)

Creek Watch – Crowd Sourced Water Information Collection

As on 14 Oct 2014

104 Tutorial on 27 July 2015 @ IJCAI 2015

Page 105: AI for Smart City Innovations with Open Data (tutorial)

Location: http://creekwatch.researchlabs.ibm.com/call_table.php

~3120 data points in 4 years from around the world

As on 14 Oct 2014

105 Tutorial on 27 July 2015 @ IJCAI 2015

Page 106: AI for Smart City Innovations with Open Data (tutorial)

Analytics: Potential use cases S. No.

Stakeholder

Use case Data Analytical techniques

1 IT Identifying and removing outliers, data validation

Sensor data Data mining (outlier detection)

2 Individual Which bathing site to use? Sensor data, ghat data

Rule-based decision support

3 Individual/ Economy

What crops can I grow that will flourish in available water?

Sensor data, crop data

Distributed data integration, co-relation

4 Institution Determine trends/anomalies in pollution levels

Sensor data, weather data

Time series analysis, anomaly detection

5 Institution Attribute source of pollution at a location

Sensor data, demographics, industry data

Physical modeling, inversion

6 Institution Sewage treatment strategy and operational planning

Sensor data, demographics data, STP data

Multi-objective optimization

7 Institution Promoting wildlife/ dolphins Sensor data, wildlife data

Rule-based decision support

106 Tutorial on 27 July 2015 @ IJCAI 2015

Page 107: AI for Smart City Innovations with Open Data (tutorial)

Air Pollution Analytical Models – A Birds Eye View

107

w

w

w

w

w

w

w

w

w

Meteorological Data

Hills High rise buildings

Forest

Water storage

Canyons

Sources of Air Pollutants Topography

Polluting gas emission

Industrial parks Metal firms

Motor Vehicle pollution

Industrial Wastes

Forest fires

wind

smog

rain

Temperature Humidity

Air  Pollution  Dispersion  Analytical  Models  

What  is  the  air  pollution  level  at  X  (e.g.,  Jurong)?  

Singapore

Tutorial on 27 July 2015 @ IJCAI 2015

Page 108: AI for Smart City Innovations with Open Data (tutorial)

Background

�  Environmental issues such as Air Pollution and Quality (APQ) are a prominent concern for citizens and cities.

�  To monitor them and take timely action, environmental engineers collect selected data from field sensors at a limited number of locations, extrapolate them for uncovered regions.

�  The algorithms to extrapolate and analyze data are also known as analytical models (AMs).

�  An AM may be appropriate under very specific conditions - terrain type of the region, specific weather conditions, specific classes of pollutants, types of pollutant sources, data sampling rate,etc.

108 Tutorial on 27 July 2015 @ IJCAI 2015

Page 109: AI for Smart City Innovations with Open Data (tutorial)

Issues Faced by Environmentalists

109

Identifying  the  right  model  based  on  the  Contextual  information  of  pollutant  sources,  met  data  and  topography  information  

CBP

BLP

Aurora

CALPUFF CTSCREEN

ADSM CTDM

AERMOD CALINE4

HIWAY2

CAR-FMI

AEROPOL

GRAL

GATOR

OSPM

STAR-CD

ARIA-Local

PBM

TAPM

SCREEN3

SPRAY

AERSCREEN

What  is  the  air  pollution    level  in  Jurong?  

Missing  Data/  NA  for    SpeciHic  ource/region/time  (precision)  

CALPUFF  

CALMET Data

Volume Source Data

User Specified

Deposition Velocities

User Specified Chemical

conversion rates Complex Terrain

Receptor Data File

MET Data

Identifying  the    Requirements  of  Execution  platform  

Execution  

Platform

Data Controllers

And formatters

Executables

CALPuff Executable

Data from raw sources

Data Access through CDOM

Tutorial on 27 July 2015 @ IJCAI 2015

Page 110: AI for Smart City Innovations with Open Data (tutorial)

110

Before and After

As-Is State Future State

Obtaining instance data

Collected from non-integrated infrastructure

Collected with integrated infrastructure

Discovering models Manual Automatic recommendation based on context (location and time)

Executing models with available data

Manual In-context invocation for select (supported) models; Manual for the rest

Note: This is a common problem in e-science . Other use-cases world-wide are in bioinformatics and geology .

Tutorial on 27 July 2015 @ IJCAI 2015

Page 111: AI for Smart City Innovations with Open Data (tutorial)

The Solution

111

1 2 3

1 Extract entities and relations from document of these AM’s

2 Use the Domain models [a semantic model] to these AM’s to produce Semantic models of the AM’s

3 Integrate with the Discovery system using the association definitions

Tutorial on 27 July 2015 @ IJCAI 2015

Kalapriya Kannan, Biplav Srivastava, Rosario U.-Sosa, Robert J. Schloss, and Xiao Liu, SemEnAl: Using Semantics for Accelerating Environmental Analytical Model Discovery, Big Data Analytics (BDA 2014), New Delhi, India, Dec 20-23, 2014.

Page 112: AI for Smart City Innovations with Open Data (tutorial)

Sample: Semantic Model – Air Pollution Concepts

112

Key Concepts: Pollutants, Pollutant Sources, Effects and Indicator.

Key Concepts Deep

Taxonomical Characterization

Reused from Existing

Scribe base

Tutorial on 27 July 2015 @ IJCAI 2015

Page 113: AI for Smart City Innovations with Open Data (tutorial)

Smarter Transportation

Details: Boston (2012), New York, (2014), India – Delhi, Bangalore (2011-2015)

Tutorial on 27 July 2015 @ IJCAI 2015 113

Page 114: AI for Smart City Innovations with Open Data (tutorial)

Press on the IBM SCC Boston team work: 1. Boston Globe, June 29, 2012 http://www.boston.com/business/technology/articles/2012/06/29/ibm_gives_advice_on_how_to_fix_boston_traffic__first_get_an_app/ (Alternative: http://bostonglobe.com/business/2012/06/28/ibm-gives-advice-how-fix-boston-traffic-first-get-app/goxK84cWB9utHQogpsbd1N/story.html) 2. Popular Science, 2 July 2012 http://www.popsci.com/technology/article/2012-07/bostons-ibm-built-traffic-app-merges-multiple-data-streams-predict-ease-congestion 3. Others: National Public Radio (USA), and a range of local TV stations on the work.

SCC Boston team with Mayor on June 27, 2012

Team at work – Source: Boston Globe article

114 Tutorial on 27 July 2015 @ IJCAI 2015

Page 115: AI for Smart City Innovations with Open Data (tutorial)

Boston  Transporta7on  :  Before  State  

GPS  

Manual  

Regional  

Video  

Road  Sensors  

Lots  of  Instrumenta7on…   Not  enough  interconnec7on…   Unexploited  Intelligence…  

Much  Data  Isolated  in  Silos  

Mul7ple  Disconnected  Camera  Networks  

Inaccessible  Data  

Manual  Opera7ons  

Insufficient  Data  

"    Boston  is  forward-­‐        thinking  &  progressive  "    Boston  recognizes        climate  &  traffic  goals        are  interconnected      Boston  is  na)onally  recognized  for  innova)on  

115 Tutorial on 27 July 2015 @ IJCAI 2015

Page 116: AI for Smart City Innovations with Open Data (tutorial)

Ecosystem  Roadmap  

Ci,zens  

Sharing   Analyzing   Forward  Thinking   Consumer  Value  

Unlocking  

Smarter Transportation Ecosystem

Industry  

Academics  

Government  

Induc,ve  Loop  Data  

Applications

Platform

Data

Ideas

Pneuma,c  Tube  Data  

Manual    Count    Data  

Automated  Data  Transfer  

Online  Access  to  Aggregated  Data  

Privacy  Considera,ons  

Ci,zen  Online  Access  

Smarter  Traffic  Infrastructure  

Environmental  Es,mates  

Mul,ple  Visualiza,ons  

City  Benchmarks  

Exploit  Video  Camera  

Advanced  Visualiza,ons  

Exploit  More  Data  Sources  

Advanced  Analy,cs  

Deliverables  "    Running  Prototype  "    Recommenda7ons  

116 Tutorial on 27 July 2015 @ IJCAI 2015

Page 117: AI for Smart City Innovations with Open Data (tutorial)

Common Model

Standards Aligned, Uniform format, Uniform Error Semantics

Mapping to Source

Data Transformation

Data Source Metadata

A Snapshot of Common Model and Mapping to Data Sources

Source Models

117 Tutorial on 27 July 2015 @ IJCAI 2015

Page 118: AI for Smart City Innovations with Open Data (tutorial)

Result  1:  Publicly  Available  Data  for  Mul7ple  Consumers  

"      Many  data  sources,  various  loca7ons  &  7mes  "      Stakeholders  can  access  data  easily  &  intui7vely    

"      Locate  available  data  sources  "      Zoom  in  to  areas  of  interest  "      Obtain  data    "      Drill  down  to  traffic  pa\erns  "      Assess  environmental  factors    "      See  what  happens  in  real  7me  

Researchers  

Prac77oners  

Planners  

Engineers  

Residents  

118 Tutorial on 27 July 2015 @ IJCAI 2015

Page 119: AI for Smart City Innovations with Open Data (tutorial)

•   Assign  different  traffic        light  pa\erns  for        different  streets,  7mes  •   Schedule  public  works        projects  to  minimize        traffic  impact  •   Detect  changes  in        traffic  pa\erns  to  drive        policy  changes        (parking,  lanes,  street)  •   Assess  traffic  impact  of        new  landmarks  •   Inform  businesses,          developers  

Result  2:  Street  Classifica7on  Based  on  Traffic  Volume  

Commuting

Going Home

Anomaly

Early-Bird

Night Owl Busy

119 Tutorial on 27 July 2015 @ IJCAI 2015

Page 120: AI for Smart City Innovations with Open Data (tutorial)

Result  3:  Birds-­‐Eye  View  of  City  Traffic  from  Aggregated  Data  

120 Tutorial on 27 July 2015 @ IJCAI 2015

Page 121: AI for Smart City Innovations with Open Data (tutorial)

New York: All Taxi Rides

taxi.imagework.com NYC taxi trips originate at various NY airport terminals (JFK and LGA) over the holiday season (Nov 15th to Dec 31st). Data Source: NYC Taxi & Limousine Commission Taxi Trip & Fare Data 2013 Stats 173.2M Rows | 28.85GB Tools Hadoop | Mapbox | Leaflet | jQuery | d3 | polyline | MapQuest Open Directions API

http://taxi.imagework.com/

121 Tutorial on 27 July 2015 @ IJCAI 2015

Page 122: AI for Smart City Innovations with Open Data (tutorial)

New York: Single Taxi Ride

http://nyctaxi.herokuapp.com/

122 Tutorial on 27 July 2015 @ IJCAI 2015

Page 123: AI for Smart City Innovations with Open Data (tutorial)

Top Cities Tourists Visit (by money spent)

Figure Courtesy: MasterCard 2014 Global Destination Cities Index, At http://newsroom.mastercard.com/digital-press-kits/mastercard-global-destination-cities-index-2014/

Top cities are getting money from tourists that countries in Middle East/ Africa / Latin America are planning by 2020

123 Tutorial on 27 July 2015 @ IJCAI 2015

Page 124: AI for Smart City Innovations with Open Data (tutorial)

Journey Planning with Open Data

124 Tutorial on 27 July 2015 @ IJCAI 2015

Page 125: AI for Smart City Innovations with Open Data (tutorial)

Promoting Public Transportation: Before and After We Seek

Many cities around the world, and especially in India and emerging ones, are getting their transportation infrastructure in shape.

–  They have multiple, fragmented, transportation agencies in a region (e.g., city) –  They do not have instrumentation on their vehicles, like GPS, to know about their

operations in real-time –  Schedule of public transportation is widely available in semi-structured form. They

are also beginning to invest in new, novel, sensing technologies –  Cities give SMS-based alerts about events on the road. Our approach seeks to accelerate time-to-value for such cities.

Kind of Information Today Available to Bus User

With IRL-Transit+ Benefit

Bus Schedule (static) Available online and pamphlets

Available from IT-enabled devices( low-cost phones, smart phones, web)

Increase accessibility

Bus Schedule Changes (dynamic)

No information Infer from city updates Increase information

Analytics (Bus Selection Decision Support)

No information Will be available (Transit)

Increase information

Standardization of information

No support Will be supported (SCRIBE, Transit)

Increase information’s interoperability

125 Tutorial on 27 July 2015 @ IJCAI 2015

Page 126: AI for Smart City Innovations with Open Data (tutorial)

A Quick Review of Related Work ¡  Bay Area, USA has : http://511.org

÷  Multi-agency public authorities consortium, has advanced instrumentation ÷  It is the model to replicate

§  Google has state-of-the-art from any non-public organization. It has separate services ¡  Maps for driving guidance ¡  Transit for public transport, more than 1 mode ¡  Gaps:

÷  Considers only time, not other factors like frequency, fare and waiting time ÷  Does not integrate across their services for different mode categories ÷  Does not publish their data

¡  Acknowledgement: We use their GTFS format to consolidate schedule data

§  Many experimental systems with capabilities less than Google, ¡  DMumbai: Go4Mumbai (portal)- A http://www.go4mumbai.com/ ¡  Delhi: Disha on DIMTS (local agency) website by IIT-D, Mumbai Navigator by IIT-B; links no longer work

§  Shortest route finding algorithms from mapping companies

126 Tutorial on 27 July 2015 @ IJCAI 2015

Page 127: AI for Smart City Innovations with Open Data (tutorial)

Journey Planning Problem �  Invariant Inputs:

¡  The person ÷  has a vehicle (e.g., car), and ÷  can also walk short distances

¡  The city has taxis, buses, metros, autos, rickshaws ÷  Buses and metros have published routes, frequency and stops ÷  Autos and rickshaws can be available at stands, or opportunistically, on the road ÷  Taxis can be ordered over the phone

�  Input: ¡  A person wants to travel from place A to B

�  Output ¡  Suggest which mode or combination of modes to select

�  Observation: Using preferences over factors that matter to users to keep commuting convenient, while making best use of available public and para-transit commute methods

127 Tutorial on 27 July 2015 @ IJCAI 2015

Page 128: AI for Smart City Innovations with Open Data (tutorial)

Background: Public Transportation Schedule Information

�  Is widely available for public transportation agencies around the world

�  Gives the basic, static, information about transportation service

�  Usually in semi-structured format with varying semantics

�  Can have errors, missing data

Delhi Bus and Metro Data

128 Tutorial on 27 July 2015 @ IJCAI 2015

Page 129: AI for Smart City Innovations with Open Data (tutorial)

Multi-Mode Commuting Recommender in Delhi And Bangalore

Highlights •  Published data of multiple authorities used; repeatable process • Multiple modes searched •  Preference over modes, time, hops and number of choices supported; more extensions, like fare possible •  Integration of results with map as future work; already done as part of other projects, viz. SCRIBE-STAT

129 Tutorial on 27 July 2015 @ IJCAI 2015

Page 130: AI for Smart City Innovations with Open Data (tutorial)

Solution Steps �  Use the widely available schedule information from individual operators

(agencies) �  Clean and consolidate it across agencies and modes to get a multi-modal

view for the region ¡  Optionally: Convert it into a standard form ¡  Optionally: Enhance (fuse) it with any real-time updates about services

for the region �  Perform what-if analysis on consolidated data

¡  Path finding using Djikstra’s algorithm ¡  Analyses can be pre-determined, analyses can also be user-created

and defined �  Make analysis results available as a service

¡  On any device ¡  To any subscriber

130 Tutorial on 27 July 2015 @ IJCAI 2015

Page 131: AI for Smart City Innovations with Open Data (tutorial)

Handling Dynamic Updates �  Invariant Inputs:

¡  The person ÷  has a vehicle (e.g., car), and ÷  can also walk short distances

¡  The city has taxis, buses, metros, autos, rickshaws ÷  Buses and metros have published routes, frequency and stops ÷  Autos and rickshaws can be available at stands, or opportunistically, on the road ÷  Taxis can be ordered over the phone

�  Input: ¡  A person wants to travel from place A to B ¡  [Optional] City provides updates on ongoing events, some may affect

traffic �  Output

¡  Suggest which mode or combination of modes to select

�  Observation: Using preferences over factors that matter to users to keep commuting convenient, while making best use of available public and para-transit commute methods

City Notifications as a Data Source for Traffic Management, Pramod Anantharam, Biplav Srivastava, in 20th ITS World Congress 2013, Tokyo

131 Tutorial on 27 July 2015 @ IJCAI 2015

Page 132: AI for Smart City Innovations with Open Data (tutorial)

Number of SMS messages for bus stops in Delhi for 2 years (Aug 2010 – Aug 2012)*

•  344 stops with updates •  3931 total stops

* using Exact Matching

132 Tutorial on 27 July 2015 @ IJCAI 2015

Page 133: AI for Smart City Innovations with Open Data (tutorial)

IRL – Transit in Aug 2012

Key Points • SMS message from city •  Event and location identified •  Impact assessed •  Impact used in search

133 Tutorial on 27 July 2015 @ IJCAI 2015

Page 134: AI for Smart City Innovations with Open Data (tutorial)

Increase Accessibility and Availability of Bus Information to Passengers

Kind of Information

Today Available to Bus Users

With Solution over Phone

Mysore ITS (for reference)*

Benefit

Bus Schedule (static) Available online and pamphlets

Available from low-cost phones (Spoken Web – Static)

Available online and pamphlets

Increase accessibility

Bus Schedule Changes (dynamic)

No information today

Will be available (Spoken Web - Human)

No information but in plan

Increase information

Bus Location No information today

Will be available (GPS)

Will be available (GPS)

Increase information

Bus Condition No information today

Will be available (Spoken Web - Human)

No information today

Increase information

Analytics (Bus Selection Decision Support)

No information today

Will be available (Transit)

No information but in plan

Increase information

Last –mile Connectivity to/ from nearest stop

No information today

Will be available (Spoken Web - Human)

No information today Increase information

Standardization of information

No support Will be supported (SCRIBE, Transit)

Some support due to GPS

Increase information’s interoperability

* Opinion based on only public information; Accurate as of Jan 2014. Spoken Web is an Interactive IVR technology. SCRIBE is a ontology models for city events.

134 Tutorial on 27 July 2015 @ IJCAI 2015

Page 135: AI for Smart City Innovations with Open Data (tutorial)

A Flexible Journey Plan Pushing the Boundaries: Information to Commuters to Reach Destination in All Eventuality

Pilots  running  in  Dublin,  Ireland  

Tutorial on 27 July 2015 @ IJCAI 2015 135 Docit: An Integrated System for Risk-Averse Multi-Modal Journey Advising, Adi Botea, Michele Berlingerio, Stefano Braghin Eric Bouillet, Francesco Calabrese, Bei Chen Yiannis Gkoufas, Rahul Nair, Tim Nonner, Marco Laumanns, IBM Technical Report, 2014

Page 136: AI for Smart City Innovations with Open Data (tutorial)

•  Traffic simulation is a promising tool to do what-if analysis impacting traffic demand, supply or every-day business decisions •  What is the congestion if everyone takes out their vehicles? •  What is the impact if buses daily failure rate doubles? •  What happens if visitors constituting 20% of city traffic come for an event?

•  However, simulators need to be setup with realistic road network, traffic patterns and decision choices

•  Open data is an important source for •  Road network (e.g., Open Street Maps) •  Creating pattern (e.g., vehicle

Origin-Destination pairs, accidents) •  Framing and interpreting decision choices

Using Open Data with Traffic Simulation

Tutorial on 27 July 2015 @ IJCAI 2015 136

Page 137: AI for Smart City Innovations with Open Data (tutorial)

New Delhi Area Selection

Area selected from openstreetmap.org with (top)(bottom)(left)(right) co-ordinates as (28.6022)(28.5707)(77.1990)(77.2522) for our experiment.

Tutorial on 27 July 2015 @ IJCAI 2015 137

Page 138: AI for Smart City Innovations with Open Data (tutorial)

Office Timing Change Decision Choices

Tutorial on 27 July 2015 @ IJCAI 2015 138 Last second of morning commute by different strategies

Page 139: AI for Smart City Innovations with Open Data (tutorial)

Traffic References

�  Tutorial on AI-Driven Analytics In Traffic Management, in conjunction with International Joint Conference on Artificial Intelligence (IJCAI-13), Biplav Srivastava, Akshat Kumar, at Beijing, China, Aug 3-5, 2013 (tutorial-slides).

�  Tutorial on Traffic Management and AI, in conjunction with 26th Conference of Association for Advancement of Artificial Intelligence (AAAI-12), Biplav Srivastava, Anand Ranganathan, at Toronto, Canada, July 22-26, 2012 (tutorial-slides).

�  Making Public Transportation Schedule Information Consumable for Improved Decision Making, Raj Gupta, Biplav Srivastava, Srikanth Tamilselvam, In 15th International IEEE Annual Conference on Intelligent Transportation Systems (ITSC 2012), Anchorage, USA, Sep 16-19, 2012.

�  Mythologies, Metros & Future Urban Transport , by Prof. Dinesh Mohan, TRIPP, 2008 �  A new look at the traffic management problem and where to start, by Biplav Srivastava, In 18th ITS

Congress, Orlando, USA, Oct 16-20, 2011. �  Arnott, Richard and K.A. Small, 1994, “The Economics of Traffic Congestion,” American Scientist, Vol.

82, No. 5, pp. 446-455. �  Chengri Ding and Shunfeng Song , Paradoxes of Traffic Flow and Congestion Pricing,

139 Tutorial on 27 July 2015 @ IJCAI 2015

Page 140: AI for Smart City Innovations with Open Data (tutorial)

Smarter Tourism

Details: Europe (2014), India (2014-) https://sites.google.com/site/biplavsrivastava/research-1/intelligent-systems/smart-tourism

140 Tutorial on 27 July 2015 @ IJCAI 2015

Page 141: AI for Smart City Innovations with Open Data (tutorial)

Why Tourism Matters

�  Pros ¡  Promotes services jobs ¡  Helps upgrade infrastructure ¡  Gives alternative revenue source to government beyond

traditional agriculture and manufacturing ¡  Helps take local culture world-wide ¡  Promotes country image

�  Cons ¡  Can lead to environmental impact if not planned well ¡  Can dilute local traditions and culture if unplanned

141 Tutorial on 27 July 2015 @ IJCAI 2015

Page 142: AI for Smart City Innovations with Open Data (tutorial)

World Tourism in Numbers

Key Points •  In 2013, >1 billion people spent overnight in

another city and spent > 1 trillion USD •  France has highest visitors, USA gets the most

money and Chinese spend the most •  Among oldest civilizations (> 5K years) in the world,

of China, Egypt and India, only China gets and sends tourists in top-5 by numbers and money spent.

•  Tourists go beyond language and history to spend their money for novel experiences

Tables Courtesy: http://en.wikipedia.org/wiki/World_Tourism_rankings (Accessed 20 Oct, 2014)

142 Tutorial on 27 July 2015 @ IJCAI 2015

Page 143: AI for Smart City Innovations with Open Data (tutorial)

Possible Strategy to Promote Tourism

�  Increase quality of experience for USPs using better information availability. Examples: ¡  Increase Service quality – Information on what is happening

and what to expect, when, at what cost; make it easy to consume offerings

¡  Remove barriers to travel and spending - Remove perception of lack-of-safety, increase transparency about supporting services like roads, hospitals, taxis

�  Promote domestic tourism in addition to international tourism ¡  Helps natives inculcate service-industry culture, build capacity

143 Tutorial on 27 July 2015 @ IJCAI 2015

Page 144: AI for Smart City Innovations with Open Data (tutorial)

City Concierge (CC): Serving People by Design

�  Target users ¡  Citizens wanting to know more about their city ¡  Travellers planning to visit new cities with memorable experiences ¡  People (e.g., business, government) wanting to compare cities

�  Group information along a small set of easy-to-follow categories ¡  We selected - Traffic, health, building, city image, others ¡  Easy to change to any set of categories

�  Languages supported – English, Portuguese, Spanish, German ¡  Easy to extend to any

2nd place winner in Europe’s CitySDK App Hackathon in June 2014 Details: http://www.slideshare.net/biplavsrivastava/city-concierge-presentation10june2014

144 Tutorial on 27 July 2015 @ IJCAI 2015

Page 145: AI for Smart City Innovations with Open Data (tutorial)

Serving People by Design �  Target users: Citizens, Travellers, People

Citizens, Travellers Most events – Helsinki Most open service requests - Lisbon

145 Tutorial on 27 July 2015 @ IJCAI 2015

Page 146: AI for Smart City Innovations with Open Data (tutorial)

Check Services of Your Favorite City – Chicago, in example

Lisbon (in Portuguese) Bonn(in German)

People, Travellers Most city services – Lisbon; Traffic most common category in cities

146 Tutorial on 27 July 2015 @ IJCAI 2015

Page 147: AI for Smart City Innovations with Open Data (tutorial)

CC Design Principles

�  Focus on features that promote usage of city data ¡  Overcoming language barriers ¡  Overcoming API and data diversity barriers ¡  Highlight commonalities, promote comparison

�  Follow standards ¡  CitySDK for tourism events upcoming ¡  Open 311 for city’s non-emergency services and service requests

�  Programming level approach ¡  Overcome (City API) errors to stay useful ¡  Be resource efficient to promote mobile apps ¡  Standardize on output formats

147 Tutorial on 27 July 2015 @ IJCAI 2015

Page 148: AI for Smart City Innovations with Open Data (tutorial)

Prototype: Bharat Khoj – Searching Events on Mobile and Web

148 Tutorial on 27 July 2015 @ IJCAI 2015

Page 149: AI for Smart City Innovations with Open Data (tutorial)

Research Challenges

�  ML Problems ¡  Event attendance prediction ¡  Event recommendation

�  Apply and innovate on analytics (AI) ¡  Handle data ambiguity ¡  Build reusable models

�  Focus on value (services science, AI) ¡  What metrics are being improved? Who are the agents and their

incentives? ¡  What processes will be impacted? How to boost adoption?

�  Build usable systems (software engineering, HCI) ¡  Bug-free, low-footprint, Apps ¡  Test human-comupter interfaces

�  Use (government) open data and publish output it too, preferably in semantically enriched form (data integration, AI)

149 Tutorial on 27 July 2015 @ IJCAI 2015

Page 150: AI for Smart City Innovations with Open Data (tutorial)

Cross Domain City Comparison: Exploring a Pair of Cities

http://city-explorer.mybluemix.net/

150 Tutorial on 27 July 2015 @ IJCAI 2015

Page 151: AI for Smart City Innovations with Open Data (tutorial)

City Comparison Functions

Example:

151 Tutorial on 27 July 2015 @ IJCAI 2015

Page 152: AI for Smart City Innovations with Open Data (tutorial)

Exploring All Cities with Comparable Data

152 Tutorial on 27 July 2015 @ IJCAI 2015

Best

Page 153: AI for Smart City Innovations with Open Data (tutorial)

Discussion

Tutorial on 27 July 2015 @ IJCAI 2015 153

Page 154: AI for Smart City Innovations with Open Data (tutorial)

Smart City Challenges

Tutorial on 27 July 2015 @ IJCAI 2015

�  From resource angle, decrease waste/ inefficiency while improving service delivery to citizens

�  Problems are old but accentuated today by population growth and reducing resources

�  Open Data, Effectiveness of AI Methods hold promise

�  Challenges ¡  Provide value quickly ¡  Use value synergies from different domains (e.g., health,

environment, traffic, corruption …) ¡  Grow to scale

154

Page 155: AI for Smart City Innovations with Open Data (tutorial)

Common (Descriptive) Analytics Patterns with Open Data

Tutorial on 27 July 2015 @ IJCAI 2015

�  Correlation of outcomes across ¡  Data sources in same domain ¡  Different domains

�  Return of investment analysis ¡  Money invested v/s Metrics to measure improvement in

domain ¡  Comparison of performance with history ¡  Comparison of performance with other regions

155

Page 156: AI for Smart City Innovations with Open Data (tutorial)

Helping Publish Good Quality Open Data is Key

Tutorial on 27 July 2015 @ IJCAI 2015

�  Have data policy in place �  Publish with best practices, have semantics, promote reuse

Figure courtesy: http://www.w3.org/TR/2015/WD-dwbp-20150625/

156

Page 157: AI for Smart City Innovations with Open Data (tutorial)

Building Community for Innovations

Tutorial on 27 July 2015 @ IJCAI 2015

�  Multi-disciplinary ¡  In AI ¡  In Computer Science ¡  In science: domain (health, transport, …), techniques (CS, engg.) and

evaluation (public policy, …) �  Multi-stakeholder

¡  Citizens ¡  Government ¡  Academia ¡  Business/ Industry ¡  Non-profits, …

�  Getting to scale is key

157

Page 158: AI for Smart City Innovations with Open Data (tutorial)

Employing All Data – Data Fusion

Tutorial on 27 July 2015 @ IJCAI 2015

�  Open Data is one source ¡  Often easiest to get but with issues (e.g., at aggregate level, with gaps,

imprecise semantics)

�  Social is another promising data ¡  People are anyway generating it (People-as-sensors) ¡  However, social sites have varying data reuse permissions,

license costs, access limits ¡  Big data techniques already being used here

�  Use sensor data if available ¡  Internet of Things (IoT) and big data techniques are relevant ¡  Most prevalent in health, environment and transportation

�  Key is to release the fused data also for reuse

158

Page 159: AI for Smart City Innovations with Open Data (tutorial)

Building a Technical Environment Problem Solving Community

159 Tutorial on 27 July 2015 @ IJCAI 2015

Page 160: AI for Smart City Innovations with Open Data (tutorial)

Thank You

Merci Grazie

Gracias Obrigado

Danke

Japanese

French

Russian

German Italian

Spanish

Portuguese

Arabic

Traditional Chinese

Simplified Chinese

Hindi

Romanian

Korean

Multumesc

Turkish

Teşekkür ederim

English

Dr. Biplav Srivastava, [email protected]://www.research.ibm.com/people/b/biplav/

160 Tutorial on 27 July 2015 @ IJCAI 2015