open local data: challenges and opportunities

WMRO Open Data Event, Birmingham, July 15, 2010

Open Local Data Challenges & opportunities

It’s been a good year for open government datadata.gov.uk, Ordnance Survey, MPs expenses...

But what about local data?


• Frankly it’s a mess



• Sporadically published by central government




• Inaccessible & impenetrable council websites





• Opaque local public bodies and quangos (goodbye RDAs, hello LEPs)






• At best unclear & at worst unusable legal situation







• Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?







• Start with the basics. Who are the councillors, where do they represent and what committees do they sit on?

• How easy it that information to find & reuse?

Simple questions, hard to answer


• How does the budget of my council compare with similar ones?



• What’s the background of the chief executive of my council?




• Is my route to work going to affected by roadworks next week?





• What are the connections between the companies awarded contracts and the councillors & senior management?





• What are the connections between the companies awarded contracts and the councillors & senior management?

• How will cuts in services be decided and what can I do to influence the decisions?

TransparencyA brief example.


• A story in Private Eye



• An investigation by the District Auditor




• A report buried in the nether recesses of the website...





• ...with no relevant heading






• Accessible only as a PDF of a scan of a document






• Accessible only as a PDF of a scan of a document

• Is it any wonder suspicions are raised?

Engagement

Engagement

• Local Authorities now have a duty to engage. This means• Reducing barriers to understanding

and involvement

Engagement


and involvement

• Accessible for all

Engagement


and involvement


• Available to use when, where & how we want to

Engagement


and involvement


• Available to use when, where & how we want to

• Publishing as data means easy to be repurposed – for mobile, for mashups, for offline use (e.g. Postcode Paper)

Equality

Equality

• At the moment, all this information is available... at a cost

Equality


£££

Equality


• Huge asymmetry of information (and thus power)

£££

Equality


• Huge asymmetry of information (and thus power)

• Raises the barriers to involvement, and, critically, to challenge

£££

EfficiencyNow: cumbersome, wasteful, opaque, error prone

Typical local data flows



Council



Council

Govt Dept A

Govt Dept B

Govt Dept C

email

web

upload



Council

Govt Dept A

Govt Dept B

Govt Dept C

email

web

upload

public (maybe)

Enter OpenlyLocal...


• Inspiration was a Manchester project, MCC Work For You. Doing something concrete to solve a problem.



• Screen-scrapes council websites, to turn web pages into structured data. Now over 150 councils done, with basic information on all 434




• Pulls info from about local councils, plus ONS, OS, NPIA, CLG...





• Increasingly working with local authorities to help them publish their information as open data






• All open data – free for reuse, including commercial reuse






• All open data – free for reuse, including commercial reuse

• 100% accessible. Data first; bells & whistles later

RDFJSON

RDFJSONXML

OpenlyLocal: where next?


• More data.


• More data. • Election data


• More data. • Election data✓


• More data. • Election data• Spending data

✓


• More data. • Election data• Spending data

✓✓


• More data. • Election data• Spending data• Where I live...

✓✓


• More data. • Election data• Spending data• Where I live...

✓✓✓


• More data. • Election data• Spending data• Where I live...• Cabinet members

✓✓✓


• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors

✓✓✓


• More data. • Election data• Spending data• Where I live...• Cabinet members• Mayors• Police authorities

✓✓✓



• More councils.

✓✓✓



• More councils.

• More connections.

✓✓✓



• More councils.


• More ways for councils to give us data.

✓✓✓



• More councils.


• More ways for councils to give us data.

• More tie-ups with hyperlocal news/community sites.

✓✓✓

The OpenElectionData project


• Tackling the open local data problem, one set at a time & learning lessons on the way



• Succeed, or fail forward




• No public database of local election results, only a commercial one (subsidized by the Electoral Commission)





• Allows even those with no prior knowledge of linked data/RDF/semantic web (choose fave buzzword here) to take part






• Just need HTML competence






• Just need HTML competence

• Over 20 councils took part, and invaluable lessons learned (see report)

http://data.gov.uk/blog/2897

http://data.gov.uk/blog/2897

<tr rel='openelection:candidacy'>  <th scope="row" rel="openelection:candidate">Keith William Cockroft</th> <td rel="openelection:party" resource="http://openelectiondata.org/id/parties/6">Labour</td>  <td property="openelection:candidateVoteCount" datatype="xsd:integer">330</td> <td>16.6%</td> <td property="openelection:elected" datatype="xsd:boolean" content="false">No</td> </tr> <tr rel='openelection:candidacy'> <th scope="row" rel="openelection:candidate"> Brenda Lilian Constable 37 Morley Road, Burntwood, Staffordshire WS7 2DE </th>

The future for local data? Local government for local people

Publish once, consume many times

Council open dataGovt Dept A

Govt Dept B

Govt Dept C

publicpublic

interest/media/

community/commercial

sites

may

be

Open data opportunities


✤ Massive efficiency savings (somewhat important when the cuts are so severe) – reducing duplication, management layers, automating regular routine tasks



✤ Reduction in bureaucracy (and associated frustration of frontline staff and citizens)




✤ Improved online services for citizens at zero cost (provided by websites/apps that work with any council)





✤ Increased transparency





✤ Increased transparency

✤ More freedom from central government targets, auditing

Challenges

Challenges

✤ Lack of ‘corporate’ awareness/understanding of open data issues

Challenges


✤ There is a lack of even basic web skills at some councils

Challenges



✤ Many councils lack web publishing resources, never mind the resources to implement open, linked data publishing

Challenges




✤ The understanding of even the basics of linked data and the steps to publishing public data in this way is very, very limited, even outside of the public sector

Challenges





✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills

Challenges





✤ Getting councils to publish data in open, linked formats requires dedication, and a range of resources and a mix of skills

✤ Outsourced web publishing contracts often make it difficult and expensive to publish open, linked data

More Challenges

More Challenges

• IDs – councils don’t generally use them; central government uses many... inconsistently

More Challenges


• Data tied up in PDFs (will we ever get this back?)

More Challenges



• Legacy systems. That nobody now understands

More Challenges




• Private companies/JVs. No FoI. No access to data

More Challenges





• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)

More Challenges





• Outsourcing (councils have outsourced the skills & knowledge and now are often dumb consumers)

• Silos, especially of budgets. How do you get money from one budget to another (even if the whole benefits)

Yet More Challenges

Yet More Challenges

✤ The size of the cuts mean they will be blunt, eliminating useful services, sparing wasteful ones. Who & what will be left & what will it mean for local public services?

Yet More Challenges


✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)

Yet More Challenges


✤ Few resources to invest in doing this, and even less understanding of where to invest it. Danger of using usual procedures, suppliers (procurement, Capita, Serco etc)

✤ Little incentive for those organisations and people threatened by open data – for them it is a Darwinian fight for survival

An example dataset: local spending data (deadline jan 2011)

An example dataset: local spending data (deadline jan 2011)✤ Technical challenge: how to extract the data from the accounts system

without including non-suppliers (e.g. foster parents, children)✤ How many councils are working on this (hopefully all of them)

✤ How many have blogged, posted the steps/issues/solutions – zero?

✤ What are the key fields that MUST be included, and do these need to be listed somewhere?





✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?

✤ How will we compare with other similar bodies?

✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)





✤ Organizational challenge✤ What is this going show about our spending/processes/contracts?

✤ How will we compare with other similar bodies?

✤ What will be the consequences (FoI requests, changed relationship with suppliers, unpopular services)

✤ Systemic challenge✤ How will this affect the relationship with the citizens?

✤ What are the implications for our structure, processes, power...?

Power in the public sector was...

✤ Access to people

✤ Access to information

✤ Understanding structure/how things worked





undermined by social networking






democratized by open data






democratized by open data

crowdsourced? bypassed?

All this is a proxy for the real battle: relevance


✤ How relevant is your local government to you?



✤ How well do the ‘proper channels’ work for you?




✤ What is your ability to achieve things outside the existing structures?




✤ What is your ability to achieve things outside the existing structures?

✤ What difference does your council make to your life?

Open data is local government’s chance to stay relevant to their

citizens/economy(but it’ll have its work cut out)

open local data: challenges and opportunities

News & Politics