What public data is already available?

To give the more technically minded some raw material and a headstart on getting any prototypes up and running, we have brought together here a huge quantity of information for reuse.

Several exciting new datasets are being make widely available for the first time, including  the Royal Mail's Postcode PAF file, Neighbourhood Statistics from the ONS, Health care information, from NHS Choices, a list of all schools in England and Wales from the DCSF and the Official Notices from the London Gazette.

Below you'll also find listed a wide range of other sources of public data, as well as some other useful APIs.

Please leave a comment if you know of any more sources of useful public data, and we'll add them to the list below. Have fun, and good luck...

---

NEW! Neighbourhood Statistics API from the Office of National Statistics

What’s there?
Neighbourhood statistics about almost everything.

More specifically, Neighbourhood Statistics (NeSS) data including 2001Census, Access to Services, Community wellbeing/Social Environment, Crime and Safety, Economic Deprivation, Education, Skills and Training, Health and care, Housing, Indicators, Indices of Deprivation, People and Society, Physical Environment, Work Deprivation.

Where do I find it?
http://neighbourhood.statistics.gov.uk/dissemination/Info.do?page=nde.htm

You may want to look at http://hublog.hubmed.org/archives/001689.html for advice on what's in there and how to use it.

What do I need?
Contact the ONS for an API key at better.info@ons.gov.uk
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Transport information from Transport Direct

What’s there?
Location of bus stops, stations etc and descriptions of the locations
The National Car Park Register
Journey planning information
A carbon calculator
The National Public Transport Data Repository feed

Where do I find it?
Access to the data is via a secure FTP service or API. To support the competition, Transport Direct have agreed to help users access the data on an individual basis. Some restrictions apply as not all the data available is owned by the government.

What do I need?

Email Nick Illsley at Transport Direct to access the data, e.g. for the FTP address and password or to access the API.

Given some of the restrictions, it will help Nick if you can say what you're planning to do.

Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Health care services and information from the NHS

What’s there?
Information about health care service providers, and “live well” health information from the NHS.

Where do I find it?
http://www.nhs.uk/nhscwebservices/Pages/Webservices.aspx

What do I need?
Fill in the Web Services Registration Form and agree to the terms of use.

{Warning: it may take a few days for the NHS syndication team to get back to you. We're working on reducing this lag.}

Obtain a free Click-Use PSI Licence to re-use Crown copyright information from http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Notices from the London Gazette

What’s there?
All notices published in the London Gazette, the Government’s Official Journal and newspaper of record, in XML from February 2007 to May 2008. Notices types covered include State, Parliament, Ecclesiastical, Public Finance, Transport, Planning, Health, Environment, Water, Agriculture & Fisheries, Energy, Post & Telecom, Competition, Corporate Insolvency, Companies & Financial Regulations, Partnerships, Societies Regulation.

Where do I find it?
http://www.londongazette.co.uk/mashup/gazettesrdf.htm

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

NEW! Address Data from the Royal Mail

What’s there?
Sample raw data from the Postcode Address File. This will give you a comprehensive overview of the type, structure and format of the address data held by the Royal Mail, as well as the different data sets Royal Mail maintain.

The full PAF file is available, but you will need to check out the sample first (the full Postzon file is not available as part of this competition)

Where do I find it?
www.royalmail.com/addressproducts

What do I need?

You should email the Address Management Unit at address.management@royalmail.com 

There are few things they need to deal with your request quickly:

• 'Show Us A Better Way' in your subject heading
• Your address (so they can post the CD straight away)

Once you have had a look at what's on the sample CD, you can get in touch to request the full data set if required. This is a commercial product so you will only get one copy per person and there will be some terms and conditions to sign. We have spoken with Royal Mail and they will accept a scanned signature. Royal Mail will also ask you to link to the idea or fill in a short form so that they can get some feedback on this initiative.

---

NEW! A list of all the schools in England and Wales from the Department of Children, Schools and Families

What’s there?
A list of schools from Edubase. The data is in Excel format and contains a list of the schools, the address, the Local Authority, type of school and the age range served.

Update: Now available in SQL form thanks to the kind people at Free Our Data.

Where do I find it?
http://www.opsi.gov.uk/advice/poi/edubase300608.zip (3MB)

http://www.opsi.gov.uk/advice/poi/SUABW_schools_list_02072008.sql.zip

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

Mapping from Ordnance Survey

What’s there?

More detailed maps than are available via the popular Google Maps API– though these OS maps are strictly for non-commercial use only and access is limited to a few thousand queries per day. (If OS's licence or usage restrictions prove too onerous, you can also try OpenStreetMap, which does what it says on the tin and gets better by the day.)

Where do I find it?

http://openspace.ordnancesurvey.co.uk/openspace/

What do I need?

Register for an API key, using the form provided by Ordnance Survey.

---

Carbon Footprint Information from Defra (via AMEE – a 3rd party aggregator)

What’s there?
Carbon Footprint information from Defra, various UK Government agencies and others around the world.

Where do I find it?
http://www.amee.cc/
http://www.dgen.net/amee/signup.php (to get an AMEE API key)

What do I need?
An AMEE developer API key. Within reason, AMEE doesn’t aim to charge for access to the platform for projects that have no budget and are non-commercial or won’t have significant scale.

---

Public Spending information from HM Treasury

What’s there?
Tables and Statistics showing:

Public expenditure overview

Analyses of budgets

Differences from previous plans

Trends in public sector expenditure

Functional and economic category analyses of public sector expenditure

Central government own expenditure

Local authority expenditure

Public corporations

Analysis of public expenditure by country and region

Country expenditure by sub-function

Where do I find it?
http://www.hm-treasury.gov.uk/economic_data_and_tools/finance_spending_statistics/pes_publications/pespub_pesa08.cfm

What do I need?
Obtain a free Click-Use PSI Licence to re-use Crown copyright information from: http://www.opsi.gov.uk/click-use/system/online/pLogin.asp

---

Other Public Information Sources

BBC 3 Day Weather Forecast
http://backstage.bbc.co.uk/data/WeatherFeeds?v=p12
(List of all weather station forecasts as OPML feed http://www.dracos.co.uk/play/bbc-weather/countries.opml )

UK Road Travel Data from the BBC
http://www.bbc.co.uk/travelnews/xml/tpegml_en/rtm.xml

UK Public Transport Data from the BBC (Currently not working - 2/7/2008)
http://www.bbc.co.uk/travelnews/xml/tpegml_en/pti.xml

National Public Transport Access Node database (locations for all bus stops, train stations etc.)
http://www.naptan.org.uk/

National Public Transport Gazetteer (names of towns and places etc.)
http://www.nptg.org.uk/

TheyWorkForYou.com API (Parliamentary debates, MP details etc.)
http://www.theyworkforyou.com/api/

Parliament Parser project (List of all MPs, constituencies, names of Lords etc.)http://ukparse.kforge.net/parlparse/

DowningStreetSays.com Feed ("Every day the Prime Minister's Spokesperson meets a small coterie of political journalists known as 'the lobby' for a topical chat, or 'briefing'." This RSS feed is a transcript of these lobby briefings.)
http://downingstreetsays.com/feed

TellThemWhatYouThink.com (Government Consultations API, courtesy of Harry Metcalfe)
http://tellthemwhatyouthink.org/api/

FixMyStreet.com (geoRSS Feeds of problems reported by the public to their local council)
http://www.fixmystreet.com/alert

The Government Says (feeds of announcements and press releases from dozens of Government departments and other public bodies).  By Sam Smith and friends.
http://www.thegovernmentsays.com/

PlanningAlerts.com API (planning applications submitted to most local authorities, courtesy of Richard Pope)
http://www.planningalerts.com/apihowto.php

BBC Backstage Feeds (BBC News feeds, BBC Search API, TV & Radio schedules etc.)
http://backstage.bbc.co.uk/data/Data

BBC Programmes data
http://www.bbc.co.uk/programmes/developers#findoutmore

Other useful APIs

Google APIs (Google Maps is particularly useful)
http://code.google.com/more/#products-geo-maps

Yahoo! APIs
http://developer.yahoo.com/everything.html
(FireEagle is particularly useful http://fireeagle.yahoo.net/)

Wikipedia Data
http://download.wikimedia.org/

Wikipedia API
http://api.futef.com/apidocs.html

Flickr API (Photo sharing API)
http://www.flickr.com/services/

MOO.com API (Printing API)
http://www.moo.com/api/

Programmable Web Directory (all APIs known to man or beast)
http://www.programmableweb.com/apilist

Comments

[Tom Taylor]

I recently tried to access the NHS Web Services, but decided that I didn't feel comfortable signing a contract or service level agreement for a personal/hobby site. If there's any way you can open that up, it would be very useful.

[Tom Taylor]

Oh, and the link to edubase returns Page Not Found for me.

[Tom Loosemore]

Thanks for the heads up, Tom - we're still tidying bits of the site up and will fix the Edubase link asap.

[Tom Loosemore]

The EduBase link to data listing all schools in England and Wales should be working now - it's http://www.opsi.gov.uk/advice/poi/edubase300608.zip

Many thanks for the tip-off, Tom...

[Mark]

It's good to see OS opening up OUR map data of OUR country, but based on some examples I've seen the map data is less, and certainly no more, detailed than your average A-Z. I appreciate that OS has to cover its costs but surely we can be privy to more detailed mapping than that at no cost? What does it offer that Google maps doesn't?

[Anthony Cartmell]

@Tom Loosemore: The OS OpenSpace API, when zoomed in, uses 1:25,000 mapping which has a great deal more information than Google Maps: for example, footpaths, bridleways, etc. Sadly the OpenSpace API also has extremely limiting Terms of Use, making it impossible to use for anything even faintly commercial.

[Charles Arthur]

The UK Road Travel Data from the BBC page doesn't seem to work - gives "XML Parsing Error: undefined entity
Location: http://www.bbc.co.uk/travelnews/tpeg/en/local/rtm/rtm_tpeg.xml
Line Number 7, Column 1:
^

"

[Alaric Snell-Pym]

OpenStreetMap is becoming a great source of mapping information - under a Creative Commons licence, IIRC!

Nice to see you're getting back to your roots, Tom - this project looks right Up Your Street ;-)

[Alex Collins]

A really useful source of information for many applications is the Post Code Database (PAF files). This allows you to know a place's location on map from the postcode. You can then calculate distances between locations, enabling applications to determine choices that are best for you; recycling, or freecycling for example.

Sadly, unlike America's ZIP codes, we do not make this available freely.

http://www.guardian.co.uk/technology/2007/apr/26/freeourdata.guardianweeklytechnologysection

[darrylxxx]

There is a lot of very useful Northern Ireland data available here http://www.ninis.nisra.gov.uk/mapxtreme/datacatalogue.asp - most of it is in Excel spreadsheets. How quaint! :-)

[William at IdealGov]

Wow. It is marvellous to see this public data finally opening up and becoming available. And the visionaries who first asked for it a decade ago are now in charge of opening it up!

Well done the team. Cant wait to see what sort of stuff people get up to with it...

[Rob]

@Tom \ Charles: The Openspace API is a move in the right direction, but it's only raster imagery. Access to vector mapping data would allow far more spatial analysis, answering questions like...

What places are near here?
What properties intersect this polygon of crime data?
What is the total area of these playing fields?

None of those question can be answered with the mapping imagery from the Openspace API, we need vector mapping data.

Open Street Map is a great community, and IMHO has a big role to play in mapping areas of the world that don't have world class mapping agencies, and supporting aid agencies - but the UK does have a world class mapping agency, and does have accurate vector mapping data, we just need access.

In someways the OS have mimicked Google and Microsoft with their Openspace API, they need to be encouraged to innovate, providing a vector mapping API would be a real start - unfortunately, I think the OS have their hands tied with the trading fund model that governs them.

Rob

[Andrew]

This extra data is great, one other request - would it be possible to provide access to traffic accident data ?

I know the STATS19 data is already available through the UK Data Archive, however there isn't clear advice on permitted reuse - it would be usful if the datasets could be added to this page with guidelines on what re-use would be allowed and what steps developers should take to avoid identifying individual victims.

[Wayne Boucher]

"Show Us a Better Way" references the great website
http://www.planningalerts.com.
They get hold of planning information the hard way, by "scraping" it
from the proprietary format of each authority. I did this starting in
2000 for Cambridge city planning data (because they only kept the data
online for a few weeks) but gave up in 2005 because they changed the
format to a version which was harder to parse and at the same time their
data became less reliable. (It's possible it's better now.) This is a
crazy way to access this important information. Nobody should be
parsing semi-random HTML for this data. The LGA should knock heads
together and force every local authority to publish their planning data
in some well-specified standardised XML format. And not just planning
data but other data (e.g. budgets, etc.). If the LGA does not have the
expertise to do this there are no doubt plenty of people (e.g. see the
people who maintain the planningalerts website) who could help.

[Neil]

Can someone discuss with the NHS:

"Unfortunately this service is not available for individual use. Please register through your university or agency for access to the web service. Regards NHS Choices team."

I don't want this to be involved with my work etc, as its nothing related!

[Tom Loosemore]

@Neil: Thanks for the heads up. We'll chase.

[Richard]

@Neil: Following this up as I type. Thanks for the heads up.

[Tony Hirst]

The Open University OpenLearn project makes several hundred units from Open University courses available under a Creative Commons license and in a mashable format (RSS or XML feeds of course content, as well as various VLE package formats): http://openlearn.open.ac.uk

Several RSS feeds also publish details of the courses that are available. If you look hard, you'll also see that various URLs for different feeds can be constructed from unit codes.

For examples of hacks around openlearn, search on http://ouseful.info

[Tony Hirst]

The Open University OpenLearn project makes several hundred units from Open University courses available under a Creative Commons license and in a mashable format (RSS or XML feeds of course content, as well as various VLE package formats): http://openlearn.open.ac.uk

Several RSS feeds also publish details of the courses that are available. If you look hard, you'll also see that various URLs for different feeds can be constructed from unit codes.

For examples of hacks around openlearn, search on http://ouseful.info

[darrylxxx]

My company, Banjax, has just created a Northern Ireland Crime Map from publicly available crime statistics http://banjax.com/sandbox/crime/2007/ Feel free to contact me if anyone wants to discuss how or why we did it! darryl [at] banjax [dot] com

[Charles Arthur]

Request: school league table results in downloadable format indexed by school URN (as used by Edubase for the E&W schools list above) going back to, well, as far as can be done.

[Jason]

Don't forget Multimap.com has a great Open API (http://www.multimap.com/openapi/) for displaying and mashing-up data with maps.

Rather than just standard vector maps (TeleAtlas/Navteq), they also have beautiful Bartholomew and OS maps in London/GB.

http://www.multimap.com/maps/#map=51.50859,-0.11725|16|4&dp=902

http://www.multimap.com/maps/#map=51.50827,-0.10442|15|4&dp=904

[Malcolm Channing]

I'm looking for a breakdown of 2007 GCSE results by subject for each school in England. For example

Hay Comprehensive
Mathematics
Grade A* A B C D E F G N
% 5 5 10 20 20 10 10 10 10

I know that such documents exist but they seem to be out of the public domain.

[Allan Thompson]

Great idea.... and will surely lead to lots of lateral thinking.
I used to be responsible for Road Safety and Traffic Management in an area and endlessly find myself arguing in favour of speed cameras.
I regard them as a very important safety tool.
Speed camera locations are available for commercial devices - but their locations are surely within the government datasets. How about showing numbers of tickets issued at various locations and 80 percentile speeds on the roads that they cover?

[David Smith]

If it were possible to get access to a database containing the mapping co-ordinates of UK postcodes, so many location-based services become possible. The construction of effective Google maps requires this data...it is available for the US, why can't we have it?