The trend for open data that can be accessed by third parties such as developers has been growing for a number of years.
The government has positioned itself as a big fan of open data and has given its backing to the Open Data Institute (ODI) to help encourage others to release their information. But releasing large data sets is no easy matter, as it comes with several challenges, not least for storage.
Jeni Tennison, the technical director at the ODI, explained to V3 that any organisation that wants to offer data access must consider storage.
“In order to provide developers with access to data in flexible ways then your storage solution needs to be flexible too. That means if it makes it hard to run new kinds of queries or access certain parts of the data, it’s bad for developers,” she said.
This can be a problem for many public sector organisations. Tennison explained: “Usually government departments have a contract with an outsourced systems integrator. If you then want to access the information you have to get the systems integrator to add that kind of access.
“One of the things we try to do now is to make people aware that if you’re procuring a new system, to build into the contract the ability to access open data from the system so you can be flexible to open data requirements.”
One organisation that has shown the benefits of open data use is Transport for London (TfL). It provides its live feed of traffic data for developers to build applications that then serve information on travel routes around the capital.
This has proven a big success but plans are now in progress to try and consolidate the number of different APIs in use at TfL, as head of online at the organisation Phil Young explained to V3.
“Because the data comes from operational and back-end systems, the challenge is to cache that information and accept a high load on the APIs. We currently use content delivery networks but in the future we will be doing it in the cloud,” he said.
“This will enable us to provide a single API on the cloud hosting solution we use [Amazon Web Services], so it will be easier for developers to build multi-mode applications, that incorporate all our data from buses, tubes, river services and so on.”
In order to ensure data access through the API in the cloud remains stable, Young said TfL has considered all the necessary backup and failover requirements.
“The architecture is highly resilient, with multiple instances so that if one fails it moves over to another, and if one site fails, it can move to another data centre in another region. It’s about making sure you’ve considered everything,” he said.
No doubt the government will be hoping the success at TfL and the push by other organisations, such as The National Archives, to open up their data sets will lead to wider benefits for the public and the economy.
Electronics and computer chain the latest high street retailer to fall into difficulties
Incisive Media and Investec Asset Management supported fundraiser crosses Atlantic in 40 days
Alphabet's health sciences division Verily have been messing with AI algorithms
North Korea's cyber attack capabilities are expanding fast - and turning their fire on a wider range of targets