Big data use in government is suffering from a lack of data scientists and locked down IT systems that stifle the skills of the talented individuals being recruited.
Chris Fleming, head of data and analytics at the Office for Science, said that the government lacks analysts with the IT and data science skills to make the best use of big data across departments.
"What we have in government is a huge cohort of what I refer to as traditional analysts: statisticians and researchers," he said.
"They have really good maths and problem solving skills but they don't really have the IT skills, so I think there's a challenge there to marrying those communities."
Fleming added that this problem has not gone unnoticed outside government, with someone at Google telling Fleming they too have noticed there is a 'massive skills gap' with regards the lack of skilled people who can work with big data sets.
The problem is further exacerbated by the inflexibility of the IT systems used in government, which Fleming said has an adverse effect on the skills of data scientists recruited by the government.
"The slightly sad fact is that we at the entry levels of government are recruiting people via our Fast Stream scheme who have really top skills," he said.
"They're straight out of university and they have good data science skills but, because of the toolset and locked down IT systems we have in government, those skills get blunted, and blunted quickly."
Fleming explained that the government needs a strong use case for big data that delivers benefits to the services used by citizens rather than simply being used for the sake of it at taxpayers' expense.
This will require the creation of a big data culture across government departments, which will require building awareness of the ways big data can be used in a cost-effective way.
"We've improved this at the senior levels of government, but the challenge for us is engaging with the vast swathes of middle management and helping them to be intelligent customers for the insights from big data analytics," he said.
Fleming explained that big data is being put to use in certain areas, despite the challenges faced by the government and wider civil service in using the information.
One area is working with Ordinance Survey to use big data in crisis management, where data on the location of transport links and vulnerable people can be interlaced with a local area map.
Fleming said that the government is looking to scale up its big data operations by looking into cloud-powered analytics, creating an accelerator programme for training traditional analysts in data science.
He explained that the government is also taking general action to address the problems of privacy and data protection in the use of big data in the public sector.
"I think the future looks bright for data in the government," concluded Fleming.
Evidence of the government's interest in data use can been seen with the appointment of Government Digital Service head Mike Bracken as the government's first chief data officer.
Darktrace pushes machine learning to take some of the pressure off of IT and security teams
Google also gets its hands on HTC's IP in a non-exclusive deal
Microsoft, Google and Samsung all targeted as Avast admits to the scale of the CCleaner compromise
Not all loose ends tied yet, admits Bain backer SK Hynix