Organisations are playing "Russian roulette" as a result of the pressure to speed up development, according to Simon Poulton, continuous delivery evangelist at CA Technologies.
Poulton said at Computing's DevOps Summit 2016 this week that a lack of coordination between development and operations when approaching data analytics can lead to missed opportunities and even privacy problems, while thorough testing is thrown completely out of the window in some companies.
Poulton championed what he called a requirements-based approach to DevOps using data analytics.
"In most organisations I've worked with, the approach to test data is just going off to production and copying production. If [the dataset] is over 40TB people think maybe not even copy it at all, and all this to me is just playing Russian roulette with test data," he said.
Poulton explained that CA found vast amounts of data when doing an analysis on data for a UK high street bank that didn't even need to find its way into pre-production environments to be integrated into new projects.
"Ninety-six per cent of investment data covered trade in the same currency. Imagine they had 100,000 records, 96,000 of which tested the same thing. Why bring those over to a pre-production environment?" he said.
Poulton reminded delegates that "all of that production data has PII [personally identifiable information] stuff", meaning that compliance with the EU General Data Protection Regulation could be a problem.
Finally, he argued that new app functionality in new projects will try to draw on existing data, but that the migration is again a waste of time and resources if there isn't any in production that can be used.
"Testers also tend to find in data the data they want to use specifically, and use it again and again, but in terms of depth, that's not a good result," he said.
The alternative, according to Poulton, is to use discovery tools to define logical relationships using a requirement-based approach.
"Identify the data specifically needed for this user's story, and use the data only that matches that requirement. So what we're really doing is ensuring through and through quality in that process," he said.
IBM and Technical University of Munich team demonstrate how Shor's algorithm, which can't be cracked by conventional computers, can be solved quickly with quantum computing
Hubble Space Telescope finds superflares from young red dwarfs could strip away planetary atmosphere
Younger stars are 100 to 1,000 times more energetic than when they're older
Two of the big four supermarkets will use the system to control sales of restricted products
PUBG news and updates: November's Update #23 to bring new Skorpion pistol and changes to blue zone visibility
Genuinely useful side-arm coming to PUBG in Update #23