Companies facing information overload

Storage Expo 2007

Delegates at Storage Expo in Olympia have been hearing today how information overload is creating a potentially unsolvable problem for storage systems.

The amount of data being stored by businesses is rising exponentially, but systems to manage the information are not in place. The problem is exacerbated by regulations demanding ever more data retention.

"This idea of just keeping everything is not going to work," said Bob Plumridge, member of the board of directors for the Storage Industry Networking Association (SNIA).

"Companies doing this will run out of steam in the next few years. You cannot just keep throwing more hardware at this problem."

Plumridge explained that an averaged-sized company now stores around two petabytes of data.

If this had to be migrated it would take about six months of data processing. In 10 years' time, based on current predictions, this will rise to eight petabytes.

Plumridge warned that the hardware platforms on which much of this data is stored are proprietary, and that there is a danger that many companies selling these devices will have ceased trading in the future.

He described the experience of an oil company which found it cheaper to resurvey some areas because the cost of getting the data off old file systems and devices that were no longer supported was so high.

The answer lies not with IT, but with standards and processes, according to Plumridge.

The SNIA is working on a 100-year standard for data storage and retrieval based on logical standards and processes that allow interoperable data storage between devices.

"People in IT have a belief that IT itself will make things better," said Jon Collins, an analyst from Freeform Dynamics.

"Process is a prerequisite for successful storage, not just technology. I see too many companies complaining that they have been lied to by IT vendors, but they drank the Kool Aid and they cannot expect processes to just come out of nowhere."