IBM scans 10 billion files to claim storage performance record

IBM is reporting record performance figures for its large-scale data management and storage cluster system which is able to scan 10 billion documents in just 43 minutes.

The cluster runs IBM's General Parallel File System (GPFS) software, and comprises 10 systems running eight-core processors with 6.8TB of solid-state storage.

The figure smashes IBM's previous record for document scanning in 2007 when one billion files were scanned in three hours.

IBM credited the breakthrough to GPFS, which allows better use of the multi-core processors and solid-state hardware, making for more efficient data transfers and I/O operations.

"Today's demonstration of GPFS scalability will pave the way for new products that address the challenges of a rapidly growing multi-zettabyte world," said IBM vice president of storage platforms Doug Balog.

"This has the potential to enable much larger data environments to be unified on a single platform and dramatically reduce and simplify data management tasks such as data placement, ageing, backup and migration of individual files."

The ability to handle large-scale data processing and analytics, or 'Big Data', has become a top priority for system vendors in recent years.

IBM has enlisted its famous Watson supercomputer to help, while rival vendors such as EMC and SAS have developed Big Data systems of their own.