Security and data governance are the main challenges facing firms embracing big data projects in 2014, according to key players in the industry speaking at a roundtable event hosted by Rackspace and attended by V3.
However, data security and governance are likely to become significant issues, as cloud services figure prominently in most big data strategies.
There is still plenty of hype around big data, with analyst firm IDC predicting that the market for this will be worth an estimated $16.1 billion in 2014 and experience a growth rate six times that of the rest of the IT market.
However, this year we can expect to see the emergence of a true big data ecosystem, according to Chris Harris, technical director for Europe the Middle East and Africa (EMEA) at Hadoop specialist firm Hortonworks.
"Last year and before, it was all about early adopters, such as Spotify and Zing, looking at Hadoop as a way to extract data to drive their business strategy. Enterprises were also looking carefully at what was happening last year, and this year are starting to break down their information silos to become data driven," he said.
John Glendenning, vice president of EMEA at DataStax, the firm behind the Apache Cassandra distributed database, agreed with this point, claiming that big data will prove more and more disruptive as those firms using it start to gain an advantage.
"That train is now starting to leave the station. More and more services will be customer driven and interact with us, and the successful ones will be those that make our lives easier," he said.
Glendenning predicted that "we will see winners in this market, and dramatic failures like Blockbuster", referring to the DVD and games rental chain that recently closed down after failing to adapt its business model in a changing marketplace.
But the growing big data ecosystem faces several challenges, one of which is governance with regards to the use of customer data, according to Mat Keep, principal marketing manager at MongoDB, developer of the NoSQL database of the same name.
"Many of the services to support big data projects are in the cloud, and so you've got customer data being moved out to the cloud and stored who knows where," he said.
The cloud is proving a frictionless way for organisations to acquire this technology and start to pilot big data projects, he added, but now they are finding that they have to get serious when one-off projects are expanded and "the governance people start to get involved and insist that everything meets compliance regulations".
Traditional data management systems that have been around for decades have mature governance and auditing controls, and this is something MongoDB is now investing in, he said.
However, big data is still at an early phase in its development and many of the tools for making use of it are somewhat crude, a point made recently by big data analytics specialist Actian.
The problem is that applications for big data tend to be highly specific to the questions that each individual organisation wants to ask about the data they have collected.
For this reason, many big data vendors tend towards offering tools – such as Hadoop – that form part of the solution, and require the customer to build the finished product themselves. This is in contrast with the rest of the IT market, where companies can purchase more or less off-the-peg solutions for requirements such as customer relationship management (CRM).
Keep acknowledged this issue, but told V3 that this would be addressed as the market matures. "Eventually, you will start to see applications available that target common usage cases," he said.
Meanwhile, Rackspace was talking up its own big data credentials. The cloud and hosting firm recently launched its ObjectRocket NoSQL database as a service (DBaaS) delivered from the cloud, and is also working with European physics laboratory Cern on using the cloud to handle data generated by the Large Hadron Collider.
Could be used for everything from search-and-rescue robots to wearable tech
Don't require the rare material being mined from the mountains of South America
IBM hopes that its new tool will avoid bias in artificial intelligence
Found by calculating the strength of the material deep inside the crust of neutron stars