Facebook has released an open source embeddable data storage platform called RocksDB that it uses to store, retrieve and serve information to its billion-strong user base.
Facebook said it hopes that by releasing the code, which it has published on the GitHub website, other developers and businesses will help refine and improve the platform, as Hadoop engineer at Facebook Dhruba Borthakur explained in a blog post.
“We hope that software programmers and database developers will use, enhance, and customise RocksDB for their own use cases. We’re looking forward to hearing your feedback and continuing to improve RocksDB and add more features,” he said.
The RocksDB tool is a key-value store that builds on the LevelDB, released by Google. Facebook said it has taken this and extended its capabilities to help it store and serve data direct from its servers. This helps improve the speed with which information is served to users.
“Applications traditionally access their data via remote procedure calls over a network connection, but that can be slow – especially when we need to power user-facing products in real time,” explained Borthakur.
“With the advent of flash storage, we are starting to see newer applications that can access data quickly by managing their own dataset on flash instead of accessing data over a network. These new applications are using what we call an embedded database.”
Borthakur said moving to the use of embedded databases had several benefits. “When database requests are frequently served from memory or from very fast flash storage, network latency can slow the query response time," he said.
“Accessing the network within a data center can take about 50 microseconds, as can fast-flash access latency. This means that accessing data over a network could potentially be twice as slow as an application accessing data locally.”
Facebook said it had to build on LevelDB as the Google platform did not work well for the scale of input/output operations per second (IOPS) on the site.
"We found that LevelDB's single-threaded compaction process did not work well for certain server workloads, and that RocksDB outperforms LevelDB for these IO-bound workloads. We saw that applications were not able to derive the full potential of the flash hardware because of data bandwidth bottleneck[s] caused by LevelDB's high write amplification," Borthakur wrote.
"By increasing write rate and lowering write amplification, we were able to avoid many of these issues and improve performance with RocksDB."
Facebook said with the improvement it has made, RocksDB can be used for a variety of key functions, such as storing and retrieving user history on a service to present relevant content (as Facebook does), spam detection, graph searches and Hadoop-based queries.
The move is yet another example of Facebook taking its technologies open source, as it continues to push its Open Compute Project, which focuses on improving the efficiencies of servers used in data centres.
Earlier this week the firm urged European firms to back this goal, claiming so far interest and input has been far behind that seen in the US and Asia.
Microsoft receives a 30 per cent cut of all purchases on the Xbox digital store
Credit card thieves used Apple ID accounts to buy and sell virtual currency for Clash of Clans and Clash Royale and Marvel Contest of Champions
$5.1bn fine further evidence that the EU is anti-US, claims Trump
New cable will connect Virginia to France