Riak has come a long way since the unfortunate demise of Basho Technologies. With the release of Riak KV 2.2.5 (https://github.com/basho/riak/blob/riak-2.2.5/RELEASE-NOTES.md), which is the first community-led release of the database, the incredible success over the last few months is clear.
A significant number of pull requests have been created, from organisations that operate large-scale Riak deployments, and also individual contributors. The community has continued to grow steadily, in both size and activity, across social channels and in terms of code contributions.
The progress is staggering, considering that it was only a little over 6 months ago that the first community meeting took place, in which the direction and roadmap of Riak was discussed and outlined.
Looking forward, we will now begin to lay out the foundations for the next major release of Riak – 3.0, in which one of the most significant changes will be the move to the latest version of Erlang/OTP, along with many other features. And again, this will be a community-led initiative, building on top of the great work that has been done so far.
bet365 has a stream of new features and enhancements that are approaching the point of being released and pushed upstream. Some of the upcoming work includes:
Selective bucket replication between clusters – The current ability to globally exclude buckets from replication across all clusters was at best a blunt tool suited to simple replication topographies. This has been enhanced to include support for per cluster bucket level inclusion. This more granular control allows for more complex replication topologies and use cases without the previous data transmission overheads.
Better real-time connection management between clusters – A number of changes have been made to eliminate replication hot nodes. Replicating between clusters of disparate sizes will now grow the connection pool to distribute replication load more evenly across the destination cluster. Rebalancing of connections has been improved to resolve issues where over time connections collapsed back to a small number of nodes.
Bidirectional full sync (keylist) between clusters and enabling tombstone reaping in the backend – The approach to the reaping of tombstones has been re-thought. Instead of tombstones being reaped by Erlang timers on each node (which proved to be fragile as well as adding unnecessary overhead and complexity), tombstones keys are now written to the bitcask backend with absolute expiry times. This approach allows for complete replication of tombstone key between clusters with tombstone keys subsequently being reaped as part of the bitcask compaction process.
Coordinated GET (Reduced intra-cluster network overhead) – Applications with high GET:PUT ratios can, under load, saturate the network with unnecessary retrieval of objects from remote vnodes. This has been optimised to limit the amount of inter cluster network traffic, recognising that with the workload profile, the remote vnodes are mostly consistent.
There are also a numerous features, enhancements and fixes that have been merged as part of the Riak KV 2.2.5 release, for which a separate release note will be published by the community.
We would like to give a special thanks to all of the community members, organisations and individual contributors who have helped to make Riak KV the success that it is, now as a fully-fledged open-source project.