Riak Pipe:Riak的任务管道功能

Riak PipeRiak刚刚发布的一项新功能,目前在Beta版。Riak Pipe简单的说就是一个类似于UNIX 管道的东西,不同之处在于Riak Pipe还定义了输入将在何处(哪一个机器节点)被接收执行,这一功能通过类似于一致性hash算法的手段,能够尽量将任务在集群中进行平均分配。

Riak Pipe 可以被用在Riak 的MapReduce任务中,以实现更为方便的任务控制和系统监控等功能。甚至在未来Riak Pipe还有可能将MapReduce接口扩展出更为独立的接口。


Hello again, Community.

I’m excited to announce the opening of a new beta-status Basho project
today: Riak Pipe.


Riak Pipe is a new way to distribute work around a Riak cluster.

The README explains much more than I can here, but essentially Riak
Pipe allows you to specify work in the form of a chain of function
pairs. One function of that pair describes how to produce output from
input, and the other describes where in the cluster an input should be
processed. Riak Pipe handles the details of ferrying data between
workers by building atop Riak Core’s distribution power.

At this point in time Riak Pipe is BETA-status software. We’d like
anyone who is interested in it to take a look and send us feedback.
Please do not put it into production. We will be continuing to
improve Riak Pipe toward a future release date.

We have two plans for Riak Pipe. The first is to power Riak’s
MapReduce system with it. We think Riak Pipe provides a cleaner, more
manageable subsystem that will provide much easier monitoring,
debugging, and general use of MapReduce in Riak. You can see our work
toward that goal in the “pipe” branch of Riak KV (start at


Our second plan for Riak Pipe is to expand Riak’s MapReduce system
with more abilities (imagine a keyed-reduce phase, or additional
processing languages), possibly to the extent of providing an entirely
separate interface (new query syntax? offline/asynchronous
processing?). But for this part, we need your help.

We have some ideas about what external client interfaces might look
like. We also have some ideas about what an external processing
interface might look like. We’re still in the early phases of
creating these, though, so if exploring the riak_pipe repository gives
you ideas, please don’t hesitate to get in touch.

And, again, Riak Pipe is BETA software. Basho does not support
running it in production at this time.


Bryan Fink
Senior Software Engineer
Basho Technologies



分类 未分类 · tag , ,