Setting up a 3 Node Riak Cluster with EC2 Cluster Compute Instances

Posted by on June 6, 2011 Leave a comment (2) Go to comments

I’ve been doing some investigation into a few NoSQL databases lately for use in a project at Symantec.  One of the NoSQL DBs I really like and wanted to test some specific use cases for is Riak.  I already think that Riak is pretty awesome so i’m not going to get into evangelizing.  This blog is purely about the test setup I created on Amazon EC2.

For my tests I decided to create a 3 node Riak cluster.  I wanted some beefy hardware that would be roughly analogous to the machines we would be putting Riak on in our real datacenters so I opted to use EC2 Cluster Compute instances to reduce latency.  The nodes are Quad-XL (because it was either that or a GPU instance, which I don’t need.)

So without any further delay… here is how to set up a 3 node Riak cluster on EC2:

Creating the First Node

Start by spinning up an EC2 Cluster Compute Instance:

Security Groups

One very important part of this process is Security Groups.  AWS uses security groups to define who can access your instances, and how.  Here is what my Security Group looks like for my Riak cluster:

  • TCP 0-65535 (within the Riak Security Group) is the only thing here that i would justify as a hack :)  When i was trying to get my cluster to join with just 8099 open it failed.  Perhaps somebody from Basho (or a Riak expert) can fill me in on exactly which ports are needed.  I’m not too worried about this since its confined to chatter between my Security Group, but I would never have a production deployment like this.
  • TCP 8099 (within the Riak Security Group)  This is the port that Riak uses for intra-cluster data handoff
  • 22 (SSH) – So i can get to the box :)
  • 80 HTTP – Not actually necessary
  • TCP 8098 (Public) – exposing the Riak HTTP interface to the world.  In a production environment this would probably be restricted to the IP-space of the servers that would be interacting with Riak.

PS — I just learned while doing this that you can define a port to be available to all instances in a security group.  How long has that been around?!  It is an awesome feature! <3

Installing Riak

Once your instance is up and running, installing Riak is a breeze (much of this is borrowed from the excellent Riak RHEL install documentation).  The only thing I had to poke around a little bit to figure out was

wget http://downloads.basho.com/riak/riak-0.14/riak-0.14.2-1.el5.x86_64.rpm
sudo yum install openssl098e
sudo rpm -Uvh riak-0.14.2-1.el5.x86_64.rpm

Note: I got a few errors from chcon, but they didnt seem to cause any issues.

Okay!  We’ve got Riak installed.  Next I edited the Riak app.config file to bind to all IP address, just for external testing:

in /etc/riak/app.config:

{http, [{ "127.0.01", 8098 }]},

becomes

{http, [{ "0.0.0.0", 8098 }]},

Now you can run sudo riak start to start Riak.  If your Security groups are set up correctly you should be able to hit it via port 8098.

Now, shut down that instance.  In AWS create an AMI from the instance (right click the instance, choose “Create Image (EBS AMI”)  Once your AMI is created launch two more EC2 instances from the AMI.  This spares you the effort of having to install Riak on each node in your cluster.  It also makes scaling out easier in the future since you can create a new instance, edit some config (preferably via Puppet or Chef in an production environment) and run riak join to join the cluster.

Once your two instances are up and running, I literally followed the Riak documentation at http://wiki.basho.com/Basic-Cluster-Setup.html.  It was really that easy.  The was one gotcha, though. Make sure that when you choose the IP address to bind (and name) for the nodes in your cluster that you use the EC2 Private IP Address (or DNS name should be fine too.)

I do have some concerns that this will cause problems on reboots, but I haven’t tested it yet.  Anyone else done this and determined that for sure?

There you have it.  At this point you should have a fully functional 3 node Riak cluster running on AWS EC2 Cluster Compute Instances.  It should be pretty damn fast!  The only thing I added on top of this was an Elastic Load Balancer that would round robin the traffic between the three nodes so that I could test how it is really scaling.

For the record, I have made my AMI public (just a base Riak install.)  The AMI ID is: ami-981ee7f1

 

Uncategorized
  • Andrew

    You should only need port 4369 for erlang inter-node communcation. See http://www.erlang.org/doc/man/epmd.html

  • http://twitter.com/_jared Jared

    To keep your port open listing small, see the following from doc/admin.org in riak:

    p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 10.0px ‘Lucida Grande’; background-color: #5f9ea0}

        + 4369: Erlang Port Mapper Daemon’s (EPMD) port, for connecting      Riak nodes.    + 5001-6024: Port range for the listener socket of a distributed      Erlang node. This range can be modified by specifying values for      =inet_dist_listen_min= and =inet_dist_listen_max= in the =kernel=      section of Riak’s =app.config=. The network interface on which      these ports are exposed can be configured by specifying a value      for =inet_dist_use_interface= (see example below).    + 8098: Default HTTP interface port.  This port can be modified by      specifying a value for =web_port= in the =riak_core= section of      Riak’s =app.config= (see example below).  The network interface      on which this port is exposed can be configured by specifying a      value for =web_ip= in the =riak_core= section of Riak’s      =app.config=.    + 8097: Default Protocol Buffers interface port. This port can be      modified by specifying a value for =pb_port= in the =riak_kv=      section of Riak’s =app.config=. The network interface on which      this port is exposed can be configured by specifying a value for      =pb_ip= in the same section (see example below).    As noted, several of these ports are configurable.  The default    configuration in =app.config= would look something like this: