A Work in Progress

This blog post is going to be fairly light on content, but it was something I googled quite a bit this weekend until i realized that Rails 3.1 provides a pretty awesome option.

The problem I had is that i have an existing Devise model and I wanted to add the :confirmable trait to that model so that when a user registers they have to provide a valid email address in order to use the site. One suggestion online was to just regenerate the model and use the newer migration but that seemed like overkill.  Another answer was to add the columns yourself.  The “up” migration was easy and just involved modifying the table to add “t.confirmable” which would generate the proper columns. The “down” migration was ugly though because you had to have knowledge of each column that devise would add when you put confirmable.

A better solution? Rails 3.1 Reversible Migrations to the rescue! If you are using rails 3.1 instead of having “up” and “down” methods you can just have a “change” method and for the most part, Rails and ActiveRecord are smart enough to figure out how to migrate that up and down (there are a few caveats, such as removing columns when migrating up.) So in the end, the code to add confirmable columns to the DB was quite simple:

Yesterday, one of my teammates who does a lot of work in automating our deployments came to me and asked me for some in parsing and updating a JSON config file so that he could automate the setting of configuration data during automated deployments.  I took one look at the config file in front of me and said “Bro, that’s not JSON.  Those are Erlang terms.”

He was working on automting the Riak deployment we are setting up for some new features.  He had never touched erlang (or any functional language) so i offered to show him the ropes and write a script that could read and update Riak app.config. I figured i would post it here for a couple of reasons:

• I’m no Erlang expert myself.  I’d love to get feedback on easier/better ways to accomplish this
• Somebody else probably wants to solve this problem and the script may help.

I’ve been doing some investigation into a few NoSQL databases lately for use in a project at Symantec.  One of the NoSQL DBs I really like and wanted to test some specific use cases for is Riak.  I already think that Riak is pretty awesome so i’m not going to get into evangelizing.  This blog is purely about the test setup I created on Amazon EC2.

For my tests I decided to create a 3 node Riak cluster.  I wanted some beefy hardware that would be roughly analogous to the machines we would be putting Riak on in our real datacenters so I opted to use EC2 Cluster Compute instances to reduce latency.  The nodes are Quad-XL (because it was either that or a GPU instance, which I don’t need.)

So without any further delay… here is how to set up a 3 node Riak cluster on EC2:

Creating the First Node

Start by spinning up an EC2 Cluster Compute Instance:

Security Groups

One very important part of this process is Security Groups.  AWS uses security groups to define who can access your instances, and how.  Here is what my Security Group looks like for my Riak cluster:

• TCP 0-65535 (within the Riak Security Group) is the only thing here that i would justify as a hack  When i was trying to get my cluster to join with just 8099 open it failed.  Perhaps somebody from Basho (or a Riak expert) can fill me in on exactly which ports are needed.  I’m not too worried about this since its confined to chatter between my Security Group, but I would never have a production deployment like this.
• TCP 8099 (within the Riak Security Group)  This is the port that Riak uses for intra-cluster data handoff
• 22 (SSH) – So i can get to the box
• 80 HTTP – Not actually necessary
• TCP 8098 (Public) – exposing the Riak HTTP interface to the world.  In a production environment this would probably be restricted to the IP-space of the servers that would be interacting with Riak.

PS — I just learned while doing this that you can define a port to be available to all instances in a security group.  How long has that been around?!  It is an awesome feature! <3

Installing Riak

Once your instance is up and running, installing Riak is a breeze (much of this is borrowed from the excellent Riak RHEL install documentation).  The only thing I had to poke around a little bit to figure out was

wget http://downloads.basho.com/riak/riak-0.14/riak-0.14.2-1.el5.x86_64.rpm
sudo yum install openssl098e
sudo rpm -Uvh riak-0.14.2-1.el5.x86_64.rpm

Note: I got a few errors from chcon, but they didnt seem to cause any issues.

Okay!  We’ve got Riak installed.  Next I edited the Riak app.config file to bind to all IP address, just for external testing:

in /etc/riak/app.config:

{http, [{ "127.0.01", 8098 }]},

becomes

{http, [{ "0.0.0.0", 8098 }]},

Now you can run sudo riak start to start Riak.  If your Security groups are set up correctly you should be able to hit it via port 8098.

Now, shut down that instance.  In AWS create an AMI from the instance (right click the instance, choose “Create Image (EBS AMI”)  Once your AMI is created launch two more EC2 instances from the AMI.  This spares you the effort of having to install Riak on each node in your cluster.  It also makes scaling out easier in the future since you can create a new instance, edit some config (preferably via Puppet or Chef in an production environment) and run riak join to join the cluster.

Once your two instances are up and running, I literally followed the Riak documentation at http://wiki.basho.com/Basic-Cluster-Setup.html.  It was really that easy.  The was one gotcha, though. Make sure that when you choose the IP address to bind (and name) for the nodes in your cluster that you use the EC2 Private IP Address (or DNS name should be fine too.)

I do have some concerns that this will cause problems on reboots, but I haven’t tested it yet.  Anyone else done this and determined that for sure?

There you have it.  At this point you should have a fully functional 3 node Riak cluster running on AWS EC2 Cluster Compute Instances.  It should be pretty damn fast!  The only thing I added on top of this was an Elastic Load Balancer that would round robin the traffic between the three nodes so that I could test how it is really scaling.

For the record, I have made my AMI public (just a base Riak install.)  The AMI ID is: ami-981ee7f1

Last night I had the pleasure of going to listen to Atul Gawande speak about medicine, healthcare, his latest book (The Checklist Manifesto), and more generally about making systems, processes, and procedures work as complexity increases beyond the capabilities of the individual.

As a surgeon, researcher, and writer Dr. Gawande has tackled some really difficult problems with regards to reducing deaths as a result of surgery and done an impressive amount of research across many fields (including with Boeing engineers, and people who build skyscrapers) to determine how to deal with complexity in a way that reduces error but also allows members of a team to exercise judgement in the heat of the moment.

The most important thing that I took away from his talk was his emphasis on the need to shift our thinking in terms of how we get things done.  Medicine and Software Engineering have deeply entrenched concepts of lone geniuses.  The Brilliant Doctor.  The Rockstar Developer.

Dr. Gawande emphasizes the need to shift from the “hero” culture to more of a “pit crew” mentality where there is a team of people who all have set responsibilities, where everybody knows everybody else’s responsibility, and every team member feels confident enough to speak up if something is going off the tracks.  A team where everybody is moving forward, pointed in the same direction.  I have seen this same sort of shift in software engineering.  I think that in some ways the agile movement reflects this mentality.  The best teams I have been on were the teams where everybody (regardless of level) felt like they had a say, and where we were all driving towards the same goal (not just a ship date, but a level of quality, a level of service to the teams we interacted with, and a commitment to doing what is right for the customer.)

The tool that Dr. Gawande introduced to his operating room (and others around the world) with stunning success is a simple one.  The checklist.  His argument is that while people’s intuitive reaction to the idea of a checklist is that it means shutting down your brain, but that a well-written checklist for a team of people can spark the memory and bring out better performance.  The idea of introducing checklists into some of the processes we do at work is intriguing me.  I’m analyzing where we fail (in field bugs, broken builds, failed build acceptance tests) and why the failures occur in order to find a place where a simple checklist could be effective.

Off the bat i’ve come up with a few ideas:

• Code reviews — On my team we already have an informal checklist for code reviews:  Does the code look right (match standards, appear to be logically correct?)  Does it build?  Does it work?  Do the unit tests run?  Do our automated integration tests run?  I wonder if this can be extended or formalized, and how?
• Deployments — When we are deploying a service and the deployment has to be coordinated with several teams does it make sense to form a “pit crew” with well known responsibilities and a checklist of things that we are all accountable for?  I’ll admit here that i’d rather try to go the devops route and automate deployments with something like Puppet but my group is just not there yet.

What do you think?  Has anyone read the book and thought of how it applies to software engineering?  Where else would a checklist be appropriate for preventing errors?  Are checklists something we can codify?  Are we already doing this to some extent with automation and unit-testing?

Over the past few weeks we’ve witnessed a couple of big outages in the cloud computer world.  First there was the downtime on Amazon’s EC2 infrastructure, and then this week there was the Blogger outage on Google’s infrastructure.  These high profile outages, coupled with the slightly different but orders of magnitude more damaging Playstation Network outage have gotten some people questioning the wisdom of building or moving services into the cloud, including the New York Times.

I’m still incredibly bullish on cloud-hosted services and the power and potential of the cloud.  There are two fundamental reasons why.  First, the cloud is The Great Leveler.  Not since the advent of the personal computer has the playing field been leveled so significantly.  If you come up with the idea for the next Facebook or Google you can bootstrap it for a tiny fraction of what buying the required hardware would have cost you.  As your service grows, you can scale.  You can compete with anybody in the world for handling user traffic on day 1.

The second reason is that while I discovered that AWS Availability Zones aren’t quite as separated as I thought (EBS control plane is shared per-region, but distributed across AZs in the region) there were many websites who survived the outage came to little or no harm (namely Twilio, Netflix.)  The outage at EC2 makes the case for cloud computing, it doesn’t damn.  High Availability 101 says that you can’t have a single point of failure.  The services that went down were primarily in a single Availability Zone on AWS.  If you have everything running in a single AZ then you are asking for trouble.  You have to assume failure.  Netflix takes this to the extreme with their Chaos Monkey.

Furthermore, the cloud is still evolving.  One thing that I think that we will see developing as an extension of some current endeavors, and as a result of these recent outages, is the “meta-cloud” provider.  Companies like RightScale are already playing in this space.  I think that they should start building technology to abstract away the cloud provider entirely and create a market for compute resources.  They can run your service on EC2 if thats where they are getting the best combination of cost and performance (think: spot instances) or they could bring it up on RackSpace’s cloud if the best price and performance is there.  They could utilize many providers to get better performance (routing users to datacenters closest to them.) They could also bring your service up in multiple cloud providers to add an even higher level of availability and redundancy, or switch your provider if an anomaly or outage is detected where you are currently hosted.