A few weeks ago I saw a thread on HN announcing the release of a new A/B testing tool for Ruby on Rails called Vanity. Since we at AppStoreHQ had been experimenting with A/B testing tools like Google Website Optimizer but didn’t really like them all that much, I thought I’d check out Vanity. Here’s my quick, 2-minute summary of my reactions since that time:
- A/B testing is fantastic. Coming from Amazon, I’ve been doing it for a few years, and now that AppStoreHQ is at the point where we can get meaningful statistics, it’s great to be able to try new things and see exactly the impact they’re having.
- Many A/B frameworks are surprisingly heavyweight. Even Website Optimizer — a great tool, no doubt — is fairly heavyweight.
- The other problem is that most frameworks expect a “target page” but in our case, we’re often trying to drive people off our site and into iTunes by purchasing apps, so we don’t have a target page necessarily. We do, however, redirect them through a page in our site, but we don’t want users loading JavaScript in the meantime. So, as it turns out, a server-side A/B testing framework is sometimes simpler and better.
- Vanity is extremely lightweight. Literally, it takes 4-5 lines of DSL in a new file to create an experiment, 1 line in your view to show the experiment, and 1 line in your controller to track the action. In fact, with Vanity 1.2, you don’t even need that last line if your goal is just to create an ActiveRecord model.
- The Vanity dashboard is great. We put it directly into our admin interface so we can link to it easily next to all the rest of our admin duties and graphs. It’s very straightforward and tells you exactly how your test is doing.
Here’s an example result from our dashboard. We ran an experiment on our app detail pages which compared the effects of 
vs 
as the buy button. Our theory was that while the latter conformed to the style of our other buttons, people would be more comfortable with the Download on iTunes button that can be found elsewhere on the web. Here are the results — Option A is the red Buy Now button, while Option B is the Download on iTunes button. I’ll let the results speak for themselves (I’ve blacked out some of the data just because we don’t necessarily need folks seeing how many detail page visits we get):
At first we were surprised by our conversion rate (not shown) being so low, but that’s just because of spiders like Googlebot. But those theoretically hit each option approximately equally, so should have no effect on the experiment. As you can see, our theory was indeed confirmed. I love the humanized “With 99% probability this result is statistically significant.”
This test took all of 2 hours to set up, including the initial setup of Vanity (which includes installing Redis). We started another experiment today which took 30min to set up. If you’re in the market for a good, easy-to-install and easy-to-use A/B testing framework for Rails, I highly suggest giving Vanity a good look.
