Announcing Evan's Awesome A/B Tools

By Evan Miller

May 13, 2013

If you've ever needed to do basic statistical calculations for an A/B test, your options traditionally have been:

None of these options are very intuitive, as they all just spit out numbers without providing much in the way of context. (“What's a t-statistic again?”)

Today I am happy to announce a new suite of online statistics calculators, which I am hereby christening Evan's Awesome A/B Tools. I am calling these tools awesome because they are intuitive, visual, and easy-to-use. Unlike other online statistical calculators you've probably seen, they'll help you understand what's going on "under the hood" of common statistical tests, and by providing ample visual context, they make it easy for you to explain p-values and confidence intervals to your boss. (And they're free!)

These tools assume some background in statistics; as great as they are, I'm afraid they're not a substitute for learning. But, if at one point in your life you knew the difference between a significance level and null hypothesis, I think you're going to love them.

So without further ado, I give you…

Evan's Awesome Sample Size Calculator

I get emails quite frequently asking me to interpret the equations in my article How Not To Run an A/B Test. Well now there's a web page that will interpret it for you:

Punch in the expected conversion rate and the smallest difference you want to detect, and voila! A big fat number tells you how many subjects you'll need in each branch of the experiment. Sliders let you adjust the desired power and significance levels; the answer updates automatically as you move the sliders, so you can intuitively get a feel for how the statistical power affects the scale of the experiment.

Then of course you're going to need…

Evan's Awesome Chi-Squared Test

So you've finished running your A/B test and it's time to interpret the results. That's where the chi-squared test comes in handy:

Key in the number of conversions and the number of trials in each branch, and presto! You'll instantly see the confidence intervals around the success rate in each group, and by manipulating the confidence slider, you can watch the intervals grow and shrink accordingly.

Below the bar chart, you'll get the result of a chi-squared test telling you whether or not there's a statistically significant difference between the two success rates. Couldn't be easier.

That is, unless you're talking about…

Evan's Awesome Two-Sample T-Test

If you're measuring a continuous outcome such as time spent or money saved, a chi-squared test won't help you. The two-sample t-test comes to the rescue:

This interface is a bit more involved than the other two, but it's packed with power:

What are you waiting for?

Check 'em out! Enjoy the awesomeness for yourself and send the link to your A/B testing friends!

Sample Size Calculator

Chi-Squared Test

Two-Sample T-Test

As always, if you encounter a bug or have a suggestion, just send me an email.


You're reading evanmiller.org, a random collection of math, tech, and musings. If you like these tools, you'll probably also like Wizard, my intuitive desktop statistics program for Mac.