What is A/B Testing?

A historical explanation of the experimentation and testing method that powers the internet.

14th June 2015, by Mac Cordrey & Yash Kothari

In the 13th century, a man named Roger Bacon—inspired by the work of Aristotle and Grosseteste—described a repeating cycle of observation, hypothesis, and experimentation. Unbeknownst to Roger he was laying the framework and foundation for the scientific method. The scientific method is the umbrella term for all the techniques and practices we have to uncover new knowledge and verify previous knowledge. This notion of systematic observation, measurement, experimentation, and testing didn’t pick up steam until the 17th century, but since then it has become a cornerstone of our understanding of the world. The polio vaccine, Argon gas, and Penicillin are all examples of discoveries that came to be using the scientific method.

To provide an illustrative example, let’s say we wanted to determine whether bean plants will grow more quickly indoors versus outdoors. We construct a hypothesis by stating a belief about what might occur. In this case, our hypothesis is bean plants will grow faster outdoors, likely due to increased sunlight. We then form our test: we observe and measure the growth of a bean plant over a three week interval. To do this, we decide to plant four bean plants in same-sized pots using the same type of soil and fertilizer. We try to keep every variable constant (economists call this ceteris paribus) except we place two plants outdoors and two plants indoors.

Each day, over the three weeks, we measure the growth of the plants and ensure that each plant receives the same amount of water. After the three weeks, we collect and analyze our data. Based on the results, we determine if there is a conclusive answer to the question: “Do bean plants grow more quickly indoors or outdoors?” Then we check this against our original hypothesis and communicate our results.

This process of testing is not limited to the sciences - it can be incredibly valuable from a business perspective as well. For example, if we are in the business of growing beans, we can continue to test our assumptions until we find the optimal conditions for growing beans. This will increase the company’s productivity leading to higher margins and a significant impact on the bottom line.

As it has become easier to track customer data online, this process has become especially prominent on the Internet. So much so that it now has its own name: A/B Testing. A/B testing is simply a name to describe the quick randomized testing of two or more variations (like the bean plants growing indoors versus outdoors) in order to discover the most effective variation. A/B testing on the web is unique for a few reasons. A/B testing gives you the ability to test one variation of a page against another, and understand which variation produces positive results at a massive scale—thousands and thousands of cycles of observation, hypothesis, and testing at an extremely low cost. The marginal cost of these tests is incredibly small relative to the marginal benefit they produce. All they require are a few lines of code to implement.

Amazon is one of the pioneers of A/B testing and website optimization. One example is how Amazon tested color variations of its “Buy It Now” button. To maximize the likelihood of a customer’s purchase, or sales conversion rate, Amazon tested different shades of orange of the button, trying to understand which one works best. After thousands of tests, they were able to settle on the shade that we see today. Often times, Amazon optimizes for features that influence our subconscious - making us more likely to purchase without even us consciously realizing. This attention to detail and focus on A/B testing has made Amazon the leading shopping platform.

Historically, our ability to test and experiment has been dictated and shaped by the tools we have available to us. For example, 100 years ago we could not prove the existence of the Higgs boson particle because the Large Hadron Collider was not in existence. Today, the Internet has enabled us to leverage this type of quick, continuous, and accurate testing in ways we never could have imagined. But it isn’t stopping there, at Prayas Analytics we are hoping to leverage the rapid evolution of technology to bring A/B testing to the offline world, creating completely new ways to understand and interact with the world around us.

Want insights delivered to your inbox?