What is A/B Testing?

A/B testing can be used for email marketing, testing two different landing pages, display advertising, and other marketing tactics. This is different from multivariate testing, which can be more than two factors…and a topic for another blog.

1. Understand your audience, by applying the basics of User Experience (UX) so you can design your test around that audience’s potential conversion
   • Make sure you choose a critical mass so you can get statistically significant results.
   • Define the timeframe of your test—some say you need at least 1,000 visitors to get enough critical mass.
   • Determine what your drivers of traffic will be—dig deep to understand what will make your audience act.
2. Design your test
   • What are you trying to learn? Keep in mind your objectives might take different approaches and you narrow your test as you learn. Make sure you are isolating your variables to determine what factor is driving your results. If you have too many variables, it will be difficult to attribute results to just the driving factor.
   • Consider what you are trying to learn: is it about the headline, benefits, images/videos, calls-to-actions (CTAs)?
     • A few questions you might ask about your CTA: does it stand out? Is it in the right position or location on the page? Depending on the conversion: does it sound like too much of a commitment? Is it clear what the value of the action will be?
3. Some additional considerations
   • Consider different tests for desktop vs. mobile as they are each different experiences and potentially different mindsets for your audience.
   • Consider if you’ve provided the “path of least resistance” to a conversion: is there too much copy? Too little copy? Can the ask be simplified? Did you make it relevant by personalizing: you can add on name, location, context, time, date, etc.
4. After you’ve run the test, see what your Conversion Rate Optimization or CRO might be
   • Check if you have achieved statistical significance – check out this calculator from Optimizely.
   • Keep testing: how can you narrow down your results further and achieve better results?
   • You can also supplement a standard A/B test with more qualitative data like an exit survey.
5. Specifics for email marketing
   • For a split test, you can use HubSpot or other marketing automation tools to split your list or if you’re working with a media partner, you can also ask them to split your list equally, as long as it is done randomly, so as not to skew your results.
   • Keep in mind you can always email 20% of your list first, then use the winning creative to the remaining 80% of your list to try and get the best results.

Something that both consistently astonishes and delights me is how testing has often proven the appeal of animals in advertising. As some of you may know, we have our own celebrity dog – Walter. In fact, he has his own blog.

Now let’s put the classic rivalry to the test of dog vs. cat.

If you enjoyed this blog, you might want to get more information like this on a regular basis. Please help us by selecting which CTA would motivate you to sign up for the KHJ blog. I’ll report back on results in a few weeks and share our learnings.

If you want to dig a little deeper, check out these resources and blogs:

Optimizely’s free A/B Test Sample Size Calculator

How to do A/B Testing: A Checklist You’ll Want to Benchmark from Hubspot

36 essential A/B best practices to boost your conversions from WP Curve

How to Build a Strong A/B Testing Plan That Gets Results by Conversion XL