When you're deciding how your website or mobile app should look, or when you're fine-tuning a marketing strategy or online advertising plan, you are better served by conducting an A/B test than by guessing the best option. A/B testing allows you to try various options to see which one resonates most with your target audience.
What is A/B testing?
A/B testing is a process in which two or more variables are tested against each other to compare results and move forward with the better solution. Typically, this means showing two or more different versions of a webpage, email marketing message or page element to different segments of customers simultaneously. From there, businesses can compare the results of their findings and either choose the better performing of the options or combine elements of both versions to create the most optimized solution.
How does A/B testing work?
An A/B test is designed to create a better user experience for customers and drive more positive actions based on the version you're testing. A testing program can make marketing efforts more profitable in regard to what problems need to be addressed and what customers best respond to.
Businesses should research the type of A/B testing they want to use. Once they know the method they'll use, they can conduct the A/B test and analyze the results after it is completed.
Why should you perform A/B tests?
Businesses unhappy with the leads they are receiving – or concerned about a lack of engagement from customers – should use A/B tests to help pinpoint solutions. A/B tests also remain a reliable way to compare two versions of a marketing campaign, mobile app or website.
1. It solves customer pain points.
People visit your website with a specific purpose in mind. Perhaps they've come to discover more about your product or service, or to educate themselves by reading your industry blog. Regardless of their goal, they may face some pain points in pursuing it. A/B tests can provide solutions and insights that solve bad user experiences.
2. It reduces bounce rate.
If your website has a high bounce rate, it may be because you provide visitors with too many options, or because expectations are mismatched. To reduce a high bounce rate, A/B test multiple variations of an element on your website until you find the best solution.
3. It increases ROI.
An A/B test allows you to enhance the engagement of your existing traffic while providing insights on how to increase conversion rates for returning and new customers. Comparing the results from an A/B test and implementing the winning solution can give you a high ROI, which will result in an increase in your business's website conversion rate.
What data can be A/B tested?
- Copy: Headlines, body text, subject lines and blog text can be analyzed in an A/B test to see which versions users respond to more and which ones have a higher click-through rate.
- Design: Businesses sometimes struggle with the design and layout of their website, and how that affects user experience. An A/B test can discover the most optimized version of critical pages, such as a homepage or campaign landing pages.
- Forms: Website forms that allow prospective customers to contact you are vital. Businesses can test to determine which form style works better for their audience and which has a higher response rate.
Types of A/B tests
Split URL testing
Commonly mistaken with A/B testing, split URL testing refers to a process where a new version of an existing webpage is tested in order to analyze which version performs better. This method is for businesses looking to make significant changes to an existing page's design or layout. During split URL testing, your website traffic is flipped between the original webpage and the variation. Compare the conversation rates to determine the winner.
Multivariate testing
A multivariate test (MVT) is a testing method where multiple page variables are simultaneously tested to analyze which ones perform best. This type of test is more complicated than a standard A/B test, and businesses should consult marketing or web development professionals to set one up.
Multi-page testing
Multi-page testing allows you to test changes to particular elements across multiple pages. This method enables you to create a consistent experience for your target audience across all relevant pages on your site. This form of A/B testing creates a focused user experience that doesn't confuse your visitors.
How to conduct an A/B test
There are many ways to go about A/B testing and many elements that you can test for. So, what approach do you need to make the most of the testing process? Here are some of the most effective A/B testing strategies:
1. Begin with a hypothesis.
Generate a hypothesis based on a regular analysis of your site to spot potential problems. You can also use qualitative polling, surveys and usability tests to better understand what your customers struggle with. Once you compile the issues, prioritize them and decide what solutions should be tested.
2. Define success markers.
It's important to test one variable at a time, but with a reasonable measure of success. You won't completely solve the problem by changing one variable. Figure out how much improvement would represent success for the variable you're testing. Shift your measure of success for each variable depending on its importance and test it. Use your results to decide which version performs best.
3. Do research.
Before implementing any A/B testing on your site, research websites that are similar to yours or companies that have tried the same testing metrics you want to use. This additional research helps determine and prioritize A/B tests so that you are not going down a path that has already been explored and yielded lackluster results.
4. Take an all-encompassing approach.
The best comprehensive method for A/B testing is to understand the demographic you're trying to reach, create specific goals for your tests, and develop an accurate and on-point hypothesis to test. Then analyze your results and put them into action.
5. Test for mobile.
Never forget to run A/B tests for mobile components of your website, campaign or landing page. Many users will scour your content via their mobile phones or other devices, so ensure that those are always optimized for mobile. Otherwise, you could lose conversions, subscribers and customers.
6. Use Google Optimize.
Google Optimize is a top option for conducting A/B testing. It is free and already integrated with your Google Analytics. There is little to no learning curve to set up your tests, and you can measure A/B tests using your current analytics goals. This tool gives you a complete picture of the results so you can make the best decision.
7. Give yourself time to gather complete results.
Marketers love looking at data from just a couple of days and assuming they have split-test answers. The truth is that you need much more time to ensure you're looking at complete results. Let your A/B tests run for weeks or months to better gauge what your customer base wants.
8. Segment your results.
Companies often make the mistake of calling A/B test data in favor of the winning arm without digging deeper into their results. To avoid this, segment your test data by other attributes or events. For example, you may find that some test arms win on desktop, but lose on mobile. Take the time to segment the data to pull out additional insights and maximize conversion rates.
9. Use visuals and insights.
While data is incredibly useful, it can be misleading at times. Whenever you're running an A/B test, use a heatmap such as Crazy Egg and use resources like Upwork or UserTesting to gather insights from people with some familiarity with your industry. This eliminates many inherent biases you may naturally have and provides valuable insight into optimizing landing pages.
10. Remain unbiased.
The purpose of A/B testing is to find what happens with an average, neutral user. If you create the test with a bias, you're likely to skew the results by creating an option that points the user toward your preferred outcome. Be sure that you're not designing your study to find the answer you want, but rather to discover what works.
11. Follow the 45/45/10 rule.
Myriad people with disabled cookies, slow connections and other technical details will skew the results of traditional 50/50 web testing. Instead, you want to run 10% of the original page, 45% to test A and 45% to test B. The people who cannot load the test will automatically be shown the original copy (10% segment), and your data will be more accurate.
12. Don't be afraid of complexity.
When running A/B tests with landing pages or ads, many users run sequential tests that only vary one factor at a time (i.e., color, font or placement of a web button). If you do this, you'll quickly realize that multivariate testing is more accurate. By adding complexity, you can understand how a combination of factors works together to drive the preferred option. Start slow, then build up to multivariate tests.
13. Test both variations at once.
When A/B testing, if you run the first variation during the first two weeks of the month and the second variation during the last two weeks of the month, those different times of the month will give you different results. The month, the day of the week, and the time the test is run are all significant. Make sure you're testing both variations simultaneously to get accurate results.
14. Start with a clear idea.
Begin with a good control version of your ad or landing page, then create multiple new variations to test against that. These variations have to perform better than the baseline results of your control version. Test anything, such as your messaging, call to action, button color and the location of the call to action to see if variations drive better results than your control.
15. Use the right software.
The right tools can make or break your company's progress, so it is important to invest in tools and software that will give you the results you're looking for, whether that's now or further down the road when your company has grown more.
16. Collect user feedback.
While A/B testing will show you what works and what doesn't, it doesn't tell you why. Along with A/B testing, get feedback from real users as well. Gathering feedback from your customers through an online survey will allow you to better understand why something didn't work for them. The results of your A/B tests and customer feedback surveys will give you the full picture.
Scott Gerber contributed to the writing and research in this article.