It’s not a secret various customer groups require unique marketing approaches. A/B testing, or split testing is a fundamental method to achieve business goals through improving conversion and other online marketing KPIs and parameters.
For e-commerce, the main page you would want to test is the product page. How to perform an A/B test on a Magento product page? Read on and follow a step-by-step guide.
Things you should know before starting your Magento A/B test
Note: there are many kinds of testing software available. But writing this I assume you’re using Google Analytics experiments feature.
- Google lets you test performance of the whole pages, not separate blocks on a single page. This means you should create a duplicate page for each test. For example, if you need to test product images, you have to create a new product page which is similar to the original one but has different images.
- You should have an access to the <head> section as you will need to insert some code to a single product page but not to the template. If you don't know how to do it, ask a developer to insert the code.
- You should test one element at a time. This means that if you aim to test your product description font, you shouldn’t change anything else. Otherwise you won’t be able to determine which element caused increase or decrease of the conversion rate. However, there may be exceptions to this rule if you want to test a new design.
- You should have a detailed plan of split tests instead of setting up them randomly. Remember: your aim is not conducting the experiment itself. Start from your goal. Say, you have to increase conversions on the product page, so choose elements that have the main influence on the conversion rate: Buy button or its alternative, product image, description, price. (For more ideas, check this article: 6 Things to A/B Test on Your Magento Product Page) Sketch a plan of the elements you want to test, think of which alternative you're going to test along with original elements, and create a timeline, taking into account seasonal traffic and your marketing plan with discounts and offers.
- One of the most important things in A/B tests is choosing the objective: a metric according to which the winner page is chosen. In most cases you will use one of your Google Analytics goals.
Creating a duplicate product for A/B testing in Magento
- Copy the product you want to test.
Navigate to the product editing page and hit the Duplicate button.
The duplicate product settings must be identical to those of your original product.
- Change SKU
An SKU field should be unique for each product, so change it by adding any number, character or word. For example:
- Change URL
The URL of your test page should also differ from the original one. Note that you shouldn’t let Magento automatically create a permanent redirect for your old URL:
- Change visibility
Make your test product accessible by direct URL only. You can achieve that by setting its visibility to Catalog (i.e. it won’t be seen in search results on your site):
Catalog visibility means your product can be accessed via categories it is assigned to. So the next step is to exclude your test product from all the categories. This can be done on the Categories tab:
- Set meta robots for the test page
You don’t need your test page to be indexed by Google, so restrict access for spiders by adding meta robots to the <head>:
<meta name="robots" content="noindex, nofollow">
- Copy product reviews
Product reviews have direct influence on sales, so there is no point in testing anything if your test page doesn’t show reviews that are seen on the original page. The exceptions are:
- Your aim is to see reviews influence on sales by removing reviews from the test page.
- The original product page doesn’t have any customer reviews.
- Duplicate all the possible settings
There is no universal solution for that, because each Magento shop has its own design and collection of extensions and customizations. You will need to check field after field, tab after tab to make sure you have two identical products. Except for the single element you're testing, of course.
NB: You may have some product options bound to SKUs or URLs, so don’t forget them, too. For example, if your images are automatically loaded from the folder named just like product SKU, copy this folder and change its name in accordance to the SKU of your test product. This way all the needed images will be loaded on the test page.
Setting up A/B test for your Magento shop in Google Analytics
Once your test page is ready, it’s time to set up a Google Analytics experiment.
- Log in to your GA account and navigate to Behavior – Experiments
- Click 'Create Experiment', and you will be redirected to the page with experiment details:
Choose a descriptive name for your test, this will help you analyze tests in the future.
Now it’s time to choose the objective, this metric can vary from experiment to experiment; from store to store.
As for the traffic percentage, I normally use 100% of visitors and have them distributed dynamically across page variations based on their performance. The latter means that the better page converts, the more visitors are sent to it during the experiment. This helps not to lose conversions if one of the pages fails to convert. You can use this method on popular pages to receive results faster, but at the same time you won't lose a lot of conversions thanks to this mechanism.
You can also choose to distribute traffic evenly if you want to. This can be done in the advanced options (you can see it on the screenshot above).
- Once you completed this step, save the changes to be redirected to the next step:
Add your original page as well as the test one, you can also add descriptive names for them here.
- The next step is to copy the experiment code and insert it in the <head> of your original page
How does it work? This code is responsible for distributing visits between the original and the test pages. All customers visit the original page first and then some of them are redirected to the variation. At the same time, the variation can be accessed through the direct URL but in this case the experiment data will be inaccurate. That’s why I wrote above how to make test page invisible for regular users and search engines.
- Review the experiment and launch it.
Tip: the number of running Google Analytics experiments is limited to 12. If you happen to plan more simultaneous experiments, you can copy Google Analytics view, add the same filters (if any), goals, etc. and set up 12 more experiments.
Analyzing A/B tests results
Traffic segmentation
Once your experiment is over, you can analyze the results and act according to the data you received. Don’t forget to segment your traffic to see the whole picture. Usually I segment visitors the following way:
by behavior:
New vs Returning
and by traffic source:
- Direct
- Organic
- Paid
- Referral
You can also compare performance of mobile and desktop visits as well as use other segments:
Are the results valid?
Normally, GA or any popular A/B testing tool does the calculation for you. If you want to double check the results, here is an A/B tests significance calculator that may be of help. You can also find other calculators or mathematical formulas to dig into details.
Important things you should know
Avoid quick decisions
Say, the experiment has been running for 2 days and you already see that the original page converts better so you decide to stop the test. That’s the wrong decision.
The experiment period greatly depends on your traffic: the more visitors you have, the shorter your experiment can be. But try to wait at least 2 weeks before taking any decisions, you will see that the results may change from day to day and at the end they may be completely different from those you observed at the beginning.
Don’t make anything special with the tested pages
Say, you are A/B testing product images for you cupcakes page. The experiment is running and you decide to launch a promo campaign with 30% discount on the cupcakes. I’m sure you’ll increase sales but the experiment results will be inaccurate, as your promotion but not images, will be the driving force for sales increase.
Also, think of the time you will be conducting the test. Results you receive from seasonal Christmas traffic will be applicable for seasonal visitors, but not for regular traffic.
Lesson learned: don’t try to use anything special to enhance sales of the tested product.
Test different variations at once
You need to test three fonts: the original one and two others. So you decide to create one test page and test two fonts at first. Once the winner is found, you set up a new experiment to test the next font. Wrong! You won’t get accurate results as the conditions and the time of your experiment have changed. That means you’d better test one thing with as many variations as you need. This is also known as multivariate testing.
Summing up
I’ve tried to give you a full overview of A/B testing in Magento. But you should understand that there’s no single guideline that provides all the answers for super successful tests. You’ll see that some of them fail and some turn out to be great conversion boosters. Moreover, you will face new challenges each time you will set up tests, but you’ll definitely learn something new about Magento and your audience. That’s really interesting! If you've tried this guide and feel like you need more for your Magento store, feel free to have a look at Magento A/B Testing extension from Amasty.
In a nutshell, you should test, test and test different things in your Magento store. That will help you to stand head and shoulders above the competition.