A/B TESTING: STEP BY STEP

How to use an A/B Testing and the MVT Framework to increase your conversion rate by up to 463%

Growth Marketing Cases

Marketeers love A/B testing. Some even have it for breakfast in the morning.

That’s why we made this practical guide, from a real case done with one of our best clients, Pontomais.

What you won’t find here are miracle theories or “5 Tips on how to do an A/B Test”.

What you will find is the complete strategy that we apply with Pontomais (and other clients) to achieve results such as increasing the conversion rate of a page by up to 463%.

With this guide, we want you to be able to apply the ideas and concepts in your A/B tests from now on.

Good read, isn’t it?

A/B Testing on Landing Pages: Step By Step

How to run a successful A/B Test, from start to finish.

WHAT DOES AN A/B TEST MEASURE?

If you already have experience with A/B tests and know the concept, you can jump to How to run more effective A/B tests using the MVT methodology..

A/B tests are one of the main tools of Growth Marketing.

An A/B test is a method of comparison between two versions of a web page to determine which has the best performance to achieve the targeted outcome.

These 2 versions are:

  1. Control Group: This is the original, allows for comparison of the status quo of the experiment’s output.
  2. Test Group: is the modified version of the control group (from a hypothesis) to test if there is a significant difference between them.

The methodology consists of displaying the different versions at random to different visitors of the website. Then, a statistical analysis is done to understand which is the best version for the given objective.

Imagine you’re a company that sells coconuts online. You have a landing page with the title Buy from the best in the market” and your competitor is using Quench your thirst with a fresh coconut.

Will changing the headline of your page to resemble your competitor increase sales?


An A/B test is perfect for this situation.
Post a page with a different headline and measure the results.

The 5 Steps of an A/B Test

An A/B test isn’t complicated. Conceptually, it is similar to the scientific method and consists of 5 steps:

With these 5 steps, you can run A/B tests that work:

  1. Observation
  2. Hypothesis
  3. Tests
  4. Analysis of results
  5. Validation of the hypothesis

Although simple, many teams continue to commit mistakes by not following this method and fail in the process.

The consequence is to waste time on tests that make no sense and without statistical validation.

We will explain how the A/B testing process at Traktor works and how we use the MVT methodology to achieve results such as increasing the conversion rate of a page by 463%.


HOW TO RUN MORE EFFECTIVE A/B TESTS USING THE MVT METHODOLOGY


#1 Finding test opportunities

How to generate a hypothesis from the information you already have?

The answer is data and observation. That’s how you create good tests. And of course, you should pay attention..

That’s how an opportunity came up with Pontomais.

Navigating through this page, we realized that there were too many elements.

The site had many different CTAs. Of course, this confuses the user and makes it difficult to understand what they should do on this page.

As soon as we noticed that, our team went to take a look at the page’s conversion rate – the rate could improve, a lot. So the A/B test was performed on this page.

I imagine you’ll think it’s not always easy to find all these opportunities for testing..

Indeed, there are many possibilities to explore possible tests. 

We’ll give you 8 ways to collect ideas:

#1 Use your website data

Use the tracking tool to your advantage: find pages with considerable traffic and low conversion rates, high bounce rates, etc. 

In addition, having recording tools like Hotjar are useful for understanding where users click most and what they ignore the most.

#2 Check out your competitors’ websites

Research your main competitors and see what they do differently. 

Do they exploit different customer needs? Content in different formats? Run a trial test.

#3 Support, Customer Success (CS) and Sales

These are the people who have direct contact with the customer.

Discuss, understand if there are sentence patterns, objections and frequently asked questions by customers and test with these terms.

#4 Talk to your customers

In surveys and feedbacks, customers share from the complaints to what they like most about the product.

Use the phrases they use in the copy of their pages.

#5 Successful Ads

Successful paid media campaigns have phrases, concepts, value propositions, and visuals that can be reused on your site.

Become inspired by your best ads and test yourself.

#6 Simplify

Often, we try to persuade users of too many things on one landing page. 

How can you make a page focused and clear to the user?

O Julian Shapiro has developed an equation that explains that point very clearly: 

Conversion rate = Desire – Work – Confusion

#7 Copywriting Tactics

You can use modelos de copywriting para fazer testes. to run tests. Choose one and do a test to see if it improves the conversion rate.

#8 Value Propositions

Don’t forget to test different value propositions. They can have significantly different conversion rates.

Value propositions are different from the qualities, benefits and features of your product. 

For example, let’s assume that we have a product that automates invoicing:

Quality / Feature Benefit Value Proposition
Automate invoice issuance Less Manual Work Never do the boring part again
Automate invoice issuance Regularity in delivery Never forget to issue invoices again
Automate invoice issuance Increased efficiency Gain time to work on what’s important
  • Quality / Feature: Automate invoice issuance
    • Benefit: Less Manual Work
      • Value Proposition: Never do the boring part again
  • Quality / Feature: Automate invoice issuance
    • Benefit: Regularity in delivery

      • Value Proposition: Never forget to issue invoices again
  • Quality / Feature: Automate invoice issuance
    • Benefit: Increased efficiency

      • Value Proposition: Gain time to work on what’s important

2. Formulate the Hypothesis: What is the origin of the problem?

Once we’ve decided where we’re going to take the test, it’s time to formulate the hypothesis..

As the page had many different CTAs, the hypothesis developed was: “if the page had only one CTA of focus, the conversion rate would increase”.

Assumption: The page with only 1 conversion focus will increase the conversion rate.

The goal is to give less options to the user, so far so good. The important question is: which CTA will we put on this page?

The options are endless. From “talk to a consultant” or “talk to sales”, to “buy now”.

To make that decision, it is important to ask two things:

  • What kind of traffic is coming to that page?
  • At what stage of the purchase process is the customer?

From these questions, you will understand how to guide the user experience within this page and it is easier to define where to direct users.

In this test, the page was a paid media landing page (Google Ads and Facebook Ads). That’s the answer:

What kind of traffic is coming to that page? Cold, paid media traffic
At what stage of the purchase process is the customer? Top of the funnel.

That is, most likely where these people were having their first contact with Pontomais when they landed on this page..

Therefore, the decision was to direct all possible actions to a single CTA:

“Take a free trial.”

Hypothesis formulated, all set. Time for action,  run the test.

#3 Elaborate the test for the hypothesis (MVT)

It is common for oneself (or your team, or boss) to have a belief that something will work or not, based on absolutely nothing even before validating a hypothesis.

Choices of what will be developed from this assumption can be disastrous.

In that case, with Pontomais, even before the test began, we strongly believed it would work.

Even if the users’ desire to convert and get to know Pontomais’ product was great, the Work they had to do to understand what they were supposed to do was very great. 

Not to mention the Confusion that was generated by so many possible and different actions that users could take.

Even so, we like to respect the statistics and what the figures tell us, but getting that answer with a high level of certainty takes time.

Time, especially in fast growing companies, is essential..

That’s why we use a framework to run tests and understand what works fastest (and continue with our consideration for statistics).

It is the Minimum Viable Test which we will call MVT. This concept is derived from Eric Ries’ model in Startup Enxuta, the MVP.

MVT is simple, it consists of a question:

What’s the least I need to do to validate this hypothesis?

Simple, yet effective.

In that case, we simply cleared the page of everything that was unnecessary. 

We directed all actions towards “free trial” and made some changes to the page copy.

  

And when I say everything, I mean everything – including the header.

The user only has one mission on this page, turn into a lead or leave..

The goal here was clear to increase desire, decrease effort and confusion.

Doing all this took about 30 minutes, but these simple changes already left us in a comfortable position to perform the test.

How to decide which elements to change

Could we just move it or the headline? Or just a button?

Yes, all options were feasible.

But as I said above, we like to have agility in the tests we run. 

You can test by changing the color of a button, but what is the real impact of this on your business in relation to the resources to run the test?

Aiming at small increases in the conversion rate (5%, by changing the color of a button, for example) ends up generating inconclusive tests in many cases. Would you leave an A/B test running for 488 days?

Change the color of a button Test value proposition

(you can calculate your test time here)

This is the example of a micro test, which can take a long time and have a low impact result (5% variation in the conversion rate).

In addition, this type of test requires a high volume of traffic to confirm the hypothesis.

Already doing a macro test, besides taking less time, serves as learning to generate new hypotheses of what is working, and thus perform new tests. 

Micro test

Change small elements within the page, such as the color of a button, a phrase, or an image.

Macro test

Make radical changes, such as the value proposition of the page and remove elements.

Macro tests are more labor intensive, from the moment of conception to the application. One way out is to gather several micro tests at once, turning them into one macro test.

Micro test: change small elements within the page, such as the color of a button, a phrase, or an image.

Macro test: Make radical changes, such as the value proposition of the page and remove elements.

Macro tests are more labor intensive, from the moment of conception to the application. One way out is to gather several micro tests at once, turning them into one macro test.

#4 Using Google Optimize to run a test efficiently

Changes made, it’s time to go live with the test.

That is an extremely important part of doing A/B tests. This is where most teams fail.  

In the rush to go live with the test quickly, many may resort to shortcuts, such as putting the test online without the proper tools.

What’s the worst that can happen? 

You can blow up your test results. Without the right tools, it is difficult to measure and compare the results in a satisfactory way.

Without reliable data to analyze you continue to be in the dark – decisions are made based only on the beliefs of those who are executing them. So…

Simply don’t do it!

The Google Optimize A/B Test Tool is free and helps you run a test the right way. Remember that here we are talking about an A/B test on a landing page..

If you are going to perform other types of testing, such as a test involving pricing your product, you will probably need an extra effort with internal development for the test to work.

In this case, we use Google Optimize.

I will explain how to install and set up a Google Optimize test on a WordPress site.

How to do an A/B test on a WordPress landing page

f you already know how to install and use Google Optmize, skip this walkthrough by clicking here.

If you have WordPress, the best A/B test tool is Google Optimize. A/B testing in Google Analytics though is not the ideal solution.

Google Optimize will split your page traffic and measure user behavior on each one. In real time, it shows you the performance of each page and the difference between them.

How to create and set up a Google Optimize account

Create a Google Optimize account, create a container, and register your domain.

You can follow Google’s own guide on how to conduct your first test. Remember that if you install the plugin Google Optimize from, you can visually manipulate the elements on your website page.

Don’t forget to link to Google Analytics and select a conversion target that matches the experiment.

Now let’s explain the installation of the Google Optimize Snippet.

This installation is different depending on how you installed Google Analytics.

Direct installation by Google Tags Manager:

You can install Snippet directly through Tag Manager by following the Google instructions. However, this is not the recommended way to install, according to Google itself.

To ideally install the snippet by Tag Manager, pause the Google Analytics script in Tag Manager. Then, enter a modified version of the code directly on your website (the modified version is here).

Installation for those who installed Google Analytics manually:

If you have installed the Google Analytics script manually, just replace it with a modified version that we’ll show you below.

Before you do that, you need 2 IDs:

  1. Google Tag Manager container code
  2. Google Analytics container code

To find these 2 IDs, you must go to the Google Optimize homepage.. The black ID is from Optimize; the blue, one from Analytics.

Check the location of both IDs in the image below:

Write those IDs down on a notepad!

You can use the Insert Headers and Footers Plugin to insert the IDs into WordPress.

Another option is to install WordPress manually, and click Options > Insert Headers and Footers.

Now adjust and paste the following script passages:

Section 1

<!– Anti-flicker snippet –>

<style>.async-hide { opacity: 0 !important} </style>

<script>(function(a,s,y,n,c,h,i,d,e){s.className+=’ ‘+y;h.start=1*new Date;

h.end=i=function(){s.className=s.className.replace(RegExp(‘ ?’+y),”)};

(a[n]=a[n]||[]).hide=h;setTimeout(function(){i();h.end=null},c);h.timeout=c;

})(window,document.documentElement,’async-hide’,’dataLayer’,4000,{‘GTM-XXXXXX‘:true});
</script>

Replace the red with the Google Optimize ID.

Section 2

<script>

(function(i,s,o,g,r,a,m){i[‘GoogleAnalyticsObject’]=r;i[r]=i[r]||function(){

(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),

m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)

})(window,document,’script’,’https://www.google-analytics.com/analytics.js’,’ga’);

ga(‘create’, ‘UA-XXXXXXX-Y’, ‘auto’); // Update tracker settings

ga(‘require’, ‘GTM-XXXXXX‘); // Add this line

ga(‘send’, ‘pageview’);

</script>

In Blue, replace with Google Analytics ID.

Already in Red, by Google Optimize ID.

Put the two script passages together.

Paste these settings into the first field, and save.

Ready! Your Google Optimize will be up and running.

#5 How to learn from MVT: analyzing results and making decisions

That’s the time that data scientists start to run away.

As I said before, we want statistics + speed, simultaneously.

Ideally, when running a test, you should have a 95% statistical confidence interval. That means that if I repeated that test 100 times, in 95 of them I would have the same result.

As we were testing the MVT, for the minimum level we can use to validate our hypothesis, an 85% confidence interval was used. We have defined that confidence interval as acceptable. We will be wrong 15% of the time, but it is a price we are willing to pay for agility.

Remember, you should always keep an A/B test as short as possible, because it slows down the webpage. This is detrimental to the user experience, which usually causes a lower conversion rate.

That was the result of our MVT test:

The conversion rate rose from 1.19% to 1.80%. 51.2% increase the conversion rate of the page – quite a lot!

What does that mean?

That means our hypothesis is, at first, proven.

eff Bezos, Amazon’s founder and CEO, says that to maintain agility in decision-making and innovation happening in the company, he encourages his team to make decisions with 70% of the information. Waiting to get 90%, confidence levels in most cases means significa que you’re being too slow.

That was the main reason why we, together with the Pontomais team, decided that we had enough information to make the implementation of a new page without header and with the changes made.

We knew of the need for a new page, because in MVT it is natural that you work a little below ideal. At that time, our team started building the new Landing Page for paid media, using the learnings we got from the test.

The result? It was way above expectations.

#6 Turning the key

The new Landing Page was made from scratch by our team.

As we realized that making the page cleaner in the test improved the conversion, we decided to redesign the new page design with the same idea in mind as well.

In addition, it has become easier to reshape the page’s value proposition and emphasize the product’s benefits in the best possible way. 

Basically, we have reduced the amount of Work and Confusion for users on the page.

Below is an example of how the page looked, but you can check it out in full here:

After all the learnings from the test and the production of the new page, we put it to run in the paid media campaigns and wait for a week. 

What happened?

The page had a conversion rate of 5.51%.

This is a relative 463% increase in the page’s conversion rate. If before, for every 1000 visitors we turned 18 leads, now we generated 55 leads.

This is one of the benefits of doing an A/B test, achieving results of this level in a short time.

Of course, even after optimizing just one page (increase the conversion rate by more than 400%), the tests should continue. 

Especially with regard to the value proposition of the product. Testing different approaches with different audiences can have a surprisingly different outcome.

We have reached the end of our case!

We explained how the MVT methodology works and the results it brought to our client, Pontomais.

Learn more about our work at TraktorCast – sincere episodes on growth marketing to accelerate your business. 

Want to know more about our conversion methodology? Talk to us through those channels: 

              

Sources: 

Who made that beautiful thing up here:

Adamante, Gabriel

Adamante, Gabriel

Technical Marketer, coding and stuff

Our technical manager, Adamante takes care of performance, SEO, Growth Hacking, Business Intelligence and the like. He’s a marketeer who’s learned to program, but he’s always been bored as a developer.
Oliveira, Aleph

Oliveira, Aleph

Client Owner

Although he doesn’t look like it, Aleph is the son every mother dreamed of: organized, intelligent and productive.

He’s the one in charge to structure marketing and sales processes and to arrange things to run like clockwork.