A/B Testing is an experiment to compare two or more variants of campaigns and find out which one performs better. In this article, you'll learn how to create an A/B Testing campaign.
Things to note:
- It is recommended to start your A/B Testing campaign with at least 5,000 recipients in order to get the most useful data from the experiment.
- You can send the test campaign to remaining recipients during the test phase by manually picking the winner from the Campaign Details.
- The recipients for each variant of campaign are picked randomly by the system.
Create A/B Testing campaign
To create a new A/B Testing campaign:-
- Go to Campaign tab.
- Click Add New Campaign button.
- Enter the campaign details and enable A/B Testing, toggle the switch next to the A/B Testing.
- Click Next.
- In A/B Test view, choose the variable you want to test.
- You can add up to 5 variants for each variable you pick.
- When you have completed the A/B Test, click Next to configure your A/B Test metrics.
A/B Test Variables
Under the experiment, your campaign variants will be identical except for the variable you pick. In Enginemailer, you are able to pick 3 variables as below.
- Subject Line
You can write different subject lines and see which subject line get the most attention. For example, you can also try to add 'Merge Tags' to the subject line to see if it gets better response. - From Name
To experiment if your recipients engage higher when the sender's name is a person's name or when is a company name. - Content
Create few versions of content to see which content has higher conversion rates. Try to experiment on the Email Section Titles, Length, CTA buttons, images and more.
Configure test metrics and Send
The last step is to define the size of the test group, how you want the winner to be chosen and how long for the test campaign to run. A/B Test metrics section will be disabled if your variant combinations are more than available subscribers.
As default, we will select a winner by click rate after 4 hours running the test campaign.
Once you have define the test metrics, click Send or Schedule button.
Define the Send Date and Time
In here, you are allowed to set different Send Date and Time for each variant. When completed, click Schedule Campaign or Send Now button to begin your A/B Testing.
Monitor A/B Test results
While the A/B test is still in progress, you are able to monitor the results in Campaign Summary screen by clicking the A/B Testing campaign name in the campaign listing.
The table below is an example how the result will looks like in Summary. To see the detailed reports, simply click on the View report button. (View report button will be temporary disabled until your campaign variants completed the test)
If you would like to send out the campaign manually without waiting for the system to decide the winner. Click on the circle checkbox next to the campaign and click Send to remaining recipients to end the test early.
After winner has been decided, through test completion or manual selection, the remaining recipients will be sent the winning version and full campaign report will become available.