I see... so A/B testing actually works then?

If you’re anything like me you will find the psychology/science/voodoo magic behind what makes visitors convert on or interact with websites fascinating. If you share another trait with me it may be that you have tried A/B tests quite a bit in the past for various mediums and have found that on the whole the differences, or more precisely uplift in conversion rate are small.

So I was delighted to analyse a recent paid advertising campaign we managed for Brandshank.

Disclaimer: I am not giving away the numbers here, percentages only however the sample size was decent enough to make some conclusions, i.e. over 150 unique visitors.

The Campaign

Brandshank custom build websites for the music industry and we are piloting a few potential suitors for regular, large scale display advertising. The sample size of traffic for this particular campaign was the largest, and came from Mixcloud – a music sharing/streaming service. We worked with Mixcloud to only target users that would be interested in the type of solution Brandshank offer (it’s custom built so a little more expensive than free site builder services) but I will hold back the gorey details of the filtering.

On the other pilots we tested though, the results were the same but we will focus on this one as the source of the case study.

We use Unbounce as our weapon of choice for creating landing pages, split testing and monitoring the outcome.

The Bits we tested

So as you can see above, we mainly tested the “above the fold” area for this campaign.

Version A

For version A we wanted to test the theory that having a video improves conversions. So we gave that prime spot, and catered the content around it, adding in a focus to the Mixcloud offer strapline and pointing out the ease at which the prospect could interact.

Version B

We flipped the content in version B, dropped the video to below the fold and allocated more prominence to the testimonials and the obligatory “tick box” layout of benefits. We also looked to shorten the headline and add the word “Trusted” as the first word of copy.

Interestingly, both versions featured the video but only version B featured 3 testimonials. Version A carried just one, below the fold.

Conclusions & Takeaways

I was delighted to see such a large uplift from an A/B test. It is definitely the most convincing test to date and we have already started using version B against other, more refined versions of the same landing page. As far as takeaways goes from this particular campaign (of course you may find otherwise in your own testing):

- Trust prominence outperforms Video prominence.
- Multi-testimonials + Avatar above the fold = win
- Features and benefits seem to help, for above the fold “interest retention”
- Shorter headlines seem to outperform longer ones.

So that’s it – a short study but a useful one nonetheless for us and hopefully for you.

Keen to gather any of your experiences, please share in the comments!

Speak Soon

Liam

http://twitter.com/dr_pete Dr. Peter J. Meyers

Interesting stuff. It’s always tough to speculate, but I wonder if the B version just tells the story faster/better. With “Web 2.0″ services, I find that I’m so saturated that I want to know right away what a tool is and why I should care. I’m not going to sit through even a short video, most of the time, but I will read a few bullets to make a go/no-go decision (and that decision happens fast, unless I’ve got tons of social proof telling me to give the site a chance).
http://twitter.com/dr_pete Dr. Peter J. Meyers

Interesting stuff. It’s always tough to speculate, but I wonder if the B version just tells the story faster/better. With “Web 2.0″ services, I find that I’m so saturated that I want to know right away what a tool is and why I should care. I’m not going to sit through even a short video, most of the time, but I will read a few bullets to make a go/no-go decision (and that decision happens fast, unless I’ve got tons of social proof telling me to give the site a chance).
http://twitter.com/dr_pete Dr. Peter J. Meyers

Interesting stuff. It’s always tough to speculate, but I wonder if the B version just tells the story faster/better. With “Web 2.0″ services, I find that I’m so saturated that I want to know right away what a tool is and why I should care. I’m not going to sit through even a short video, most of the time, but I will read a few bullets to make a go/no-go decision (and that decision happens fast, unless I’ve got tons of social proof telling me to give the site a chance).
http://twitter.com/dr_pete Dr. Peter J. Meyers

Interesting stuff. It’s always tough to speculate, but I wonder if the B version just tells the story faster/better. With “Web 2.0″ services, I find that I’m so saturated that I want to know right away what a tool is and why I should care. I’m not going to sit through even a short video, most of the time, but I will read a few bullets to make a go/no-go decision (and that decision happens fast, unless I’ve got tons of social proof telling me to give the site a chance).

Anonymous

Yeah I think you’re right Peter. It was a really interesting test for me particularly though as there is a common theory that videos aid conversions. I think ultimately (i.e. further down the page) videos do help a user make a more concrete decision to convert but I think these days you’re right, a visitor will look for the signals that tell a “go/no-go” story right away, and above the fold.

http://www.facebook.com/profile.php?id=709841496 Isaac Hill

I think video vs. bullet points/testimonials has more to do with the type of client you are converting than anything else. Dr. Meyers, Iiamv and myself are avid internet browsers and users. We have adjusted to the better ways of gathering information and the best way to process that information as we immediately go into the decision making (whether or not to convert). Quick bullet points are the best way to get to users like us. However, the average user and the average buyer of web services look at the internet in a different way. Videos are an important part of their decision making. they dont often visit the web as much as we do. When they browse – they read and take things slowly. They will sit through a 3 minute video without thinking twice. I would like a/b split testing done a group of average users vs. advanced internet users using the same a/b landing pages for both groups. Those numbers will be very different than what you see above.