During his election, Barack Obama raised $690 million from his supporters in online fundraising. His team was obsessed with Split testing everything around his campaign. In one A/B test, for example, his team learned that one subject line generated $403,600 while the other $2,540,866. That’s a $2.1 million difference caused by the subject line itself!
There was one particular experiment that is believed to earn $60 million. This experiment served at the very beginning of the campaign to validate multiple hypotheses about the best positioning and media type of Obama’s campaign. Altogether, the team tested 6 messages and 4 CTA buttons in a rigorous multivariate test.
The metric used to measure success was the signup rate. There were a total of 310,382 visitors to the website during the experiment, which meant roughly 13,000 people saw each variation. The results were as follows:
The winner was this combination of a family-oriented image and the “Learn more” call to action button. This variation had a sign-up rate of 11.6%, while the original page had a sign-up rate of 8.26%. That’s an improvement of 40.6%.
It’s also worth noting that all video versions performed worse than those with static images. That was quite surprising for the team that favored video content.
So how did this experiment earn $60 million? Throughout the campaign, roughly 10 million people signed up. Since the experiment improved the conversion rate by 40.6%, it would have been closer to 7,120,000 without it. That’s a difference of 2,880,000 sign-ups.
Roughly 10% of email addresses donated $210 on average throughout the campaign. This means that every email is worth $21, which translates into $60 mil for the 2,880,000 emails that the experiment helped to bring in.
Share your experience
Have you tested an interesting experiment that you’d like to share? Publish it on the largest online directory for experimentation-led growth ideas! Influence thousands of growth experts and get recognition for you and your company.