Amazon A/B Testing Quantified How Much I Suck at Improving Conversions (A Case Study).

Subscribe to blog via email.

These days, the majority of our Marital Bliss sales come through Amazon, so I spend a fair bit of time tweaking the landing page and praying to Jeff Bezos to maximize our conversions. (I know, I know, Andy Jassy is the CEO of Amazon now, but what am I supposed to do with my basement Bezos shrine that I built during the pandemic?)

Back in July, I started playing with Amazon’s A/B testing features. If you’re unfamiliar with the concept of A/B testing, it’s functionality that lets me create two separate pages for our game and then over the course of several weeks, it shows half our visitors version A, and it shows the other half version B, and it tracks which one performs better.

…by the way, this happens all around us every day, and we rarely notice it. Have you ever read a news article that pops up again later in the day with a different headline? Congratulations, you’ve been A/B tested, and the headline you read the first time likely didn’t get as much engagement as the second one you saw later.

Anyway, back to Amazon. The functionality has a few quirks that gave me some headaches. On the setup side, I never found a way to save a draft of the page I was updating. This is a nightmare for someone like me who likes to constantly tweak things before publishing it.

Secondly, the site enforces a TWO WEEK lag between when I submit the B content for approval, and when the test actually starts running. A real person somewhere reviews the B content to make sure I haven’t put anything outrageous in there, and then it gets approved (in a day or two in my experience) and then it just sits there, waiting, for two weeks before it actually goes live. The minimum amount of time that an A/B test can run is 4 weeks… so in total, it takes 6 weeks to run a test, which feels like an eternity to me.

Fortunately, once it does go live, the results are really interesting.

Version A

Here’s our current page (Version A) and reigning champion. In my hubris, I thought improving my page would be easy. My opening sentence is so hilariously stuffed with keywords that it barely make sense. Keywords are highlighted for easy reference:

Traditionally, if you are looking for a gift for your wife or husband or date night ideas and want to play an adult game you are usually stuck with two options: uncomfortable and cheesy relationship games focused on romantic conversation cards OR a sex card game with some over-the top dice.

Version B

For Version B, I opted for the “kitchen sink” approach, and I paid the price.

Yes, Version B is really long. “More” has now been quantifiably demonstrated to NOT be better.

Additionally, I used some new imagery and wanted to test them in black and white, thinking it might help the page stand-out in an otherwise color-dominated environment. Spoiler alert: It didn’t.

The Results

The results are pretty compelling. At the end of the experiment, Amazon assigned a 74% probability that Version A was better, which feels right to me, and gave me some helpful metrics to reference if I ever need to quantify how much I suck at trying to improve our product page (for now!).

The outlier week 4 is interesting, but there’s nothing substantive to say why that week was different. Plus there’s a lot of variables at play. Here’s one example – we get a decent amount of repeat buyers who purchase the game as a gift for weddings, anniversaries, etc. Those people obviously already know that they’re going to buy the game when they hit the page, so even though those show up as units sold in the final metrics, it didn’t matter which version they landed on.

Next Steps

The next A/B experiment is already locked and loaded. Starting on 9/28, feel free to hop on our landing page and buy 30 units of Marital Bliss. Oh boy, you’ll really skew the numbers.

Our updates, your inbox

After subscribing, look for our confirmation email. If it goes to your junk or spam folder, add us to your safe senders list!