What is A/B testing?

In a digital environment where everything can be tracked and measured, testing your strategies is a no brainer. Content strategy plays a key role in driving organic traffic and converting sales, so by leveraging and optimizing your content, you can increase the chances of shoppers purchasing your product over your competitors. A/B testing is one of a handful of the best practices you can use to enhance content on the platform.

Also known as split testing, A/B testing on Amazon is a method of determining the best-performing product listing variation. By comparing two versions of the same content, you can identify the exact element that is driving purchase-ready shoppers to act, taking into account metrics such as conversion rate, sessions, and total sales. That being said, you should have a strong understanding of your current metrics, performance and challenges before building an A/B testing plan. This will allow you to have a solid base foundation before making any changes.

Start by running an A/B test experiment on Amazon.

Amazon launched its own A/B testing tool in 2019 called Manage Your Experiments, which allows US brand owners to test two variations of one product listing element. As a brand owner wanting to test different A+ content elements, you must have eligible ASINs, otherwise they will not be displayed in the A/B test experience. To be eligible, ASINs must belong to your brand, and they must have high enough traffic in their respective categories to determine content winners with confidence. 

Keep best practices for A/B testing in mind.

By following a few tips, you can begin your test experiment with confidence. Here are some recommendations:

• Strategically create a hypothesis to learn something from the experiment regardless of the outcomes.
• To get a larger sample size, experiment on high-traffic ASINs.
• Stick to making one change at a time to avoid confusion on which variables influenced the experiment outcomes. 
• Don’t stop early – you can run experiments for four, six, eight or ten weeks to ensure accurate results.

Get to know the elements of A/B testing.

Nearly any element of your A+ content can be A/B tested. However, you should use your own judgment to determine what to test based on your current performance. Consider working with an Amazon Ads Partner to strategically outline the testing method. Some variation ideas for your A/B experiment: 

• Utilize a comparison chart.
• Rearrange and update your modules and images.
• Present the same images and text using a different module layout.
• Include lifestyle images.
• Highlight one set of product features versus a different set.
• Add your brand name to the product title.
• Use different headlines to engage and motivate shoppers to learn more about your product.

Keep in mind that your experimental content should differ from the existing content, otherwise it is less likely to affect consumer behavior and you may not be able to confidently determine a winning variation. The key to a unified branding strategy is consistency, which applies to your color scheme, icons, layout, and even text fonts.

Wait for optimal data.

Patience is key here as most tests need several weeks to gather enough data to determine a winning variation. While Amazon recommends testing for eight to ten weeks, you can adjust the schedule or even turn off the test while it is running. You will start to see data within one or two weeks, although this preliminary data is not representative of the true impact of your experiment. Be sure to let your experiment run its full course before interpreting the results and making a decision. 

Gauge the effectiveness of your experiment.

While launching new marketing initiatives on Amazon can lead to an increase in traffic, you should know how to gauge the effectiveness of each initiative to dial in on what works best. 

Once your A/B test ends, you will get the following from Amazon: 

• Recommendations on which content variation is more effective
• A confidence level of the recommendations
• A confidence interval of likely outcomes from that content
• Estimated 12-month impact on sales

These insights will highlight the most effective content for your product detail pages to boost total sales and conversions for your Amazon products. By learning from these experiments, you can optimize and improve your other product listings. You might find it beneficial to run experiments during different seasonal periods as this will provide you with a better understanding of your consumers and their expectations. 

When analyzing your results, be sure to keep your audience in mind to make the most informed decision. Although it takes time and patience to run A/B tests, the optimization is worth it if you want to dominate your market. Happy testing and selling!

CMOs understand this need to meet audiences where they are more effectively: research from WARC shows that half of marketers say they are spending more on performance this year than last. If marketers are not marrying the performance levers of media and measurement to that of creative, they are not leveraging the most-malleable potential for aptly fueling for performance.

Adopting the performance creative mindset does more than enhance the rigor of ideas; it helps brands develop creative that is authentic to each channel and made for a target-specific audience. If done strategically, it educates and informs consumers, meeting them where they are—at the right place and right time, motivating them to engage with the brand speaker and (eventually) move down-funnel to purchase. Let’s dig into the tenets for performance creative success.

Be data-first, always.

Rethink your approach to creativity to make room for data inputs and outputs. Challenge yourself to view each new idea as potentially iterative, malleable for specific audiences and the different ways they interact across the user journey. This also gives ideas legs to stay relevant in the long term.

The old agency practice is antiquated, but many still place longform TV spots on digital. Rather, we must put in the time and energy to meet people where they are, creating meaningful experiences for audiences in a personalized and culturally conscious fashion. When we think with our audience truly in mind, they accept our invitation and welcome us as a speaker.

In this respect, performance creative isn’t about ticking off the boxes to optimize for the lowest common denominator in the audience. In an interview with SoDA Executive Director Tom Beck, Media.Monks Co-founder Wesley ter Haar put it this way: “It’s about how we message and personalize in ways that are actually meaningful for people and fun… understanding where the content lives and whether the effectiveness of the work you’re doing actually delivers the results you want.”

We aim to create the most-refined “fuel" for the media engine through rigorous iterative testing. With data as our driver, we continue to learn virtuously through an ever-evolving approach where performance metrics inform future creative briefs. Because we have the data, we understand what messaging and visuals perform best with our audience. And, in turn, we can more-confidently invest in larger creative production efforts.

Produce at scale to extend campaign lifespans.

To fuel experimentation and maximize the lifespan of any campaign, it's important to capture as much raw content as possible when investing the time, energy and capital in live production. This yields a healthy reservoir of raw content that you can continually draw from in post-production to craft an ecosystem of deliverables that are channel-specific in terms of best practice and intended audience consumptive behavior. And, past 2-3 weeks of placement, we can continue to adjust iteratively based on performance data indicators.

Here's how it worked for Hatch, a brand whose devices help people achieve better sleep. We used an out-of-the-box creative approach combining feedback data and content production to iterate fresh content at speed. We identified top-performing creative placements based on key metrics and began our efforts by iterating messaging and intro visuals as our baseline to quickly improve rising CPAs. Initial learnings led to wider, net-new messages and visual testing utilizing post-production only, and also instilled confidence in briefing live production efforts. Following this plan, Hatch was able to meet larger audiences more effectively, all with a greater variety of data-driven creativity.

Meet people where they are.

An insights-based approach to production also helps teams focus on which creatives best appeal to audiences on different platforms. Think about it: people turn to YouTube to lean back and relax, Pinterest for shopping inspiration, or TikTok for user-generated content and snackable reels—and content that performs best on these channels is designed to meet those expectations. We must, therefore, always begin all efforts with our audience in mind.

It’s important to be purposeful in how you show up across channels and play by the best practices to reflect the content each audience seeks. A look that fits the feed means all the difference between a viewer who’s engaged versus one who’s pining to hit the “skip” button as soon as five seconds are up. And be aware of what you’re up against: on social media, for example, your content competes with the personal relationships and connections that people care most about. Making a strong opening hook is crucial to capture attention, stopping audience thumbs and eyes and driving them to engage.

Also consider creative’s place in the funnel. Awareness content is most powerful when it’s educational and informative, while consideration content lends itself well when focused on a product’s single-minded, USP-specific message. Conversion content should be quick to incentivize the viewer.

Turning fear of failure into data-driven success.

If you don’t see improved metrics out of the gate, you must go back and keep trying. A lack of performance doesn’t necessitate a lack of learnings to inform future creative efforts. Even when a creative placement “fails,” we’re able to glean data on what does not work and devise new approaches. Follow the mantra: “Always be testing.”

Once you have built an ecosystem of deliverables that enable creative testing at scale, you’ll be able to iterate content at scale with data as your guide. So don’t be afraid to fail. It takes the best scientists time to synthesize the purest fuels and ignite transformative, new compounds. Likewise, performance creative begins by making a commitment to being data-driven, producing at scale and being willing to test, iterate and learn to meet audiences meaningfully where they are. We must be creative scientists in the endless pursuit to synthesize better and better assets to fuel media efforts.

Making greater use of data and analytics than with the simple (yet effective) example above, A/B testing can be used to measure the impact of a range of different tests, such as email marketing, PPC campaign testing, website content and more. 

But here, we’re going to discuss a very simple scenario. Let’s say we want to run a regional marketing test, and therefore wish to measure its impact to inform whether it’s worth running again in the future or not. But what exactly is A/B testing, and how can it help in this case? How should we go about running the test itself? And how do we actually measure its impact?

What is A/B testing?

A/B testing, sometimes otherwise known as split testing or test and control, is used to compare two versions of something and measure the impact from one element that is changed. In our regional marketing test example, we are looking to measure the uplift (if any) across the regions that the marketing ran, compared with the regions where it didn’t.

How do you run an A/B test?

Start with the hypothesis to be tested. For example, you believe a certain piece of marketing will outperform the norm. This could be prompted by a question, such as “Why isn’t a particular channel working?” or “We’d like to explore this strategy in the future, how can we test if it works?” The latter fits well with our regional marketing example.

From here, make sure you have the data required to measure the test and set it up in the correct way. For example, if running a regional test, ensure you are able to collect regional/store level data. And be sure to discuss the test with the relevant parties to ensure that the spend, flighting and regions chosen for the test are sufficient. 

At any one time, there will be a whole range of factors that influence sales. However, keeping everything the same except for the marketing that is being tested, we can confidently assume that any uplift is coming from the test itself.

How do you measure A/B test results?

First, we have to collect all required data. In our example of regional testing, split the sales data into two distinct sets: a test set and a control set. Collecting store level sales data would inform the split of sales data in our case. 

Next, we must index the test and control sets to have the same baseline so that a fair comparison can be made. Given that they may be of different absolute values, this is the best way of comparing the two when looking at the change in sales.

As mentioned previously, by keeping all other factors the same (i.e. availability of product, no differences in offering between stores, etc.), any difference seen in the test set can be attributed to the marketing activity we have run. See the chart below for a nice visual example of this.

What A/B testing achieves.

A/B testing allows us to calculate ROI more accurately than we would be able to otherwise, for example by modelling total national sales with traditional MMM. Store level modelling is another solution, but for smaller ad-hoc tests it is often not cost efficient due to the resource required and relative spend behind the test.

A/B testing doesn’t just provide data-driven proof that a test has or hasn’t worked. It also provides the opportunity to apply insights into consumer behavior across other areas of your marketing. If a regional test has worked when marketing a particular product, then it may well work again by using a different channel.

Carrying out regular testing works best, as it helps to reach the optimal marketing spend and laydown quicker within each channel, making for more fruitful marketing. And of course, similar to our favorite seed experiment, shedding more light on marketing will only help its impact grow! Learn more about how we can help you now.

How Do I Know It’s Working?

I recently talked to an advertiser for an enterprise software product who was leveraging target account marketing on LinkedIn to reach executive-level decision makers. He told me it “wasn’t working.” When I asked him, “what do you mean when you say it isn’t working?” he responded that it wasn’t generating any leads. I looked at his campaigns, and saw that he was showing ads to VPs and CXOs, driving them to a free demo signup page.

Put yourself in the prospect’s shoes for a moment: A senior executive isn’t interrupting their day to click on an ad from a company they’ve never interacted with, and then immediately signing up for a demo with a salesperson without interacting with any other content on the site. Heck, even a junior employee doing research for a senior executive probably wouldn’t request a sales demo without doing significantly more research than reading one ad and one landing page. Saying that these LinkedIn campaigns “don’t work” is like throwing a cheetah in a swimming pool with a dolphin and calling it slow when it loses a swim race.

Think Like a Customer

Last year we invested in a single sign on software (SSO) to enhance our company’s digital security. Our CEO was the ultimate decision-maker on the purchase and gave signoff on the investment, but there was no chance that he would take 30 minutes listening to a demo pitch. He assigned our Director of Technology to lead the evaluation process. That director did demo four different software platforms before ultimately making a recommendation that our CEO approved.

Ok fine, but then why worry about showing our CEO an ad at all? When our CEO tasked that director with leading the evaluation process, he mentioned that he knew Company A and Company B had offerings that were worth evaluating. So before any serious research took place, two of the four spots for consideration were decided. Influencing our CEO’s likelihood of mentioning, remembering, or just being aware of Company A and Company B is exactly why you show him an ad even if he’s never going to convert on a demo offer.

Evaluate the Right Way

So how should you evaluate success? That depends on what marketing tactic you’re using and what it is meant to accomplish. Or, continuing the metaphor from before, whether it was designed to sprint in the plains or swim in the ocean, so to speak.

If you imagine yourself as the SSO software marketer trying to influence our CEO, you should first divvy up targeting between prospecting and remarketing for evaluators and influencers. Each of the four segments would have a different measure for success.

If you’re getting in front of CXOs and driving heavy site engagement and lots of return visits, you’ve won. How much would you pay for the CEO of one of your top 500 target accounts to come to your site, spend five minutes clicking around, seeing that you are a leader on a recent Gartner report (or any other prominent award), and then download an ungated asset? For many B2B companies, that’s probably worth $500+. So who cares if you’ve spent $5,000 with two conversions if you’re driving killer engagement left and right?

The purchase funnel continues to get more complex. You’re rarely going to hit a home run on the first pitch, so start to weigh the walks and singles that ultimately lead to scoring runs. If you’re evaluating the right metrics and not giving up on channels and tactics for the wrong reasons, you’re going to end up with more deals and revenue, and those are the ultimate success metrics.

Learn more on how our performance.monks can help with these tough challenges as they arise.