Providing a digital experience that’s more closely aligned with what a user wants is a fairly obvious answer to why we experiment, but the way we approach it as an industry is changing.

Brands are now focused on delivering a well-timed cross-channel journey, to a wider volume of users with increasingly high expectations for personalized experiences. Isolated tests and experiments aren’t providing enough information on the entirety of a user's journey. 

So is experimentation really “dead,” as data thought leaders have been saying? Are we ready to move on to the next latest and greatest framework that generative and agentic AI have to offer?

Recently, reading Avinash Kaushik’s latest TMAI newsletter made me pause and reflect on where we’re heading as an industry. It hit the spot on many things I’ve been thinking about, specifically in how experimentation practices are evolving, and what that means for the future.

For years, A/B testing has been a cornerstone for optimization, and it’s a method that has taught the industry a lot. 

The truth is, A/B testing itself isn’t dead; it’s evolving. Or, better yet, it has to evolve. It’s not the method itself that’s to blame for the limitations we sometimes see, it’s the mindset attached to it. When A/B testing is treated as a checkbox activity, focused only on incremental changes, or siloed from other disciplines like search engine optimization (SEO), paid media, or product, its potential impact is greatly diminished.

The future of experimentation is more holistic, integrated and focused on delivering real-time, user-centric experiences. This doesn’t mean abandoning A/B testing, but expanding the lens to incorporate more sophisticated strategies that blend multiple disciplines, like SEO, conversion rate optimization (CRO), paid media and analytics.

It’s important to clarify that while experimentation as a function is typically iterative and methodical, its role extends beyond testing. Experimentation should inform and support the delivery of dynamic, real-time brand experiences, though delivering in real time itself is more closely tied to optimization, personalization and UX. 

The two complement each other: experimentation provides the precision and insights to optimize, while real-time strategies act on those learnings to adapt experiences dynamically.

This distinction matters because experimentation, while not inherently “real-time,” lays the foundation for modern, adaptable and user-centered approaches that make brands more agile. By focusing on how these disciplines work together, teams can move beyond traditional silos to deliver more innovative, more integrated strategies.

I’m not a classically trained CRO practitioner. My background is in digital marketing, product and growth for over 14 years. For me, experimentation has always been one tool in a much bigger toolbox: a way to make better, more precise decisions with proactivity and pragmatism at the forefront, always focused on ROI and accomplishing business goals.

When I approach experimentation, it’s never been in isolation. It’s always been CRO + Product, CRO + SEO, or CRO + Growth. I don’t see it as just tweaking elements or optimizing individual touchpoints. I’ve always viewed it as part of a bigger strategy to align business goals, user intent and data. This perspective has shaped how I think about the future of experimentation and how we do not need better tools, we need better thinking.

It’s important to recognize the incredible work already happening. Many teams and practitioners are pushing the boundaries of experimentation by adopting innovative approaches. They’re rethinking the entire user journey, using data in smarter ways and aligning optimization efforts with broader business goals.

And let’s give credit to the experimentation community as a whole: the content, tools and thought leadership that the community shares are invaluable for helping others grow and adapt. From blog posts to podcasts  or conferences to open discussions on social platforms, the spirit of collaboration and knowledge-sharing is what drives this industry forward.

One of the ongoing challenges with experimentation is how we define and approach it. The rapid evolution of testing has unintentionally muddied the waters with organizations vying for the latest and greatest way to provide better connected user journeys.

Experimentation, CRO, A/B testing: these terms often get used interchangeably, and sometimes the definitions create unnecessary silos. The truth is, good experimentation should already encompass the broader strategies we associate with "advanced" practices, like behavioral targeting or dynamic personalization.

The issue isn’t the name we give it, it’s how we implement it. When experimentation is reduced to tweaking surface-level elements like button colors or headlines, it misses the bigger picture. True experimentation involves connecting the dots across the customer journey, testing bold hypotheses, and focusing on business outcomes like lifetime value, retention, and revenue growth. 

While A/B testing remains a valuable tool, A/B testing’s greatest impact comes when it’s part of a larger ecosystem. 

Imagine a hypothesis born out of an SEO content strategy, validated through paid ad experiments and optimized for conversion through CRO. Creating and running tests now includes greater customer sentiment, considers how it fares against public interest and search volume, or creates the foundation for analyzing the entirety of a customer funnel. This kind of collaboration doesn’t focus on improving one metric, but aims to deliver a cohesive, end-to-end experience that aligns with both user intent and business objectives.

Moving toward this holistic approach requires more than just tools; it requires nurturing a culture of collaboration, curiosity and a strong foundation in data within your teams.

Based on my experience and the clients I have been servicing, I noticed that the teams that excel at experimentation have a few things in common:

1. They integrate data as a central pillar for decision-making, ensuring insights drive hypotheses and actions.
2. They embrace cross-functional collaboration, involving SEO, Paid Media, Product and UX teams in the process.
3. They prioritize big-picture hypotheses tied to business outcomes like customer lifetime value, retention and revenue growth.
4. They balance qualitative insights (customer voice, user research) with quantitative analysis to uncover deeper truths about their users.

This maturity doesn’t happen overnight, but the teams that focus on nurturing and growing this culture are the ones driving the future of experimentation—and there are many companies that do that very well, like Starbucks and 
Nissan to name a few.

The future of experimentation lies in creating Real-Time Brand experiences, which means dynamic, adaptive interactions that resonate with users in the moment. While this might sound similar to concepts like personalization or CRO, it’s essential to clarify their differences. Their overlap often contributes to the definitional confusion in our industry, but when understood, can unlock an exponentially optimized user experience.

Personalization focuses on tailoring content or interactions to individual users based on data signals, such as first-party data or behavioral patterns. Conversely, CRO is about optimizing specific touchpoints to drive better outcomes, often through structured experimentation. Real-Time Brands combine these elements into a cohesive, adaptive system that evolves with the user’s journey, leveraging insights from experimentation and the execution capabilities of personalization.

What makes Real-Time Brands distinct is their focus on delivering seamless interactions at scale, powered by advanced technologies like machine learning. These experiences aren’t about static “best practices” or one-size-fits-all solutions. With a dynamic, and increasingly personalization-driven audience, each individual user is constantly creating and adapting what they want. 

Instead, they are about continuously adapting to user needs in their journey’s context, whether recommending the next best product, tailoring messaging based on previous interactions or optimizing navigation flows based on intent.

This approach goes beyond personalization; it creates context-driven experiences that feel intuitive to the user while driving measurable outcomes for the business. The key is to view experimentation, personalization and real-time capabilities as complementary, not interchangeable, with each playing a critical role in building connected, user-centric strategies.

Real-time brand experiences are meant to create a sense of connection. 

Users are more online than ever before, and the trend is increasing every year. The average adult in the US spent over 11 hours a day interacting with media in 2018. 

In 2025, the same segment is estimated to spend over 12 and a half hours 

Making your brand, content, platform or campaign stand out against the collective trillions of individual hours spent seeing content requires providing the “perfect” interaction: a digital experience that matches the exact criteria a user needs to interact, engage and complete the ideal key action. 

Providing a Real-Time Brand experience shows users that the brand understands them, values their time and is committed to meeting their needs in ways that feel both seamless and meaningful.

For teams, this shift requires thinking beyond isolated experiments and embracing the complexity of modern user journeys, and necessitates the ability to orchestrate an ecosystem of touchpoints that work together to build trust, drive engagement and deliver measurable business outcomes.

At its core, A/B testing was designed to do just that: find an optimal version of a touchpoint that more-closely resonates with what users respond to. With AI, machine learning and a holistic approach to create iterative experiences from real-time data, brands can essentially create individualized A/B tests on a segmented and individual level that automatically self-adjust as interaction data comes in. 

As technology evolves, Real-Time Brand experiences will increasingly become the standard for experimentation. They represent the future of connecting with users in a way that feels personalized, proactive and perfectly aligned with their expectations.

The future of experimentation isn’t a rejection of the past but an evolution toward something bigger.

Remember, A/B testing is not dead, but the mindset around it must evolve. To unlock its full potential, teams need to move beyond isolated, surface-level tests and embrace holistic approaches that connect experimentation across SEO, paid media, product and CRO.

Modern experimentation prioritizes dynamic, real-time brand experiences that adapt to individual user needs. By blending personalization, behavioral targeting and experimentation, brands can create seamless, meaningful interactions that drive both user satisfaction and business outcomes.

The success of experimentation lies not in the tools alone, but in the mindset and culture behind it. Teams must nurture and reward collaboration, prioritize bold hypotheses tied to business goals and focus on delivering real value to both customers and the organization.

To everyone in the industry, whether you’re just starting or leading the way, thank you for pushing the boundaries and keeping this space vibrant.

Experimentation will always be a cornerstone of digital optimization. The core components of why we experiment are exactly the same: provide an opportunity for interaction that works a little bit better. Its future, though, lies in the connections we build, the problems we solve and the meaningful experiences we create together. 

Want to explore how to take your experimentation program to the next level? Let’s get in touch below.

Understanding cohort data.

The use of customer cohort level data allows us to examine and predict the value of repeat purchases over time. A cohort level analysis typically seeks to answer a question along the lines of, “If I bring a new customer in with these characteristics, how much value will they bring to my business in terms of continued purchases?” Typical characteristics that may be factored into the analysis could include type of goods or services purchased, payment method, device type and time of year the purchase was made.

One such example can be shown in the chart below, where we see new customers that enter in Month 1 (initial purchase month) and the value that we gain from repeat purchase of that cohort of customers over time.

This allows us to apply an average multiplier to any future new customer revenue based on the characteristics provided—showing us that for instance, new customers driven at a particular time of year, or through a particular category or payment type, are more valuable than others. One such example is shown below, where regardless of category, driving credit customer revenue is much more valuable than customers paying upfront in full. We can also see that it pays off much more in the long term to drive new customer revenue through category A in Q1 and Q4, whereas we drive more long-term value from new customer revenue through category B in Q2.

Leveraging MMM to drive profit.

These insights are interesting themselves, but drive meaningful action when combined with media costs, effectiveness and profit margins to optimize a media laydown. Rather than simply spending on media when our traditional “busy periods” are and adapting to that, we can instead optimize our media laydown to take advantage of customer lifetime value instead. 

This allows us to not only run scenarios that seek to drive profit but also scenarios to maximize long-term customer value in the most cost effective way.

For more information on how we can help with your Marketing Effectiveness measurement or Market Mix Modelling, visit our Measurement page or contact us.