Choose your language

Choose your language

The website has been translated to English with the help of Humans and AI

Dismiss

Your Metrics Are Lying: How to Manage the Impact of Bot Traffic on Your Data

Your Metrics Are Lying: How to Manage the Impact of Bot Traffic on Your Data

Data Analytics Data Analytics, Data maturity 4 min read
Profile picture for user Francisco Regoli

Written by
Francisco Regoli
Analytics Project Manager

Image depicting a robot typing on a computer

It's estimated that around 40% of all internet traffic is generated by bots, according to Cloudflare’s “Radar Report.” For those of us in marketing and data analysis, this is a big deal—bot traffic can skew our reports and lead us to trust inaccurate metrics, often without us even knowing. With data, it’s hard enough to make sure everything is being collected properly; but with traffic bots, how can you get real insights?

As a response to this issue, the most popular digital analytics tools have started to offer bot filtering features. While it is advisable to activate them, they have shown very low effectiveness against the different types of bots that abound on the web; or worse, it could filter out real, and beneficial bot traffic, skewing data in favor of inaccurate traffic. 

Bottom line, our data is at risk.

bot filtering check box image

How does bot traffic impact your business?

For companies making business decisions based on data, bot traffic can have detrimental consequences on their digital strategies. It skews various metrics like conversion rates, bounce rates, total users, and sessions, leading to unexplained fluctuations. Additionally, increased traffic can raise costs for digital analysis tools, as many pricing models are based on the number of visits. AI tools and implementations that train on data impacted by traffic bots can produce inaccurate insights. Bot traffic can also bog down websites by overloading servers, resulting in slow page load times or, in severe cases, making the site inaccessible to users. In extreme situations, allowing unwanted traffic can create security vulnerabilities and lead to leaks of sensitive information.

Recently, one of our clients asked us for assistance in reviewing certain sudden increases in traffic originating from Frankfurt during the early hours of the morning, which did not align with their historical data. After analyzing the reports and cross-referencing the different available dimensions, we discovered that, during certain periods, 90% of the total users recorded in the reports exhibited behavior that was difficult to attribute to humans. This not only seriously affected the data quality but also incurred significant expenses due to the volume of visits the website was receiving.

However, it’s not just extreme situations that can affect our data quality. Even a small percentage of anomalies can lead to unreliable reports. So, how can we stop this and keep our data reliable?

Know the enemy

The first step to effectively counteract bots is to understand them. Not all bots are alike; each type demands a unique strategy. A common classification distinguishes between malicious and non-malicious bots. Let’s examine some typical examples of malicious ones.

Types of malicious traffic bots

1. Scalper bots:

These programs snap up tickets and other limited-availability goods at lightning speed, only to resell them later at higher prices.

2. Spam bots:

Designed to flood your inbox or messages with junk, often laden with malicious links. Who hasn't been on the receiving end of annoying spam?

3. Scraper bots: 

These bots automatically extract data from websites, often copying content from competitors to gain an edge.

On the other hand, non-malicious bots are the ones that can quickly handle tedious tasks. They gather large amounts of data that would otherwise take days or even months to retrieve, easing the burden on humans for repetitive tasks.

Types of beneficial traffic bots

1. Spider (web crawler): 

Google's bots are some of the most advanced. They relentlessly search the web for videos, images, text, links, and more. Without these crawlers, websites wouldn't get any organic search traffic.

2. Backlink checkers: 

These tools help you find all the links a website or page receives from other sites. They’re crucial for SEO.

3. Website monitoring bots: 

These bots watch over websites and can alert the owner if, for instance, the site is under attack by hackers or goes offline.

My goal isn’t to exhaustively detail every type of bot out there, as they are constantly evolving. Instead, I want to highlight the various behaviors that influence our filtering and removal strategies, as well as the complexity involved. In the end, whether they are good or bad, all bots are unwanted in our reports, and we need to minimize their impact on our data.

Countering bot attacks with the right tools

Nowadays, you can find both automated and manual strategies to tackle this challenge. In the case of automated solutions, bot filtering programs stand out, either integrated into analytical tools or specialized software for AI-driven bot detection. However, as mentioned earlier, their effectiveness tends to be low, and in many cases, they come with associated costs.

On the other hand, we have non-automated solutions that provide better results, and we can categorize them based on the filtering approach they adopt:

Reactive Approach: Apply custom filters at the report level. This method is simple and flexible, requiring no development-level changes. It's an effective first step for early detection. Utilizing tools available in analytics platforms—like GA4 segments, Looker Studio filters and data warehouse queries—makes it easy to implement, though it’s less robust.

Preventive Approach: Implement filters before collecting data. Although this can be challenging and resource-intensive, it effectively prevents the impact on reporting and restricts bots from accessing the website and its servers.

Establishing a data quality review cycle

To keep our data free from bot traffic and ensure optimal results, it’s best to use a comprehensive strategy combining both preventive and reactive measures. This is known as the data quality review cycle, a model of continuous monitoring designed to constantly detect anomalies. It involves collaborative efforts from analysts, developers, and product owners to find efficient solutions that safeguard the integrity and reliability of the data.

graph illustrating the data quality review cycle

Although we can’t entirely eliminate bot traffic from our reports, proactively implementing data quality review strategies offers us practical and effective ways to address this issue.

In summary

  • Bots can serve both harmful and benign purposes; in both cases, it's crucial to keep them out of reports.
  • Bot traffic has negative consequences for both digital and commercial strategies.
  • While analytics platforms have features that automatically block some bot traffic, their effectiveness is limited.
  • Constantly monitoring anomalies in the reports is essential for identifying bot traffic.
  • To avoid unwanted traffic influence and ensure that the data is not biased or contaminated, it’s necessary to implement both preventive and reactive measures.
  • Including a data quality review cycle in the workflow is crucial for keeping the reports free from bot traffic.
Learn how to manage the impact of bot traffic on your data and safeguard the integrity of your metrics with effective bot filtering strategies and continuous data quality review. Learn how to manage the impact of bot traffic on your data and safeguard the integrity of your metrics with effective bot filtering strategies and continuous data quality review. bot data analytics Google Analytics Data Analytics Data maturity

Watch Your Mouth: Key Considerations for Developing Chatbots

Watch Your Mouth: Key Considerations for Developing Chatbots

4 min read
Profile picture for user mediamonks

Written by
Monks

Watch Your Mouth: Key Considerations for Developing Chatbots

Today’s consumers demand relevant, personalized content and instantaneous access to information at all hours of the day. With closer access to brands through social and messaging channels, chatbots have proven to be an effective way for organizations to strike a deeper connection with users, whether they be consumers or employees.

There are several different use cases for a chatbot; it can help you provide always-on customer service, provide personalized content to users in regular intervals, help your organization proactively screen job applicants and do much more. In essence, the main benefit that chatbots provide is the automation of routine, repetitive and simple tasks to make processes more efficient. They’re also an excellent source of user behavioral data, including finding patterns in terms used, most popular queries, user demographics and more. All these benefits help brands maintain a more direct, constant connection with consumers—if they’re designed with some key considerations in mind.

Before Building, Balance Benefits and Demand

Unlike a human, a chatbot is available at a moment’s notice, 24/7. Think of bots as modern, more interactive and relevant FAQ lists at its simplest level, but be aware that they are capable of doing much more, like engaging with users based on their surroundings. Whether it be providing entertainment or self-service troubleshooting, chatbots allow brands to provide services without the need for human intervention (though in some cases a human takeover is recommended, like solving more complex tasks or providing support in emotionally charged scenarios).

Screen Shot 2019-04-17 at 12.57.38 PM

To meet consumer need, this chatbot by Johnsville makes it easy for customers to order food quickly.

That said, chatbots aren’t the right fit for everyone. Before you invest in building one for your brand, consider your target demographic and the value you wish for the bot to provide. A lack of desire for automation can cause frustration for users who must use a chatbot. There may also be a learning curve to adapt to a new technology depending on your demographic, which can lead to more problems than solutions overall. A good method for determining whether a chatbot is right for your organization is to weigh the potential benefits with user desire or demand.

Know How to Set the Tone

A chatbot serves as a notable channel for representing a brand voice. Far from a frivolous thing, an attractive voice and personality can be incredibly beneficial for brands. Microsoft’s Xiaoice chatbot, for example, employs advanced emotional intelligence to carry humanlike, nuanced conversations with users. With the persona of a teenage girl, the AI is so popular in China that she has achieved celebrity status, according to Microsoft.

image06

This chatbot for Absolut employs a fun (if not a little disconcerting) voice to entertain the user.

But Xiaoice is just one fraction of a larger AI framework, and her underlying mechanisms power branded, third-party characters as well. So, what’s the value in these bots’ trademark small talk and chit-chat that has made them so popular with Eastern users—and what does it mean for chatbots that are designed to accomplish a specific task or organizational goal? The value lies in providing social capital by keeping users engaged, allowing for deeper emotional connections.

Given the power of a good voice, brands interested in the technology should consider the tone of voice and identity that fits their brand. While a consumer-facing bot has the freedom and flexibility to speak in a more casual tone, one that’s intended for employee use should take on a more professional persona. Will your bot speak to users in gifs? Will it offer emoji-based button responses? Is it lazy, or energetic? These are some questions you can ask to envision the personality your bot can take. Have fun with it!

Earn Users’ Trust

Chatbots are excellent at providing relevancy and personalization in their messaging to users—and they accomplish that by leveraging data gathered across the course of conversation or even through external sources (more on that below). For users to feel comfortable sharing their data with organizations, the value that data provides must be clear.  Chatbots are ideal for this because they can walk users through an onboarding process that asks permissions for data, clearly explaining why it’s necessary at each step.

Screen Shot 2019-04-17 at 12.44.37 PM

This Lufthansa bot offers value before asking for added input, gaining user trust in the process.

As users interact with a chatbot, they get instant feedback about how that data informs the user experience. For example, a bot with knowledge of a user’s home and work addresses can prove lifesaving for finding one’s way at rush hour when transit services change. While users might find most data collection and practices to be esoteric and opaque, the question-and-answer approach (not to mention the personality) of chatbots makes this process more transparent. And once that data is in their hands, organizations can also use it to discover new trends or forecast emerging user needs, thereby improving the experience even more.

Architecture and Maintenance

Speaking of data collection, an effective chatbot requires an architecture that plugs into one or several data sources. This might include data you already have about the user (for example, a retailer pulling from a user’s purchasing history), knowledge bases that troubleshoot common questions, partner data or other sources. Whatever data sources you pull from, you must ensure your chatbot’s architecture supports it—and be prepared to add more if and when it becomes necessary. When in doubt, consider partnering with a developer who can audit your data sources and build an architecture equipped to plug into these forms of data.

On that note, to develop a chatbot is to commit to the long haul: it’s important to iterate and optimize the bot for a better user experience based on the feedback collected, whether it be explicit comments from users or implicit usage data. One major example of this is expanding your market and localizing chatbot content to match. Brands must be sure they’re ready to scale up the growing capabilities of a chatbot to accommodate emerging user behaviors—though if they don’t have the resources, a creative partner experienced in tooling assets at scale for a global audience can be of help.

A chatbot can make for a valuable service to your audience, whether its focus is on consumers or employees. But conversation is an artform, and just like any artist, you need a vision and tools in place to deliver the experience you seek for your users. Having established that, your brand is ready to say “Hello” to deeper, closer relationships with your audience.

Chatbots are a great way to inject brand engagement with a little added personality. But just like with human interaction, first impressions matter. Keeping these guidelines in mind, brands and developers alike can confidently pursue one-to-one interactions with chatbots. Watch Your Mouth: Key Considerations for Developing Chatbots Before developing that bot, brush up on some rules of netiquette.
chatbots bot assistant facebook messenger emerging tech emerging technology

Choose your language

Choose your language

The website has been translated to English with the help of Humans and AI

Dismiss