Overview about bot traffic, types and their purposes

1. What Is Bot Traffic?

Bot traffic describes any non-human traffic that visits a website. Whether the website is a hugely popular news site or a small, newly published startup, the site is bound to be visited by a certain number of bots over time.

While the term ‘bot traffic’ is often misconstrued to be inherently harmful, this is not always the case. There is no doubt that some bot traffic is designed to be malicious and can negatively affect analytics data. These web crawlers can be used for credential stuffing, data scraping, and in some cases, even launching distributed denial of service (DDoS) attacks.

However, web robots are also essential for the operation of specific web services such as search engines and digital assistants. Therefore, digital publishers need to use their analytics data to discern between human behaviour and the good, the bad, and the ugly of bot traffic.

2. Types of Bots to Watch Out For

As mentioned, specific bots are required for the operation and optimal performance of search engines and digital assistants. However, specific search engine bots are explicitly designed to cause damage to sites and user experience.

The types of bots to watch out for include:

Click Bots

Click bots are used for click spamming by making fraudulent ad clicks. For most web publishers, particularly those using Pay Per Click (PPC ads), this is considered the most damaging type of bot. This is because click bots cause a skew in the data analytics, replicating web traffic and consequently eroding the budget with no benefit to the publisher.

Download Bots

Similar to click bots, download bots also interfere with genuine user engagement data. However, rather than affecting the ad click count, they create a fake download count. This is most pertinent when a publisher uses a marketing funnel, for example, a free ebook download. Download bots create a phoney download, leading to false performance data.

Spam Bots

Spambots are the most common bot. The purpose of a spambot is often to scrape contact information, including email addresses and phone numbers, create fake user accounts, or operate stolen social media accounts. They also disrupt user engagement through the distribution of unwarranted content, such as:

- Spam comments, including referral spam
- Phishing emails
- Ads
- Website redirects
- Negative SEO against competitors

Spy Bots

Spy bots are so named because they act in precisely such a manner – as spies. They steal data and information, such as email addresses from websites, chat rooms, social media sites, and forums.

Scraper Bots

Scraper bots visit websites with the sole malicious intent of stealing publishers’ content. Scraper bots can create a real threat to a business and its web pages. Created by third-party scrapers, they are employed by business competitors to steal valuable content, such as lists of products and prices which are then repurposed and published on competitor sites.

Imposter

Bots Imposter bots replicate human behaviour by appearing as genuine website visitors. They intend to bypass online security measures, and they are the bots most often responsible for DDoS activity.

3. What Is Good Bot Traffic?

While the above examples are undoubtedly cases of harmful bot traffic, what are some instances of good bot traffic? The following bots are legitimate and are there to provide helpful solutions for websites and applications.

Search Engine Bots

Search engine bots are the most obvious and well-known of the ‘good’ bots. Search engine bots crawl the web and help website owners get their websites listed in search results on Google, Yahoo, and Bing. These bots are helpful SEO tools.

Monitoring Bots

Monitoring bots help publishers ensure their website is healthy and accessible while operating at peak performance. Monitoring bots operate by automatically pinging the site to ensure it is still online. If anything breaks or the site goes offline, the publisher will be automatically notified, making these bots very useful to site owners.

SEO Crawlers

SEO crawlers are software that crawls a website and its competitors to provide data and analytics on page views, users, and content. Web admins can then use these reports to plan their content to improve their referral traffic, search visibility, and organic traffic.

Copyright Bots

Copyright bots crawl the internet, scanning for copyrighted images to ensure no one is illegally using copyrighted content without permission.

4. How Does Bot Traffic Affect Websites?

Bot traffic can effectively destroy businesses if they don’t learn how to identify and manage bot traffic. Sites that rely on advertising alongside sites that sell products and merchandise with limited inventory are particularly vulnerable.

For sites that are running ads, bots that land on the site and click on various page elements can trigger fake ad clicks. This is known as click fraud, and while it may initially increase ad revenue, once online advertising networks detect the fraud, it will usually result in the site and the owner getting banned from their network.

For eCommerce sites with limited inventory, inventory hoarding bots can virtually shut down their shop by filling carts with tons of merchandise, making it unavailable for purchase by genuine shoppers.

5. How to Identify Bot Traffic

As we move into an increasingly tech-driven future, search engine crawler bots are getting smarter by the day. A report released by Imperva in 2020 found that bots comprised almost 40% of Internet traffic, out of which bad bots were the most significant offenders.

Web publishers and designers can identify bot traffic by examining the network requests to their sites. Utilizing an integrated analytics tool such as Google Analytics will further help website owners identify traffic bots in their website traffic. The hallmarks of bot traffic include the following characteristics:

Abnormally High

Abnormally High Bounce

Abnormally High Pageviews

Abnormally High Bounce Rate

Junk Conversions

Spike in Traffic From an Unexpected Location

6. How to Stop Bot Traffic

Once a company or agency has learned how to identify bot traffic, it is imperative that they gain the knowledge and tools needed to stop bot traffic negatively affecting their site.

The following tools will help minimize threats:

Legitimate Arbitrage

Traffic Arbitrage is the practice of paying to bring traffic to a website to ensure high-yielding PPC/CPM-based campaigns. By only purchasing traffic from known sources, site owners can reduce the risk of bad bot traffic.

Use Robots.txt

Placing a robots.txt file will assist in keeping bad bots away from a site.

JavaScript for Alerts

Site owners can place a contextual JavaScript (JS) to alert them whenever a bot appears to enter the website.

DDOS Lists

Publishers can compile a list of offensive IP addresses and deny those visit requests on their website, thereby reducing the number of DDoS attacks.

Use Type-Challenge

Response Tests: One of the simplest and most common ways to detect bot traffic is to utilize CAPTCHA on the sign-up or download form. This is particularly useful in stopping download and spambots.

Scrutinize Log Files

For web admins who have a sophisticated understanding of data and analytics, examining server error log files can help find and fix website errors caused by bots.

7. How to Detect Bot Traffic in Google Analytics

For publishers using Google Analytics, there are some simple ways to set up your site to filter out bot traffic.

Firstly, visit the Google Analytics Admin Panel.
Next, Navigate to View Settings in the View tab.
Scroll down to the Bot Filtering checkbox.
Click Check in the checkbox if unchecked.
Finally, hit Save.

8. Why Is It Important to Protect Your Ads?

Any website that is running Pay Per Click ads will at some point be hit by bot traffic of one form or another. It is imperative that publishers take steps to protect their ads, or bot traffic will eventually cause the following issues:

Website data and analytics may become skewed
Website load time and performance may begin to deteriorate
Websites become vulnerable to botnets, DDOS attacks, and ultimately negative SEO results
CPC is negatively affected, and ultimately revenue may be lost

9. Are you a digital publisher who needs help monitoring bots?

At Netlink, we pair simplified, cutting-edge programmatic advertising technology with impartial guidance to help our clients understand the ad tech landscape and get the most out of the ads on their websites. Contact our friendly team to learn more today via: sale@appmatic.sg.