What is Ahrefsbot and How To Block it?

Are You Using Blacklisted Backlinks? Check Now

Is the Ahrefsbot giving your site trouble?

Ahrefsbot crawls your website regularly. It acts just the same as Googlebot – coming and going as it pleases.

This isn’t usually a problem but sometimes it can cause problems especially if it’s overloading your server and slowing your site down.

The good news?

You can quickly and easily block Ahrefsbot from visiting your website. You just need to choose the right method that works best for you.

In this article, I’m going to share everything you need to know about Ahrefsbot and the 2 best methods to block it if you need to.

What is Ahrefsbot?

Ahrefsbot is the Ahrefs web crawler that powers the huge link index and database that Ahrefs has. Simply Ahrefsbot is at the heart of what makes Ahrefs such a powerful SEO tool.

Over the years Ahrefsbot has collected and added over 12+ trillion active links to the Ahrefs database. You can see why Ahrefsbot is an integral part of the Ahrefs tool.

How Does Ahrefsbot Work?

Ahrefsbot crawls as many websites as possible. It collects a range of quality link data on each crawl and sends that data back to the Ahrefs database.

Ahrefs can display accurate data to their users, providing detailed information for marketing and optimising websites.

According to Ahrefs, Ahrefsbot…

That’s a lot of power!

Curious about Ahrefs SEO tools? Check out my complete Ahrefs review.

Ahrefsbot: A Good or Bad Robot?

There are a number of bots used online.

The four most common types are:

All of these bots power different applications and make them more useful for users. These are called ‘good‘ bots.

But there are also ‘bad‘ bots out there. These bots do nasty things like:

They can also use up tons of your server bandwidth and cause a lot of havoc on your site.

While Ahrefsbot is doing good things most of the time – not everything it does is perfect. There is also a potential that it could be used for bad.

Ahrefsbot Offers A Few Benefits…

Ahrefsbot crawls your website and analyses both:

This data is added to the Ahrefs ever-growing database. But what are the benefits of letting Ahrefs crawl your website?

Practical SEO Data You Can Use

The Ahrefsbot data offers a huge amount of value to the SEO and marketing community.

The data that it collects allows SEO’s and marketers to understand how to better optimise their websites for large search engines like Google.

Allowing Ahrefsbot to crawl your website ensures that it is always up to date in their database. You get the most accurate data on your own website and you can even compare it to your competitors.

This helps you:

…But It Can Also Cause Several Issues To Your Site

As with anything, there are a few potential downsides to Ahrefsbot that you need to be aware of.

Each of these can cause unwanted issues on your site that may not be worth the risk.

Ahrefsbot Can Make Your Site Slower

No one likes a slow loading website, right?

Whenever Ahrefsbot crawls your site it loads like it would for any other person which uses up bandwidth.

If Ahrefsbot is crawling your website too regularly, it can take up too much bandwidth which means the site will load slower for other visitors.

We all know that speed is a Google ranking factor and affects user experience negatively.

Does Ahrefsbot Really Respect robot.txt?

Ahrefs is very strong on their claim that Ahrefsbot respects robots.txt. They have said multiple times that Ahrefsbot will follow both disallow and allow rules.

Here’s the problem:

There have been a few users that have said this isn’t true. Even after adding robots.txt to their sites, they notice they are receiving traffic from the Ahrefsbot.

Not obeying robots.txt rules is a sign of a bad bot. Red flag!

Ahrefsbot Can Be Used By Spammers

I hate spammers too. Unfortunately, they do exist and as website owners, they are something we need to be aware of.

There is nothing worse than receiving tons of unwanted spammy referral traffic that not only clogs up your Google analytics report but also affects your website performance.

Spammers can immitate the Ahrefsbot user agent to:

If you notice you are receiving tons of traffic regularly from Ahrefsbot, you might want to review and compare with the official Ahrefs IP Address ranges.

GROWTH HACK: Sign up for my free video training series and get a copy of our internal private SEO tool.

How to Block Ahrefsbot?

You can block or limit AhrefsBot using your robots.txt file or htaccess file. Both methods should work but take a look at each option below to see which works best for you.

Method 1: Block Ahrefsbot With robots.txt

Ahrefs says that Ahrefsbot follows robots.txt rules. You can block Ahrefsbot by adding new rules to your robots.txt file.

You have two options here:

1. Change the frequency Ahrefsbot can visit your site
2. Completely block Ahrefsbot from visiting your site

To change the frequency of Ahrefsbot visiting your site, add the following code to your robots.txt file:

User-agent: AhrefsBot
Crawl-Delay: 5

Crawl-Delay value is the time of the delay recorded in seconds. You will need to change this value to delay the number of seconds between each crawl from Ahrefsbot.

Want to block Ahrefsbot completely?

To do that you will need to add a disallow tag to your robots.txt file. Simply copy and paste the code below:

User-agent: AhrefsBot
Disallow: /

That’s it! Ahrefsbot should obey the robots.txt rule you just set up.

Method 2: Block Ahrefsbot Using the .htaccess file

This is a very effective method for blocking Ahrefsbot and gives you more control.

Unlike method 1 where you rely on the fact that the Ahrefsbot will obey the rules you set – with method 2, you will be taking complete control at the server level.

You will be adding the range of IP addresses that the Ahrefsbot uses and blocking it based on those IP addresses.

Here is a small example of the code you could add to your .htaccess file to block IP addresses:

Order Allow,Deny
Deny from 51.222.152.133
Deny from 54.36.148.1
Deny from 195.154.122
Allow from all

You want to make sure you block all of the current Ahrefs IP addresses that are listed here. That means you will need to edit the code above to make sure it blocks the current IP’s used by Ahrefs.

Follow this tutorial if you need help with your .htaccess file.

Should You Block Ahrefsbot?

This is a good question.

Using the Ahrefs SEO tools?

Then don’t block the Ahrefsbot because you want to ensure your website is being updated in the Ahrefs index.

Don’t use the Ahrefs bot and it’s causing more harm than good?

Then it might be best to block it.

Ultimately – this question will help you decide on the Ahrefsbot: “Does this benefit my website or does it hurt my website?” and you can only figure that out by looking at yours sites access logs.

Wrapping It Up

Ahrefsbot is the second most powerful SEO bot on the planet.

It has helped Ahrefs amass a huge link index of over 12+ trillion links and powers all of their awesome SEO tools.

But…

Ahrefsbot can in some cases, cause some issues and even worse be used by spammers that refer unwanted traffic to your site.

Use any of the 2 methods to block the Ahrefsbot if you want to and add a layer of protection for your website and don’t forget to check out my full Ahrefs review.

Frequently Asked Questions?

What is Ahrefs bot?
Ahrefs Bot is a web crawler that powers the 12 trillion link database for Ahrefs online marketing toolset. The data collected by Ahrefs Bot directly from the web is then used by thousands of SEOs and digital marketers around the world to manage their online marketing campaigns.

Should I block Ahrefs?
It depends. If you’re using any of the Aherfs tools, then, you should not block it- But, if you aren’t using Aherfs tools (and don’t plan on using their tools) then, the best is to block it.