Should You Block AhrefsBot, SemrushBot and GPTBot?

Somebody has told you to block the bots. Maybe a developer, maybe a forum thread, maybe an agency that wanted to show it was doing something technical. Or it has already happened: your robots.txt has lines in it you did not write, and nobody can tell you what they cost.

On 21 October 2026 we read the robots.txt of nine Singapore corporate gift supplier websites. Eight blocked nothing. One blocked 25 crawlers by name: six that feed AI systems, and nineteen belonging to SEO and site-audit tools.

That one file is the whole decision in miniature, and most of it was a reasonable call. The six AI blocks were aimed almost entirely at training crawlers, not at the bots that put a business in front of a buyer. The nineteen SEO blocks hide one specific thing from competitors, and the same thing from the owner and from whoever they hire next.

Three different jobs, one text file

The confusion is not stupidity. It is that robots.txt makes every crawler look like the same kind of visitor, when they are doing three unrelated jobs.

Training. Collecting text that may go into a model. Blocking this changes nothing about whether you show up anywhere today.

Retrieval. Fetching your page at the moment someone asks a question, so your business can be named in the answer. Blocking this takes you off the shelf.

Measurement. Crawling you so a tool can sell your data to anyone, including your competitor and you. Blocking this is a privacy decision, not a visibility one.

Nearly every bad robots.txt decision we see comes from applying a training-crawler reflex to a retrieval crawler. The vendors are clear about which is which, if you read their own pages rather than a blog post about them.

What the vendors say their own bots do

Bot	Job, in the vendor's words	Block it?
GPTBot	Crawls content that may train OpenAI's foundation models	Costs you nothing visible
OAI-SearchBot	"Surfaces websites in ChatGPT's search features"	Takes you off the shelf
ChatGPT-User	Visits when a user interacts with ChatGPT; "not used for automatic crawling, and robots.txt rules may not apply"	The block may not hold
OAI-AdsBot	Checks the safety of landing pages submitted as ads; not used for training	Only if you never advertise there
ClaudeBot	Collects content for model training	Costs you nothing visible
Claude-User	Retrieves pages when a user asks Claude a question	Takes you off the shelf
Claude-SearchBot	Supports search quality	Takes you off the shelf
AhrefsBot	Builds the Ahrefs database and feeds Yep, its search engine	Hides your backlinks, from everyone

Those descriptions come from OpenAI's crawler documentation, Anthropic's support article on ClaudeBot and the AhrefsBot page, all read on 21 October 2026.

Note the shape of it. Each of the two assistant vendors runs a training crawler and separate retrieval crawlers. One line in a text file can refuse training and keep retrieval, and that distinction is where the money is.

Blocking a training crawler costs nothing and protects nothing you can prove

This is the honest version, and it cuts both ways.

If you block GPTBot and ClaudeBot, nothing about your current visibility changes. The retrieval bots still fetch you, the assistants can still name you, Google is unaffected. Anyone who tells you that blocking training crawlers will hurt your AI visibility is overselling.

But the reverse claim is just as oversold. Blocking a training crawler does not get your content back, does not remove it from models already trained, and gives you no enforceable right to anything. Anthropic's page is candid that blocking by IP address "may not work correctly or persistently guarantee an opt-out", partly because the bots need to read your robots.txt in the first place. OpenAI says robots.txt rules may not apply to ChatGPT-User at all, because that fetch is a person clicking, not a crawl.

So treat training blocks as a statement of preference that costs you nothing. If you want to make it, make it. Do not pay anyone to make it, and do not expect it to defend anything.

The one site in our sample did exactly this: it blocked ClaudeBot and left Claude-User and Claude-SearchBot alone. Whether by design or by copying a good list, that is the surgical version. The full data is in our study of nine corporate gift websites.

Blocking a retrieval crawler takes you off the shelf

This is the one that costs real money, and it is the one that gets done by accident.

OAI-SearchBot, Claude-User, Claude-SearchBot and PerplexityBot exist to fetch a page when somebody is asking a question right now. Block them and you are not refusing to be trained on. You are declining to be an available answer.

Two things make this error common. First, the blocklists that circulate are often a single bundle of "AI bots", with training and retrieval agents mixed together and no explanation. Second, the word GPTBot looks like the general case and OAI-SearchBot looks like a detail. It is the other way round.

OpenAI adds one softening note worth knowing: a site that opts out of OAI-SearchBot can still appear as a navigational link. You lose the substance, not the name. That is not a reason to block.

Our wider robots.txt census of 219 Singapore sites found this kind of block to be rare here, which is good news. The more common Singapore problem is not robots.txt at all but the wall behind it, which we cover in checking the bot gets in before you buy AI visibility.

What blocking AhrefsBot actually hides

Nineteen of the twenty-five blocks on that one site were SEO and audit crawlers: Ahrefs, Semrush's family of agents, Moz, Majestic, Screaming Frog, Botify, OnCrawl and others.

The usual pitch for this is "stop competitors spying on you". Here is what it really does and does not do.

It does hide your backlink profile. Those tools only know about links they have crawled, so a block genuinely degrades what a competitor can learn about where your links come from.

It does not hide your rankings, because anyone can search and look. It does not hide your prices, your pages, your titles, your schema or your lead times. It does not hide you from Google, which uses its own crawler and ignores all of this. Ahrefs' own page makes no claim about Google rankings either way, and notes that the same crawler feeds its Yep search engine, so there is a small visibility cost tacked onto the measurement one.

And it hides the same data from you. The day you hire anyone to improve your search visibility, the first thing they will want is your link profile, and a historic block means there is less of it recorded. You will be paying someone to work with a thinner picture of your own site.

Here is the part where we argue against our own interest. Your staying open is convenient for us. Our audits are faster and better when the tools have years of your history. That is our convenience, not your benefit, and it is not a reason for you to do anything. If you genuinely do not want your link profile sold to your competitors, block the audit crawlers and accept that your next agency works with less. It is a legitimate trade and we will not pretend otherwise.

What we would push back on is an agency that blocks these crawlers on your behalf without telling you, because the effect is to make your account harder for the next agency to assess. That is a practice worth asking about, whoever your current provider is.

Four questions before you add a line

Run any proposed block through these, in order:

1. Which of the three jobs is this bot doing? If the vendor's own page says training, the block is free and toothless. If it says retrieval or search, stop.
2. Would a buyer ever meet me through this bot? ChatGPT search, Claude, Perplexity and Yep are all places a Singapore buyer might ask a question. If the answer is yes, the block has a price.
3. Who else loses this data? If the answer includes "me and anyone I hire", decide whether the competitor's loss is worth your own.
4. Will the block hold? Repeat it on every subdomain, as Anthropic's page instructs. Accept that user-initiated fetches may ignore it, and that IP blocking is worse than useless because it stops the bot reading your robots.txt.

What I would do

For a typical Singapore SME with a website that needs to generate enquiries:

• Leave every retrieval crawler open. OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot. No exceptions I can think of for a business that wants to be found.
• Block the training crawlers only if you actually mind. GPTBot, ClaudeBot, CCBot. It is a preference, it is free, and it changes nothing you can measure. Do not let anyone bill you for it.
• Decide the SEO crawlers deliberately, once, in writing. Either you accept the visibility of your link profile as the price of being measurable, or you block it and tell your next agency you did. Both are defensible. Drifting into it is not.
• Then go and check the wall. A perfectly tuned robots.txt means nothing if your firewall serves a challenge page to anything without a browser engine. One of the nine sites we measured did exactly that, while its robots.txt was almost entirely open.

Open your own file first. It is at yourdomain.com/robots.txt, it takes ten seconds, and about half the owners we do this with find at least one line nobody can account for.

What it costs

Editing robots.txt costs ten minutes and no money. You do not need us, or anyone, for the edit itself.

The real costs are elsewhere. Blocking retrieval bots costs you enquiries you will never see and cannot count, which is the worst kind of cost. Blocking audit crawlers costs you and your future agency a clearer picture of your own link history, in exchange for a competitor's inconvenience. Blocking training crawlers costs nothing and buys nothing but a position you may hold for good reasons of your own.

The cost of doing nothing is usually zero, which is worth saying plainly. If your robots.txt currently blocks nothing, as eight of our nine sites did, you can close this page and spend the afternoon on your lead times instead. That will make you more money.

When blocking is the right call

Three situations where we would block, and mean it.

If your site publishes material you licence commercially — a price database, original photography, a members' library — refusing training crawlers is consistent with how you make your money.

If you have a staging or client-portal subdomain, block everything on it. Most of the accidental indexing we find in a technical audit is a staging site nobody thought about.

And if a crawler is costing you real server load, use Crawl-delay rather than a block. Anthropic's page says it supports the non-standard Crawl-delay extension, with a one-second example. Slowing a bot keeps you on the shelf. Blocking it does not.

One next step

If you want a second pair of eyes on your own robots.txt, send us the domain. We will tell you which of the three jobs each line is refusing and what we think it is costing you, in plain words, as part of the free visibility check through our contact page. If the answer is "your file is fine, leave it alone", that is what we will say. It often is.

What we will not do is promise that leaving the door open gets you cited. As our piece on AI citations puts it, assistants choose their own answers. Access is a precondition, not a result.

FAQ

Does blocking GPTBot hurt my visibility in ChatGPT?

No. OpenAI's documentation separates the two: GPTBot crawls content that may train its models, while OAI-SearchBot is the crawler that surfaces websites in ChatGPT's search features. Blocking GPTBot alone leaves ChatGPT's search able to fetch and cite your pages normally.

Which AI crawler should a Singapore business never block?

Any retrieval crawler: OAI-SearchBot, Claude-User, Claude-SearchBot and PerplexityBot. These fetch your page at the moment somebody asks a question, so blocking them removes you as a possible answer rather than protecting anything. The training crawlers are a separate, far cheaper decision.

Does blocking AhrefsBot affect my Google ranking?

No. Google uses its own crawler and ignores rules written for other bots. Ahrefs' own page makes no claim about Google rankings. What a block does change is how much of your backlink profile is recorded, for your competitors and equally for you and any agency you hire.

Should I block SEO crawlers to stop competitors researching me?

It hides your link profile and little else. Competitors can still see your rankings, pages, prices and markup by looking. Decide it once, deliberately, and tell your next agency you have done it, because they will otherwise read the thin data as a thin site.

Does blocking an AI crawler by IP address work?

Anthropic advises against it, saying IP blocking may not work correctly or persistently guarantee an opt-out, partly because the bots need to read your robots.txt and IP blocks stop them doing so. Use the robots.txt token instead, and repeat it on every subdomain you want excluded.

How do I check what my site currently blocks?

Open yourdomain.com/robots.txt in a browser. Read every "User-agent" line and the "Disallow" under it. A "Disallow: /" means that bot is refused the whole site. If a line names a bot you cannot identify, look the name up on the vendor's own documentation before you keep or remove it.