Before You Buy AI Visibility, Check the Bot Gets In

You open your website on your phone. It loads. The photos are there, the WhatsApp button works, the enquiry form sends. Whatever is wrong with your marketing, the site is clearly fine.

Then a machine asks for the same page and gets an error.

Over two days this week we measured twenty ranked Singapore business pages across two unrelated industries. Five of the twenty did not return a page to us at all: a certificate error, or a bot check standing where the page should have been. All five looked perfectly normal in a browser.

If you are about to pay someone to make your business visible in AI answers, check this before you sign anything. It is also, usually, not a marketing problem.

Why a website answers a machine differently

Because the thing serving your page is not only your website. It is your website behind a stack of other services, several of them designed to treat unfamiliar visitors with suspicion.

A content delivery network sits in front of your server to make pages fast and absorb attacks. A bot-protection layer sits in front of that to keep scrapers out. A security plugin does a third version of the same job inside your site. None of these was installed by someone thinking about ChatGPT. They were installed to stop bad traffic, and they work by asking one question: does this visitor look like a browser?

A search crawler does not. Neither does an AI crawler, nor the tool your prospective agency uses to audit you. They arrive without a mouse, without a screen size, often from a data centre address, and a protection layer reasonably concludes they are not a customer.

So your site gives them something other than your page, silently. Nothing appears in your inbox. Your analytics look normal, because analytics run on JavaScript these visitors never execute. The failure is invisible from every place an owner would think to look.

This is the gap our own AI search readiness study declared it had not measured. It checked robots.txt and said so in its limits: firewalls and CDN blocking were out of scope. This article is the part we left out.

What we found, twice, in two days

Five of twenty ranked pages, across two industries that have nothing to do with each other.

On 21 September 2026 we measured ten ranked Singapore tuition centre pages. Three of the ten returned a bot check instead of the page.

On 22 September 2026 we measured ten Singapore moving companies for a study of their websites. Two of the ten did not return a page:

• One returned HTTP 526, a Cloudflare error. Cloudflare's own documentation describes it: "This error indicates that Cloudflare is unable to verify the SSL certificate on your origin server, preventing a secure connection from being established." In plain terms, the certificate on the origin server has expired or does not match, and the edge refuses to pass the request through.
• One returned HTTP 202 with a body of 169 bytes: a redirect into a captcha challenge. The same wall covered its robots.txt, so even the file that tells crawlers what they may read was unreachable.

Both of those businesses were ranking on the first page of results for competitive buying searches when we measured them. Ranking today and being readable today are separate facts, and the first can outlive the second for a while.

One more detail from the same run, because it cuts the other way. A third site refused our first request with a connection reset, and served a perfectly good page on all three retries a minute later. That was a transient network fault, not a defect, and if we had recorded the first attempt we would have published a false finding about a business that had done nothing wrong. Test more than once. We did, and it changed the number we are reporting.

What Google actually does with each answer

It depends entirely on which kind of failure you have, and the difference matters more than most owners expect.

For server errors, Google's documentation on HTTP and network errors is direct: "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling. For Google Search, already indexed URLs are preserved in the index, but eventually dropped." And on the content itself: "Any content Google receives from URLs that return a 5xx status code is ignored."

Read "eventually dropped" carefully. A 526 does not remove you this week. It starts a clock. Your rankings look untouched right up until they are not, which is why this failure mode is so often discovered months late and blamed on an algorithm update.

For the 4xx family, Google is blunter: "Google doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index."

The failure that is worse than an error

The captcha wall, and it is worse precisely because it is not an error.

A 526 is at least honest. Google's crawlers recognise a 5xx as a server problem, slow down, and preserve the existing index entry for a while. The system is built to forgive a server having a bad day.

A challenge page returning HTTP 202 gets none of that forgiveness, because it asks for none. 202 is a success code. Nothing in the protocol says a problem occurred. As far as any crawler is concerned that site answered correctly, and its answer was 169 bytes of redirect markup. No error to slow down for, nothing to preserve, no bad day to wait out. Just a business whose page, to a machine, contains almost nothing.

If you must have one of these two problems, have the error.

Why we cannot tell you what Googlebot sees

We measured from a data centre address with a research user-agent. Googlebot does not look like that, and the difference is the honest limit on everything above.

Google publishes the IP ranges its crawlers use, and many bot-protection services allowlist them by default. So it is entirely possible that the five pages in our sample serve Googlebot the real page and serve everyone else a wall. That is not a contradiction of our finding. It is the finding, stated precisely.

"Everyone else" is a growing list. It includes the AI crawlers assistants use to fetch pages, most of them newer than the allowlists sitting in front of these sites. It includes the audit tool of any agency you ask for a quote, and a person on a VPN. Trusting Google and distrusting unfamiliar machines was a reasonable setting in 2019 and is a narrowing one now. Our guide to getting cited by ChatGPT explains why the crawler has to read the page before anything else can happen.

We have had this problem ourselves. Our method page records a review on 12 August 2026 that found AI crawler requests to our own site not getting through. We are not describing someone else's mistake.

How to check your own site in ten minutes

Four steps, in order. You need a terminal, and nothing else.

1. Ask for your homepage as a plain client. Run curl -I https://yourdomain.com. You want a 200. A 526, 403, 503 or a redirect into something with "captcha" or "challenge" in the path is the problem this article describes.
2. Ask again as an AI crawler. Run curl -I -A "GPTBot" https://yourdomain.com, then repeat with PerplexityBot and OAI-SearchBot. A different answer from step 1 means something is filtering by user-agent.
3. Ask for your robots.txt the same way. If that file is behind the wall too, crawlers cannot even read your instructions.
4. Repeat each check three times, a minute apart. One failure is a network fault. Three is a configuration.

If every check returns 200 and real HTML, this is not your problem and you can stop reading. If any of them does not, take the output to whoever runs your hosting. Not to an SEO agency.

When this is not a marketing problem

Most of the time, which is the uncomfortable part of writing this.

If your curl checks come back wrong, the fix is a certificate renewal, a CDN setting or a rule in a security plugin. It is an afternoon of work for whoever administers your hosting, and it is often included in a support plan you already pay for. Nobody should charge you a monthly retainer for it.

We would rather say that plainly, because "your site blocks AI" makes an excellent sales opening and a poor invoice. If an agency leads a pitch with this finding and then quotes a twelve-month content programme to solve it, the finding and the quote are not connected. Ask what the fix is. If they cannot name the specific setting, they found it with a tool and do not understand it. We have written more about reading an agency's claims before you pay.

There is a real marketing problem behind this one, and it only becomes worth paying for after the wall comes down. A page a crawler can now read is not automatically a page worth citing. Most of the fifteen pages we could read had plenty else wrong with them, and those are the problems an agency is actually for.

One next step

Run the four curl checks above on your own site this week. They cost nothing and they answer a question no dashboard will.

If you would rather we ran them, send us your domain through our contact page and we will tell you what your site returns to a plain client, to three named AI crawlers, and whether your robots.txt is reachable. If the answer is that everything is fine, we will say so and there is nothing to buy. If the answer is a hosting fix, we will name the setting and you can take it to your host.

If you want the search and AI work done after that, our SEO, AEO and GEO programme starts from S$500 a month with a three-month minimum. It is the wrong purchase while your server is answering machines with a certificate error, and we will tell you so rather than take the order.

What nobody can promise you is a citation. AI platforms choose their own answers. All any of this does is make sure they can read the page before they choose.

Measurements describe twenty Singapore business pages sampled on 21 and 22 September 2026 from a data centre address. This page names no business and is not a rating of any company.

FAQ

How do I know if my website is blocking AI crawlers?

Run curl -I -A "GPTBot" https://yourdomain.com and compare the response with a plain curl -I. A different status code, a redirect into a challenge page, or a 403 means something is filtering by user-agent. Check robots.txt the same way, and repeat each test three times before concluding anything.

What is a Cloudflare 526 error?

Cloudflare describes it as an error indicating that Cloudflare cannot verify the SSL certificate on your origin server, which prevents a secure connection being established. It usually means the certificate at your own server has expired or does not match the domain. Visitors may still see a working site depending on configuration.

Why is an HTTP 202 captcha wall worse than an error?

Because 202 is a success code, so no crawler treats it as a problem. There is no slowdown, no preservation of the existing index entry, and no bad day to wait out. The crawler simply records that your page answered correctly and contained almost nothing, in our sample 169 bytes.

My site ranks fine on Google. Does this still matter?

Possibly. Google publishes its crawler IP ranges and many protection services allowlist them, so a site can serve Google the real page and serve everything else a wall. The growing list of everything else includes AI crawlers, audit tools and visitors on a VPN. Ranking now does not mean readable now.

Should I pay an agency to fix this?

Usually no. The fix is typically a certificate renewal, a CDN setting or a security plugin rule, and it belongs to whoever runs your hosting. Ask anyone who raises it to name the specific setting. If they cannot, they found it with a tool. Marketing work is worth buying after the page is readable.