How to block LLMs from accessing your website?

Published 6 months ago • 1 min read

This question came up during last week's session of the SEO for LLMs course, and it’s something a lot of SEOs are still unsure about:

Is it possible to block LLMs from accessing your content? Or are we all just putting rules into robots.txt and hoping for the best?

Let’s break it down based on what we discussed in the session.

1. Robots.txt can work

You can exclude LLM-related bots in robots.txt the same way you block any crawler. However, as discussed in the session:

Some bots respect robots.txt
Some… don’t
And some do until they don’t

So, robots.txt is helpful, but it’s not airtight.

2. Cloudflare is a great option

Cloudflare allows blocking at the server level, not just in robots.txt.
Crystal specifically mentioned:

Cloudflare has strict, LLM-focused rules
They even have an option that allows you to charge bots to access your content
Overall, it’s currently one of the strongest solutions if you truly want to block access

Blocking LLMs' access to paywalled content

If your content is visible when JavaScript is disabled (e.g., HTML includes the paywalled content by default), LLMs can still see it.
The session example noted:

New York Times, Financial Times, other major publishers - all blocking content at the logged-in experience level.

This approach prevents access even if someone tries to use dev tools, switch user agents, or render without JS.

in the next few weeks, I'll be sending three nuggets from our SEO for LLMs sessions. And if you want to get a full deep dive into this topic, you are still welcome to join the course.

Insights from Google Search Central Live Canada

Last Tuesday, I was at the first-ever Canadian edition of Google Search Central Live. with Martin Splitt It was a full house, and I was told not all people got in, so I am very grateful that I was invited there. Hopefully they'll do more events like this in Toronto. I could tell you that everything the Googlers shared was very new. But it's wouldn't be true. Most of the things they were talking about are what you're already doing at your daily SEO job. And here is why it's a great thing: We...

23 days ago • 2 min read

Technical SEO for LLMs?

Someone has recently asked me about the most important technical SEO considerations for AI visibility. So I thought I'd share them with you. Here you go 👇🏻 1. Allow AI Crawlers in your robots.txt Make sure your robots.txt does not block major AI bots: GPTBot (ChatGPT / OpenAI) ClaudeBot (Claude AI / Anthropic) PerplexityBot (Perplexity AI) Bingbot (used by Microsoft Copilot) Googlebot + Google-Extended (used by Google Gemini and AI Overviews) 💡 But if you want to block AI crawlers, robots.txt...

3 months ago • 1 min read

Master SEO communication skills in 4 Steps

Heads up, I'm doing a series of super valuable tips to help you improve SEO <> Dev communication (and grow your paycheck). If you're not interested, click here to opt out but stay on the list. In the previous email, we agreed on this: The results of your work depend on your SEO recommendations being implemented by devs. That’s why SEO + Devs should 🤝 These 4 steps will help you collaborate with developers much more effectively by speaking the same language with them. Let’s go 🙌 Step 1: Create...

3 months ago • 2 min read