How to block LLMs from accessing your website?

Published 3 months ago • 1 min read

This question came up during last week's session of the SEO for LLMs course, and it’s something a lot of SEOs are still unsure about:

Is it possible to block LLMs from accessing your content? Or are we all just putting rules into robots.txt and hoping for the best?

Let’s break it down based on what we discussed in the session.

1. Robots.txt can work

You can exclude LLM-related bots in robots.txt the same way you block any crawler. However, as discussed in the session:

Some bots respect robots.txt
Some… don’t
And some do until they don’t

So, robots.txt is helpful, but it’s not airtight.

2. Cloudflare is a great option

Cloudflare allows blocking at the server level, not just in robots.txt.
Crystal specifically mentioned:

Cloudflare has strict, LLM-focused rules
They even have an option that allows you to charge bots to access your content
Overall, it’s currently one of the strongest solutions if you truly want to block access

Blocking LLMs' access to paywalled content

If your content is visible when JavaScript is disabled (e.g., HTML includes the paywalled content by default), LLMs can still see it.
The session example noted:

New York Times, Financial Times, other major publishers - all blocking content at the logged-in experience level.

This approach prevents access even if someone tries to use dev tools, switch user agents, or render without JS.

in the next few weeks, I'll be sending three nuggets from our SEO for LLMs sessions. And if you want to get a full deep dive into this topic, you are still welcome to join the course.

Technical SEO for LLMs?

Someone has recently asked me about the most important technical SEO considerations for AI visibility. So I thought I'd share them with you. Here you go 👇🏻 1. Allow AI Crawlers in your robots.txt Make sure your robots.txt does not block major AI bots: GPTBot (ChatGPT / OpenAI) ClaudeBot (Claude AI / Anthropic) PerplexityBot (Perplexity AI) Bingbot (used by Microsoft Copilot) Googlebot + Google-Extended (used by Google Gemini and AI Overviews) 💡 But if you want to block AI crawlers, robots.txt...

11 days ago • 1 min read

Master SEO communication skills in 4 Steps

Heads up, I'm doing a series of super valuable tips to help you improve SEO <> Dev communication (and grow your paycheck). If you're not interested, click here to opt out but stay on the list. In the previous email, we agreed on this: The results of your work depend on your SEO recommendations being implemented by devs. That’s why SEO + Devs should 🤝 These 4 steps will help you collaborate with developers much more effectively by speaking the same language with them. Let’s go 🙌 Step 1: Create...

about 1 month ago • 2 min read

SEOs vs Devs

As an SEO, you are always doing your best, even when it feels hard. That’s why it’s so painful when developers just ignore your SEO recommendations. It’s like they don’t even care. 😭 And it hurts even more when you realize that the results of your work depend on your SEO recommendations being implemented by devs. (So your performance is basically judged based on the things that are out of your control.) I’m sorry, my friend. It does feel hard to speak the same language with developers. But...

about 1 month ago • 1 min read