If ChatGPT, Claude, or Perplexity never cite your site when users ask questions in your space, the cause is almost always one of six things. This guide walks each in order of likelihood, with how to test for it and what to fix.

The short answer

AI assistants only cite pages they can crawl, extract, and trust. If any of those three break, you’re invisible. The six issues below are the specific ways each one breaks for most sites.

1. AI crawlers can’t reach your site

ChatGPT’s crawler is GPTBot, Claude’s is ClaudeBot, Perplexity’s is PerplexityBot, Google’s AI overview crawler is Google-Extended. If your robots.txt disallows any of them, you opt out of that engine entirely.

The other common block is at the edge. Cloudflare and some firewall vendors ship default rules that 403 these user-agents even when robots.txt allows them. The same applies to rate-limiting and bot-detection systems.

How to check: try fetching a page with each user-agent. If you get a 403 or a captcha page, the crawler does too.

2. Your content is hidden behind JavaScript

If your page is a client-side React/Vue/Svelte app that fetches its content after load, most AI crawlers see an empty shell. GPTBot and ClaudeBot don’t execute JavaScript at the same fidelity as Google’s rendering pipeline.

The fix is server-side rendering (or static generation) for anything you want cited. View the page source — not DevTools — to see what crawlers actually receive.

3. There’s no structured data for the AI to extract

AI retrievers favour pages with explicit signals: JSON-LD for articles, FAQs, products, organizations. A page with <script type="application/ld+json"> declaring its author, publish date, and topic is easier to extract and trust than an unmarked one with the same prose.

At minimum, add Article schema to posts, Organization schema sitewide, and FAQPage schema anywhere you have Q&A content. Validate with Google’s Rich Results Test before shipping.

4. Your content isn’t answer-shaped

AI assistants synthesise answers. They prefer pages that already answer in the same shape. The reliable pattern:

• H1 = the question, verbatim.
• First paragraph = direct 2-3 sentence answer.
• H2s = sub-questions, in question form.
• Under each H2 = a direct answer, then expansion.

A 2000-word essay that buries the answer in paragraph 12 loses to an 800-word page that answers in paragraph 1 — even if the essay is more thorough.

5. You lack E-E-A-T signals AI engines weigh

Experience, Expertise, Authoritativeness, Trustworthiness. The specific signals AI grounding looks for:

• A real author byline that links to an author page.
• Visible datePublished and dateModified — both in human-readable form and in JSON-LD.
• Outbound citations to primary sources (research, official docs, named experts).
• An organization with an Organization schema and consistent sameAs links to your social profiles.

Anonymous, undated pages without citations are demoted across every AI engine that we’ve tested.

6. You don’t have an llms.txt

llms.txt is to AI crawlers what sitemap.xml is to search crawlers — a plain-text index of your most important content, in a format optimised for LLM consumption. It’s a new convention from llmstxt.org.

You can see ours at /llms.txt. It takes minutes to add and is one of the lowest-effort, highest-signal moves available.

Per-engine quirks worth knowing

The big four behave a little differently. None of these override the six issues above, but they help when diagnosing why you appear on one engine and not another.

• ChatGPT — combines historical training data with live retrieval via GPTBot. If you launched recently, historical training doesn’t help you; you need live retrieval to work.
• Perplexity — scrapes live on every query. Blocking PerplexityBot is the fastest way to disappear from Perplexity completely.
• Claude — Anthropic uses ClaudeBot for retrieval grounding. Tone and clarity matter; sites with clear prose tend to be cited more than keyword-stuffed pages. See our Claude-specific guide.
• Gemini & Google AI Overviews — pull from Google’s index, but opting out via Google-Extended removes you from Gemini answers without affecting regular Google ranking. See our Google AI guide.

How to check if your site is the problem

The fastest way is to scan it. The form below runs the same audit described in this article and returns a grade for each of the six issues, with specific fixes.

Frequently asked questions

Ready to fix what’s blocking you? Create a free account or see pricing.