43 pts
Fetchability
Cloudflare/WAF challenges, 403 walls, JS-only SPAs, robots.txt that locks out OAI-SearchBot or Perplexity-User. If LLM crawlers can't load your page, nothing else matters.
NEW Now checking 12 AI crawlers · llms.txt support
How well can AI find your site?
A free auditor that scores any URL 0–100 on whether ChatGPT, Claude, Perplexity, and Google AI Overviews can crawl, parse, and cite it. 27 weighted checks, run in seconds.
Works with every major AI search engine
- OpenAI
- Anthropic
- Perplexity
- Meta AI
- Apple
Why it matters
AI search is the new front page
ChatGPT alone reaches more than 800 million weekly users. When an AI picks 1–3 sources to cite, it picks from sites it can actually crawl and parse — not just whatever ranks #1.
800M+
ChatGPT weekly active users
17%
Of Google searches show AI Overviews
+41%
Citation lift from adding statistics
+28%
Citation lift from adding quotations
Generative engines reshape how visibility is earned online — well-cited, statistic-rich, quotation-rich content sees a substantial boost in citation rate, even from sources that don't rank in the top 5 of traditional search.
Aggarwal et al., GEO: Generative Engine Optimization · Princeton, 2024
What we check
Five categories. 27 checks. One score.
Every audit runs the same weighted checks across five signal categories that determine whether AI engines can find, parse, and cite your site.
21 pts
Core SEO
Title length, meta description, canonical URL, OpenGraph/Twitter, sitemap.xml, html lang. The traditional SEO foundation that AI search still expects.
13 pts
Semantic HTML
One h1, no skipped heading levels, header / nav / main / article / footer landmarks, image alt text. The shape AI uses to parse your page.
22 pts
Answer Engine Signals
llms.txt manifest, JSON-LD (Article, Organization, FAQ, Person, LocalBusiness), Mozilla Readability extraction, author byline + sameAs links, dateModified freshness.
6 pts
Content Quality
Princeton's GEO study found front-loaded answers, statistics density (+41%) and quotations (+28%) are what actually move the needle in generative answers. We check for them.
12 bots
AI crawler matrix
GPTBot, ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended — we check robots.txt against every one.
How we handle bot-protected sites
We always try a vanilla direct fetch first. If your site responds with a Cloudflare
challenge, 403, or other WAF block, we retry through BrightData's Web Unlocker so we
can still grade the page — but you lose the major fetch_direct
credit, because that's exactly what an AI crawler would also fail on.
Real example
This is happening right now in every AI assistant.
Users no longer scroll through 10 blue links — they get a synthesized answer with 1–3 citations. Whether yours is one of them depends on signals you can actually fix.
Crawler can fetch your HTML directly
JSON-LD structured data is present
Content uses statistics and citations
AI Assistant
FAQ
Frequently asked questions
What is Answer Engine Optimization (AEO)?
AEO is the practice of structuring a website so that generative AI search engines —
ChatGPT, Claude, Perplexity, Google AI Overviews — can crawl it, understand it, and
cite it as a source. It overlaps with SEO but adds signals like an
/llms.txt manifest, robots.txt allowlists for AI crawlers
(OAI-SearchBot, Claude-User, PerplexityBot,
Google-Extended), JSON-LD structured data, and content patterns from
Princeton's GEO research (front-loaded answers, statistics, quotations).
Read the full AEO primer → How does the score work?
The auditor runs 27 weighted checks across five categories — Fetchability (43 points),
Core SEO (21), Semantic HTML (13), Answer Engine signals (26), and Content quality (6) —
then normalizes to 0–100. Letter grade: A ≥ 90, B ≥ 80,
C ≥ 70, D ≥ 60, F otherwise.
See the full rubric →
Does it run my site through a headless browser?
No. The auditor reads server-rendered HTML directly via
undici, the same
way a real LLM crawler would. If your site needs JavaScript to render content, it
fails the ssr_content check — that's intentional, because AI crawlers
also don't run JS reliably.
Why AI crawlers don't run JS → Which AI crawlers does the tool check robots.txt against?
GPTBot, ChatGPT-User, OAI-SearchBot,
ClaudeBot, Claude-User, Claude-SearchBot,
PerplexityBot, Perplexity-User, Google-Extended,
Applebot-Extended, Meta-ExternalAgent, and CCBot.
The complete robots.txt for AI crawlers
guide → What is /llms.txt and do I need one?
/llms.txt is a proposed convention for a markdown file at your site root
that gives AI agents a curated index of your most important pages — like a sitemap, but
optimized for LLMs to read. It's worth 5 of the 100 points in this audit.
/llms-full.txt is an optional full-content dump worth 1 bonus point.
How to write an llms.txt → See how AI engines see your site.
Free. No signup. Results in seconds.