A plain-text version of this site for AI assistants is at https://contengi.com/llms.txt, with the full product documentation in one file at https://contengi.com/llms-full.txt. Every page is also available as markdown by adding .md to its URL.
Four questions decide whether an AI engine can quote your page: can its crawler get in, can it read the content, does it understand what the page is, and can it find the rest of your site.
We load your page twice: once as GPTBot, with no JavaScript, and once in a real browser. Every heading, paragraph, and link a visitor sees gets matched against the bot’s copy. The score is the share of words that made it across.
Your robots.txt, read the way crawlers read it, for 14 bots across AI search and AI training. The scan also catches firewalls that turn GPTBot away before robots.txt comes into it.
Title, meta description, H1, canonical, language, internal links, alt text, and link previews. All read from the HTML a bot downloads, so a title that only appears once scripts run gets flagged.
Schema.org structured data, llms.txt, and your sitemap. Schema injected by a tag manager or a script shows up as a warning, because crawlers that don’t render never see it.
Google renders JavaScript, given time. Most AI crawlers skip it. When Vercel studied crawler traffic across its network in late 2024, none of the major AI crawlers ran JavaScript, OpenAI’s and Anthropic’s included. They downloaded script files and never executed them.
So a page assembled in the browser can look perfect to you and blank to ChatGPT. Single-page apps, reviews pulled in by a widget, pricing that loads on click, and FAQs hidden behind a script all fall into the gap.
The fix is to put the content in the HTML your server sends, through server-side rendering or a static build. After that, the job is giving bots something worth quoting.
FAQ
GPTBot, OpenAI’s crawler, using its published user agent. If your server refuses that request, we fetch the page again as a normal browser so you still get a result, and flag the refusal as a warning.
Usually. Googlebot renders pages in a headless Chrome, though rendering can trail the first crawl. ChatGPT, Claude, and Perplexity crawlers download the HTML and stop there, which is why the score is based on the raw HTML.
That’s your call. Blocking GPTBot or ClaudeBot keeps your pages out of future training data. Blocking the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) keeps you out of the answers themselves, which is rarely what anyone wants. The scan lists the two groups separately for that reason.
Any heading, paragraph, list item, link, or button a visitor can see that isn’t in the raw HTML. Cookie banners, chat widgets, and pop-ups are left out of the score, since every site injects those with JavaScript and nobody wants them quoted.
A plain-text file at yoursite.com/llms.txt that summarises your site and links to its key pages, written for AI models to read. It’s still a proposal and support across AI engines is patchy, but it’s a small file to write, so the scan suggests one.
Sometimes. Some firewalls let the real GPTBot through from OpenAI’s IP addresses while blocking everyone else using its name, and a few sites serve bots a different page altogether. The scan shows what a bot gets from an ordinary request, which is the right thing to fix against.
No. We fetch it, analyse it, and send you the result. The only thing we log is your IP address and the time, so one person can’t run hundreds of scans an hour.
Also free: the LLM visibility check and the AI slop cleaner.