The AI-Crawlable SaaS Implementation Checklist

This is the sequenced version of everything else in this cluster - five phases, each with a concrete pass/fail check, in the order that actually works. The most common mistake isn't skipping steps, it's doing them in the wrong order: teams add llms.txt first because it's the easiest box to check, then never get to the rendering fix that actually matters. This checklist puts things in the order the evidence supports, not the order that feels easiest.

Phase 1 - Diagnose

1. Run the log-based self-test on your three to five highest-priority pages - pricing, feature/comparison pages, your docs homepage, and any landing pages carrying real commercial intent. Record four separate outcomes per page, not one pass/fail: whether a verified bot was observed at all; the final HTTP delivery status and any redirect or challenge result; whether the critical, crawler-relevant content is actually present in the initial response body (not a size comparison to what a browser renders - fetch and inspect the raw response); and whether a root cause is established or still under investigation. Access logs record status, bytes, and user agent, not the response body a bot parsed, and response size can vary with compression, cache variants, personalization, redirects, or WAF responses - treat it as an anomaly signal that tells you where to look closer, never as the pass criterion itself. Fail: document which specific pages failed on which of the four outcomes - don't move to Phase 2 without a concrete list.
2. Identify the root cause of every failure. Is this a plain client-side-rendered SPA with no server-rendering layer at all, or a specific component inside an otherwise SSR-capable framework that's fetching its data client-side - or is it not a rendering problem at all (blocked access, a redirect loop, personalization, geolocation, a backend error)? Pass: each failing page has a stated, specific cause - "the whole app is CSR," "this one pricing-table component uses a client-only fetch," and "this page is blocking the bot at the WAF" require different fixes, and conflating them wastes engineering time in Phase 3.
3. Confirm which crawlers are real. Cross-reference the user-agent strings in your logs against each operator's published IP ranges. Pass: you have a confirmed list of genuine, IP-verified bot traffic - not a raw count of anything claiming to be "GPTBot" in a log line.

Phase 2 - Decide

4. Choose a fix path per failing page or component using the decision framework: what does the content need (static, periodically refreshed, or genuinely per-request), are you already on an SSR-capable framework, how much engineering time do you actually have this quarter, and is this a permanent fix or a bridge you'll revisit. Pass: every failing page or component is assigned to one of: build-time static generation; cached/revalidated rendering (ISR/SWR); request-time server rendering; a hybrid static/cached shell with a request-time component; or bot-targeted dynamic rendering as a monitored bridge - not left as "we'll figure it out." Reserve request-time or hybrid request-time rendering for content that genuinely varies by request, and never assign a bot-targeted prerendering snapshot to personalized or authenticated content - a cached crawler snapshot is a public, generic representation, not request-specific user rendering, and can't substitute for it.
5. Scope the project to public pages only. Exclude authenticated app routes - no crawler reaches them regardless of rendering method, so fixing them isn't part of this project. Pass: a written scope line (marketing site, docs, any public content) that a reviewer could hold you to.

Phase 3 - Fix

6. Implement the chosen fix for each page or component per its assigned path from Phase 2.
7. Check every content-bearing component's data-fetching method specifically. This is the step migrations most often skip: adopting Next.js or Nuxt doesn't automatically fix anything if the component that renders your pricing table still fetches its data with a client-only hook instead of the framework's server-aware one. Pass: no content that needs to be crawler-visible depends on a fetch that only runs after the page has already loaded in a browser.
8. Fix your sitemap's indexability, and lock down staging, while you're already in the code. A sitemap that merely returns no 404s isn't enough - a listed URL can still 200 while redirecting elsewhere, carrying a noindex directive, being blocked, duplicating another page's canonical, or serving a soft 404, and it would still pass a 404-only check. Pass: the production sitemap contains only the intended absolute canonical URLs, each returns its expected content without an unwanted redirect, and each URL's robots/indexing directives are consistent with being included (per Google's own sitemap guidance: include the URLs you want to appear in search, and generally list canonical URLs - and per Google's own noindex documentation, a noindex tag only works if the crawler can actually reach the page, so don't block a URL in robots.txt and expect noindex on it to do anything). Separately: exposed staging or non-production environments are an access-control and confidentiality problem, not a crawl-budget one - pass means staging is behind authentication or network access controls, not merely "not linked anywhere" or excluded via robots.txt, which Google's own guidance is explicit is not a substitute for actually restricting access to confidential or private content.

Phase 4 - Verify

9. Run a controlled initial-HTML regression test immediately after deployment - record final delivery status, the redirect chain, response headers, and whether the page-specific critical-content marker is present in the raw response body (not a browser-DOM comparison). Pass: the regression test confirms the fix at the HTTP level for every page that previously failed, on the same four-outcome basis as Phase 1 - this is a synthetic check you control, not proof a real bot has revisited yet.
10. Separately, monitor verified provider traffic for a defined observation window - 2 to 4 weeks is reasonable for most crawlers, or until each target crawler has revisited the priority URLs, whichever comes first; low-volume crawlers may take longer, so set the window based on that crawler's typical frequency in your own logs rather than a fixed deadline for all of them. Pass: IP-verified revisits from the target crawlers return clean results consistent with the regression test. Not yet observed (distinct from fail): no verified revisit has happened yet within the window - that's a reason to keep waiting or investigate crawl frequency, not to declare the fix broken. Reserve an actual fail for a verified revisit that still comes back with the content problem.

Phase 5 - Deliberately Low Priority

11. Add llms.txt last, and only if it's genuinely trivial to add. The evidence is direct on this: Ahrefs' May 2026 analysis of 137,210 domains found about 38,000 with a valid llms.txt file, and 97% of those received zero requests of any kind - no bots, no humans - that month. Ahrefs itself notes its customer base skews more technical and SEO-aware than the web generally, so treat that as an upper bound on adoption, not a random sample. Separately, Google's own Search documentation says new machine-readable or AI text files aren't needed to appear in AI Overviews or AI Mode - a narrower statement than "Google never uses them," and Chrome's Lighthouse tooling now includes an optional llms.txt audit, which is worth acknowledging even though "emerging and optional" isn't the same as "adopted." Taken together: it costs almost nothing and won't hurt anything, but it does not belong ahead of any step above it, and it should never be presented as a meaningful AI-visibility fix on its own. Worth rechecking this periodically - adoption and provider support are exactly the kind of thing that can shift with one announcement.

What to Do Next

← Back to the full pillar page
Phase 1: the self-test, step by step →
Phase 2: the SSR vs. prerendering decision guide →
Phase 3: the client-fetch failure mode, explained →
Phase 4: the UA and IP-verification reference →
Phase 5: the full llms.txt verdict →

---

Sources: Most of this page synthesizes implementation guidance already sourced, claim by claim, in the pillar page and Pages 1 through 5 and 7 of this cluster; see each linked page for the sourcing behind the claim it covers (crawler behavior, framework rendering behavior, and so on). This revision added direct citations for the acceptance criteria specific to this page: response-body-vs-log-metadata behavior is documented at NGINX's access-log module documentation, response-variation-by-request is documented at RFC 9110's content-negotiation section, and curl's own defaults are documented at curl's HTTP scripting guide. Sitemap and indexability guidance is from Google's own sitemap-building documentation and noindex documentation. The staging/confidential-content guidance draws on Google's control-what-you-share guidance (password-protecting private content) together with Google's own robots.txt introduction, which states directly that robots.txt "is not a mechanism for keeping a web page out of Google" and that password-protection is the right tool "if you want to keep information secure from web crawlers." The llms.txt usage figures (137,210 domains, ~38,000 with a valid file, 97% zero-request rate, May 2026) are from Ahrefs' own study, including its own stated sample caveat; Google's "not needed for AI Overviews or AI Mode" language is from Google's AI-features documentation; the optional llms.txt check is from Chrome Lighthouse's llms.txt audit documentation.