Next.js SEO for AI Search: How to Get Your Site Seen in ChatGPT and Google AI Mode

Your Next.js site can rank well on Google and still be invisible to ChatGPT, Claude and Perplexity. The crawlers behind those assistants fetch your raw HTML and never run your JavaScript. Anything that appears only after hydration, a client-side fetch or a late streamed chunk may never reach them.

Here is the short answer. To make a Next.js site visible in AI search:

• Serve your real content in the first HTML response.
• Keep content pages fully static, especially if you use Cache Components.
• Allow AI search crawlers in robots.txt, and make a separate decision about training crawlers.
• Make sure your firewall agrees with your robots.txt.
• Prove it works with crawler tests and server logs, not assumptions.

This guide covers each step with working code, plus several issues most Next.js SEO guides for AI search have not caught yet.

Quick Answer: What AI Search Needs From a Next.js Site

Generative engine optimization (GEO) and answer engine optimization (AEO) sound like new disciplines. On a Next.js site, they mostly come down to one question: can a machine that reads only HTML find, understand and quote your page? To answer yes, your site needs six things:

• Readable HTML: headlines, body text, internal links and key facts are in the first HTTP response, not injected later by JavaScript.
• Complete metadata in the head: title, description, canonical URL and Open Graph tags arrive in the head of the document, not in a streamed chunk.
• Crawler access: robots.txt allows the AI search crawlers you want, and no firewall rule quietly overrides it.
• Honest responses: real pages return 200, missing pages return 404, and rate limits return 429 instead of a silent challenge page.
• Citable content: direct answers, specific facts and clear headings that an engine can lift as a passage.
• Discovery signals: a sitemap, Bing indexing and IndexNow, so search indexes pick up new and updated pages quickly.

Next.js gives you a head start, because Server Components render HTML on the server by default. But several common patterns undo that advantage without any visible error, and the newest one comes from the framework's own recommended defaults.

How AI Crawlers Read Your Next.js Pages

Do AI Crawlers Render JavaScript?

For the most part, no. A server log study by Vercel and MERJ found that none of the major AI crawlers render JavaScript. OpenAI's and Anthropic's crawlers do download JavaScript files (around 11.5 percent of ChatGPT crawler requests and 23.8 percent of Claude crawler requests), but they never execute them. SEO consultant Glenn Gabe tested fully client-rendered pages and found they looked blank to ChatGPT, Perplexity and Claude.

Google is the exception. Googlebot renders pages with a headless Chromium browser, so content built by JavaScript can still reach Google Search, AI Overviews and AI Mode. Browser-based agents also run a real browser during a user's session.

The safe rule is simple: build for the crawlers that cannot render. Anything a non-rendering crawler can read, a rendering crawler can read too.

The volume is not small. Across Vercel's network, GPTBot generated 569 million requests in a single month, and Anthropic's Claude crawlers generated 370 million. Together with AppleBot and PerplexityBot, these crawlers accounted for roughly 28 percent of Googlebot's volume over the same period.

What a Controlled Test Revealed

Vercel and the Ora research lab served one documentation site in 19 different configurations. They then tested whether two clients could retrieve a planted answer: a plain fetch client and a JavaScript-executing client. Only two kinds of change stopped retrieval:

• Answer available only through JavaScript: the plain fetch client failed every probe (0 of 5), while the JavaScript client passed every one.
• Server returning 403 to agent user agents: both clients failed every probe.

The more surprising result was this: adding llms.txt, JSON-LD, sitemaps or Markdown mirrors did not change whether the clients could retrieve an answer that was already in server-rendered HTML.

That sets the priority order for everything in this guide. Rendering and access come first. Everything else helps only once those two are right.

The AI Crawlers That Matter

Each AI company runs several user agents, and each has a different job. Mixing them up is the most common robots.txt mistake.

The Cache Components Trap: Why AI Crawlers May Get an Incomplete Page

This is the biggest gap in Next.js GEO advice, and it is growing quickly. Starting with Next.js 16.4, the Next.js team recommends Cache Components as the best choice for every Next.js app. New apps created with create-next-app have it enabled by default, and it will become the default in Next.js 17.

Cache Components is an excellent model for performance. But it changes what crawlers receive, and AI crawlers are not part of the default plan.

How Next.js Decides What a Bot Receives

With Cache Components, regular visitors and DOM-capable crawlers receive the prerendered shell immediately, and dynamic content streams in as it resolves.

A second group, called HTML-limited bots, is handled differently. These bots skip the prerendered shell and get a dynamic render, so their metadata can be placed in the head. The htmlLimitedBots config option, introduced in Next.js 15.2, decides which user agents belong to that group. Its default list includes:

• Google crawlers such as Mediapartners-Google, AdsBot-Google and Google-PageRenderer
• Bingbot
• Twitterbot
• Slackbot

The default list was built for search engines and social preview bots. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot don't match it, so under Partial Prerendering they receive the streaming shell instead of a complete document.

A streamed response delivers dynamic content in later chunks, and inline scripts move those chunks into place. A crawler that doesn't run scripts, or that stops reading early, can miss content and find metadata outside the head.

You have two ways to fix this. The first is safer.

Fix 1: Make Content Pages Fully Static (Recommended)

If a page is fully prerendered, every client receives the same complete HTML from the cache. Metadata is resolved at build time, there is no streaming to miss, and bot detection doesn't matter at all. Blog posts, service pages and case studies rarely need request-time data, so this fits most marketing and content sites.

Next.js 16.4 added ensureStatic to enforce this. Set to 'navigation' , it fails the build if any dynamic content is included in the page, so a request-time component can never sneak into your blog by accident.

// app/blogs/[slug]/page.tsx
import { cacheLife } from 'next/cache'
import { getPost, getAllSlugs } from '@/lib/posts'

export const ensureStatic = 'navigation'

export async function generateStaticParams() {
  const slugs = await getAllSlugs()
  return slugs.map((slug) => ({ slug }))
}

async function getCachedPost(slug: string) {
  'use cache'
  cacheLife('days')
  return getPost(slug)
}

export async function generateMetadata({
  params,
}: {
  params: Promise<{ slug: string }>
}) {
  const { slug } = await params
  const post = await getCachedPost(slug)

  return {
    title: post.metaTitle ?? post.title,
    description: post.metaDescription ?? post.excerpt,
    alternates: { canonical: `/blogs/${slug}` },
  }
}

export default async function Page({
  params,
}: {
  params: Promise<{ slug: string }>
}) {
  const { slug } = await params
  const post = await getCachedPost(slug)

  return (
    <article>
      <h1>{post.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: post.html }} />
    </article>
  )
}

After building, check the route in the build output. If it is marked as Partial Prerender ( ◐ ), something on the page is still dynamic.

Also handle new posts. Slugs that were not known at build time are served from a fallback shell on their first request, which is exactly the route type affected by the bug described below. When you publish, trigger a rebuild or call revalidatePath for the new URL and the blog index. Then test the new URL as a crawler before you share it.

You can also add ensureStatic to a layout. For example, setting it on app/blogs/layout.tsx protects every post at once.

Fix 2: Add AI Crawlers to htmlLimitedBots (Test First)

Some pages genuinely need request-time data. For those pages, you can extend htmlLimitedBots so AI crawlers receive the blocking render:

// next.config.ts
import type { NextConfig } from 'next'

const nextConfig: NextConfig = {
  cacheComponents: true,
  htmlLimitedBots:
    /[\w-]+-Google|Google-[\w-]+|Bingbot|Applebot|Twitterbot|Slackbot|Discordbot|LinkedInBot|facebookexternalhit|GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i,
}

export default nextConfig

Known Issues to Check Before You Ship This

1. Custom bots can lose all metadata or trigger server errors. A reported Next.js issue ( #95406 ) describes this on non-prerendered PPR routes: user agents that match your custom htmlLimitedBots list but not the built-in list receive pages with no title, description or canonical tag. In one production report on Vercel, GPTBot requests returned HTTP 500 errors, producing more than 100 errors within hours of deployment, and the team reverted the config. The cause appears to be that some parts of Next.js still check the built-in list directly. Test this on your exact Next.js version in a preview deployment before going live.
2. Your list replaces the default; it doesn't add to it. Carry over every built-in entry you rely on. Check the current default list in the Next.js source file html-bots.ts . A proposed fix ( #98749 ) would add AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and ChatGPT-User to the built-in list. If your version includes that change, you may not need a custom list at all.
3. Blocking renders cost response time. Every bot on the list waits for generateMetadata and everything it awaits. Keep metadata queries fast and cached.
4. The crawler path renders separately. Bots on this list re-render the page dynamically, so a page that loads for a person can fail for a crawler. A shell component that depends on build-time-only data is the usual cause.

Because of these risks, use Fix 1 wherever you can. Use Fix 2 only for pages that must be dynamic, and only after testing.

Streaming Metadata on the Edge Runtime

A separate reported issue ( #98748 ) affects Edge SSR. Title, description and Open Graph tags arrive in later Suspense chunks, and HTML-limited bots close the connection after the first chunk. As a result, social previews and AI crawlers see empty metadata.

If any of your content pages run on the Edge runtime, test them with the crawler checks later in this guide, or move them to static generation.

Client-Side Rendering Leaks That Hide Your Content

Even a site built entirely on the App Router can leak client-side rendering into important places. These four patterns are the most common.

Lists Fetched in useEffect

When I fetched my own blog index as plain HTML, the response contained the layout, navigation and footer, but not one link to an article. The post list was being loaded in the browser. Google could still render it. To GPTBot, my blog index was an empty page with no path to the posts.

The pattern usually looks like this:

'use client'

import { useEffect, useState } from 'react'

export default function BlogList() {
  const [posts, setPosts] = useState([])

  useEffect(() => {
    fetch('/api/posts')
      .then((res) => res.json())
      .then(setPosts)
  }, [])

  return (
    <ul>
      {posts.map((post) => (
        <li key={post.slug}>
          <a href={`/blogs/${post.slug}`}>{post.title}</a>
        </li>
      ))}
    </ul>
  )
}

The fix is to fetch on the server so the links ship in the first response:

import Link from 'next/link'
import { getPosts } from '@/lib/posts'

export default async function BlogList() {
  const posts = await getPosts()

  return (
    <ul>
      {posts.map((post) => (
        <li key={post.slug}>
          <Link href={`/blogs/${post.slug}`}>{post.title}</Link>
        </li>
      ))}
    </ul>
  )
}

You can keep filters, search and entrance animations. Put them in a small client component that wraps the server-rendered list, rather than letting the list itself depend on the browser.

This matters beyond the index page. In the Vercel and Ora study, agents reached the homepage in 69 percent of runs, and it was their first step 92 percent of the time. Agents navigate through links just as people do. If your index has no crawlable links, your posts become orphans.

Accordions, Tabs and Read More Toggles

If a click handler mounts the text, a crawler that doesn't run JavaScript never sees it. FAQ accordions, pricing tabs and "read more" buttons often hide the most quotable content on a page.

The fix is to render the full text on the server and collapse it with CSS. The native details element does this with no JavaScript at all:

<details>
  <summary>What does a Next.js migration include?</summary>
  <p>The full answer lives in the HTML, so crawlers can read it even though it starts collapsed.</p>
</details>

Animated Text, Canvas and 3D Scenes

Animated sites carry extra risk. Some text animation setups render empty spans until JavaScript splits the words. Some heroes put the main headline inside a WebGL canvas, where no crawler can read it.

Keep the real text in the HTML and animate it after load. An element that starts at zero opacity is fine, because the text still exists in the markup. A headline that only exists as pixels in a Three.js scene is not.

If you're choosing between animation libraries with this in mind, my guide to animated website libraries for Next.js and React covers which ones work with Server Components.

Code Shown as Images or Client-Side Embeds

For developer content, this matters more than most people expect. In the Vercel and Ora study, 61 percent of agent answers that included code contained lines copied verbatim from a page the agent had fetched.

Code shown as screenshots, or loaded through client-side embeds, can't be reused that way. Put code in real pre and code elements in the server-rendered HTML.

robots.txt for AI Crawlers: Allow AI Search, Block AI Training

OAI-SearchBot vs GPTBot

OpenAI's crawler documentation says each setting is independent. You can allow OAI-SearchBot so you appear in ChatGPT search, while disallowing GPTBot to keep your content out of model training.

The reverse mistake is common and costly. Blocking OAI-SearchBot removes your pages from ChatGPT's search answers. Many sites block "OpenAI" broadly and then wonder why ChatGPT never cites them.

Two more details matter:

• ChatGPT-User is different. It visits pages when a person asks ChatGPT to do something, and OpenAI notes that robots.txt rules may not apply to it.
• Changes aren't instant. OpenAI says a robots.txt change can take around 24 hours to reach its systems, so neither a block nor a reversal takes effect immediately.

Other companies follow the same split. Anthropic uses ClaudeBot for training, Claude-SearchBot for search and Claude-User for user requests. Google uses the Google-Extended token to control Gemini training. That token doesn't affect how you rank in Google Search.

A robots.ts You Can Copy

This configuration allows AI search and user-triggered fetches, blocks the main training crawlers, and keeps private routes closed to everyone:

// app/robots.ts
import type { MetadataRoute } from 'next'

const SITE = 'https://yoursite.com'
const privatePaths = ['/admin', '/api/', '/drafts']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: privatePaths,
      },
      {
        userAgent: [
          'OAI-SearchBot',
          'ChatGPT-User',
          'Claude-SearchBot',
          'Claude-User',
          'PerplexityBot',
          'Perplexity-User',
        ],
        allow: '/',
        disallow: privatePaths,
      },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot'],
        disallow: '/',
      },
    ],
    sitemap: `${SITE}/sitemap.xml`,
  }
}

The AI search group repeats what * already allows. That's deliberate: it documents your decision in code, and it protects you if someone later tightens the wildcard rule.

The Group Matching Rule Most People Miss

Under the robots.txt standard (RFC 9309), a crawler follows only the most specific group that matches its user agent. Once OAI-SearchBot finds its own group, it ignores the * group completely.

So if you give AI crawlers a named group with only allow: '/' , they're also allowed into /admin and /api/ . Always repeat your private paths inside every named group.

Adding Content Signals to robots.txt

Content Signals is a robots.txt extension, introduced by Cloudflare, that states how you want your content used after it's crawled. It has three signals:

• search : building a search index and showing results
• ai-input : real-time AI answers, such as grounding and retrieval
• ai-train : model training or fine-tuning

Cloudflare's managed robots.txt sets search=yes, ai-train=no as the default for millions of domains, and leaves ai-input for each owner to decide. If you want to appear in AI answers, set ai-input=yes explicitly.

The MetadataRoute.Robots type in robots.ts has no field for custom lines like this. To add one, replace app/robots.ts with a route handler that returns plain text (you can't have both files):

// app/robots.txt/route.ts
const SITE = 'https://yoursite.com'

const body = `User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /admin
Disallow: /api/
Disallow: /drafts

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: ${SITE}/sitemap.xml
`

export function GET() {
  return new Response(body, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  })
}

In this version, AI search crawlers have no named group, so they follow the * group, private paths included. That avoids the group matching problem described above.

Two caveats. Compliance with Content Signals is voluntary, so treat them as a statement of preference, not a lock. Google Search Console may also report the line as "Syntax not understood". Cloudflare says it has seen no effect on crawling or SEO from those reports.

Should You Block Training Crawlers at All?

That's a business decision, not an SEO decision. Blocking GPTBot doesn't change whether ChatGPT cites you in search, because training affects future models, not the live search index. Blocking it also doesn't remove data that was already collected.

If you want maximum reach and don't mind training use, allow everything. If you want citations without contributing training data, use the split shown above.

Firewall and CDN Rules That Override robots.txt

robots.txt is a request, not an access control. Your firewall decides who actually gets through. In the controlled probes described earlier, a server returning 403 to agent user agents stopped every client, no matter how the page was built.

Vercel's AI Bots Managed Ruleset

Vercel's AI bots managed ruleset identifies known AI crawlers and can deny them at the firewall, regardless of what robots.txt says. It's inactive by default and shows as Allow in the dashboard. If it has been switched to Deny , either set it back or add a bypass rule for your public routes.

The separate bot protection ruleset is less risky. In Challenge mode, it automatically excludes crawlers in Vercel's verified bot directory.

Cloudflare and Other WAFs

Cloudflare, AWS WAF and similar services offer one-click "block AI bots" rules. Those rules can drop search crawlers before they ever read your robots.txt.

If you use one, check exactly which user agents it covers. Make sure the rule that blocks training crawlers doesn't also catch OAI-SearchBot, Claude-SearchBot or PerplexityBot.

Return Honest Status Codes

Crawlers and agents rely on HTTP status codes to understand your site:

• Missing pages: return a real 404 (in Next.js, call notFound() ) or redirect to a replacement. Never return 200 with a "page not found" message. These soft 404s remove the signal that tells crawlers the page is gone.
• Rate limiting: return 429 Too Many Requests with a Retry-After header, not a silent challenge page. This gives well-behaved crawlers a clear signal to back off and return later.

How to Test Your Next.js Site as an AI Crawler

Most AI visibility problems are invisible in a browser. Testing works on two layers:

• The application layer: what your Next.js app sends to each user agent. curl tests cover this.
• The network layer: whether real crawlers get through your firewall. Only server logs can confirm this.

Run the following checks on every page template, not just the homepage.

Step 1: Check the Status Code

curl -iL -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yoursite.com/blogs/your-post

The request should end with a 200 . Redirects along the way are fine. A 403 or 429 usually points to a firewall or CDN rule, not robots.txt. A 500 on a dynamic route can point to the htmlLimitedBots issue described earlier.

Step 2: Read What the Crawler Reads

curl -s -A "GPTBot/1.2 (+https://openai.com/gptbot)" https://yoursite.com/blogs > bot_view.html

grep -c 'href="/blogs/' bot_view.html

Open bot_view.html and confirm your headline, body text and internal links appear as plain text. On a blog index, the grep count should roughly match your number of posts. If it returns zero, the list is client-rendered.

Step 3: Confirm Metadata and JSON-LD Arrive in the First Response

grep -o '<title>[^<]*</title>' bot_view.html
grep -c 'application/ld+json' bot_view.html

If the title is missing, review the Cache Components section above. If the JSON-LD count is zero, check whether a tag manager is injecting your schema after load. You can confirm the same thing by opening view-source: on any page and searching for application/ld+json .

Step 4: Loop Through Every AI User Agent

Different user agents can hit different firewall rules and different rendering paths. Test them all:

for ua in "GPTBot" "OAI-SearchBot" "ChatGPT-User" "ClaudeBot" "Claude-SearchBot" "Claude-User" "PerplexityBot"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$ua" https://yoursite.com/blogs)
  echo "$ua: $code"
done

Step 5: Confirm Real Crawler Access in Your Logs

This is the step most guides skip. A curl test can't prove that a real crawler gets through. Bot blockers judge requests by where they come from as much as by the user agent string, so your laptop can pass where a real crawler is blocked, or fail where a real crawler gets through.

Your server logs are the real check:

• OpenAI publishes IP ranges for its crawlers, including searchbot.json for OAI-SearchBot and chatgpt-user.json for ChatGPT-User. Anthropic publishes ranges for its crawlers too.
• Find requests from those ranges and confirm they reach the intended pages with a 200 .
• On Vercel, open Observability , then CDN Requests , and filter by crawler.
• A run of 404s usually means your sitemap lists URLs that no longer exist.

Matching IP ranges also protects you from scrapers that fake an AI crawler's user agent.

Next.js JSON-LD: What It Does and What It Doesn't

Structured data helps engines understand what a page is, who wrote it and how it fits into your site. It also powers rich results in Google. But it is not a magic AI lever.

Google's official generative AI guide lists AI-specific schema among the tactics you don't need. In the Vercel and Ora probes, adding JSON-LD didn't change whether clients could retrieve an answer. Ship it because it describes your page accurately, not because it forces citations.

Schema Types Worth Shipping

A note on FAQPage: Google shows FAQ rich results only for a small set of authoritative sites. Treat FAQPage markup as context for machines, not as a way to win a visual search feature.

Render JSON-LD in the App Router

Render the script tag inside the page component, using the same data that renders the visible content. JSON-LD injected after load, for example through a tag manager, never reaches non-rendering crawlers.

const SITE = 'https://yoursite.com'

function ArticleJsonLd({ post }: { post: Post }) {
  const url = `${SITE}/blogs/${post.slug}`

  const jsonLd = {
    '@context': 'https://schema.org',
    '@graph': [
      {
        '@type': 'BlogPosting',
        '@id': `${url}#article`,
        headline: post.title,
        description: post.excerpt,
        image: post.coverImage,
        datePublished: post.publishedAt,
        dateModified: post.updatedAt ?? post.publishedAt,
        mainEntityOfPage: url,
        author: {
          '@type': 'Person',
          name: post.authorName,
          url: `${SITE}/about`,
        },
      },
      {
        '@type': 'BreadcrumbList',
        itemListElement: [
          { '@type': 'ListItem', position: 1, name: 'Home', item: SITE },
          { '@type': 'ListItem', position: 2, name: 'Blogs', item: `${SITE}/blogs` },
          { '@type': 'ListItem', position: 3, name: post.title, item: url },
        ],
      },
    ],
  }

  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{
        __html: JSON.stringify(jsonLd).replace(/</g, '\\u003c'),
      }}
    />
  )
}

Render <ArticleJsonLd post={post} /> inside your article. The replace call escapes any < character, so content inside the JSON can't break out of the script tag. For type safety, type the object with the schema-dts package.

Keep Critical Facts in Visible Text

Some tools that convert pages to Markdown for AI agents drop script blocks entirely. Anything that exists only in JSON-LD, such as a price, a service area or a credential, can disappear in that conversion.

• Repeat every important fact in the visible text.
• Show both the published date and the updated date on the page, and make them match datePublished and dateModified .
• Keep your name, services and contact details identical across your site, Google Business Profile, LinkedIn and directories. When sources disagree, AI engines have to pick one version, and it may not be yours.

Next.js AEO and GEO: Writing Content That Gets Cited

Technical access gets you into the room. Content decides whether you're quoted. AI engines cite passages, not whole pages, so write each section so it can stand on its own.

Lead Each Section With the Answer

Put a direct, complete answer in the first two or three sentences under each heading, then add detail. A paragraph that only makes sense after three paragraphs of context is hard to quote.

Use Question-Shaped Headings

Google has said AI Overviews and AI Mode can use a "query fan-out" technique: they run several related searches across subtopics, then combine the results. Headings that match those sub-questions, such as "Do AI crawlers render JavaScript?", give the engine a clear passage for each one.

Add Specifics Worth Quoting

Engines prefer content with something concrete to cite:

• Numbers with a named source
• Comparison tables
• Step-by-step instructions
• Working code
• Your own test results

A page that says "AI crawlers have trouble with JavaScript" gives an engine little to work with. A page that shows real curl output from a real site gives it evidence.

Show Who Wrote It and When It Was Updated

A named author with a linked about page is a stronger trust signal than an anonymous byline. Pair it with a visible updated date. Between two pages that answer the same question equally well, the more clearly maintained one has the edge.

Build Topical Depth

One strong article helps. A cluster of related, interlinked articles helps more, because it shows depth on a subject.

For example, this guide connects naturally to native view transitions in Next.js and to using WordPress as a headless CMS . Both involve deciding what renders on the server and what renders in the browser.

Google AI Mode and AI Overviews: What Is Different

Google Says AEO and GEO Are Still SEO

Google's official guide to optimizing for generative AI features says these features run on Google's core ranking systems. It also includes a mythbusting section listing tactics you don't need for Google Search:

• llms.txt files
• "Chunking" content into small pieces
• Rewriting content specifically for AI
• Special AI schema or Markdown versions of pages
• Pursuing inauthentic mentions

For Google, the work is the same work that has always mattered: helpful content, crawlable pages and a trustworthy site.

Googlebot Renders JavaScript, but Don't Lean on It

Because Googlebot renders pages, client-rendered content can still appear in Google's AI features. But rendering adds an extra processing step, and the same content stays invisible to ChatGPT, Claude and Perplexity.

Server-rendered HTML works for every engine at once. Treat Google's rendering as a fallback, not a strategy.

Eligibility and the Search Generative AI Setting

To be cited in AI Overviews or AI Mode, a page must meet three conditions:

• It is indexed.
• It is eligible to show with a snippet.
• The site is included in Search generative AI features, controlled in Search Console under Settings , then Search generative AI .

Check that this setting hasn't been switched off. Also check that you haven't added nosnippet to important pages by accident. Use data-nosnippet only on sections you genuinely want excluded.

Measure With the Search Console Generative AI Report

Search Console's Generative AI report shows impressions from AI Overviews and AI Mode, broken down by page, country, device and date. It doesn't show clicks or queries. Those impressions are also already included in the Web search totals of the main Performance report.

Here's a practical way to use it:

1. Export the list of pages with generative AI impressions.
2. Pull clicks for the same pages from the main Performance report, over the same dates.
3. Look for pages with many AI impressions but few clicks. Those pages are being used in answers but aren't earning visits. Give them a clearer reason to click, such as a tool, a template or a deeper worked example.

Bing, IndexNow and ChatGPT Search

Bing deserves more attention from Next.js developers than it gets. Bing's index powers Copilot, and it's widely reported as a key retrieval source for ChatGPT search. If Bing hasn't indexed a page, that page has fewer ways to reach AI answers.

Start by verifying your site in Bing Webmaster Tools and submitting your sitemap. Then add IndexNow, a push protocol that notifies Bing and other participating engines the moment a URL changes. Google doesn't use IndexNow, so treat it as an addition to your Google setup, not a replacement.

Add IndexNow to a Next.js Site

Generate a key and save it as a text file in public/ (for example public/your-key.txt , containing only the key). Then add a helper:

// lib/indexnow.ts
const HOST = 'yoursite.com'
const KEY = process.env.INDEXNOW_KEY!

export async function submitToIndexNow(urls: string[]) {
  if (!urls.length) return

  const res = await fetch('https://api.indexnow.org/indexnow', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json; charset=utf-8' },
    body: JSON.stringify({
      host: HOST,
      key: KEY,
      keyLocation: `https://${HOST}/${KEY}.txt`,
      urlList: urls,
    }),
  })

  if (!res.ok) {
    console.error('IndexNow submission failed', res.status)
  }
}

Call it wherever you publish or update content, right after revalidatePath . Submit both the post URL and the blog index URL, so the new link is discovered from both places.

llms.txt and Markdown: Optional Extras, Not Foundations

What the Evidence Says About llms.txt

Many guides still present llms.txt as essential. The evidence is more nuanced:

• Google: Google Search does not use llms.txt, and Google says it's fine to maintain one for other services that do.
• Adoption: an Ahrefs analysis of 137,000 sites found that 97 percent of llms.txt files were never fetched.
• Agents: in the Vercel and Ora study, agents reached llms.txt mostly through links (86 percent of fetches) rather than by guessing the path. In 36 percent of the runs that reached it, the agent's final answer drew on its content.

So llms.txt can help agents and coding tools once they find it, but it does nothing for Google, and it only works if it's linked. If you add one, keep it a short, described index of your key pages, and link it from your footer. Do this only after your HTML is solid.

Serve Markdown Through Content Negotiation

In the Vercel and Ora study harnesses, 65.1 percent of agent web fetches requested Markdown. A server that supports content negotiation can return clean Markdown from the same URL that serves HTML to browsers. This is better than publishing separate .md URLs, which create duplicate pages that crawlers may fetch twice.

In Next.js, a header-based rewrite handles it:

// next.config.ts
import type { NextConfig } from 'next'

const nextConfig: NextConfig = {
  async rewrites() {
    return {
      beforeFiles: [
        {
          source: '/blogs/:slug',
          has: [{ type: 'header', key: 'accept', value: '(.*)text/markdown(.*)' }],
          destination: '/md/blogs/:slug',
        },
      ],
    }
  },
}

export default nextConfig

// app/md/blogs/[slug]/route.ts
import { getPost } from '@/lib/posts'

export async function GET(
  _req: Request,
  { params }: { params: Promise<{ slug: string }> }
) {
  const { slug } = await params
  const post = await getPost(slug)
  const body = `# ${post.title}\n\n${post.excerpt}\n\n${post.markdown}`

  return new Response(body, {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      Vary: 'Accept',
    },
  })
}

Three details to get right:

• The Vary: Accept header stops your CDN from serving cached Markdown to browsers.
• Add /md/ to your disallowed paths in robots.txt, so the internal route isn't crawled directly.
• If your posts are stored as HTML, convert them with a library such as turndown .

Serving Markdown won't make ChatGPT cite you on its own. It makes your content cheaper and cleaner for agents to read once they arrive.

Next.js AI Search Visibility Checklist

• Content pages are fully static, enforced with ensureStatic , and not marked as Partial Prerender in the build output.
• New posts trigger a rebuild or revalidatePath , and are tested as a crawler after publishing.
• Any htmlLimitedBots override is tested on your exact Next.js version, with no 500 errors and no missing metadata for AI user agents.
• Blog indexes, category pages and navigation links are present in the raw HTML.
• No key content sits behind click-to-load accordions, tabs, canvas-only headlines or code screenshots.
• robots.txt allows OAI-SearchBot, Claude-SearchBot and PerplexityBot, and repeats private paths in every named group.
• Training crawlers are allowed or blocked as a deliberate business decision, optionally stated with Content Signals.
• The Vercel AI bots ruleset and any WAF "block AI" rules don't block AI search crawlers.
• Missing pages return 404, and rate limits return 429 with Retry-After.
• curl tests as each AI user agent return 200 with your title, content and JSON-LD.
• Server logs show requests from OpenAI's and Anthropic's published IP ranges reaching your pages.
• BlogPosting, BreadcrumbList and Person or Organization JSON-LD render on the server, and key facts also appear in visible text.
• Each section opens with a direct answer under a clear, question-shaped heading.
• The author and updated date are visible and match your schema.
• The site is verified in Bing Webmaster Tools, and IndexNow runs on publish.
• The Search generative AI setting is on, and the Generative AI report is reviewed alongside the main Performance report.
• llms.txt and Markdown negotiation are added only after everything above is in place.

Frequently Asked Questions

Is Next.js Good for AI Search?

Yes, when it's configured carefully. Server Components deliver complete HTML by default, which is exactly what AI crawlers need. The risks come from client-side fetching, content behind interactions, and Partial Prerendering responses that AI crawlers don't receive in full.

Does Cache Components Hurt AI Search Visibility?

Not if your content pages are fully static. Prerendered pages send complete HTML to every client. The risk is on partially prerendered routes, where AI crawlers outside the HTML-limited bot list receive a streaming shell. Use ensureStatic on content routes to guarantee they stay static.

Should I Add GPTBot to htmlLimitedBots?

Only for pages that must render dynamically, and only after testing. A reported issue shows custom entries can strip metadata or cause 500 errors on non-prerendered PPR routes in some versions. Making content pages fully static is the safer fix.

Does Blocking GPTBot Remove My Site From ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, which is controlled separately. You can block GPTBot and still appear in ChatGPT search, as long as OAI-SearchBot is allowed in robots.txt and through your firewall.

Can ChatGPT Read Content Loaded With JavaScript?

Not through its crawlers. GPTBot, OAI-SearchBot and ChatGPT-User fetch raw HTML without running scripts. Browser-based agents can see rendered pages during a user's session, but the crawlers that build ChatGPT's search index can't, so server-render anything you want cited.

Does llms.txt Help Rankings in Google AI Mode?

No. Google's official guidance says Google Search doesn't use llms.txt. It may help some AI agents that find it through a link, so treat it as an optional extra once your HTML is solid.

How Do I Check if ChatGPT Can See My Site?

Fetch your pages with curl using the GPTBot and OAI-SearchBot user agents. Confirm a 200 status, and check that your content and JSON-LD appear in the response. Then confirm in your server logs that requests from OpenAI's published IP ranges reach those pages. Finally, ask ChatGPT and Perplexity questions your pages answer, and see whether they cite you.

Will AI Assistants Cite My Pages as Soon as I Fix These Issues?

There's no fixed timeline. OpenAI alone notes that robots.txt changes take around 24 hours to apply, crawlers have to revisit your pages, and some assistants answer from cached data. Sitemaps and IndexNow speed up indexing. Citations still depend on whether your content is the clearest answer available.

Make Your Next.js Site Readable to Every Engine

AI search doesn't replace SEO; it raises the standard for technical SEO. The sites that get cited serve complete HTML to every crawler, control access deliberately, describe themselves accurately, and answer questions directly.

Most of the fixes in this guide are small config changes. The hard part is knowing where to look, because none of these problems show up in a browser.

If you want to know exactly what GPTBot, ClaudeBot and PerplexityBot see on your site, start with a free website audit . I'll test your pages as each AI crawler and show you what to fix first. For a full Next.js build or migration with AI search visibility built in from the start, get in touch .