Pagination vs Infinite Scroll for AI Search: What Crawlers Actually See

Infinite scroll hides the majority of your archive content from AI crawlers because those crawlers do not execute JavaScript, do not scroll, and therefore only see the first batch of items rendered in the initial HTML response.

That first sentence matters because most teams assume the pagination problem is an SEO problem from the last decade. It is not. The same architectural issue that made infinite scroll risky for Googlebot in 2015 is even more consequential today, because the crawlers feeding ChatGPT, Claude, and Perplexity are less capable of handling dynamic content than Googlebot was then.

What the crawlers actually do

In December 2024, Vercel and MERJ published a study analyzing crawler behavior across Vercel’s network. The finding on JavaScript rendering was unambiguous: none of the major AI crawlers currently render JavaScript. This includes GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot. ChatGPT and Claude crawlers fetch JavaScript files as text (ChatGPT at 11.5% of requests, Claude at 23.84%), but they do not execute them.

The exceptions are Gemini (which uses Google’s own infrastructure and inherits Googlebot’s full rendering pipeline) and AppleBot (which uses a browser-based crawler with full JavaScript execution). Every other major AI crawler is operating on raw HTML.

Anthropic and Perplexity do not publish explicit documentation on their crawlers’ JavaScript rendering behavior, but the Vercel/MERJ data covers real traffic patterns across two different technology stacks (Next.js and a custom monolithic framework), making the finding robust. Google’s Gemini is a documented exception because Google publishes that Gemini uses Googlebot infrastructure.

Why infinite scroll specifically fails

Infinite scroll relies on two browser capabilities that AI crawlers do not use: the IntersectionObserver API (or scroll event) to detect when the user reaches the bottom of the page, and a fetch or XHR call to retrieve the next batch. Without a browser executing that JavaScript, the crawler receives only the first 10 to 20 items in the initial HTML. Everything below that is unreachable.

This is directly relevant to our content chunking framework: content not present in the initial HTML response is not chunked, not embedded, and not retrievable. An article buried on page 4 of an infinite-scroll archive has no path to a citation. Even if it appears in your XML sitemap, the sitemap only tells the crawler the URL exists. The archive page itself, which is often the highest-authority entry point for category content, provides no link to that article for a crawler that cannot scroll.

Pagination patterns that work

Three patterns provide full crawl coverage for all crawlers including non-JS ones:

• Numbered pagination with real hrefs (/blog/page/2/, /blog/page/3/): every page is a real URL with server-rendered content and a real anchor tag pointing to the next page. This is the simplest and most universally supported pattern.
• Cursor-based pagination (/blog?after=post-id): functional for AI crawlers if the link is rendered server-side as an <a href>. Slightly harder for users to bookmark. Preferred by some APIs.
• Load-more button with URL update: acceptable if the URL changes when new content loads (pushState) and that URL is server-renderable. The button itself must be a real <a href>, not a <button> with a click handler, so crawlers can follow it without executing JavaScript.

The common requirement across all three: every paginated state has a unique URL that returns full server-rendered content when fetched directly. JavaScript-only state changes break this requirement entirely.

Progressive enhancement: keeping infinite scroll for users

If your team has good UX reasons for infinite scroll, the correct approach is to build pagination first and layer the infinite scroll experience on top. The pattern works like this:

1. The server renders /blog/page/1/ with the first batch of items and a real <a href="https://organikpi.com/blog/page/2/"> link to the next page.
2. On the client, JavaScript intercepts that link and uses a fetch to load the next batch, appending it to the current view.
3. The browser URL updates via history.pushState to /blog/page/2/.
4. Direct navigation to /blog/page/2/ still works server-side and returns full content.

This pattern, sometimes called pjax or Turbo (in Rails and related frameworks), gives users the seamless scroll experience while keeping every page addressable and crawlable. Frameworks like Next.js with server components, Hotwire, and Astro support this architecture without custom wiring. The key principle is that the client-side enhancement is strictly additive: remove the JavaScript and the pagination still works.

rel=prev/next: retired by Google, still worth including

Google retired rel=prev/next as an indexing signal in March 2019. John Mueller confirmed the decision plainly: “We don’t use link-rel-next/prev at all.” Google had stopped using it for years before the announcement and removed the documentation when the retirement was made official.

That said, other crawlers including Bing, and potentially some AI crawlers, still parse these hints. They cost nothing to implement and add useful pagination context for any crawler that does use them. Our recommendation: include them as a secondary signal, but do not treat them as a substitute for proper paginated URLs. The real structural requirement is the URL itself.

One canonical mistake to avoid: many SEO plugins canonicalize all paginated pages back to page 1. This actively signals to crawlers that pages 2 through N are duplicates of page 1, suppressing their independent indexation. Each paginated page should self-canonical to its own URL. For example, /blog/page/2/ should have <link rel="canonical" href="https://organikpi.com/blog/page/2/" />, not <link rel="canonical" href="https://organikpi.com/blog/" />.

Testing methodology: checking what crawlers actually see

The fastest way to test what a non-JS crawler sees on your archive pages is to fetch the page with curl using a crawler user agent, then count the items in the response:

# Fetch as GPTBot and count list items
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot"   https://yoursite.com/blog/ | grep -c "<li"

# Fetch as ClaudeBot
curl -s -A "claudebot"   https://yoursite.com/blog/ | grep -c "<li"

# Check if a "next page" link exists in the HTML
curl -s -A "GPTBot/1.3" https://yoursite.com/blog/   | grep -i 'rel.*next\|href.*page.2'

If the item count matches only your first batch (typically 10 to 12) and no “next page” link appears, your archive is invisible beyond that first batch. The second test is to open the page in a browser with JavaScript disabled (Chrome DevTools: Settings, Debugger, Disable JavaScript). If the “Load More” button or auto-fetch stops working and no fallback links appear, crawlers are seeing the same broken experience.

The table: comparing archive patterns for AI crawlers

How far do crawlers follow pagination?

Crawlers do not follow pagination infinitely. Crawl budget limits and diminishing relevance signals mean deeper paginated pages get progressively deprioritized, so you cannot rely on crawlers to reach the bottom of a long archive. The practical implication: your most recent and most linked content should appear on the early pages. Older archive content depends on the XML sitemap and on internal links from high-authority pages, not from deep pagination alone.

For large archives, we recommend combining paginated URLs with a robust internal linking strategy that creates direct paths to evergreen content without requiring a crawler to paginate 50 pages deep. Category landing pages, best-of lists, and related-posts widgets all serve this function.

Headless and hybrid rendering frameworks

If you are building on a headless CMS or a modern JavaScript framework, the question is which rendering strategy you are using. Headless CMS setups that rely entirely on client-side rendering for archive pages will have the same problem as plain infinite scroll: the first batch is all that crawlers see. Next.js with App Router and server components, Nuxt with SSR mode, and Astro (which is static by default) all generate server-rendered paginated HTML correctly when configured to do so.

The Shadow DOM and Web Components problem is adjacent: components that render their content inside a shadow root are not visible to crawlers that only parse the light DOM. Archive items built as Web Components need a server-rendered fallback or a server-side rendering step to expose their content to AI crawlers.

What we do for clients

When we run a GEO audit, archive pagination is one of the first things we check using the curl methodology above. The most common finding is a WordPress or Shopify archive that uses a “Load More” button but the button is a <button> element with a JS click handler rather than an <a href> to the next page. Fixing this is usually a 30-minute template change.

The second most common finding is a headless frontend where the pagination API call is entirely client-side. In these cases, we recommend adding a server-rendered static page layer (Next.js generateStaticParams, for example) so that paginated URLs exist as real server-rendered documents, independent of the JavaScript application state.

For canonical and redirect configuration, the rule is simple: each paginated page self-canonicalizes, uses HTTP 200, and is linked from the previous page via a real anchor. Redirecting pages 2+ to page 1 is a critical mistake that actively hides paginated content from indexation.

The semantic HTML of the paginated page also matters. Archive items marked up as <article> elements with <h2> headings and descriptive anchor text give crawlers a structured list of what the page contains. An archive that renders items as nested <div>s with no semantic structure is harder for AI crawlers to parse and chunk correctly.

Finally, our open-source GEO/AEO Tracker can be used to monitor whether archive pages are being cited in AI engine responses over time. A sudden drop in archive citation rate, without a corresponding drop in individual article citations, is a leading indicator that your pagination structure has broken and crawlers have lost access to deeper pages.