How to Audit Any Site With Serpent, Our Free Open-Source SEO Crawler

TL;DR

• Serpent audits any site free, with no URL caps.
• We crawled 200 Smashing Magazine pages live.
• It found 1,502 issues and three broken internal links.
• No page on the site ships structured data.
• Every page gets a 0-100 AI readiness score, and embeddings flag cannibalisation.

Serpent is a free, MIT-licensed desktop crawler that audits technical SEO and AI search readiness in one pass. This guide walks through a real crawl of smashingmagazine.com: 200 URLs, 1,502 issues, an average AI readiness score of 44.9 out of 100, and three internally linked URLs that return a hard 404. Every screenshot below comes from that one crawl.

Full disclosure: we built Serpent, and we run it on this site. That cuts both ways. We know the tool well, and you can verify every claim by running the same crawl yourself. It is open source, it has no URL caps, and the download is free.

What Serpent is

Serpent is a desktop app built on Electron, React, and SQLite. Crawl data stays in a local database on your machine, API keys live in the OS keychain, and there is no telemetry. The crawler respects robots.txt by default, and there is no license server to phone home to.

Grab an installer from the GitHub releases page, which ships a Windows .exe, macOS .dmg builds for both Apple Silicon and Intel, and a Linux .AppImage. You can also run it from source with Node 20 and three commands. Windows SmartScreen warns on first launch because builds are unsigned while a code signing application is pending: click More info, then Run anyway. Every release ships SHA256 checksums and GitHub build provenance, so you can confirm the binary came from the repo before running it.

git clone https://github.com/danishashko/serpent.git
cd serpent
npm install
npm run dev

Set up the crawl

Paste a seed URL, pick an engine, set the limits. For this walkthrough we pointed Serpent at smashingmagazine.com with a 200 URL cap, depth 3, and a rate limit of 2 requests per second. The cap is manners: it is someone else’s production site, and two pages per second never reads as an attack.

The Local engine is free and covers most sites. Two more exist for harder targets: a JS Browser engine that renders JavaScript-heavy pages with Electron’s built-in Chromium at roughly 1 to 2 seconds per page, and a Web Unlocker engine that routes requests through Bright Data for bot-protected sites. That last one spends real money, so Serpent tracks spend per crawl and stops at a hard cost limit you set.

One setting worth knowing: the crawl User-Agent. Serpent identifies as Serpent by default, and you can crawl as Googlebot or Bingbot instead. The robots.txt rules follow the agent you pick, so a Googlebot crawl obeys Googlebot’s rules. How your own host treats crawlers deserves the same attention. Ours was returning 429s to GPTBot without telling us, and Cloudflare now offers per-bot control over AI crawlers.

Keep it focused with exclude patterns

Our first attempt at this crawl wasted most of its budget. Smashing Magazine hosts its desktop wallpaper archive under /files/, and 114 of the 200 slots went to PNG downloads instead of pages. Four regex lines in the advanced panel fixed it: /files/, /author/, /category/, and one that matches asset file extensions. Every image is still inventoried in the Images tab, so nothing is lost by keeping them out of the page queue.

The same panel holds URL parameter stripping for utm tags, HTTP Basic auth for staging environments, custom headers, and session cookies for crawls behind a login.

Run it

At 2 requests per second, 200 URLs takes a little under two minutes of fetching. Pages stream into the table as they land, and pause, resume, and stop all work mid-crawl. Our run reported an average response of 494 milliseconds when it finished.

Read the results

The Pages tab is the raw material: status code, title and its pixel width, meta description, H1, word count, page size, crawl depth, link score, GEO and performance scores, an accessibility score, and an indexability verdict for every URL. Three things stood out in this crawl.

• The homepage has no text H1. Its only h1 element wraps the logo image, so extraction comes back empty. Crawlers and screen readers see the same nothing. The paginated article archives have no h1 element at all.
• 34 of the 200 crawled URLs carry noindex. For a magazine most of that is deliberate archive hygiene, but a number that size is always worth eyeballing on your own site.
• Not one page on the site ships JSON-LD. No crawled page carried structured data of any kind, which the GEO tab later scores at zero.

Issues, grouped and ranked

The Issues tab groups every finding into eleven categories and rates each one critical, warning, info, or opportunity. One of those categories is accessibility: every crawl runs a static WCAG pass over the same parsed HTML, scoring each page 0 to 100 across 14 rules that use axe-core’s rule names, with no second request and no separate tool. The biggest block here is security: 170 pages are served without a Content-Security-Policy header.

The critical row is the interesting one. Three URLs return a 4xx, and two of them are the same bug twice. On the CSS generators article from 2021 an image link was written with a quote inside the href, so the browser resolves it against the page path and lands on a 404. The third is a June 2025 article on optimizing SVGs that is gone, still linked from an August 2026 post under the anchor “optimizing and structuring”. We opened all three in a browser to confirm.

The other tables

Four more tables sit beside Pages, and each answers something Pages cannot.

• Links holds all 27,680 links found, 19,071 internal and 8,609 external, with anchor text, rel attributes and crawlability. When a crawl finishes, Serpent checks the status of every unique external link in the background, so rot in your outbound links surfaces without a second tool.
• Images lists 4,010 assets with alt text and size. 170 of them have no alt attribute.
• Redirects reconstructs the chains, 28 of them here, with a hop count per chain. The pattern is easy to read: the site’s category URLs are linked as /categories/ and every one of them 301s to /category/. That is a single find and replace in a template, and it recovers a hop on 29 internal links.
• Duplicates runs exact and near-duplicate matching on every crawl. On your own site that check matters for a second reason: two pages covering the same ground split your AI citations.

A Hreflang tab sits alongside them and stays empty on a monolingual site like this one.

Pull any field off the page

Custom extraction rules take a name and a CSS selector, and every crawled page gets scraped for them. We asked for three: author, published date, and the article section from Open Graph. That produced 519 values across the crawl, and one small finding fell out of it. Every page that carries the tag reports its section as “General”, all 173 of them, so the site’s own category metadata carries no signal at all.

Selectors read the element text and fall back to a content attribute, so meta tags work as targets alongside ordinary markup. Prices, bylines, product codes, review counts: anything with a stable selector comes back as a column you can export.

Compare the crawl against the sitemap

The Sitemap tab fetches a sitemap or a sitemap index, expands child sitemaps, handles .xml.gz, and diffs the result against what you crawled. Smashing Magazine’s sitemap lists 7,879 URLs. Every URL we crawled appears in it, which is the answer you want. Two things do not look right: seven URLs are listed more than once, and one is non-indexable but still submitted.

Read the orphan count with the crawl cap in mind. It reports every sitemap URL your crawl did not reach, so on a 200 URL sample of a 7,879 URL site the number is large by construction. It only means something once your crawl covers the site.

The same tab generates a sitemap from the crawl, built only from same-origin URLs that returned 200 and are actually indexable, splitting into a sitemap index past 50,000 URLs. Useful when a CMS emits a sitemap that disagrees with reality.

Score the speed without a second crawl

The Perf tab scores every page from the response Serpent already has: time to first byte, page weight, image optimisation, and content efficiency. No new requests, so a 200 page site scores in seconds. This crawl averages 81.1 out of 100, with 192 pages good and 8 needing work.

Image optimisation is the weak axis at 65, and the worst pages are the wallpaper posts, which is what you would expect from a page whose job is to serve large images. Time to first byte averaged 542 milliseconds across the crawl.

Static scores are a triage tool, not a lab measurement. For real user data the same tab pulls Core Web Vitals from the PageSpeed Insights API, which works without a key at low volume and needs a free key past five URLs per run.

Score it for AI search

Classic crawlers score a page for Google’s index. Serpent’s GEO tab also scores every crawled page from 0 to 100 for answer engines, across four axes: entity clarity, answer readiness, citation signals, and structured data completeness. The scoring is static analysis. It needs no API key and sends no extra requests.

Smashing Magazine averages 44.9 out of 100, and no page in the crawl clears 70. The best reaches 56. Entity clarity sits at 60, answer readiness at 57, citation signals at 61, and structured data completeness at zero, because the site ships no JSON-LD at all. We hand-checked the homepage and two articles to make sure that zero was the site and not the tool.

The table sorts worst first, which puts the redirect URLs at the top on a straight row of zeros. They have no content to score, so read past them: 170 of the 200 land between 40 and 69, and 30 sit below 40.

Read that last axis carefully. Nothing in our own citation data says JSON-LD alone earns AI citations, so treat it as a machine-readability check, not a citation lever. Where markup still earns its keep is entity and author signals. For the page-by-page fixes, our 50-point AI search readiness checklist pairs well with this tab: sort by overall score, work the worst pages first.

The llms.txt check

The same tab validates llms.txt against the llmstxt.org proposal: whether the file exists, whether the structure holds (an H1, a blockquote summary, H2 link sections), whether llms-full.txt sits alongside it, and whether the links inside it were reachable in your crawl. Smashing Magazine returns a 404 for it, which the checker reports with a suggested fix.

Our position on the file itself is lukewarm, and the llms.txt adoption data explains why: adoption is thin and evidence of impact is thinner. But if you publish one, publish a valid one. A link an LLM cannot fetch is worse than no file.

Find the pages that compete with each other

Exact and near-duplicate hashing catches pages that share wording. It cannot catch two pages that cover the same subject in completely different words, which is the version of cannibalisation that actually hurts. The Semantic tab handles that with embeddings: pick a provider, embed the stored page text, and compare meaning instead of strings. Gemini, OpenAI, and a local Ollama model are all supported, and vectors are normalised before storage so similarity is a plain dot product.

We embedded 170 pages with Gemini. At a 0.92 threshold the tab flagged 58 semantically similar pages and 41 low relevance outliers, pages sitting far from what the rest of the site is about. The clearest pair was commercial rather than editorial: the printed book and the ebook for the same title score 97% against each other, two URLs competing to be the answer to one question.

The third view is semantic search over your own crawl. Type a question and get the pages that answer it, ranked by meaning, which is a fair approximation of how an answer engine picks a page to cite.

The treemap

The Map tab draws the whole crawl as a treemap. Tile size is link score, Serpent’s internal PageRank-style equity metric, and color is issue severity. You read it in five seconds: the biggest tiles are where fixes pay off first.

Test robots.txt before you ship it

Serpent carries its own robots.txt parser, and the settings panel exposes it as a tester. Paste a robots body, give it a URL and a user agent, and it tells you whether the URL is allowed and which line decided it. It implements the parts of Google’s spec that cause real accidents: user-agent grouping, longest-match precedence with Allow winning ties, wildcards, and end anchors.

The same field overrides the live file for a crawl, so you can test a rule change against your own site without deploying it first. That is the cheap way to find out that a Disallow you were about to ship also blocks a template you care about.

Schedule it, then read the diff

A crawl you run once is a snapshot. Schedules re-run a saved crawl on an interval, and with auto-compare on, each run is diffed against the previous one. What you read afterwards is a list of what changed: pages added, pages gone, and fields that moved. On a site that ships weekly that is a two-minute read instead of a full audit.

Any two crawls can also be compared by hand from the Compare button. Exports cover the same ground in every direction: CSV and JSON per tab, a bulk export of everything at once, a per-URL export, and a PDF report with selectable sections and your own brand color.

The parts that cost money

Everything above ran free, on one machine, with no account. Four features need something extra, and all four are off unless you turn them on.

• SERP checks live rankings for a keyword list through the Bright Data SERP API at $0.003 per query.
• Competitors and Gaps use the same account’s Discover API to find who else ranks for your topics and which subjects they cover that you do not.
• AI Insights sends a page to a model you choose for content and technical review. Ollama runs it locally for nothing; OpenAI, Anthropic, Gemini, and OpenRouter all cost per call.
• Search Console connects with your own Google OAuth credentials and brings impressions and clicks next to the crawl, which is also what makes orphan page detection work.

Every paid path is metered. Serpent tracks spend per crawl and per day, and stops at the limit you set.

Drive it from Claude

While the app is open, Serpent runs a Model Context Protocol server at http://127.0.0.1:7777/mcp. It accepts connections from localhost only and is never exposed to the network. Claude Desktop, Claude Code, or any MCP client can drive it through eleven tools, from start_crawl and get_crawl_status to get_issues and export_csv.

{
  "mcpServers": {
    "serpent": {
      "type": "http",
      "url": "http://127.0.0.1:7777/mcp"
    }
  }
}

Add that to claude_desktop_config.json while Serpent is running, restart Claude, and ask in plain language for a crawl and its critical issues. We exercised the server against the crawl above: get_issues returns the three 404s, the missing H1s, and the title length warnings as structured JSON an agent can act on. If you already track your brand with AI visibility tools, this closes the loop: the assistant that reads your visibility numbers can now run the technical audit too.

What it does not do

• JavaScript rendering is opt-in and costs 1 to 2 seconds per page, so JS-heavy sites crawl slower.
• The accessibility pass is static. Color contrast needs computed styles, so a browser-based checker still owns that one.
• Builds are unsigned while the SignPath Foundation application is pending, so Windows and macOS each warn once at first launch. The Linux AppImage needs libsecret installed for the keychain integration.
• There is no log file analysis.
• Semantic analysis needs an embedding provider. Ollama makes it free, but it is still a setup step.

None of these blocked an audit for us.

A 15-minute starter workflow

1. Install Serpent and crawl your own site, capped at 500 URLs with a rate limit of 2. Exclude your asset directories first.
2. Fix the criticals first: 404s and server errors, then any noindex that should not be there.
3. Open Redirects and look for a repeated pattern. One template fix usually clears a whole family.
4. Run the GEO analysis, sort by overall score, and rewrite the three worst pages so each answers its core question in the first paragraph.
5. Embed the crawl and read the semantically similar list. Merge or differentiate anything above 0.95.
6. Schedule a weekly recrawl with auto-compare and read the diff instead of the full report. It slots straight into the rest of your SEO automation.

Everything above ran on the free local engine, with one Gemini key for the semantic pass and nothing else. The code lives at github.com/danishashko/serpent, issues and pull requests are open, and Serpent’s tool page tracks releases. For the paid half of a GEO stack, the GEO tools we tested cover tracking and citation monitoring.