# How to Audit Any Site With Serpent, Our Free Open-Source SEO Crawler **URL:** https://organikpi.com/blog/technical-seo/serpent-seo-crawler-guide/ **Published:** 2026-09-09 **Modified:** 2026-10-02 **Author:** Daniel Shashko > Serpent audits any site free, with no URL caps. We crawled 200 Smashing Magazine pages live. It found 1,502 issues and three broken internal links. No page on the site ships structured data. Every page gets a 0-100 AI readiness score, and embeddings flag cannibalisation. --- > Serpent audits any site free, with no URL caps. We crawled 200 Smashing Magazine pages live. It found 1,502 issues and three broken internal links. No page on the site ships structured data. Every page gets a 0-100 AI readiness score, and embeddings flag cannibalisation. Serpent is a free, MIT-licensed desktop crawler that audits technical SEO and AI search readiness in one pass. This guide walks through a real crawl of smashingmagazine.com: 200 URLs, 1,502 issues, an average AI readiness score of 44.9 out of 100, and three internally linked URLs that return a hard 404. Every screenshot below comes from that one crawl. Full disclosure: we built Serpent, and we run it on this site. That cuts both ways. We know the tool well, and you can verify every claim by running the same crawl yourself. It is open source, it has no URL caps, and the download is free. ## What Serpent is Serpent is a desktop app built on Electron, React, and SQLite. Crawl data stays in a local database on your machine, API keys live in the OS keychain, and there is no telemetry. The crawler respects robots.txt by default, and there is no license server to phone home to. Grab an installer from the [GitHub releases page](https://github.com/danishashko/serpent/releases), which ships a Windows .exe, macOS .dmg builds for both Apple Silicon and Intel, and a Linux .AppImage. You can also run it from source with Node 20 and three commands. Windows SmartScreen warns on first launch because builds are unsigned while a code signing application is pending: click More info, then Run anyway. Every release ships SHA256 checksums and GitHub build provenance, so you can confirm the binary came from the repo before running it. git clone https://github.com/danishashko/serpent.git cd serpent npm install npm run dev ## Set up the crawl Paste a seed URL, pick an engine, set the limits. For this walkthrough we pointed Serpent at smashingmagazine.com with a 200 URL cap, depth 3, and a rate limit of 2 requests per second. The cap is manners: it is someone else’s production site, and two pages per second never reads as an attack. The crawl form. Seed URL, engine and limits on the left, Start Crawl at the bottom. The Local engine is free and covers most sites. Two more exist for harder targets: a JS Browser engine that renders JavaScript-heavy pages with Electron’s built-in Chromium at roughly 1 to 2 seconds per page, and a Web Unlocker engine that routes requests through Bright Data for bot-protected sites. That last one spends real money, so Serpent tracks spend per crawl and stops at a hard cost limit you set. One setting worth knowing: the crawl User-Agent. Serpent identifies as Serpent by default, and you can crawl as Googlebot or Bingbot instead. The robots.txt rules follow the agent you pick, so a Googlebot crawl obeys Googlebot’s rules. How your own host treats crawlers deserves the same attention. Ours was [returning 429s to GPTBot](https://organikpi.com/blog/technical-seo/gptbot-429-host-block/) without telling us, and Cloudflare now offers [per-bot control over AI crawlers](https://organikpi.com/blog/technical-seo/cloudflare-ai-crawl-control-guide/). ### Keep it focused with exclude patterns Our first attempt at this crawl wasted most of its budget. Smashing Magazine hosts its desktop wallpaper archive under /files/, and 114 of the 200 slots went to PNG downloads instead of pages. Four regex lines in the advanced panel fixed it: /files/, /author/, /category/, and one that matches asset file extensions. Every image is still inventoried in the Images tab, so nothing is lost by keeping them out of the page queue. Exclude patterns, one regex per line. The same panel holds User-Agent, auth and custom headers. The same panel holds URL parameter stripping for utm tags, HTTP Basic auth for staging environments, custom headers, and session cookies for crawls behind a login. ### Run it At 2 requests per second, 200 URLs takes a little under two minutes of fetching. Pages stream into the table as they land, and pause, resume, and stop all work mid-crawl. Our run reported an average response of 494 milliseconds when it finished. 58 of 200 URLs in, at 2 per second. The H1 column already shows MISSING on the homepage. ## Read the results The Pages tab is the raw material: status code, title and its pixel width, meta description, H1, word count, page size, crawl depth, link score, GEO and performance scores, an accessibility score, and an indexability verdict for every URL. Three things stood out in this crawl. - **The homepage has no text H1.** Its only h1 element wraps the logo image, so extraction comes back empty. Crawlers and screen readers see the same nothing. The paginated article archives have no h1 element at all. - **34 of the 200 crawled URLs carry noindex.** For a magazine most of that is deliberate archive hygiene, but a number that size is always worth eyeballing on your own site. - **Not one page on the site ships JSON-LD.** No crawled page carried structured data of any kind, which the GEO tab later scores at zero. 200 URLs. Every measured column sorts, and the filter box narrows the table. The homepage H1 reads MISSING in the first row. ### Issues, grouped and ranked The Issues tab groups every finding into eleven categories and rates each one critical, warning, info, or opportunity. One of those categories is accessibility: every crawl runs a static WCAG pass over the same parsed HTML, scoring each page 0 to 100 across 14 rules that use axe-core’s rule names, with no second request and no separate tool. The biggest block here is security: 170 pages are served without a Content-Security-Policy header. The critical row is the interesting one. Three URLs return a 4xx, and two of them are the same bug twice. On the CSS generators article from 2021 an image link was written with a quote inside the href, so the browser resolves it against the page path and lands on a 404. The third is a June 2025 article on optimizing SVGs that is gone, still linked from an August 2026 post under the anchor “optimizing and structuring”. We opened all three in a browser to confirm. Category, issue, then affected URLs. The three 4xx URLs are the critical row. ### The other tables Four more tables sit beside Pages, and each answers something Pages cannot. Every link with its anchor text, type and crawlability. The totals sit in the footer. - **Links** holds all 27,680 links found, 19,071 internal and 8,609 external, with anchor text, rel attributes and crawlability. When a crawl finishes, Serpent checks the status of every unique external link in the background, so rot in your outbound links surfaces without a second tool. - **Images** lists 4,010 assets with alt text and size. 170 of them have no alt attribute. - **Redirects** reconstructs the chains, 28 of them here, with a hop count per chain. The pattern is easy to read: the site’s category URLs are linked as /categories/ and every one of them 301s to /category/. That is a single find and replace in a template, and it recovers a hop on 29 internal links. - **Duplicates** runs exact and near-duplicate matching on every crawl. On your own site that check matters for a second reason: two pages covering the same ground [split your AI citations](https://organikpi.com/blog/seo-strategy/keyword-cannibalization-ai-citations/). Every /categories/ URL 301s to /category/. One template fix clears the whole family. A Hreflang tab sits alongside them and stays empty on a monolingual site like this one. ### Pull any field off the page Custom extraction rules take a name and a CSS selector, and every crawled page gets scraped for them. We asked for three: author, published date, and the article section from Open Graph. That produced 519 values across the crawl, and one small finding fell out of it. Every page that carries the tag reports its section as “General”, all 173 of them, so the site’s own category metadata carries no signal at all. Three rules, 519 values. The section column reads General on every page. Selectors read the element text and fall back to a content attribute, so meta tags work as targets alongside ordinary markup. Prices, bylines, product codes, review counts: anything with a stable selector comes back as a column you can export. ## Compare the crawl against the sitemap The Sitemap tab fetches a sitemap or a sitemap index, expands child sitemaps, handles .xml.gz, and diffs the result against what you crawled. Smashing Magazine’s sitemap lists 7,879 URLs. Every URL we crawled appears in it, which is the answer you want. Two things do not look right: seven URLs are listed more than once, and one is non-indexable but still submitted. Read the orphan count with the crawl cap in mind. It reports every sitemap URL your crawl did not reach, so on a 200 URL sample of a 7,879 URL site the number is large by construction. It only means something once your crawl covers the site. The same tab generates a sitemap from the crawl, built only from same-origin URLs that returned 200 and are actually indexable, splitting into a sitemap index past 50,000 URLs. Useful when a CMS emits a sitemap that disagrees with reality. The sitemap diff, and the generator that builds one from the crawl. ## Score the speed without a second crawl The Perf tab scores every page from the response Serpent already has: time to first byte, page weight, image optimisation, and content efficiency. No new requests, so a 200 page site scores in seconds. This crawl averages 81.1 out of 100, with 192 pages good and 8 needing work. Image optimisation is the weak axis at 65, and the worst pages are the wallpaper posts, which is what you would expect from a page whose job is to serve large images. Time to first byte averaged 542 milliseconds across the crawl. Static scores are a triage tool, not a lab measurement. For real user data the same tab pulls Core Web Vitals from the PageSpeed Insights API, which works without a key at low volume and needs a free key past five URLs per run. Four axes per page, scored from the response already in the database. ## Score it for AI search Classic crawlers score a page for Google’s index. Serpent’s GEO tab also scores every crawled page from 0 to 100 for [answer engines](https://organikpi.com/blog/geo-ai-search/what-is-aeo-answer-engine-optimization/), across four axes: entity clarity, answer readiness, citation signals, and structured data completeness. The scoring is static analysis. It needs no API key and sends no extra requests. Smashing Magazine averages 44.9 out of 100, and no page in the crawl clears 70. The best reaches 56. Entity clarity sits at 60, answer readiness at 57, citation signals at 61, and structured data completeness at zero, because the site ships no JSON-LD at all. We hand-checked the homepage and two articles to make sure that zero was the site and not the tool. The table sorts worst first, which puts the redirect URLs at the top on a straight row of zeros. They have no content to score, so read past them: 170 of the 200 land between 40 and 69, and 30 sit below 40. Read that last axis carefully. Nothing in our own [citation data](https://organikpi.com/blog/geo-ai-search/state-of-ai-search/) says JSON-LD alone earns AI citations, so treat it as a machine-readability check, not a citation lever. Where markup still earns its keep is [entity and author signals](https://organikpi.com/blog/brand-authority/person-schema-author-eeat/). For the page-by-page fixes, our [50-point AI search readiness checklist](https://organikpi.com/blog/geo-ai-search/geo-audit-checklist/) pairs well with this tab: sort by overall score, work the worst pages first. Four axes per page, worst first. The zero rows are redirect URLs; structured data is zero site-wide. ### The llms.txt check The same tab validates llms.txt against the [llmstxt.org](https://llmstxt.org/) proposal: whether the file exists, whether the structure holds (an H1, a blockquote summary, H2 link sections), whether llms-full.txt sits alongside it, and whether the links inside it were reachable in your crawl. Smashing Magazine returns a 404 for it, which the checker reports with a suggested fix. Our position on the file itself is lukewarm, and the [llms.txt adoption data](https://organikpi.com/blog/distribution/llms-txt-adoption-impact/) explains why: adoption is thin and evidence of impact is thinner. But if you publish one, publish a valid one. A link an LLM cannot fetch is worse than no file. The llms.txt validator reports a 404, with the structure a valid file needs. ### Find the pages that compete with each other Exact and near-duplicate hashing catches pages that share wording. It cannot catch two pages that cover the same subject in completely different words, which is the version of cannibalisation that actually hurts. The Semantic tab handles that with embeddings: pick a provider, embed the stored page text, and compare meaning instead of strings. Gemini, OpenAI, and a local Ollama model are all supported, and vectors are normalised before storage so similarity is a plain dot product. We embedded 170 pages with Gemini. At a 0.92 threshold the tab flagged 58 semantically similar pages and 41 low relevance outliers, pages sitting far from what the rest of the site is about. The clearest pair was commercial rather than editorial: the printed book and the ebook for the same title score 97% against each other, two URLs competing to be the answer to one question. Pages ranked by meaning. The printed book and the ebook for one title score 97% against each other. The third view is semantic search over your own crawl. Type a question and get the pages that answer it, ranked by meaning, which is a fair approximation of how an answer engine picks a page to cite. ## The treemap The Map tab draws the whole crawl as a treemap. Tile size is link score, Serpent’s internal PageRank-style equity metric, and color is issue severity. You read it in five seconds: the biggest tiles are where fixes pay off first. The crawl as a treemap. Tile size is link score, color is issue severity. ## Test robots.txt before you ship it Serpent carries its own robots.txt parser, and the settings panel exposes it as a tester. Paste a robots body, give it a URL and a user agent, and it tells you whether the URL is allowed and which line decided it. It implements the parts of Google’s spec that cause real accidents: user-agent grouping, longest-match precedence with Allow winning ties, wildcards, and end anchors. The same field overrides the live file for a crawl, so you can test a rule change against your own site without deploying it first. That is the cheap way to find out that a Disallow you were about to ship also blocks a template you care about. ## Schedule it, then read the diff A crawl you run once is a snapshot. Schedules re-run a saved crawl on an interval, and with auto-compare on, each run is diffed against the previous one. What you read afterwards is a list of what changed: pages added, pages gone, and fields that moved. On a site that ships weekly that is a two-minute read instead of a full audit. Any two crawls can also be compared by hand from the Compare button. Exports cover the same ground in every direction: CSV and JSON per tab, a bulk export of everything at once, a per-URL export, and a PDF report with selectable sections and your own brand color. ## The parts that cost money Everything above ran free, on one machine, with no account. Four features need something extra, and all four are off unless you turn them on. - **SERP** checks live rankings for a keyword list through the Bright Data SERP API at $0.003 per query. - **Competitors and Gaps** use the same account’s Discover API to find who else ranks for your topics and which subjects they cover that you do not. - **AI Insights** sends a page to a model you choose for content and technical review. Ollama runs it locally for nothing; OpenAI, Anthropic, Gemini, and OpenRouter all cost per call. - **Search Console** connects with your own Google OAuth credentials and brings impressions and clicks next to the crawl, which is also what makes orphan page detection work. Every paid path is metered. Serpent tracks spend per crawl and per day, and stops at the limit you set. ## Drive it from Claude While the app is open, Serpent runs a Model Context Protocol server at http://127.0.0.1:7777/mcp. It accepts connections from localhost only and is never exposed to the network. Claude Desktop, Claude Code, or any MCP client can drive it through eleven tools, from start_crawl and get_crawl_status to get_issues and export_csv. { "mcpServers": { "serpent": { "type": "http", "url": "http://127.0.0.1:7777/mcp" } } } Add that to claude_desktop_config.json while Serpent is running, restart Claude, and ask in plain language for a crawl and its critical issues. We exercised the server against the crawl above: get_issues returns the three 404s, the missing H1s, and the title length warnings as structured JSON an agent can act on. If you already track your brand with [AI visibility tools](https://organikpi.com/blog/geo-ai-search/best-ai-visibility-tools/), this closes the loop: the assistant that reads your visibility numbers can now run the technical audit too. ## What it does not do - JavaScript rendering is opt-in and costs 1 to 2 seconds per page, so JS-heavy sites crawl slower. - The accessibility pass is static. Color contrast needs computed styles, so a browser-based checker still owns that one. - Builds are unsigned while the SignPath Foundation application is pending, so Windows and macOS each warn once at first launch. The Linux AppImage needs libsecret installed for the keychain integration. - There is no log file analysis. - Semantic analysis needs an embedding provider. Ollama makes it free, but it is still a setup step. None of these blocked an audit for us. ## A 15-minute starter workflow - Install Serpent and crawl your own site, capped at 500 URLs with a rate limit of 2. Exclude your asset directories first. - Fix the criticals first: 404s and server errors, then any noindex that should not be there. - Open Redirects and look for a repeated pattern. One template fix usually clears a whole family. - Run the GEO analysis, sort by overall score, and rewrite the three worst pages so each answers its core question in the first paragraph. - Embed the crawl and read the semantically similar list. Merge or differentiate anything above 0.95. - Schedule a weekly recrawl with auto-compare and read the diff instead of the full report. It slots straight into the rest of your [SEO automation](https://organikpi.com/blog/ai-marketing-automation/seo-automation/). Everything above ran on the free local engine, with one Gemini key for the semantic pass and nothing else. The code lives at [github.com/danishashko/serpent](https://github.com/danishashko/serpent), issues and pull requests are open, and [Serpent’s tool page](https://organikpi.com/tools/serpent/) tracks releases. For the paid half of a GEO stack, the [GEO tools we tested](https://organikpi.com/blog/geo-ai-search/best-geo-tools/) cover tracking and citation monitoring. ## Frequently Asked Questions ### Is Serpent really free? Yes. Serpent is MIT-licensed with no URL caps, no license server, and no telemetry. The only optional costs are your own AI API keys for the analysis features and Bright Data usage for the Web Unlocker, SERP, and competitor features. The core crawler, the issue detection, the performance scoring, and the GEO scoring cost nothing. ### What platforms does Serpent run on? All three. The releases page ships a Windows .exe, macOS .dmg builds for Apple Silicon and Intel, and a Linux .AppImage. You can also run it from source on any of them with Node 20 or newer: clone the repo, npm install, npm run dev. ### Does the GEO score need an API key? No. The GEO and AEO scoring is static analysis of the pages already in your crawl, so it runs offline with no key and no extra requests. The same is true of the performance scores and the accessibility pass. The AI analysis and semantic features are the ones that are bring-your-own-key. ### Can Serpent find keyword cannibalisation? Yes, two ways. Exact and near-duplicate hashing catches pages that share wording. The Semantic tab goes further: it embeds the stored page text with Gemini, OpenAI, or a local Ollama model and ranks pages by meaning, so two pages covering one subject in completely different words still surface as a pair. ### Can Serpent crawl JavaScript-heavy or bot-protected sites? Yes, with the right engine. The JS Browser engine renders pages with Electron's built-in Chromium at roughly 1 to 2 seconds per page. The Web Unlocker engine routes requests through Bright Data for bot-protected sites, with a hard cost limit you set per crawl. ### Where does my crawl data go? Nowhere. Crawls are stored in a local SQLite database on your machine, API keys sit in the operating system keychain, and the app sends no telemetry. The built-in MCP server accepts connections from localhost only.