The Site-Wide Crawl Sampler is the audit you reach for when you already suspect a problem in this dimension and need a fast, copy-paste-able fix list. It reuses the same chrome as every other jwatte.com tool — deep-links from the mega analyzers, AI-prompt export, CSV/PDF/HTML download — but the checks it runs are narrow and specific.
Fetches sitemap.xml, samples URLs across the site for template diversity, runs an audit on each, and shows score distribution + worst / best URLs. Detects template-level vs page-level issues.
Why this dimension matters
Indexing issues compound silently. A single <meta name="robots" content="noindex"> left in a template after staging can deindex the entire site; a sitemap that omits pagination URLs can leave half the catalog uncrawled; a Disallow: that overlaps with a Sitemap: entry creates a per-bot disagreement (Google may index the URL; Bing may not). These are the slow-leak failures that Search Console flags weeks after they start.
Common failure patterns
- Canonical tag pointing at a 404 or a redirect chain — the audit verifies that every canonical URL resolves 200-OK and doesn't redirect. A canonical that chains to /404 or that 301s to another URL is a Google Webmaster Guidelines violation.
- Mismatched hreflang cluster — locale A links to locale B with hreflang=es, but locale B does not reciprocate. Google silently drops the entire cluster from international indexing. The audit checks bidirectionality.
- Sitemap declaring URLs that
noindexvia meta or X-Robots-Tag — Sitemap entries are suggestions; noindex is authoritative. If the same URL says "index me" in sitemap and "don't index me" in the HTML, Google follows the HTML. Flag and resolve. - Soft-404s on category/tag pages with zero items — the page returns HTTP 200 but has no substantive content. Google treats these as low-quality and deprioritizes the domain. Generate a 404 response for empty tag/category pages.
How to fix it at the source
Treat Search Console as the source of truth for what Google actually thinks of your site; submit sitemap updates + changelogs there. For hreflang, use a link-graph audit to verify bidirectional coverage every sitemap regeneration. For indexing conflicts, the audit's per-bot simulation (Googlebot vs Bingbot vs per-LLM bot) catches directives that pass one crawler and fail another.
When to run the audit
- After a major site change — redesign, CMS migration, DNS change, hosting platform swap.
- Quarterly as part of routine technical hygiene; the checks are cheap to run repeatedly.
- Before an investor / client review, a PCI scan, a SOC 2 audit, or an accessibility-compliance review.
- When a downstream metric drops (rankings, conversion, AI citations) and you need to rule out this dimension as the cause.
Reading the output
Every finding is severity-classified. The playbook is the same across tools:
- Critical / red: same-week fixes. These block the primary signal and cascade into downstream dimensions.
- Warning / amber: same-month fixes. Drag the score, usually don't block.
- Info / blue: context-only. Often what a PR reviewer would flag but that doesn't block merge.
- Pass / green: confirmation — keep the control in place.
Every audit also emits an "AI fix prompt" — paste into ChatGPT / Claude / Gemini for exact copy-paste code patches tied to your stack.
Related tools
- Mega Analyzer — One URL, every SEO/schema/E-E-A-T/voice/mobile/perf audit in one pass..
- IndexNow Submission Audit — Checks IndexNow key file at root.
- robots.txt Simulator — Paste a robots.txt, a list of URLs, and a bot.
- noindex / X-Robots-Tag Conflict Audit — Probes a URL, compares HTML meta-robots directive against HTTP X-Robots-Tag header.
- Link-Graph Depth Audit — Extracts all outbound links from a page, classifies each domain by authority tier (government / academic / major publisher / known / unknown), scores outbound-domain diversity and primary-source citation depth..
Fact-check notes and sources
- Google Search Central: Robots.txt introduction
- Sitemaps.org: Protocol spec
- IndexNow: Protocol spec
- Google: hreflang annotations for localized pages
This post is informational and not a substitute for professional consulting. Mentions of third-party platforms in the tool itself are nominative fair use. No affiliation is implied.
