You write a page, publish it, confirm it's set to index, follow in Search Console. A month later it's gone from Google and you have no idea why. You check the HTML. The meta robots tag says index, follow. Everything looks fine.
What you didn't check was the HTTP response headers. Your server, your CDN, or your hosting platform added an X-Robots-Tag: noindex header that you never configured and never saw. Two directives, two layers, and the restrictive one wins.
The silent override
Google's documentation is clear on this: when a meta robots tag in HTML and an X-Robots-Tag in HTTP headers both apply to the same page, Google follows the most restrictive combination. If HTML says index and the header says noindex, the page gets deindexed.
This isn't a bug. It's by design. The logic is that if any layer says "don't index," the site owner probably meant it. The problem is that most site owners only manage one layer and don't know the other one exists.
How conflicts happen
Staging environment headers leaking to production. A developer adds X-Robots-Tag: noindex to staging to keep it out of search. The configuration gets copied to production during a deploy and nobody notices because the HTML meta tag still says the right thing.
CDN or reverse proxy defaults. Some hosting platforms add X-Robots-Tag headers to certain response types by default. Netlify, Cloudflare Pages, and Vercel all let you set custom headers in config files, and a misconfigured rule can apply noindex to pages you intended to be indexed.
Plugin or middleware conflicts. WordPress SEO plugins manage meta robots in HTML while a security plugin or server config manages X-Robots-Tag in headers. They don't coordinate, and when they disagree, the page disappears.
Per-file-type rules. A server config that adds X-Robots-Tag: noindex to all PDF files or all image responses. You intended to keep PDFs out of search results but the rule accidentally matches HTML pages too.
What the audit checks
The Noindex Conflict Audit probes a URL and reads both the HTML <meta name="robots"> tag and the HTTP X-Robots-Tag response header. It compares the two and flags any conflict where they disagree.
The audit also handles nuance. A page might have <meta name="robots" content="index, nofollow"> in HTML and X-Robots-Tag: noindex in headers. That's not just a conflict on indexing but also a disagreement about link following. The tool maps every directive from both sources and shows where they align and where they don't.
It also checks for multiple meta robots tags in the HTML, which is another common source of confusion. If a page has two meta robots tags with different values, browsers and crawlers use the most restrictive combination of both.
The scariest finding is what the tool calls a "silent kill": the HTML says index, the header says noindex, and the page is currently deindexed in search. The owner thinks everything is fine because they only check the HTML.
If you're managing multiple sites and need to catch these configuration drift problems before they cost you traffic, The $100 Network ($9.99 on Kindle) covers the operational patterns that prevent silent failures across a portfolio.
Fact-check notes and sources
- Google's robots meta tag specification states: "If there's a conflict between the HTTP header and the page-level setting, the more restrictive setting applies." Source: Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications."
- X-Robots-Tag is a de facto standard implemented by Google, Bing, and Yandex. It is not an official HTTP header registered with IANA.
- Multiple meta robots tags on the same page are combined using the most restrictive union of all directives. Source: Google Search Central documentation.
Related reading
- AI posture consistency — when crawler directives disagree across robots.txt, ai.txt, and meta tags
- Hreflang bidirectional checks — another case where two-layer directives must agree
- Canonical cluster conflict — conflicting signals that confuse search engines
- Index coverage delta — finding pages that should be indexed but aren't
This post is informational, not SEO-consulting advice. Mentions of Google, Bing, Netlify, and other third parties are nominative fair use. No affiliation is implied.
