There are two places you can tell search engines whether to index a page. The first is the <meta name="robots" content="..."> tag in your HTML head. The second is the X-Robots-Tag HTTP response header sent by your server. Most site owners only think about the first one. Many don't even know the second one exists. And when they contradict each other, Google follows the most restrictive directive.
That means if your HTML says index, follow but your server sends X-Robots-Tag: noindex, your page will be deindexed. No crawl error. No warning in Search Console. Just a slow, silent disappearance from search results over the next few weeks as Googlebot processes the conflicting signals and picks the stricter one.
How the conflict happens
The most common cause is a reverse proxy or CDN adding headers that the site developer never asked for. Nginx configurations, Apache .htaccess files, Cloudflare page rules, and Netlify headers files can all inject X-Robots-Tag without anyone touching the HTML. A hosting migration is a particularly fertile ground for this problem. Your old server had a development rule that added noindex to staging URLs. Your new server inherited that config file, and now it's applying the header to production pages.
WordPress caching plugins are another frequent culprit. Some plugins add X-Robots-Tag: noindex to cached 404 pages. If the cache layer starts serving stale 404 responses for valid URLs (which happens more than you'd expect), those pages now carry a noindex header that contradicts the perfectly fine meta tag in the HTML.
The opposite scenario exists too. You intentionally added <meta name="robots" content="noindex"> to a page to keep it out of search, but the CDN strips or overwrites that with a permissive X-Robots-Tag. Now a page you wanted private is being indexed.
Why this is hard to catch manually
If you view page source, you'll see the meta tag. That looks fine. You move on. But the HTTP header is invisible in the browser unless you open DevTools, navigate to the Network tab, find the document request, and inspect the response headers. Nobody does that routinely for their own pages.
Even Search Console won't always make this obvious. The URL Inspection tool shows crawl data but doesn't flag the specific header conflict. You'll see "Excluded by noindex" in the coverage report and spend an hour staring at your HTML source trying to find a noindex tag that isn't there, because the directive is coming from the server, not the page.
Server configuration changes don't trigger obvious site-level alerts. A DevOps engineer updates the nginx config for an unrelated reason, maybe adding CORS headers or tuning cache TTLs, and accidentally uncomments a line that adds X-Robots-Tag. Nobody tests for that in a staging environment because nobody thinks to check indexation headers during infrastructure changes.
What the audit checks
The noindex / X-Robots-Tag Conflict Audit probes a URL and reads both layers. It fetches the page through a proxy, extracts the meta robots tag from the HTML, and reads the X-Robots-Tag from the HTTP response headers. Then it compares them.
If both layers agree (both say index, or both say noindex), you get a pass. If they disagree, the tool flags exactly which layer says what, explains which directive Google will follow (the most restrictive one), and tells you which layer needs to change.
This pairs well with the Redirect Chain Audit to make sure your redirects aren't passing through a server that injects noindex headers mid-chain. It also works alongside the Meta + OG Inspector for a broader view of what your page is actually communicating to crawlers and social platforms.
If you're running a content site and your traffic has dropped without an obvious cause, check this before blaming algorithm updates. In The $97 Launch, I talk about the invisible infrastructure that can silently undermine a site's discoverability. Meta robots conflicts are one of those invisible problems that cost real money by the time you notice them.
Fact-check notes and sources
- Google follows the most restrictive robots directive when multiple sources disagree. Source: Google Search Central, "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications".
- X-Robots-Tag is a de facto standard supported by Google, Bing, and Yandex. It was introduced by Google in 2007 and is not part of any W3C or IETF specification.
- WordPress caching plugin noindex issues have been documented in WP Rocket and W3 Total Cache support forums, typically related to 404 page caching behavior.
Related reading
- Redirect Chain Audit — find chains that pass through noindex-injecting servers
- Robots.txt Audit — the other layer of crawl control that can conflict with meta directives
- Meta + OG Inspector — broader view of what your page communicates to crawlers
- Orphan Page Detector — pages that vanish from internal linking are doubly at risk when meta conflicts exist
- Sitemap Lastmod Truthfulness — stale lastmod values can delay discovery of meta robots fixes
This post is informational, not SEO-consulting advice. Mentions of Google, Cloudflare, Nginx, and WordPress are nominative fair use. No affiliation is implied.
