# Meta crawler and link preview checklist Version 1.0, September 24, 2026. Written by J.A. Watte (https://jwatte.com). Free to copy, adapt and share. No signup, no attribution required. Meta publishes no Muse crawler, user agent or website standard (checked September 24, 2026). There is nothing to make a site "Muse ready". What Meta does document for site owners is short: robots.txt tokens for five crawlers, the rules its link-preview crawler follows, the limits on preview images for Facebook and WhatsApp, and a reviewed form for Muse connectors. This file is all of it, as boxes to tick. Every item names its source; the full list is at the end. **How to use it.** Work top to bottom. Sections 1 to 4 are things you can check and fix today. Section 5 is context, not work. Run the free audit after each change: - Meta AI Crawler + Link Preview Audit: https://jwatte.com/tools/meta-ai-crawler-audit/?url=https://yoursite.example/&autorun=1 - Meta's Sharing Debugger (needs a Facebook login; press "Scrape Again" after every fix): https://developers.facebook.com/tools/debug/ If you paste this file into an AI coding assistant, tell it to change the template or generator that builds your pages, not one built page, or the next build undoes the fix. ## 1. robots.txt: Meta's five tokens Meta's crawler page lists five user-agent tokens and says robots.txt is how to express your preferences ("industry-standard practices like robots.txt rather than non-standard formats like NoAI tags"). | Token | What Meta says it does | Honours robots.txt | |---|---|---| | facebookexternalhit | Builds link previews for links shared on Facebook, Instagram and Messenger | Yes, but "might bypass robots.txt when performing security or integrity checks" | | Meta-WebIndexer | Improves Meta AI search: allowing it "helps us cite and link to your content in Meta AI's responses" | Yes | | Meta-ExternalAgent | "training foundation AI models or improving products by indexing content directly" | Yes | | Meta-ExternalFetcher | Fetches individual links at a user's request; also used for evaluating and improving agentic AI | "may bypass robots.txt rules" | | Meta-ExternalAds | Advertising and other business products | Yes | - [ ] **1.1 robots.txt answers 200 (or 404 if you want no rules at all).** A 5xx or a timeout is worse than no file: under RFC 9309 a crawler "MUST assume complete disallow" when robots.txt is unreachable because of a server error. A robots.txt that answers with your HTML home page counts as a file with no rules. - [ ] **1.2 Meta-WebIndexer is allowed on every page you want Meta AI to cite, and on /.** This is the one documented lever for Meta AI citations. - [ ] **1.3 facebookexternalhit is allowed on every page people share.** A Disallow for it means blank link previews. - [ ] **1.4 Meta-ExternalAgent is a separate decision.** Blocking it is a training and indexing opt-out. It does not block Meta AI citations (that is Meta-WebIndexer) and it does not block previews. - [ ] **1.5 Do not rely on a Disallow for Meta-ExternalFetcher.** Meta says it "may bypass robots.txt rules", because a person asked for that page. - [ ] **1.6 Any named group copies the * group's Disallow lines.** A crawler obeys only the group that names it (RFC 9309, section 2.2.1). The moment you add `User-agent: meta-webindexer` with `Allow: /`, it stops obeying everything under `User-agent: *`, including your `Disallow: /admin/` and `Disallow: /thank-you/`. Repeat every * Disallow inside each named group, or do not add the named group at all. Naming a Meta token is optional: if * already allows it, a named group adds nothing. - [ ] **1.7 No rules stranded under the wrong group.** Blank lines and comments do not end a group; only the next `User-agent` line does. A `Disallow` written below a blank line and a comment under `User-agent: FacebookBot` applies to FacebookBot alone, not to everyone. Put rules meant for everyone under `User-agent: *` (and into each named group). - [ ] **1.8 Legacy tokens are harmless but do nothing new.** FacebookBot and facebookcatalog are no longer on Meta's crawler page. Keeping them is fine; they do not stand in for the five current tokens. - [ ] **1.9 Match tokens case-insensitively, and as a substring in CDN or firewall rules.** robots.txt tokens are case-insensitive. Cloudflare Radar lists a Meta-ExternalAds user agent that is an iPhone Safari string with the token at the very end, inside a "compatible" comment, so a firewall rule should look for the token anywhere in the User-Agent header, not only at the start. - [ ] **1.10 No IP allowlists.** Meta's crawler page no longer explains how to verify its IP addresses: the WHOIS lookup it used to describe is gone from the May 21, 2026 revision. Allow by user-agent token or not at all. - [ ] **1.11 Allow 24 hours.** Meta: "crawlers may cache the contents of robots.txt for up to 24 hours." A safe pattern, when you do want a named Meta group: ``` User-agent: * Disallow: /admin/ Disallow: /thank-you/ User-agent: meta-webindexer User-agent: facebookexternalhit Disallow: /admin/ Disallow: /thank-you/ Sitemap: https://yoursite.example/sitemap.xml ``` ## 2. What the link-preview crawler gets from your server Meta's crawler page lists five requirements for facebookexternalhit (gzip and deflate, Open Graph tags in the first 1 MB, a few seconds, the Range header, and allow-listing its user agents or IP addresses) and gives one sample request: ``` curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "https://yoursite.example/" ``` - [ ] **2.1 Your CDN or firewall lets it through.** No 403, 429 or 503 and no challenge page ("Just a moment...", "Attention Required") for facebookexternalhit, Meta-WebIndexer or WhatsApp. This is a firewall setting (bot-fight modes, WAF rules), not robots.txt. - [ ] **2.2 Compression is gzip or deflate.** Meta: "Your server must use gzip and deflate encodings" and "Our crawler only accepts gzip and deflate encodings." A request that offers only gzip and deflate must never get Brotli (br) or zstd back. - [ ] **2.3 The page arrives "within a few seconds."** Serve shared pages from cache. - [ ] **2.4 The Range header is honoured exactly or ignored.** Meta: return all required properties "according to the bytes specified in the Range header ... or it should ignore the Range header altogether." Either answer 200 with the whole page, or answer 206 with exactly the bytes Content-Range promises. The failure to look for: an edge function or HTML rewrite (an injected script, a geo tag) that runs after the CDN sliced the range, so the 206 body is longer than its Content-Range says (RFC 9110, section 14.4). Skip the rewrite when the request carries a Range header. - [ ] **2.5 Optional: an AAAA (IPv6) record.** Meta's FAQ says the crawler looks for a AAAA record after a URL change. Low priority for a site that has not moved. To check 2.4 by hand, compare the byte count with the span in Content-Range: ``` curl -s -D - -o body.bin -H "Range: bytes=0-524288" -H "Accept-Encoding: gzip, deflate" -A "facebookexternalhit/1.1" "https://yoursite.example/" | grep -i "content-range" wc -c body.bin ``` For `Content-Range: bytes 0-222965/222966` the file must be 222,966 bytes. ## 3. Open Graph tags - [ ] **3.1 og:title, og:description, og:url and og:image on every page, none empty.** These are Meta's basic tags. - [ ] **3.2 Every og: URL is absolute.** Meta's FAQ: "All URLs must be absolute." Write `https://`, not `/images/card.jpg` and not `//cdn.example/card.jpg`. - [ ] **3.3 The tags sit in the head, near the top.** Facebook reads Open Graph properties only "before the first 1 MB." WhatsApp needs the whole head "within the first 300KB of the HTML", and og:title, og:description and og:url "must be inside the `` tag." Large inline scripts, styles and JSON in the head push the tags and the head boundary down. - [ ] **3.4 No og: tags after the closing head tag.** WhatsApp's documentation puts the tags inside the head, and ogp.me gives the first tag preference. - [ ] **3.5 The first og:image is the one you want.** ogp.me: "The first tag (from top to bottom) is given preference during conflicts." - [ ] **3.6 og:url equals your rel=canonical.** Meta attributes shares and likes to that URL. - [ ] **3.7 og:title carries no branding.** Meta: the title "without any branding such as your site name." Put the brand in og:site_name. - [ ] **3.8 og:description about 80 characters for WhatsApp.** WhatsApp shows one or two lines and says "80 characters will suffice." - [ ] **3.9 og:locale on any page that is not in US English.** It defaults to en_US. - [ ] **3.10 Leave fb:app_id alone unless you use Facebook Insights.** The Sharing Debugger flags it as a "missing required property" on almost every site. It does not change the preview. - [ ] **3.11 Optional: domain verification.** The facebook-domain-verification meta tag lets a Business Manager edit link previews for your domain. It has nothing to do with AI or crawling. ## 4. The og:image - [ ] **4.1 It loads.** The URL answers 200 with an image/* Content-Type when facebookexternalhit asks. A 404, a 5xx or an HTML page means a blank or white preview (Meta's FAQ: the image "is no longer available, is too big or could not be fetched"). - [ ] **4.2 It is under 8 MB, and under 600 KB for WhatsApp.** Meta: "The size of the image file must not exceed 8 MB." WhatsApp: "under 600KB in size." In my own Sharing Debugger test on September 24, 2026, a 9,430,028-byte PNG cover on one of my book sites came back "Image Too Big ... exceeded the maximum allowed size of 8Mb." A 1200x630 JPEG at quality 80 is usually under 300 KB. - [ ] **4.3 It is JPEG, PNG, GIF or WebP, not SVG.** Meta's documented og:image:type values are image/jpeg, image/gif and image/png. What I saw in the same test on September 24, 2026: - a WebP og:image on a site I run rendered as the normal large card, so there is no need to convert WebP for Facebook's sake; - an SVG og:image on a news site I run rendered only as a small, cropped square thumbnail, so replace SVG with a PNG or JPEG; - AVIF was not tested. - [ ] **4.4 It is big enough and roughly 1.91:1.** Minimum 200x200 ("The minimum allowed image dimension is 200 x 200 pixels"). Under 600x315 you get a small square instead of the large card. Meta recommends at least 1200x630 and an aspect ratio "as close to 1.91:1 ... as possible." WhatsApp wants "300px or more in width with 4:1 width/height or less aspect ratio." - [ ] **4.5 og:image:width and og:image:height are declared and true.** Meta: they let the crawler "render the image immediately without having to asynchronously download and process it", so the first share shows the picture. Declare the real pixel size, and set og:image:type to the type the server actually sends. - [ ] **4.6 A changed image gets a new URL, and the old file stays.** Meta: "Use a new URL for the new image or the image won't be updated" and "Don't remove old images." Then press "Scrape Again" in the Sharing Debugger. A complete image block: ``` ``` ## 5. Muse and Meta's other agents: what exists and what does not Context, not a to-do list. - **Muse's own browser has no token.** Meta says Muse runs "a real up-to-date Chromium based browser", that its browsing agent "sees an accessibility tree snapshot of the page", and that "When Muse browses the internet, it will appear as your activity." Meta documents no user agent, IP list or signature for it, so robots.txt cannot address it and there is nothing to allowlist. - **Accessible labels help it (my inference, not a Meta rule).** An agent that reads the accessibility tree gets a form field or button with no accessible name as a nameless control, so it cannot tell what the control is for, the same problem a screen reader user has. Label every field. - **Public phone numbers.** Meta is testing Muse calls to businesses whose phone numbers are publicly available. Whether to take those calls is a business decision; Meta publishes an opt-out form. - **Connectors are reviewed, not discovered.** The Muse connector form takes a hosted HTTPS MCP endpoint or a Raw API with an optional OpenAPI spec, plus privacy, terms and documentation URLs, a support contact and a 512x512 icon. Meta: "We'll review your connector for functional, security, and legal requirements." No file on your site gets you listed. - **Meta Business Agent reads your website and honours robots.txt.** Its website knowledge crawl can report "blocking our crawler in robots.txt." Meta does not name that crawler's token. - **What Meta does not say.** Nothing about reading llms.txt, ai.txt, AGENTS.md or Content-Signal, and no WebMCP or A2A support. Keep those files for the agents that do read them. - **If you wired up WebMCP ("AI functions") before March 2026:** the draft removed `navigator.modelContext.provideContext` on March 5, 2026. Register tools with `document.modelContext.registerTool` instead. Meta documents no WebMCP support; this matters for Chrome and ChatGPT. ## 6. Verify - [ ] **6.1** Run https://jwatte.com/tools/meta-ai-crawler-audit/?url=https://yoursite.example/&autorun=1 and clear every scored FAIL and WARN in groups A to D (rows labelled INFERRED or REPORTED are hints, not requirements). - [ ] **6.2** Paste the page URL into the Sharing Debugger and press "Scrape Again". Look for the large card and for warnings other than fb:app_id. - [ ] **6.3** For WhatsApp, type the link into a chat without sending it. A preview should appear within about 10 seconds; if it is small, recheck 4.2 to 4.4. - [ ] **6.4** After a robots.txt change, wait 24 hours before judging the result. ## Sources (all checked September 24, 2026) - Meta Web Crawlers, "Updated: May 21, 2026": https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ - Meta, sharing for webmasters (basic tags, og:image:type values, title without branding): https://developers.facebook.com/docs/sharing/webmasters/ - Meta, images in link shares (8 MB, 200x200, 600x315, 1200x630, 1.91:1, width and height tags, new URL for a new image): https://developers.facebook.com/docs/sharing/webmasters/images/ - Meta, link sharing FAQ (absolute URLs, white-box images, small square images, AAAA): https://developers.facebook.com/docs/sharing/webmasters/faq/ - Meta Sharing Debugger: https://developers.facebook.com/tools/debug/ - Meta, domain verification: https://developers.facebook.com/docs/sharing/domain-verification/ - WhatsApp link previews (head within 300KB, image under 600KB, 300 px, 4:1, 80 characters): https://developers.facebook.com/documentation/business-messaging/whatsapp/link-previews.md - The Open Graph protocol (first tag wins, og:locale default): https://ogp.me/ - Cloudflare Radar, Meta-ExternalAds (the token at the end of an iPhone Safari user-agent string): https://radar.cloudflare.com/bots/directory/meta-externalads - RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309 - RFC 9110, section 14.4, Content-Range: https://www.rfc-editor.org/rfc/rfc9110#section-14.4 - Meta, security and safety for AI agents with Muse (September 8, 2026): https://research.meta.ai/blog/security-and-safety-for-ai-agents-our-approach-with-muse - Meta help, Muse calls to businesses: https://www.meta.com/help/artificial-intelligence/4532990443643263/ and the opt-out form: https://help.meta.com/requests/1088631730352171/ - Muse connector platform: https://muse.ai/platform - Meta Business Agent, website knowledge: https://developers.facebook.com/documentation/meta-business-agent/reference/configure/agent-knowledge-websites.md - WebMCP, removal of provideContext (pull request 132, merged March 5, 2026): https://github.com/webmachinelearning/webmcp/pull/132 - The author's own Sharing Debugger test, September 24, 2026 (WebP large card, SVG small cropped thumbnail, 9.4 MB PNG refused): https://jwatte.com/blog/meta-muse-website-compatibility/ Informational only, not legal or SEO advice. Meta, Facebook, Instagram, Messenger, WhatsApp and Muse are named for identification (nominative fair use); no affiliation is implied.