Flags.com's robots.txt blocks Meta's link-preview crawler by name and then blocks every crawler it does not list, which includes Pinterest's and X's and LinkedIn's preview bots. Meta uses that crawler to build the title and image when someone shares a link on Facebook, Instagram or Messenger, and Pinterest says a blocked crawler is the usual reason Rich Pins fail. For a flag store, whose customers share a new flag on the porch before Memorial Day and the Fourth of July and pin garden and house flags all year, every shared link risks showing as a bare URL.
First action
Edit robots.txt and run five product links through Meta's Sharing Debugger.
How to measure it
Social referral sessions
- Now
- –
- Target
- +30%
- By
- 90 days
Steps
Edit robots.txt
Allow the social preview crawlers and Storebot-Google with cart/checkout exclusions.
BigCommerce Developer 1 hour
Check Cloudflare
Make sure verified bots pass the browser check.
BigCommerce Developer 1 hour
Test previews
Meta Sharing Debugger on the homepage and top products.
E-commerce Manager 1 hour
Check tags
Open Graph title, image and price on product pages.
BigCommerce Developer 1 day
Enable Rich Pins
Validate with Pinterest.
Marketing Manager 1 day
Claim the domain
Verify flags.com in Pinterest Business.
Marketing Manager 1 hour
Measure
Referral sessions and orders by network.
E-commerce Manager monthly
Why it works
Add allow rules for facebookexternalhit, Pinterestbot, Twitterbot, LinkedInBot and Storebot-Google (with the same cart and checkout exclusions used for Googlebot), and remove the Facebot and facebookexternalhit "Disallow: /" lines. Check that the Cloudflare bot settings let verified social crawlers through too. Then test five product links in Meta's Sharing Debugger and Pinterest's Rich Pin validator and enable Rich Pins.
Shares and pins with a picture and price get far more clicks than bare links, and this traffic is free: it comes from customers and their friends.
Ready to use
Copyrobots.txt rules to add
# Allow social link previews and Google Shopping checks User-agent: facebookexternalhit User-agent: Pinterestbot User-agent: Twitterbot User-agent: LinkedInBot User-agent: Storebot-Google Disallow: /account.php Disallow: /cart.php Disallow: /checkout.php Disallow: /login.php # Then delete the two "Disallow: /" groups for Facebot and facebookexternalhit
Risks and how to handle them
Unwanted scrapers return
Keep the block for scrapers; allow only named, verified social crawlers.
Cached bare previews
Re-scrape top pages in each debugger after the change.
Evidence
- Your websiterobots.txt (checked October 20, 2026): named rules let Googlebot, Bingbot, DuckDuckBot, Applebot, Semrush and OpenAI's crawler in; then "# Block all other bots" with "User-agent: * Disallow: /". Meta's crawlers are blocked by name ("User-agent: facebookexternalhit Disallow: /" and the same for Facebot). Crawlers not named, such as Pinterest's, X's and LinkedIn's link-preview bots and Google's Storebot-Google, fall under the block-all rule. A separate group lets AI assistants such as ChatGPT-User, OAI-SearchBot, PerplexityBot and ClaudeBot in, except for cart and account pages.Source
- Meta documentationMeta's web crawler documentation: FacebookExternalHit fetches a page when someone shares its link on Facebook, Instagram or Messenger, to build the link preview (title, description, image).Source
- Pinterest supportPinterest staff: a Rich Pin error "usually happens when our Pinterest crawler is being blocked by your website. Make sure to include a parameter in your robots.txt file that allows our crawler to access your site."Source
- Tech stackStack seen in the page test and DNS: BigCommerce, Klaviyo (email and SMS), Gorgias chat, Reviews.io, Lucky Orange session recording, a Search Magic site search, Google and Bing Ads and the Meta pixel; Google Workspace mail with DMARC policy "none". The site sits behind a Cloudflare browser check that turns away automated readers.Source