Groupon's new Head of SEO & AI Search is asked to make "AI crawler policy a conscious decision rather than a default", because ChatGPT, Claude and Gemini now recommend places by name. Today groupon.com's robots.txt opens with Cloudflare's managed block: it disallows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent and others, and its content signal says ai-train=no but leaves ai-input unset. Blocking training may be intended, but Google-Extended also controls grounding in Gemini Apps, and pages that answer plain clients with a 403 or challenge page cannot be fetched by assistants acting for a user.
First action
List every user agent in robots.txt with its purpose and the decision for each.
How to measure it
AI citations on tracked local queries
- Now
- –
- Target
- 2x baseline
- By
- 2 quarters
Steps
Inventory agents
Each bot, its purpose (training, search, user fetch).
Technical SEO lead 2 days
Decide policy
Per agent, with the revenue trade-off written down.
Head of SEO & AI Search 1 week
Set content signals
search, ai-input and ai-train values.
Head of SEO & AI Search 1 day
Update robots.txt
Owned rules outside the managed block.
Technical SEO lead 1 day
Check the challenge
Verified AI fetchers reach city, category and deal pages.
Platform Engineering 1 week
Baseline citations
50 city x category queries in ChatGPT, Claude, Gemini.
Head of SEO & AI Search 1 week
Re-measure
Same queries monthly, by city.
Head of SEO & AI Search monthly
Why it works
Write the policy agent by agent: keep training crawlers blocked if that is the business choice, but explicitly allow search and user-fetch agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) and decide on Google-Extended with the Gemini trade-off in view. Add "ai-input=yes" to the content signal if retrieval is wanted, move the rules out of the managed block so they are owned, and check that verified AI fetchers get past the bot challenge on city, category and deal pages.
Assistants can only recommend and cite pages they can read. A written policy also stops a platform default from deciding Groupon's AI visibility each time Cloudflare changes it.
Ready to use
ChecklistAI crawler policy
[ ] Training crawlers: GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended (decide) [ ] Search and user agents: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot (allow?) [ ] Google-Extended: Gemini grounding trade-off [ ] Content-Signal: search, ai-input, ai-train [ ] Rules moved out of the managed block [ ] Bot challenge lets verified fetchers through [ ] Monthly citation check by city
Risks and how to handle them
Content used for training without permission
Keep training crawlers blocked; allow only retrieval and search agents.
Bot traffic load
Allow verified agents only and keep rate limits.
Evidence
- HiringHead of SEO & AI Search (posted September 23, 2026): "AI Overviews now appear on the large majority of the local queries we track in North America. ChatGPT, Claude and Gemini recommend places by name, and agents are starting to search, compare and transact without ever loading a page." The role owns "crawler and rendering policy across all markets, including AI crawler policy as a conscious decision rather than a default", "machine-discovery endpoints, structured deal data, connectors to Claude, ChatGPT and Gemini", measured "geolocated rather than nationally".Source
- Your websiterobots.txt (checked October 21, 2026) opens with a "BEGIN Cloudflare Managed content" block: "Content-Signal: search=yes,ai-train=no" with no ai-input value, then "Disallow: /" for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, CloudflareBrowserRenderingCrawler, Google-Extended, GPTBot and meta-externalagent. Later rules include "Disallow: /search*".Source
- Platform documentationGoogle: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal"; it controls use for Gemini training and "grounding in Gemini Apps". Anthropic: ClaudeBot collects content for training, while Claude-User fetches pages when people ask Claude a question and Claude-SearchBot indexes for search; Anthropic's bots honour robots.txt and anti-circumvention signals such as CAPTCHAs.Source
- Access checkFetched with a plain HTTP client (no browser) on October 21, 2026: robots.txt, the sitemap index and llms.txt return 200, but the homepage returns 403 and each of the 30 sitemap files listed in sitemap.xml returns a "Just a moment..." challenge page instead of XML.Source