The new Education marketing director is asked to "maximize awareness of our marquee rankings launches", and the paid team is building high-intent acquisition. But usnews.com's robots.txt blocks the search and answer crawlers of ChatGPT (OAI-SearchBot, ChatGPT-User) and Perplexity from everything except five sponsored insurance articles, while Claude's search crawler is not listed at all. When a student asks an assistant for "best colleges for nursing", it cannot read or link the U.S. News list, so the answer and the click go to others.
First action
Ask ChatGPT, Perplexity and Claude ten ranking questions and note which sources they cite.
How to measure it
AI assistant referral sessions to ranking pages
- Now
- –
- Target
- +100,000 a month
- By
- 90 days
Steps
Baseline
Ten ranking questions in four assistants; note sources cited.
SEO and audience development lead 2 days
Agree the policy
Search and user fetchers yes on ranking pages; training no.
General Counsel 1 week
Rewrite robots.txt
One explicit group per crawler, scoped paths.
SEO and audience development lead 1 day
Mark up the lists
Clear rank, school and year on each page.
SEO and audience development lead 1 week
Tag referrals
Separate AI assistant sessions in analytics.
Data Science lead 2 days
Time it to a launch
Brief PR and social on cited lists.
Director Marketing, Education 2 weeks
Measure
Referrals, branded search, citations before and after.
SEO and audience development lead 4 weeks
Why it works
Split the policy by purpose: keep training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) blocked, and allow the search and user-fetch crawlers on the "Best" ranking pages and methodology pages only, so assistants can quote a school's rank with a link. List every vendor's crawlers explicitly so the policy is the same for all. Measure assistant referrals and branded search for four weeks around the next rankings launch.
Answer engines send a growing share of research traffic and cite pages they can fetch; a cited rank with a link brings visits, reinforces the brand at decision time and gives the launch more reach without giving away training data.
Ready to use
ChecklistAI crawler policy by purpose
[ ] Training crawlers blocked: GPTBot, Google-Extended, ClaudeBot, CCBot, Applebot-Extended [ ] Search crawlers allowed on /best-* and methodology: OAI-SearchBot, PerplexityBot, Claude-SearchBot [ ] User fetchers allowed on the same paths: ChatGPT-User, Perplexity-User, Claude-User [ ] One explicit group per crawler [ ] AI referral channel in analytics [ ] Citation check before and after the next launch
Risks and how to handle them
Content used for training anyway
Keep training crawlers blocked and review the logs monthly.
Undercuts a licensing negotiation
Allow ranking pages only and keep the full data for licensing (idea 17).
Evidence
- HiringDirector Marketing, Education, New York or Washington DC hybrid, $120,000 to $150,000, posted October 14, 2026: "Establish a marketing structure within U.S. News Education to lead demand generation"; email nurture, newsletters, webinars, social and blogs; display ads and sponsorships "for U.S. News education solutions"; events with sales; work with PR, Social and Product "to maximize awareness of our marquee rankings launches"; define the MQL, build a lead scoring matrix and "Automate the MQL alert and routing process in Zoho"; a Zoho attribution model tying bookings to marketing; weekly KPIs; three direct reports and about five contractors; campaigns across subsidiaries "(CollegeAdvisor, TeenLife, etc.)"; "Established relationships with key university executives."Source
- Your websiterobots.txt (checked November 4, 2026): GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, Anthropic-ai and Google-Extended are each blocked from the whole site except five sponsored "brandfuse" articles for the insurer Lemonade; Perplexity-User, Claude-Web, Applebot-Extended, Amazonbot, meta-externalagent and others are blocked outright; Claude-SearchBot and Claude-User are not listed, so they fall under the general rules, which only block /search and /emailprefs. Bingbot and Applebot also have the same five brandfuse lines; Googlebot-News is kept out of /sponsored/.Source
- HiringU.S. News in its job posts: "a multifaceted digital media company" publishing "independent reporting, rankings, data journalism and advice"; platforms "include Education, Health, Money, Travel, Cars, News, Real Estate, Careers and Insurance"; "We reach millions of people monthly during moments when they are most in need of expert advice and motivated to act on that advice directly on our platforms"; signature "Best" guides on colleges, graduate schools, hospitals, diets, cars and financial services.Source
- HiringPerformance Marketing Analyst (Paid Search & Social Marketing Specialist), Washington DC hybrid, $70,000 to $95,000, posted October 14, 2026: run Google, Microsoft, Meta, LinkedIn and TikTok campaigns "to support high-growth verticals", "with a heavy focus on helping to build out an A/B testing program" from creative and copy variants "to landing page CRO"; use "search intent data and organic affiliate partner performance" to inform social targeting; "track down-funnel lead quality, refine attribution models, and test incrementality"; bid on Target ROAS and tCPA "utilizing offline conversion tracking (OCT) data"; affiliate marketing experience is a must.Source