Quora's new growth lead will own top-of-funnel acquisition and is expected to understand organic search. Yet Quora's robots.txt keeps Bing (which also feeds DuckDuckGo, Yahoo and Copilot) and the crawlers behind ChatGPT search, Perplexity and Claude search away from every question page. In Bing, searches like "quora how do I learn to code" show only the Quora homepage and help center. The one exception is that OAI-SearchBot may read URLs containing "drinking-water", so a one-topic test already seems to be running.
First action
List the evergreen topics and draw the random 5% of pages.
How to measure it
Referred sign-ups per 1,000 test pages vs control
- Now
- –
- Target
- Measured lift with a go/no-go per topic
- By
- 6 weeks
Steps
Agree the question
What referred traffic and sign-ups are worth against content-licensing value.
Lead Product Manager - Growth 2 days
Pick test pages
Random 5% of pages in three evergreen topics, plus matched controls.
Growth Data Scientist 2 days
Write the rules
robots.txt Allow lines for search-and-cite crawlers only on the test paths.
SEO Engineering Lead 1 day
Legal and policy check
Confirm the licensing notice and training block still cover every crawler.
Legal 3 days
Track referrals
Referrer and crawler logs, sign-ups and digest opt-ins by page group.
Growth Data Scientist 1 week
Read the result
Lift per topic after six weeks; extend, change or roll back.
Lead Product Manager - Growth 6 weeks
Why it works
Treat it as an experiment, not a policy flip. Pick a random 5% of question pages in a few evergreen topics and allow only search-and-cite crawlers (Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) to read them, while training crawlers stay blocked and the licensing notice stays in place. Track referred visits, sign-ups and digest opt-ins from those pages against matched control pages for six weeks, then decide topic by topic.
It turns a large, unmeasured trade-off into a number the growth team can act on: either a new source of readers and sign-ups, or proof that the current block costs little. The privacy promise to writers is kept, because anonymized and deleted answers drop out of these indexes the same way they drop out of Google.
Ready to use
ChecklistCrawler test set-up
[ ] Random 5% of pages in 3 evergreen topics [ ] Matched control pages [ ] Allow lines for search-and-cite crawlers only [ ] Training crawlers still disallowed [ ] Referrer and bot log dashboard [ ] Sign-up and digest opt-in tracking by group [ ] Six-week review date
Risks and how to handle them
Content reused for AI training
Allow only the crawlers whose operators state they are for search; keep the training notice and monitor logs.
Writers worried about exposure
Deleted and anonymized answers leave these indexes on recrawl; say so in the help center.
Evidence
- HiringLead Product Manager - Growth, Quora (remote, posted October 13, 2026): will "Own the end-to-end user growth funnel, with a focus on top-of-funnel acquisition, activation, and early retention across web and mobile surfaces", "build engaging experiences that expand Quora's reach to new demographics", "Define objectives, metrics, and guardrails" and "Run structured experimentation across the funnel". Preferred: "Experience growing UGC products with heavy SEO traffic and a strong understanding of organic search as an acquisition channel".Source
- Your websiterobots.txt (read October 22, 2026): the catch-all "User-agent: *" group allows only the homepage, /about, /press, /challenges, /login and /signup and disallows everything else, with the note "If you operate a search engine and would like to crawl Quora, you can email [email protected]." Googlebot, Applebot and Google-Extended have their own groups that allow question pages; there is no group for Bingbot, so Bing falls under the catch-all. The file also lists embed widget paths (/widgets/content_iframe/ and /widgets/content_js/).Source
- Your websiterobots.txt AI rules: GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-User, Claude-SearchBot, Bravebot and Applebot-Extended are disallowed from the whole site. OAI-SearchBot (the crawler behind ChatGPT search results) is disallowed except for URLs containing "drinking-water". A notice says crawlers may not use Quora content to train AI models without a contractual licensing agreement.Source
- Search checkBing searches run October 22, 2026 (twice, a few minutes apart): "quora how do I learn to code" and "quora is it worth learning japanese" return only the quora.com homepage, /about, the help center and the es.quora.com home page, no question or answer pages; "site:quora.com what is the meaning of life" returns no Quora pages at all. Bing results also feed DuckDuckGo, Yahoo and Microsoft Copilot.Source
- News coveragePrecedent: Microsoft confirmed in July 2024 that Bing stopped crawling Reddit after Reddit's new robots.txt took effect, and Reddit results dropped out of Bing while Google kept showing them.Source
- Your websiterobots.txt explains the Wayback Machine opt-out: "People share a lot of sensitive material on Quora" and it is "important that they can delete or anonymize their previously-written answers"; Quora keeps "full edit history" in product instead.Source