The new VP of Product Marketing is hired to "own how the world understands, chooses" Astronomer and to win against "other managed Airflow services and alternative orchestrators". Data engineers now ask AI assistants exactly that question. But astronomer.io's robots.txt tells every crawler "ai-input=no", asking AI systems not to use the site in their answers, while the same site publishes llms.txt files and a docs MCP server inviting them in. The llms.txt that is there describes Astronomer as "a modern data orchestration platform" for "data availability", lists 52 blog posts and leaves out the comparison pages, case studies, pricing and Astro Observe.
First action
Agree with legal and engineering which signal to change, and edit robots.txt.
How to measure it
Citation share for tracked questions
- Now
- –
- Target
- 50%
- By
- 90 days
Steps
Baseline
Ask 30 buyer questions in four assistants and record sources.
Competitive Intelligence Manager 2 days
Decide the signal
Keep ai-train=no; set ai-input=yes for site and docs.
SVP of Marketing 2 days
Edit robots.txt
One line, plus a check that docs inherit it.
Web Team 1 hour
Rewrite llms.txt
Positioning, comparisons, pricing, cases, products.
Vice President, Product Marketing 3 days
Add proof
Case study numbers in plain text on each linked page.
Head of Content 1 week
Monitor
Same 30 questions monthly.
Competitive Intelligence Manager monthly
Report
Citations and AI-referred signups to marketing leadership.
Marketing Analytics Manager monthly
Why it works
Keep "ai-train=no" if training is the concern, but change the marketing site and docs to "ai-input=yes" so answer engines can quote Astronomer's own pages. Rewrite llms.txt around what a buyer asks: what Astro is (orchestration for data and AI, the Astro Runtime figures), Astro vs MWAA, Composer, Dagster and open-source Airflow, pricing, the Airflow 2 support offer, Private Cloud and Observe, and the 20 strongest case studies with their numbers. Then ask the leading assistants 30 buyer questions and track citations monthly.
If the site opts out, AI answers are built from rivals' pages and forum threads. Opting in and curating the file is a one-line change plus a day of writing, and it puts Astronomer's positioning into the channel where shortlists are now made.
Ready to use
ChecklistAI answer readiness for astronomer.io
[ ] robots.txt: ai-train=no, search=yes, ai-input=yes [ ] llms.txt opens with current positioning (orchestration for data and AI) [ ] Links: Astro vs MWAA / Composer / Dagster / OSS, pricing, Airflow 2 support, Private Cloud, Observe [ ] 20 case studies with one result each [ ] Comparison table dated this quarter (idea 2) [ ] 30 buyer questions tracked monthly
Risks and how to handle them
Concern about content reuse
Training stays opted out; only answer-time use changes.
Answers change slowly
Track sources cited, not only mentions.
Evidence
- Your websiteastronomer.io/robots.txt (checked October 23, 2026) applies to every crawler: "Content-Signal: ai-train=no, search=yes, ai-input=no". Under the Content Signals convention, ai-input=no asks AI systems not to use the site's content as input when generating answers, which covers the marketing pages, comparisons, pricing and case studies on the same host.Source
- Your websiteAt the same time the site publishes /llms.txt, /docs/llms.txt, an 8.8 MB /docs/llms-full.txt and a docs MCP server. The main llms.txt describes Astronomer as helping data teams "enhance their data availability with Astro, a modern data orchestration platform"; its 63 links are developer resources and 52 blog posts, with no case studies, no pricing or comparison pages (Dagster, other managed Airflow services), and no Astro Observe.Source
- HiringVice President, Product Marketing (posted October 21, 2026; reports to the SVP of Marketing): "own how the world understands, chooses, and adopts Astronomer products"; define positioning "across Astro, Astro Private Cloud, and our agentic orchestration and AI capabilities"; "Drive competitive strategy and win/loss programs so our field teams can confidently differentiate against other managed Airflow services and alternative orchestrators"; build a team covering "product, solutions and industry, competitive intelligence, and sales enablement"; "Partner with Demand Generation, Content, Events, and Community to turn the story into pipeline, and measure what works".Source
- Your website"Astro vs Other Managed Apache Airflow® Services" page: its comparison table is labelled "AS OF DECEMBER 2024"; proof points are Campspot's move from Amazon MWAA ("Reducing job runtime from 2 hours to 2 minutes") and WeWork's move from Google Cloud Composer. Other comparison content: /airflow/astro-vs-dagster/, "The top 7 alternatives to Amazon MWAA" and "... to Google Cloud Composer" blog posts, and briefings on leaving Control-M, AutoSys and Tidal.Source
- Your websiteHomepage: "ORCHESTRATION FOR DATA AND AI. Move data and agentic workflows from prototype to production. Powered by Apache Airflow®, augmented with AI."; the Astro Runtime claims "500,000 sustained concurrent tasks", "25× open-source Airflow" and "228ms p95 task start latency"; featured case studies are Booking.com, Together AI and Janus Henderson.Source