Polymath growth playbook

Add a Horizon-SWE block above the fold: one line on what it measures (a simulated software company with 73 tools, 50 tasks and live traffic), the headline result, and links to a dedicated /horizon-swe page. On that page, publish a sortable leaderboard covering pass/fail and partial credit, with error bars, run date and harness. Rerun it within two weeks of each major model release. End the page with "Want your model or agent evaluated, or trained on environments like this?" and a link to the project-request page (idea 2).

Labs "do not swap out evals casually", so the benchmark people already check is the one they adopt and buy around. A current leaderboard gives researchers a reason to cite it, the team a reason to post, and buyers a concrete picture of what Polymath builds.

[ ] Homepage block: what it measures, headline result, link
[ ] /horizon-swe page in the sitemap
[ ] Pass/fail and partial-credit tables with error bars
[ ] Run date and harness version per row
[ ] Models released since February rerun
[ ] Evaluate-or-train call to action
[ ] Rerun rule: within two weeks of a major release
[ ] Citation snippet for researchers

• Publishing scores invites contamination of the test set

  Keep the task set private and publish only scores and method (idea 8 adds a separate public sample).

• A customer lab dislikes a public ranking

  Run only public models through public APIs, and give advance notice before posting.

• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source
• Your websiteHomepage (checked November 9, 2026): headline "Simulation environments for training & evaluating autonomous agents"; "Polymath is a data lab focused on increasing the reliability and autonomy of AI agents"; says it works with the leading model labs and is backed by Base10, Cervin Ventures and Y Combinator; sections for Applications & Services, Data, Tasks, Verifiers and Agent(s). It does not mention or link Horizon-SWE or any result, and the only contact is a "Get in touch" email link, shown three times; no form, booking link or newsletter.Source
• Y CombinatorLaunch post: Polymath sells to frontier AI labs that want to acquire training environments; agents practice on work that could take humans days; "Leading models score around 25% on the benchmark"; the only ask is an email to the founders.Source
• PressStartupHub.ai profile of Polymath (June 2026): "Polymath is selling the gym equipment, not just the scoreboard"; names Applied Intuition and labs' internal teams as competition; "Labs do not swap out evals casually."Source
• Competitor researchMechanize's homepage: training environments and evals for frontier coding agents, with a public benchmark, GBA Eval, on its own site (gbaeval.com) showing model results, plus an apply page and pages about its interview process.See the research Source

Add /request with five questions: domain, training or evaluation, horizon length, tools the environment must include, and rough volume and timing. Keep the email as a fallback. Route submissions to the founders with a two-business-day reply target. Link the page from the homepage, the Horizon-SWE page and every post.

Structured requests are easier to send and easier to scope, and the answers show which domains and tools labs actually ask for, which feeds ideas 3 and 18.

• Labs prefer not to put needs in a form

  Keep every field optional and the email visible.

• Your websiteHomepage (checked November 9, 2026): headline "Simulation environments for training & evaluating autonomous agents"; "Polymath is a data lab focused on increasing the reliability and autonomy of AI agents"; says it works with the leading model labs and is backed by Base10, Cervin Ventures and Y Combinator; sections for Applications & Services, Data, Tasks, Verifiers and Agent(s). It does not mention or link Horizon-SWE or any result, and the only contact is a "Get in touch" email link, shown three times; no form, booking link or newsletter.Source
• Y CombinatorLaunch post: Polymath sells to frontier AI labs that want to acquire training environments; agents practice on work that could take humans days; "Leading models score around 25% on the benchmark"; the only ask is an email to the founders.Source
• Your websiteSitemap (7 URLs, checked November 9, 2026): the homepage, /blog, /careers, /privacy, /terms and two posts (/introducing-polymath and /blog/horizon-swe); no environment, benchmark, research, security or contributor pages, and no lastmod dates.Source

Create /environments with one spec card per environment, starting with the simulated software company: tools and services included, task types, horizon length, verifier types, how runs are isolated, and delivery formats. Add a "custom builds" section explaining how a new environment is scoped. Link the card to Horizon-SWE as proof.

A spec sheet lets a lab researcher forward Polymath internally without a call, and it turns blog detail into a page that search and AI assistants can quote.

• Revealing a customer's custom environment

  Only list environments Polymath owns and has already described publicly.

• Your websiteHomepage (checked November 9, 2026): headline "Simulation environments for training & evaluating autonomous agents"; "Polymath is a data lab focused on increasing the reliability and autonomy of AI agents"; says it works with the leading model labs and is backed by Base10, Cervin Ventures and Y Combinator; sections for Applications & Services, Data, Tasks, Verifiers and Agent(s). It does not mention or link Horizon-SWE or any result, and the only contact is a "Get in touch" email link, shown three times; no form, booking link or newsletter.Source
• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source
• Your websitePositioning change: earlier in 2026 the headline was "We build the most advanced environments for training and evaluating AI agents on long-horizon, multi-tool tasks in any domain"; it is now "Simulation environments for training & evaluating autonomous agents".Source
• Your websiteSitemap (7 URLs, checked November 9, 2026): the homepage, /blog, /careers, /privacy, /terms and two posts (/introducing-polymath and /blog/horizon-swe); no environment, benchmark, research, security or contributor pages, and no lastmod dates.Source

Add the contract role to /careers, link each role to its Ashby post, and add a short intro: what Polymath builds, the more than $8M raised from named investors, working with leading model labs, and links to Horizon-SWE and the posts. Sync the YC jobs list. Consider moving the old April posts to a dated "still open" status, so candidates know they are live.

For a six-person team, every hire is a growth lever. The contract role is the most flexible way to add environment-building capacity, and remote engineers who find it through search or YC can't see it today.

• More unqualified applicants

  State the bar and the take-home up front.

• Your websiteCareers page: three roles (Member of Technical Staff - Research and - Engineering at $150k-$300k plus equity; Strategic Projects Lead at $150k-$250k plus equity) and a hiring email; no description of the company or its work, and no listing for the contract Software Engineer role that is open on the Ashby board, which has four roles.Source
• HiringSoftware Engineer (contract, remote, posted April 14, 2026): builds high-fidelity environments, long-horizon tasks and robust verifiers, and turns research failure modes into new benchmarks. The postings say Polymath has raised more than $8M from Base10, Cervin Ventures, Founders Future, Y Combinator and angels, and works with the world's leading model labs.Source
• Y CombinatorCompany profile: Winter 2026 batch, team size 6, San Francisco; a data lab building RL environments to improve long-horizon agent performance, including for recursive self-improvement and long-horizon software engineering; the YC jobs list shows two roles.Source

Add accessible names to the icon buttons, drop unused JavaScript and CSS from the homepage bundle, and load analytics after first paint. Recheck PageSpeed afterwards. Don't spend more than a day: the field data is already good.

Engineers at labs judge a technical vendor by its craft. A clean audit costs a day and removes an easy criticism.

• Deferring analytics drops some events

  Compare event counts for a week.

• PageSpeedGoogle PageSpeed (November 9, 2026): real visitors pass Core Web Vitals (LCP 1.2 s, INP 164 ms, CLS 0). Lab mobile score 66 (desktop 88), with 1.3 s of blocking time, 829 KB of JavaScript (about 355 KB unused), an analytics script at 430 KB over 34 requests, and one accessibility issue: "Buttons do not have an accessible name".Source

For each rerun (idea 1), post a short thread: the new score, one surprising failure with a screenshot or log excerpt, and a link to the leaderboard. Add an RSS feed and a simple email signup to the blog so researchers can follow results.

Concrete failures spread further among researchers than score tables do, and each post sends people back to the leaderboard and the request page.

• Posts read as attacks on a model

  Frame failures as hard problems, not as one vendor's weakness.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard

• Your websiteBlog: two posts, dated January 10 and February 2, 2026, and nothing since; no newsletter signup or RSS feed.Source
• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source

Publish one short note a month on a single failure class: missing migrations, rollbacks, flaky tests, cross-service config, incident triage. Each note shows how often it occurs, an anonymized trace, and the kind of environment and verifier that trains it out. End each note with a link to the request page.

Lab researchers buy from people who understand their models' gaps better than they do. A steady series makes Polymath the reference on long-horizon failures and gives sales a reason to reach out.

• Giving away what customers pay for

  Publish the diagnosis, keep the environments and tasks private.

• HiringMember of Technical Staff - Research (San Francisco, posted April 14, 2026): long-horizon evaluation, agent post-training and environment design focused on where current models fail, including investigating frontier-model failure modes, building benchmarks for complex tool-use tasks and publishing research.Source
• HiringSoftware Engineer (contract, remote, posted April 14, 2026): builds high-fidelity environments, long-horizon tasks and robust verifiers, and turns research failure modes into new benchmarks. The postings say Polymath has raised more than $8M from Base10, Cervin Ventures, Founders Future, Y Combinator and angels, and works with the world's leading model labs.Source
• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source
• Your websiteBlog: two posts, dated January 10 and February 2, 2026, and nothing since; no newsletter signup or RSS feed.Source

Package five of the 50 task types as a public sample: a slimmed container of the simulated company, the tasks, the verifiers, and a one-command run script for the harness already used. State clearly that the leaderboard uses the private held-out set. Ask users to cite Horizon-SWE and link to the request page for the full environments.

A runnable sample is the cheapest proof that the environment is real and hard, and every paper or agent team that uses it becomes a warm lead and a citation.

• Labs train on the public sample

  Keep it small and separate from the scored set, and say so.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard

• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source
• Competitor researchMechanize's homepage: training environments and evals for frontier coding agents, with a public benchmark, GBA Eval, on its own site (gbaeval.com) showing model results, plus an apply page and pages about its interview process.See the research Source

Once the public sample exists (idea 8), contribute an integration to the harness project following its contribution process, with docs that point back to the full leaderboard. Offer the same integration to other open agent frameworks that the team's users ask about.

It puts Horizon-SWE in front of the people who build agents every day, at no ad cost, and every run produces a comparable score.

• Maintainers decline

  Ship it as a separate plugin with clear docs.

• #8 Release a small public sample of Horizon-SWE that anyone can run, and keep the full set held out

• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source

Define three offers, each with a typical scope, timeline and deliverable: (1) evaluation access to existing environments; (2) a custom environment built to a lab's spec, with verifiers; (3) a monthly subscription of fresh tasks in an agreed domain. Publish an "How we work" page without prices, and use the same three in every proposal.

Named packages shorten scoping, make proposals comparable, and push the business toward recurring revenue rather than one-off builds.

• Packages feel rigid to large labs

  Present them as starting points.

• Y CombinatorLaunch post: Polymath sells to frontier AI labs that want to acquire training environments; agents practice on work that could take humans days; "Leading models score around 25% on the benchmark"; the only ask is an email to the founders.Source
• HiringStrategic Projects Lead (San Francisco, posted April 5, 2026): partnerships, data and new market expansion; best practices for organizing domain-expert teams; scaling the expert contributor network (sourcing, evaluating, onboarding and managing experts); leading zero-to-one data projects from scoping and PRDs through data quality; experience with RL environments and frontier-lab data sales is a plus.Source
• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source

Publish /contributors: who Polymath looks for (senior engineers, SREs, domain experts), what the work is (tasks, verifiers, environments), how pay and time commitment work, the quality bar, and a short application. Link it from the careers page and every research note.

Supply of expert task builders limits how fast a data lab can grow, and a visible program also signals to labs that Polymath can scale with quality.

• Too many weak applicants

  A paid sample task as the first screen.

• HiringStrategic Projects Lead (San Francisco, posted April 5, 2026): partnerships, data and new market expansion; best practices for organizing domain-expert teams; scaling the expert contributor network (sourcing, evaluating, onboarding and managing experts); leading zero-to-one data projects from scoping and PRDs through data quality; experience with RL environments and frontier-lab data sales is a plus.Source
• HiringMember of Technical Staff - Engineering (San Francisco, posted April 14, 2026): builds the systems behind the RL and contributor platforms, including platforms that let contributors create RL environments, QA agents for data quality, and tooling to speed up environment and task creation.Source
• Your websiteSitemap (7 URLs, checked November 9, 2026): the homepage, /blog, /careers, /privacy, /terms and two posts (/introducing-polymath and /blog/horizon-swe); no environment, benchmark, research, security or contributor pages, and no lastmod dates.Source
• Competitor researchHalluminate's homepage: "a data research lab building benchmarks and RL environments to push the frontier of AI in knowledge work", starting with financial services and consulting; lists an Expert Network; published the Westworld Finance Diligence Bench (August 11, 2026, 88 problems); contact by email only.See the research Source

Generate a standard report for every task batch: the share of tasks solved by a human expert, the verifier false-pass and false-fail rates on seeded solutions, duplicate and leakage checks, and the difficulty spread. Publish an anonymized example report on the site and make the live report part of every contract.

Quality is the one thing a lab can't check quickly by itself. Measuring it turns Polymath's core claim into evidence, and it raises the bar for rivals.

• Numbers look worse than expected

  Fix before publishing; the report drives internal quality too.

• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source
• HiringMember of Technical Staff - Engineering (San Francisco, posted April 14, 2026): builds the systems behind the RL and contributor platforms, including platforms that let contributors create RL environments, QA agents for data quality, and tooling to speed up environment and task creation.Source
• HiringSoftware Engineer (contract, remote, posted April 14, 2026): builds high-fidelity environments, long-horizon tasks and robust verifiers, and turns research failure modes into new benchmarks. The postings say Polymath has raised more than $8M from Base10, Cervin Ventures, Founders Future, Y Combinator and angels, and works with the world's leading model labs.Source

After each delivery, agree a quarterly cadence: Polymath runs the customer's newest model on its environments (under the customer's terms), clusters the failures, and ships a fresh task family aimed at them, with the quality report from idea 12. Price it as the subscription in idea 10.

It ties renewals to the lab's own progress curve, and it is hard for a rival to copy without the same failure analysis.

• Labs won't share checkpoints

  Use their public API models, or have the lab run the evals and share only the failure logs.

• #10 Package three ways to buy: an eval license, a custom environment build, and an ongoing task subscription
• #12 Ship a data-quality report with every delivery: verifier accuracy, solvability and leakage checks

• HiringSoftware Engineer (contract, remote, posted April 14, 2026): builds high-fidelity environments, long-horizon tasks and robust verifiers, and turns research failure modes into new benchmarks. The postings say Polymath has raised more than $8M from Base10, Cervin Ventures, Founders Future, Y Combinator and angels, and works with the world's leading model labs.Source
• HiringMember of Technical Staff - Research (San Francisco, posted April 14, 2026): long-horizon evaluation, agent post-training and environment design focused on where current models fail, including investigating frontier-model failure modes, building benchmarks for complex tool-use tasks and publishing research.Source
• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source

Offer a small number of university groups hosted access to the full Horizon-SWE set under a simple agreement: no training on it, cite it, and share failure analyses back. Put an application form on the Horizon-SWE page and pick groups working on agents and RL.

Papers that use Horizon-SWE spread it into the research community labs hire from, and the shared failure reports feed the research notes (idea 7).

• Test set leaks

  Hosted runs only; no task files leave Polymath's systems.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard

• HiringMember of Technical Staff - Research (San Francisco, posted April 14, 2026): long-horizon evaluation, agent post-training and environment design focused on where current models fail, including investigating frontier-model failure modes, building benchmarks for complex tool-use tasks and publishing research.Source
• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source

Publish /security: how runs are isolated, what customer data and model outputs Polymath stores and for how long, contributor confidentiality and access controls, ownership of delivered environments and tasks, and how to request the trust center documents. Keep it factual and link the trust center for detail.

A readable summary answers the first round of security questions before a call and shortens procurement for a young vendor.

• Claiming more than is certified

  State only controls that are in place and audited.

• Your websiteThe footer's Security link opens a Vanta trust center at trust.polymathlabs.ai that shows only its title to a text reader; the main site has no page on security or data handling.Source
• Your websiteSitemap (7 URLs, checked November 9, 2026): the homepage, /blog, /careers, /privacy, /terms and two posts (/introducing-polymath and /blog/horizon-swe); no environment, benchmark, research, security or contributor pages, and no lastmod dates.Source

Offer hosted runs: an agent team connects its agent to the simulated company and gets a scored report on the held-out set, with failure traces. Charge per run or by monthly quota. Offer an optional private leaderboard, and let teams publish verified scores on the public one.

It reuses the environment already built, adds many smaller recurring customers, and makes Horizon-SWE the score agent companies quote to their own buyers.

• Distracts from lab revenue

  Gate on demand interviews, and build on the same infrastructure.

• Test-set overfitting by repeat runs

  Rate limits and a rotating held-out set.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard

• Y CombinatorLaunch post: Polymath sells to frontier AI labs that want to acquire training environments; agents practice on work that could take humans days; "Leading models score around 25% on the benchmark"; the only ask is an email to the founders.Source
• PressStartupHub.ai profile of Polymath (June 2026): "Polymath is selling the gym equipment, not just the scoreboard"; names Applied Intuition and labs' internal teams as competition; "Labs do not swap out evals casually."Source
• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source

Package a fixed-price "agent bake-off": Polymath runs two to four candidate coding agents through the simulated company, optionally adding tasks shaped like the buyer's stack, and delivers a comparison report with failure traces. Start with engineering leaders in Polymath's investor networks.

It opens a buyer group far larger than frontier labs, reuses the same environment, and makes Polymath's score part of enterprise purchasing.

• Agent vendors object to the results

  Disclose the method, and keep reports private to the buyer.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard

• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source
• HiringStrategic Projects Lead (San Francisco, posted April 5, 2026): partnerships, data and new market expansion; best practices for organizing domain-expert teams; scaling the expert contributor network (sourcing, evaluating, onboarding and managing experts); leading zero-to-one data projects from scoping and PRDs through data quality; experience with RL environments and frontier-lab data sales is a plus.Source
• HiringSoftware Engineer (contract, remote, posted April 14, 2026): builds high-fidelity environments, long-horizon tasks and robust verifiers, and turns research failure modes into new benchmarks. The postings say Polymath has raised more than $8M from Base10, Cervin Ventures, Founders Future, Y Combinator and angels, and works with the world's leading model labs.Source

Use the request-page answers (idea 2) and customer conversations to pick one adjacent long-horizon domain where Polymath's software-company simulation gives a head start, such as SRE and incident operations, data engineering pipelines, or ML research workflows. Build one environment, a 30-task benchmark and a launch post, staffed by the contributor network.

A second public benchmark proves the method generalizes, widens what each lab can buy, and avoids competing head-on where a rival already leads.

• Spreading a small team thin

  Start only once the software environment has a recurring customer base.

• #2 Replace the email-only contact with a short "Request an environment" page
• #11 A public "Build environments with Polymath" page for expert contributors

• Your websitePositioning change: earlier in 2026 the headline was "We build the most advanced environments for training and evaluating AI agents on long-horizon, multi-tool tasks in any domain"; it is now "Simulation environments for training & evaluating autonomous agents".Source
• HiringStrategic Projects Lead (San Francisco, posted April 5, 2026): partnerships, data and new market expansion; best practices for organizing domain-expert teams; scaling the expert contributor network (sourcing, evaluating, onboarding and managing experts); leading zero-to-one data projects from scoping and PRDs through data quality; experience with RL environments and frontier-lab data sales is a plus.Source
• Y CombinatorCompany profile: Winter 2026 batch, team size 6, San Francisco; a data lab building RL environments to improve long-horizon agent performance, including for recursive self-improvement and long-horizon software engineering; the YC jobs list shows two roles.Source
• Competitor researchHalluminate's homepage: "a data research lab building benchmarks and RL environments to push the frontier of AI in knowledge work", starting with financial services and consulting; lists an Expert Network; published the Westworld Finance Diligence Bench (August 11, 2026, 88 problems); contact by email only.See the research Source

Define one throughput number, verified tasks shipped per week, and the funnel behind it: drafted, verifier passed, human-checked, delivered. Build platform features against the slowest stage: task templates, automatic verifier scaffolding, QA-agent pre-review and contributor dashboards. Review weekly.

Throughput at fixed quality is the unit economics of a data lab. Doubling it doubles what each sale can deliver without more full-time hires.

• Speed lowers quality

  Count only tasks that pass the quality report (idea 12).

• #12 Ship a data-quality report with every delivery: verifier accuracy, solvability and leakage checks

• HiringMember of Technical Staff - Engineering (San Francisco, posted April 14, 2026): builds the systems behind the RL and contributor platforms, including platforms that let contributors create RL environments, QA agents for data quality, and tooling to speed up environment and task creation.Source
• Y CombinatorCompany profile: Winter 2026 batch, team size 6, San Francisco; a data lab building RL environments to improve long-horizon agent performance, including for recursive self-improvement and long-horizon software engineering; the YC jobs list shows two roles.Source

Pay a group of experienced engineers from the contributor network to attempt a sample of tasks under the same tools and rules. Record pass rate, wall-clock time and where they got stuck. Publish human and agent results side by side on the leaderboard, plus an "hours of human work per task" figure.

A human baseline makes the benchmark legible to buyers and press, backs the "days of work" claim with data, and doubles as a check that every task is solvable.

• Humans pass easily, making the gap look small

  Report time and cost per task, not only pass rate.

• #1 Put Horizon-SWE on the homepage and turn the February results into a living leaderboard
• #11 A public "Build environments with Polymath" page for expert contributors

• Y CombinatorLaunch post: Polymath sells to frontier AI labs that want to acquire training environments; agents practice on work that could take humans days; "Leading models score around 25% on the benchmark"; the only ask is an email to the founders.Source
• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source
• HiringStrategic Projects Lead (San Francisco, posted April 5, 2026): partnerships, data and new market expansion; best practices for organizing domain-expert teams; scaling the expert contributor network (sourcing, evaluating, onboarding and managing experts); leading zero-to-one data projects from scoping and PRDs through data quality; experience with RL environments and frontier-lab data sales is a plus.Source

Pick one tool category, such as error monitoring or issue tracking. Write a short study of how agents use it in Horizon-SWE runs: how often they consult it, which actions succeed, and where they go wrong. Offer it to that tool's developer-relations team to co-publish, promoted on both companies' channels.

It borrows a large developer audience, positions Polymath as the source of agent-behaviour data, and may open a second kind of buyer: tool companies that want agent-readiness testing.

• Simulated tools differ from the real product

  Say exactly how the tool is represented in the simulation.

• #7 A monthly "Where agents fail" research note, from the failure modes the team already studies

• Your website"Towards Greater Reliability and Autonomy in Software Engineering Agents" (January 10, 2026): Polymath works with frontier AI labs to customize and scale environments that simulate production systems, covering deployment, CI/CD and coordination tools such as Slack, Sentry, Linear and GCP; "RL on bad data only degrades model performance"; "The most difficult engineering problems live at the seams".Source
• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source

Build a web viewer for selected runs: a timeline of the agent's actions across repo, CI, deploy, monitoring and chat, with the environment's state alongside and the verifier's verdict at the end. Embed a few public runs on the Horizon-SWE page. Give customers a private version for their own runs.

Showing beats telling for a hard-to-picture product. The viewer becomes the demo, the social clip and a feature customers use to debug their models.

• Public runs reveal task details

  Only replay tasks from the public sample.

• #8 Release a small public sample of Horizon-SWE that anyone can run, and keep the full set held out

• Your websiteHorizon-SWE post (published February 2, 2026, updated February 6): a containerized software company with a running app, live synthetic traffic and 73 tools over MCP, a monorepo with 20,000+ human-authored commits, and 50 tasks such as multi-service rollouts, incident response, API migrations and flaky-test fixes; graded on feature correctness, deployment and DevOps, and engineering quality; eight frontier models run in the OpenHands harness, with a best pass rate of 25.5% and a best partial-credit score of 60.4; "even the strongest models complete only about one in four runs successfully end-to-end". A case study shows a deployment failing on a missing database migration. No leaderboard, dataset or repository is linked.Source
• Your websiteHomepage (checked November 9, 2026): headline "Simulation environments for training & evaluating autonomous agents"; "Polymath is a data lab focused on increasing the reliability and autonomy of AI agents"; says it works with the leading model labs and is backed by Base10, Cervin Ventures and Y Combinator; sections for Applications & Services, Data, Tasks, Verifiers and Agent(s). It does not mention or link Horizon-SWE or any result, and the only contact is a "Get in touch" email link, shown three times; no form, booking link or newsletter.Source