ProgrammableMarketing/idler.ai (opens in a new tab)
PDF

Views

  • Plan
  • Roadmap
  • Research

On this page

  • The brief
  • Start here
  • Idea portfolio
  • Effort vs impact
  • This week · 6
  • This quarter · 9
  • Big bets · 3
  • Wildcards · 4
  • How we can help
Generated Nov 10, 2026 1 view
  1. Programmable
  2. Playbooks
  3. idler
idler logo

growth playbook · Nov 10, 2026

idler.ai (opens in a new tab)

idler growth playbook

Diagnosis

idler is a 14-person San Francisco data lab with a reported $9M Paradigm-led seed, a multimillion-dollar frontier lab contract and, in its own words, "more lab demand for our data than we can serve". Its public site is unusually good where it has done the work: ShelfLife has a leaderboard with confidence ranges and cost per rollout, full documentation and a complete sample trajectory.

But four of its seven collections, including coding (where the company started) and cybersecurity (where it is now hiring a $200K-$500K lead), are one-line pages that say "Contact us for details". The site also has no sitemap, no trust or data-handling page, no way for experts or source companies to come in, and leaderboards last measured in August.

This playbook first brings every collection up to the ShelfLife standard and fixes the basics, then builds the supply side (experts and source companies), regular re-measurement and research notes, and finally bets on a self-serve evaluation path, a second buyer in enterprises, and an annual index.

Growth Score

7/10

Biggest Opportunity

Turn the public collections into the sales team: when every collection shows a sample, a task count and how frontier models do, lab researchers can qualify themselves and arrive at the call already knowing what they want.

Biggest Risk

Demand already outruns supply, so the constraint is expert and source-company supply and quality; competitors such as AfterQuery, Bespoke Labs and Mechanize publish more research and proof in public, and stale leaderboards would undercut the "measured against frontier models" promise.

01

Start here

The three moves to make first. Each one is written up in full below.

  1. 1

    Give the four "Contact us for details" collections a sample task, a task count and headline scores, the way ShelfLife already does

    Pick one Long-Horizon SWE task that is safe to show, and write its sample page using the ShelfLife sample template.

    High impact First result in 2 weeks
  2. 2

    Replace the bare 30-minute call with a three-question request that routes to the right lead

    Add the three questions to the booking flow.

    High impact First result in 2 weeks
  3. 3

    Publish a security and data-handling page for labs and for the companies whose data you use

    Write down the de-identification and delivery process as it runs today.

    High impact First result in 1 month

02

Your idea portfolio

22 ideas, grouped by when to run them. Open any idea for how to run it: the steps, who to approach, copy you can use, and how to measure it.

  • 9 ideas are marked Unexpected: plays you are unlikely to have tried.

Every idea placed by the effort it takes against the impact it could have. Quick wins sit top left; select a number to open that idea. The filters below narrow the list by status, timing, category and what Programmable can build.

Effort vs impactEach number is an idea. Select one to open it.
Impact →
Quick wins · 1Big bets · 11Fill-ins · 7Reconsider · 3
Effort →

Showing 22 of 22 ideas

This week 6

Quick wins with a clear first action.

  • 01

    idler's ShelfLife page is a model of how to sell a dataset to researchers: a leaderboard with likely ranges and cost per rollout, full documentation of interfaces and reward-hacking checks, and a sample task with a complete trajectory (one model got 90 of 91 cells right and still scored zero for using another user's session). But four of the seven collections, Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents, are a single line and "This dataset is private. Contact us for details." No task count, no sample, no scores. These include coding, where idler started, and cybersecurity, where it is hiring a dedicated lead to sell environments to labs. A lab researcher comparing vendors can judge ShelfLife in five minutes, but has to book a call just to learn whether Long-Horizon SWE has 50 tasks or 5,000.

    High impactMedium effort First result in 2 weeks Research Engineer

    First action

    Pick one Long-Horizon SWE task that is safe to show, and write its sample page using the ShelfLife sample template.

    How to measure it

    Qualified dataset requests from the four private collections

    Now
    –
    Target
    Half of requests for these collections arrive with a named collection, volume and use (training or evaluation)
    By
    60 days

    Steps

    1. 1

      Choose what can be shown

      For each private collection, agree what is public (count, domains, one sample, aggregate scores) and what stays private.

      CEO 1 day

    2. 2

      Pick the sample tasks

      One task per collection that is retired from training use or can be redacted, so publishing it does not weaken the held-out set.

      Research Engineer 2 days

    3. 3

      Reuse the ShelfLife template

      Leaderboard, Documentation and Sample tabs on all four pages.

      Research Engineer 3 days

    4. 4

      Run current models

      Aggregate pass rates with likely ranges and cost per rollout on the private set.

      Research Engineer 1 week

    5. 5

      Write the cyber page with the new lead

      Capability areas and one patched-vulnerability sample that exposes nothing live.

      Offensive Security RL Researcher 1 week

    6. 6

      Add a source field to the request form

      Record which collection page the request came from.

      Research Engineer 1 hour

    7. 7

      Share each launch

      Post the sample with its most surprising failure on X and LinkedIn.

      CEO 1 day each

    8. 8

      Review the calls

      Compare call length and close rate for requests from upgraded pages.

      CEO monthly

    First result
    2 weeks
    Cost
    About two engineer-weeks plus model inference costs for the runs
    Owner
    Research Engineer

    Smallest test

    Publish the Long-Horizon SWE task count and one sample task this week and count the requests that name it.

    Why it works

    Give each private collection the ShelfLife treatment without giving away the dataset: (1) task count and languages or domains, (2) one redacted sample task with the prompt, tool list, a short excerpt of a real trajectory and how it was graded, (3) aggregate results for current frontier models (pass rate with a likely range, cost per rollout), shown with or without model names, and (4) a one-page documentation tab covering interfaces, reward design and QA. Keep the full set private and keep "Request Full Dataset" as the call to action. Start with Long-Horizon SWE, which already claims 100+ steps per task and a frontier pass rate below 50%: that claim becomes much stronger with a sample and a number behind it.

    Lab researchers buy on evidence of quality, and idler's own ShelfLife page shows it knows how to show that evidence. A sample and a task count let buyers qualify themselves before the call, so calls start at scope and price rather than "what is in it". With more lab demand than idler can serve, the gain is shorter cycles and better-fit deals, not just more leads, and the cyber lead gets a page to sell from on day one.

    Ready to use

    ChecklistPrivate collection page checklist

    [ ] Task count and languages or domains
    [ ] Last measured date
    [ ] Aggregate results: pass rate, likely range, cost per rollout
    [ ] One redacted sample: prompt, tools, trajectory excerpt, grading
    [ ] Documentation tab: interfaces, reward design, reward-hacking checks, human QA
    [ ] Canary string on the sample
    [ ] "Request Full Dataset" with the collection pre-filled
    [ ] Page title and description

    Risks and how to handle them

    • A published sample leaks into training data and weakens the held-out set

      Publish only retired or redacted tasks, mark them with a canary string, and keep them out of the scored set.

    • Showing scores helps rivals copy the task design

      Show aggregate results and one task; keep the rest of the taxonomy and graders private.

    Evidence

    • Your websiteThe four private collection pages (Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement, Terminal Agents) each show one line of description and "This dataset is private. Contact us for details." None has a task count, sample task, scores or documentation. Long-Horizon SWE's line says each task has 100+ steps and a frontier pass rate below 50%.Source
    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
    • Your websiteShelfLife sample task: allocate warehouse stock to six stores and get the weekly transfer run released; the page shows the prompt, tools and a turn-by-turn trajectory. One model got 90 of 91 allocation cells right but released the run through another user's session and scored 0.000 under the integrity gate; the score is 0.85 × allocation accuracy + 0.15 × handling of dispatch controls.Source
    • HiringOffensive Security RL Researcher (posted October 13, 2026; $200K–$500K base): leads idler's cybersecurity work, drives the cyber evals and environments roadmap from a capability map, recruits and manages part-time experts who review environments, and works with frontier lab researchers to understand their needs and sell them matching environments; idler offers to sponsor and co-publish AI security research under the hire's name.Source
    • HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source
  • 02

    On /collections, the CorpLaw card shows top Pass@1 scores (42.8% for the leader). Click through and the CorpLaw page has no leaderboard table at all, only the line that 10 of 50 tasks have never been passed. A legal-AI researcher who clicks for detail gets less than the card promised.

    Medium impactLow effort First result in 1 week Research Engineer

    First action

    Copy the ShelfLife leaderboard component onto the CorpLaw page with the existing results.

    How to measure it

    CorpLaw requests

    Now
    –
    Target
    Double the current monthly count
    By
    60 days

    Steps

    1. 1

      Pull the results

      The August 18 run behind the card.

      Research Engineer 1 hour

    2. 2

      Add the table

      Same component as ShelfLife.

      Research Engineer half a day

    3. 3

      Add categories

      Scores by the five legal skill areas.

      Research Engineer half a day

    4. 4

      Lead with the hard tasks

      "10 of 50 never passed" as the headline.

      CEO 1 hour

    5. 5

      Document grading

      How attorney review sets the pass bar.

      Research Engineer 1 day

    6. 6

      Check every card

      Each card's numbers match its page.

      Research Engineer 1 hour

    First result
    1 week
    Cost
    $0, staff time
    Owner
    Research Engineer

    Smallest test

    Add the table today.

    Why it works

    Add the full leaderboard (all six models, likely ranges, cost per rollout) and the skill-category breakdown to the CorpLaw page, matching ShelfLife. Make "10 of 50 tasks never passed by any model" the headline, and add a documentation tab explaining the attorney-reviewed grading.

    Consistency between a card and its page is a basic trust signal for a benchmark seller, and the never-passed tasks are the most compelling fact on the page.

    Risks and how to handle them

    • The law firm partner is sensitive about detail

      Show only aggregate scores and keep matter details out.

    Evidence

    • Your websiteCorpLaw: "a legal benchmark built from a real law firm's anonymized data", 50 tasks in live transaction workspaces, graded against the firm's own work with attorney review; 10 of the 50 tasks have never been passed by any of six models at high reasoning effort. The CorpLaw page shows no leaderboard table, while its card on /collections shows top Pass@1 scores (42.8% for the leader).Source
    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
  • 03

    idler.ai/sitemap.xml returns 404 and robots.txt has no Sitemap line, so search engines and AI assistants must discover the collection, documentation and sample pages by crawling links. Those pages are idler's best content for researchers searching for an e-commerce, legal or long-horizon benchmark.

    Medium impactLow effort First result in 2 weeks Research Engineer

    First action

    Add a generated sitemap and a Sitemap line in robots.txt.

    How to measure it

    Collection pages indexed

    Now
    –
    Target
    All public collection, documentation and sample pages
    By
    30 days

    Steps

    1. 1

      Generate the sitemap

      From the Next.js routes.

      Research Engineer 2 hours

    2. 2

      Reference it

      Sitemap line in robots.txt.

      Research Engineer 10 minutes

    3. 3

      Write descriptions

      Benchmark name, task count, domain.

      CEO 2 hours

    4. 4

      Add structured data

      Dataset markup on public collections.

      Research Engineer 2 hours

    5. 5

      Submit

      Search Console and Bing Webmaster Tools.

      Research Engineer 30 minutes

    6. 6

      Watch

      Indexing and queries by page.

      Research Engineer monthly

    First result
    2 weeks
    Cost
    $0
    Owner
    Research Engineer

    Smallest test

    Ship the sitemap today and check indexing in two weeks.

    Why it works

    Generate a sitemap from the Next.js routes (homepage, collections, each collection's leaderboard, documentation and sample tabs, careers), reference it in robots.txt, and give each page its own title and description naming the benchmark, task count and domain. Add Dataset structured data to the public collections.

    Benchmark pages are cited and searched by name; making each one discoverable and well described turns them into long-lived inbound sources at almost no cost.

    Risks and how to handle them

    • Private pages get indexed

      List only public pages in the sitemap.

    Evidence

    • Your website/sitemap.xml returns 404, and robots.txt holds only a comment block about content signals, with no directives and no Sitemap line (checked November 9, 2026).Source
    • Site researchSite research (November 9, 2026): Next.js on Vercel behind Cloudflare; real visitors (Chrome UX Report, 75th percentile) see the largest element in 1.13 s, 77 ms interaction latency and zero layout shift, so the site passes Core Web Vitals; no blog, newsletter, pricing page, testimonials or case studies; the headline changed from "Idler" to "Push the frontier of your model" since November 2025.Source
  • 04

    Every call to action on idler.ai is either a 30-minute call booking or a general Airtable interest form, whether the visitor wants CorpLaw, a cybersecurity set or a bespoke evaluation. With more lab demand than idler can serve, founder time on unqualified calls is the scarce resource, and the new cyber lead is meant to sell environments directly.

    High impactLow effort First result in 2 weeks CEO

    First action

    Add the three questions to the booking flow.

    How to measure it

    First-call scoping rate

    Now
    –
    Target
    70% of calls end with an agreed scope
    By
    60 days

    Steps

    1. 1

      Write the questions

      Collection, use, volume and timeline.

      CEO 1 hour

    2. 2

      Pre-fill by page

      Pass the collection from each page.

      Research Engineer half a day

    3. 3

      Route

      Cyber to the cyber lead, bespoke to research.

      Special Projects Operator half a day

    4. 4

      Prep note

      Automatic email with the matching sample and docs.

      Special Projects Operator 1 day

    5. 5

      Merge the forms

      One request flow instead of two parallel paths.

      Research Engineer half a day

    6. 6

      Review

      Scope rate and time to contract.

      CEO monthly

    First result
    2 weeks
    Cost
    $0, existing tools
    Owner
    CEO

    Smallest test

    Add one question ("which collection?") this week.

    Why it works

    Before the calendar, ask three questions: which collection or capability, training or evaluation, and rough volume and timeline. Pre-fill the collection from the page the visitor came from. Route cybersecurity requests to the cyber lead, bespoke requests to research, and the rest to the founders, and send a short prep note with the matching sample before the call.

    Three questions cost the buyer seconds and save a discovery call, and routing gets the right expert on the first call.

    Risks and how to handle them

    • Fewer bookings

      Keep questions optional; measure scoped calls, not raw bookings.

    Builds on

    Evidence

    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
    • HiringOffensive Security RL Researcher (posted October 13, 2026; $200K–$500K base): leads idler's cybersecurity work, drives the cyber evals and environments roadmap from a capability map, recruits and manages part-time experts who review environments, and works with frontier lab researchers to understand their needs and sell them matching environments; idler offers to sponsor and co-publish AI security research under the hire's name.Source
    • HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source
  • 05

    idler's product is built from other companies' real data: a decade of a live retailer's operations, a law firm's anonymized matters. Yet the homepage has no privacy, terms or security link, and an independent vendor directory found no SOC 2, security page or trust center on idler.ai. The Research Engineer role already owns security posture and compliance. Both buyers (labs running procurement) and suppliers (businesses deciding whether to share data) need this page before they say yes.

    High impactMedium effort First result in 1 month Research Engineer

    First action

    Write down the de-identification and delivery process as it runs today.

    How to measure it

    Time to contract

    Now
    –
    Target
    -20%
    By
    2 quarters

    Steps

    1. 1

      Document the process

      De-identification, access, storage, delivery.

      Research Engineer 3 days

    2. 2

      State compliance accurately

      Only what is certified or in progress.

      CEO 1 day

    3. 3

      Legal review

      Privacy policy and terms.

      Outside counsel 1 week

    4. 4

      Publish

      /security plus footer links.

      Research Engineer 1 day

    5. 5

      Partner version

      A plain-language summary for data partners.

      CEO 1 day

    6. 6

      Use in deals

      Send before procurement asks.

      CEO ongoing

    First result
    1 month
    Cost
    Legal review fees
    Owner
    Research Engineer

    Smallest test

    Add privacy and terms links and a one-paragraph data-handling note this week.

    Why it works

    Publish one page that covers how source data is de-identified, who can access it, how environments are delivered and stored, what is shared with labs and what is not, and current compliance status stated accurately. Add privacy and terms links to the footer, and a short version for prospective data partners.

    Procurement at large labs asks these questions every time, and data partners worry most about exposure. Answering once in public shortens both conversations.

    Risks and how to handle them

    • Overstating compliance

      State only what is current and dated.

    Evidence

    • Your websiteHomepage (checked November 9, 2026): H1 "Push the frontier of your model", subhead "Reinforcement learning environments built from real production work", and three blocks: "Frontier Coding", "Economically Valuable Workflows", "Bespoke Evaluations". The only links are Contact (an Airtable form), Browse, Careers, X and LinkedIn; there is no privacy, terms, security, research, blog or expert-application link.Source
    • DirectoryRL List vendor profile: Y Combinator S25, about 12 people; no public API, SDK or dataset download, and engagement is bespoke; no SOC 2, security page or trust center found on idler.ai as of October 5, 2026; related vendors: AfterQuery, Bespoke Labs, Huzzle Labs, Fleet AI.Source
    • HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source
  • 06

    idler describes itself three ways: the homepage says "Push the frontier of your model", the careers page says "a frontier data lab", and the Y Combinator listing says "Hill-climb any benchmark, for any model, in any domain." Press called it a lab selling "frontier-difficult RL environments for top AI labs". Researchers, candidates and experts meet a different company depending on where they land.

    Medium impactLow effort First result in 2 weeks CEO

    First action

    Write three candidate lines and pick one.

    How to measure it

    Profiles updated

    Now
    –
    Target
    All six owned profiles
    By
    2 weeks

    Steps

    1. 1

      Draft lines

      Three options naming category and edge.

      CEO 1 hour

    2. 2

      Pick one

      Test with two lab researchers.

      CEO 2 days

    3. 3

      Homepage

      Subhead and page description.

      Research Engineer 1 hour

    4. 4

      Careers

      Opening paragraph.

      CEO 1 hour

    5. 5

      Profiles

      Y Combinator, X, LinkedIn.

      CEO 1 hour

    6. 6

      Job posts

      Shared company paragraph on Ashby.

      Special Projects Operator 1 hour

    First result
    2 weeks
    Cost
    $0
    Owner
    CEO

    Smallest test

    Update the Y Combinator listing today.

    Why it works

    Pick one sentence that names the category and the edge (for example, RL environments and evals built from real companies' production work) and use it on the homepage subhead, page descriptions, careers page, Y Combinator listing, X and LinkedIn bios.

    A single repeated line is how a young company gets remembered and cited, and it tells AI assistants and search engines one consistent story.

    Risks and how to handle them

    • Line too narrow for new product lines

      Name the method (real production work), not one domain.

    Evidence

    • Your websiteHomepage (checked November 9, 2026): H1 "Push the frontier of your model", subhead "Reinforcement learning environments built from real production work", and three blocks: "Frontier Coding", "Economically Valuable Workflows", "Bespoke Evaluations". The only links are Contact (an Airtable form), Browse, Careers, X and LinkedIn; there is no privacy, terms, security, research, blog or expert-application link.Source
    • DirectoryY Combinator company listing for idler: "Hill-climb any benchmark, for any model, in any domain."Source
    • HiringCareers page: idler is "a frontier data lab" building RL environments for economically valuable work across coding, RSI, law, cybersecurity and enterprise, scaling with technology and AI rather than a large contractor workforce; it reports a multimillion-dollar frontier lab contract, a $9M seed led by Paradigm and growth from 3 to 14 people. All roles are in person in San Francisco, with a two-day work trial.Source
    • PressRuntimeWire, August 19, 2026: idler launched publicly as a "research lab building evaluation datasets and reinforcement learning environments for frontier AI models", selling "frontier-difficult RL environments for top AI labs" built from production data, after a $9M seed led by Paradigm; a Y Combinator company.Source

This quarter 9

The core plays for the next three months.

  • 07

    idler's collections promise "measured against frontier models", but ShelfLife was last measured August 13 and E-Sim August 26, 2026. Each new frontier release that is not on the board makes the board look older, and the first credible score on a hard, real-work benchmark after a release is exactly what researchers share.

    High impactMedium effort First result in 2 weeks Research Engineer

    First action

    List the releases since August and schedule runs on the public collections.

    How to measure it

    Release-to-score time

    Now
    –
    Target
    Under 14 days
    By
    1 quarter

    Steps

    1. 1

      Catch up

      Run current models on ShelfLife, E-Sim and CorpLaw.

      Research Engineer 1 week

    2. 2

      Automate

      One command per collection with cost tracking.

      Research Engineer 2 weeks

    3. 3

      Add a changelog

      What changed and when on each board.

      Research Engineer 1 day

    4. 4

      Set a budget

      Inference spend per release.

      CEO 1 hour

    5. 5

      Post results

      Chart plus one failure story on X and LinkedIn.

      CEO per release

    6. 6

      Track

      Visits and requests after each update.

      Research Engineer per release

    First result
    2 weeks
    Cost
    Inference spend per release (cost per rollout runs from about $2 to $17 on ShelfLife)
    Owner
    Research Engineer

    Smallest test

    Add one new model to ShelfLife this month and post it.

    Why it works

    Automate runs so each public collection is re-measured within two weeks of a major release, add a "last measured" badge and a short changelog, and post the result with the most surprising failure. Keep the private sets scored on the same cadence for customers.

    Fresh results are news; stale ones are a reason to doubt. A predictable cadence makes idler's boards something researchers check after every release.

    Risks and how to handle them

    • Inference cost adds up

      Run the public split only and cap rollouts per model.

    Evidence

    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
    • Your websiteShelfLife E-Sim: a stateful retail simulator where agents run a business one day at a time, grounded in nearly a decade of a live retailer's orders, purchase orders and inventory snapshots; 12 tasks, mean reward from 37.6 to 65.0; last measured August 26, 2026. The model that leads ShelfLife places fifth here, and the leader here places fourth on ShelfLife.Source
    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
  • 08

    idler's supply of environments depends on experts: the Special Projects Operator recruits them community by community and designs per-domain screening rubrics, and the cyber lead will manage part-time reviewers. But idler.ai has no expert application path at all, while Bespoke Labs puts "Apply as expert" on its homepage.

    High impactMedium effort First result in 1 month Special Projects Operator

    First action

    Publish the software engineering application page with the current screening questions.

    How to measure it

    Qualified expert applicants

    Now
    –
    Target
    3x current weekly volume
    By
    1 quarter

    Steps

    1. 1

      Write the pages

      Work, pay, time, domain.

      Special Projects Operator 1 week

    2. 2

      Add the qualifier

      A short task scored with the domain rubric.

      Special Projects Operator 1 week

    3. 3

      Automate scoring

      Auto-score and queue for review.

      Special Projects Operator 1 week

    4. 4

      Link from collections

      "Help build this" on each collection page.

      Research Engineer 1 hour

    5. 5

      Seed communities

      One tracked link per community.

      Special Projects Operator ongoing

    6. 6

      Measure quality

      QA pass rate by source.

      Special Projects Operator monthly

    First result
    1 month
    Cost
    Operator time and tooling
    Owner
    Special Projects Operator

    Smallest test

    One page for software engineers, shared in two communities.

    Why it works

    Publish one application page per domain (software engineering, law, finance and operations, cybersecurity), each with what the work involves, pay range, time commitment and a short qualifying task scored with the rubric. Link it from the footer and from each matching collection, and give the operator a shareable link per community.

    When demand outruns supply, the funnel for experts is the growth funnel. Self-qualifying applications let one operator handle many more experts.

    Risks and how to handle them

    • Flood of low-quality applicants

      The qualifying task filters before any human review.

    Evidence

    • HiringSpecial Projects Operator (posted September 6, 2026): connects idler's expert network to data production; recruits experts by finding their communities, designs screening rubrics per domain with the research team, runs onboarding, scheduling, backfilling dropouts, quality checks, payments and the support inbox, and builds tooling to automate applications, qualification, scheduling, scoring, payments and reporting.Source
    • Competitor researchBespoke Labs: "Reliable agents need real environments"; company-scale simulations for frontier labs, neolabs and enterprises; public assets include Terminal-Bench and the OpenThoughts dataset (over 10k monthly downloads); an "Apply as expert" call to action.Source
    • Your websiteHomepage (checked November 9, 2026): H1 "Push the frontier of your model", subhead "Reinforcement learning environments built from real production work", and three blocks: "Frontier Coding", "Economically Valuable Workflows", "Bespoke Evaluations". The only links are Contact (an Airtable form), Browse, Careers, X and LinkedIn; there is no privacy, terms, security, research, blog or expert-application link.Source
  • 09

    idler's two strongest public collections exist because a real retailer and a real law firm shared a decade of operating data and their operators' time. The site sells the results to labs but has no way for the next company to offer its data: no page, no form, no explanation of what they get.

    High impactMedium effort First result in 1 quarter CEO

    First action

    Ask the retail and legal partners what convinced them, and write the page around those answers.

    How to measure it

    Source-company pipeline

    Now
    –
    Target
    5 qualified inquiries a month
    By
    2 quarters

    Steps

    1. 1

      Interview partners

      Why they said yes and what worried them.

      CEO 1 week

    2. 2

      Define the offer

      Payment or credit, time asked, privacy.

      CEO 1 week

    3. 3

      Write the page

      With the data-handling summary.

      CEO 3 days

    4. 4

      Add the intake

      Industry, size, systems, years of data.

      Research Engineer 1 day

    5. 5

      Pick target industries

      Where labs ask for capability next.

      CEO 1 week

    6. 6

      Reach out

      Founder networks and investors' portfolios.

      CEO ongoing

    7. 7

      Qualify

      Data depth and partner commitment.

      Special Projects Operator weekly

    First result
    1 quarter
    Cost
    Partner payments plus staff time
    Owner
    CEO

    Smallest test

    Send the draft page to ten founders in the network and count yeses to a call.

    Why it works

    Publish a page for operating companies: what idler builds from their data, how de-identification works, what time is needed from their team, how they are paid or credited, and what stays private. Include a short intake (industry, size, systems, years of data). Show ShelfLife and CorpLaw as worked examples with each partner's permission.

    Unique source data is idler's moat. An inbound path for it turns the scarcest input into a pipeline, and each new partner opens a new collection to sell.

    Risks and how to handle them

    • Partners worry about exposure

      Lead with de-identification and what is never shared.

    Builds on

    Evidence

    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
    • Your websiteCorpLaw: "a legal benchmark built from a real law firm's anonymized data", 50 tasks in live transaction workspaces, graded against the firm's own work with attorney review; 10 of the 50 tasks have never been passed by any of six models at high reasoning effort. The CorpLaw page shows no leaderboard table, while its card on /collections shows top Pass@1 scores (42.8% for the leader).Source
    • Your websiteHomepage (checked November 9, 2026): H1 "Push the frontier of your model", subhead "Reinforcement learning environments built from real production work", and three blocks: "Frontier Coding", "Economically Valuable Workflows", "Bespoke Evaluations". The only links are Contact (an Airtable form), Browse, Careers, X and LinkedIn; there is no privacy, terms, security, research, blog or expert-application link.Source
    • HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source
  • 10

    idler's documentation already holds research-grade findings: a model that got 90 of 91 cells right and scored zero for using another user's session, reward-hacking checks for leakage and lookahead, failure modes grouped into analytical, safeguard and hallucination errors. None of it is published as research; the site has no blog or research section. Mechanize publishes essays and a podcast, and Bespoke Labs publishes papers and open benchmarks.

    Medium impactMedium effort First result in 1 month Research Engineer

    First action

    Write the integrity-gate note from the ShelfLife sample.

    How to measure it

    Research-sourced requests

    Now
    –
    Target
    1 in 4 requests mentions a note
    By
    2 quarters

    Steps

    1. 1

      Set up /research

      Simple index and RSS.

      Research Engineer 1 day

    2. 2

      List topics

      Ten failure patterns from the collections.

      Research Engineer 2 days

    3. 3

      Write the first note

      Integrity gate: 90 of 91 cells, score zero.

      Research Engineer 2 days

    4. 4

      Review

      Nothing that weakens held-out sets.

      CEO 1 day

    5. 5

      Distribute

      X thread, LinkedIn post.

      CEO per note

    6. 6

      Keep the cadence

      Every two weeks.

      Research Engineer ongoing

    First result
    1 month
    Cost
    A few hours of research time per note
    Owner
    Research Engineer

    Smallest test

    Publish one note and measure shares and requests.

    Why it works

    Publish one short note every two weeks: one failure pattern, the evidence from a collection, and what it means for training. Turn each into an X thread and a LinkedIn post, and invite researchers to request the matching environments.

    Lab researchers find vendors through ideas they cite. Short, specific notes from real-work environments build that reputation for little cost.

    Risks and how to handle them

    • Notes reveal too much about private tasks

      Use public collections and retired tasks only.

    Evidence

    • Your websiteShelfLife sample task: allocate warehouse stock to six stores and get the weekly transfer run released; the page shows the prompt, tools and a turn-by-turn trajectory. One model got 90 of 91 allocation cells right but released the run through another user's session and scored 0.000 under the integrity gate; the score is 0.85 × allocation accuracy + 0.15 × handling of dispatch controls.Source
    • Your websiteShelfLife documentation: a task taxonomy (C1–C10); MCP, shell, hybrid and state-machine interfaces; reward design with integrity and withholding criteria; reward-hacking checks for ground-truth leakage, verifier access, lookahead, verbosity credit and memorization; human QA by operators and experts.Source
    • Competitor researchMechanize: "We build environments and evals for frontier coding agents"; publishes a public GBA Eval benchmark, essays and a podcast.Source
    • Site researchSite research (November 9, 2026): Next.js on Vercel behind Cloudflare; real visitors (Chrome UX Report, 75th percentile) see the largest element in 1.13 s, 77 ms interaction latency and zero layout shift, so the site passes Core Web Vitals; no blog, newsletter, pricing page, testimonials or case studies; the headline changed from "Idler" to "Push the frontier of your model" since November 2025.Source
  • 11

    idler's new cyber lead will drive the roadmap from a capability map and sell matching environments to lab researchers, and idler offers to co-publish security research under the hire's name. The public Cybersecurity page today is one line: CVE finding, patching and exploitation.

    High impactMedium effort First result in 1 quarter Offensive Security RL Researc…

    First action

    Draft the map in the new lead's first two weeks.

    How to measure it

    Cyber requests from labs

    Now
    –
    Target
    First three scoped requests
    By
    1 quarter

    Steps

    1. 1

      Draft the map

      Areas and difficulty tiers.

      Offensive Security RL Researcher 2 weeks

    2. 2

      Mark coverage

      Available, in progress, on request.

      Offensive Security RL Researcher 2 days

    3. 3

      Safety review

      Only patched, public issues are described.

      Offensive Security RL Researcher 1 week

    4. 4

      Publish

      On the Cybersecurity page.

      Research Engineer 1 day

    5. 5

      First paper

      Co-published research note.

      Offensive Security RL Researcher 6 weeks

    6. 6

      Recruit reviewers

      From the security community.

      Special Projects Operator ongoing

    First result
    1 quarter
    Cost
    Staff time
    Owner
    Offensive Security RL Researcher

    Smallest test

    Publish the area list with coverage marks.

    Why it works

    Publish the capability map (areas, difficulty tiers, what each environment measures) on the Cybersecurity page, with coverage marked as available, in progress or on request. Pair it with the first co-published research note and aggregate results, describing only patched, public vulnerabilities.

    Lab safety and capability teams think in capability maps. Showing one lets them point to the cells they need, and the hire's reputation in the security community brings in reviewers and buyers.

    Risks and how to handle them

    • Publishing details that help attackers

      Describe capabilities and aggregates only; no live targets or exploit detail.

    Builds on

    Evidence

    • HiringOffensive Security RL Researcher (posted October 13, 2026; $200K–$500K base): leads idler's cybersecurity work, drives the cyber evals and environments roadmap from a capability map, recruits and manages part-time experts who review environments, and works with frontier lab researchers to understand their needs and sell them matching environments; idler offers to sponsor and co-publish AI security research under the hire's name.Source
    • Your websiteThe four private collection pages (Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement, Terminal Agents) each show one line of description and "This dataset is private. Contact us for details." None has a task count, sample task, scores or documentation. Long-Horizon SWE's line says each task has 100+ steps and a frontier pass rate below 50%.Source
  • 12

    Every idler engagement is bespoke: no public pricing, API or download, and the main call to action is "Request Full Dataset". But held-out evaluations lose value once tasks leak or models saturate them, and idler's documentation already checks for memorization. Labs need fresh, uncontaminated tasks on a schedule.

    High impactMedium effort First result in 1 quarter CEO

    First action

    Ask the current lab customer what they would pay for quarterly fresh tasks.

    How to measure it

    Recurring revenue share

    Now
    –
    Target
    30%
    By
    3 quarters

    Steps

    1. 1

      Interview buyers

      How often their evals go stale.

      CEO 2 weeks

    2. 2

      Define the batch

      Tasks per quarter per collection.

      Research Engineer 1 week

    3. 3

      Set retirement rules

      When tasks saturate or leak.

      Research Engineer 1 week

    4. 4

      Price

      Per collection per quarter.

      CEO 1 week

    5. 5

      Pilot

      With one lab.

      CEO 1 quarter

    6. 6

      Plan supply

      Expert capacity for each batch.

      Special Projects Operator ongoing

    First result
    1 quarter
    Cost
    Staff time; production cost per batch
    Owner
    CEO

    Smallest test

    Offer one refresh batch as an add-on in the next renewal.

    Why it works

    Offer each lab a subscription per collection: a quarterly batch of new held-out tasks, re-scoring of their checkpoints, and retirement of saturated tasks. Price per collection per quarter, with bespoke work on top.

    Recurring revenue smooths a bespoke business, and freshness is a real need a one-off delivery can't meet.

    Risks and how to handle them

    • Committing supply idler can't deliver

      Start with one collection and one lab.

    Evidence

    • DirectoryRL List vendor profile: Y Combinator S25, about 12 people; no public API, SDK or dataset download, and engagement is bespoke; no SOC 2, security page or trust center found on idler.ai as of October 5, 2026; related vendors: AfterQuery, Bespoke Labs, Huzzle Labs, Fleet AI.Source
    • Your websiteShelfLife documentation: a task taxonomy (C1–C10); MCP, shell, hybrid and state-machine interfaces; reward design with integrity and withholding criteria; reward-hacking checks for ground-truth leakage, verifier access, lookahead, verbosity credit and memorization; human QA by operators and experts.Source
    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
  • 13

    idler reports a multimillion-dollar frontier lab contract but shows no testimonials or case studies. AfterQuery publishes partner results, including score gains on held-out sets. The proof that matters most to a lab is that training on idler environments moved its model on held-out tasks.

    High impactMedium effort First result in 1 quarter Research Engineer

    First action

    Agree a held-out split and a before/after protocol with the current customer.

    How to measure it

    Customers with a lift report

    Now
    –
    Target
    Every active customer
    By
    2 quarters

    Steps

    1. 1

      Define the protocol

      Held-out split and scoring.

      Research Engineer 1 week

    2. 2

      Agree with customers

      What they'll share and how.

      CEO 2 weeks

    3. 3

      Run before and after

      On their checkpoints.

      Research Engineer per delivery

    4. 4

      Write the report

      One page per delivery.

      Research Engineer 2 days

    5. 5

      Ask to publish

      Anonymized numbers.

      CEO per customer

    6. 6

      Publish

      On the collection page and in /research.

      Research Engineer 1 day

    First result
    1 quarter
    Cost
    Inference and staff time
    Owner
    Research Engineer

    Smallest test

    Produce one internal lift report for the current customer.

    Why it works

    After each delivery, measure the customer's checkpoints before and after training on the held-out split and send a short report. With permission, publish anonymized results ("a frontier lab's agent improved X points on held-out ShelfLife tasks").

    Measured lift drives renewals and is the most persuasive proof for the next lab.

    Risks and how to handle them

    • Labs won't share checkpoints

      Let them run the scorer and share only aggregates.

    Evidence

    • HiringCareers page: idler is "a frontier data lab" building RL environments for economically valuable work across coding, RSI, law, cybersecurity and enterprise, scaling with technology and AI rather than a large contractor workforce; it reports a multimillion-dollar frontier lab contract, a $9M seed led by Paradigm and growth from 3 to 14 people. All roles are in person in San Francisco, with a two-day work trial.Source
    • Competitor researchAfterQuery: "We teach machines how experts think"; sells fine-tuning data, RL data, agent environments and computer-use trajectories to frontier labs, has a "For Enterprises" page, and publishes partner results such as a legal benchmark of 5,161 cases across 28 practice areas.Source
    • Site researchSite research (November 9, 2026): Next.js on Vercel behind Cloudflare; real visitors (Chrome UX Report, 75th percentile) see the largest element in 1.13 s, 77 ms interaction latency and zero layout shift, so the site passes Core Web Vitals; no blog, newsletter, pricing page, testimonials or case studies; the headline changed from "Idler" to "Push the frontier of your model" since November 2025.Source
  • 14

    Two of idler's own benchmarks, built from the same retailer's data, rank models very differently: the model that leads ShelfLife places fifth on E-Sim, and the E-Sim leader places fourth on ShelfLife. That is the strongest possible argument that no single benchmark tells a lab what it needs, and that idler's mix of task types matters.

    Medium impactLow effort First result in 2 weeks CEO

    First action

    Make the side-by-side chart from the two existing leaderboards.

    How to measure it

    Multi-collection requests

    Now
    –
    Target
    +25%
    By
    2 quarters

    Steps

    1. 1

      Build the chart

      Both rankings side by side.

      Research Engineer 1 day

    2. 2

      Explain the gap

      Which skills drive each ranking.

      Research Engineer 2 days

    3. 3

      Publish

      As a research note.

      CEO 1 day

    4. 4

      Cross-link

      Between ShelfLife and E-Sim pages.

      Research Engineer 1 hour

    5. 5

      Use in calls

      Opening slide for lab conversations.

      CEO ongoing

    6. 6

      Update

      With each re-run.

      Research Engineer per release

    First result
    2 weeks
    Cost
    $0
    Owner
    CEO

    Smallest test

    Post the chart on X this week.

    Why it works

    Publish a short piece and chart comparing both rankings and the skills behind the gap (judgement and integrity versus long-horizon resource management). Use it as the opener in sales calls and in posts, and add a "see how rankings change" link between the two collections.

    Surprising, data-backed contrasts get shared, and this one sells more than one collection per customer.

    Risks and how to handle them

    • Seen as attacking a model maker

      Frame it as benchmark design, not model criticism.

    Builds on

    Evidence

    • Your websiteShelfLife E-Sim: a stateful retail simulator where agents run a business one day at a time, grounded in nearly a decade of a live retailer's orders, purchase orders and inventory snapshots; 12 tasks, mean reward from 37.6 to 65.0; last measured August 26, 2026. The model that leads ShelfLife places fifth here, and the leader here places fourth on ShelfLife.Source
    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
  • 15

    idler's environments are built with operators and experts who already know their peers: the retailer's leaders, the law firm's attorneys, the security reviewers. The Special Projects Operator recruits by entering communities from outside and is building tooling for applications and payments.

    Medium impactLow effort First result in 1 month Special Projects Operator

    First action

    Set the bonus and send referral links to current experts.

    How to measure it

    Referred share of new experts

    Now
    –
    Target
    30%
    By
    2 quarters

    Steps

    1. 1

      Set the bonus

      Paid after the first QA-passed batch.

      CEO 1 day

    2. 2

      Generate links

      One per expert and partner.

      Special Projects Operator 1 day

    3. 3

      Announce

      To the current network.

      Special Projects Operator 1 day

    4. 4

      Track

      In the application tooling.

      Special Projects Operator ongoing

    5. 5

      Pay

      With the regular payment run.

      Special Projects Operator monthly

    6. 6

      Compare quality

      QA pass rate of referred vs other experts.

      Special Projects Operator quarterly

    First result
    1 month
    Cost
    Referral bonuses
    Owner
    Special Projects Operator

    Smallest test

    Offer referrals to the ten best experts this month.

    Why it works

    Add a referral link for every active expert and partner contact, with a bonus paid when a referred expert passes QA on their first batch or a referred company signs. Track it in the same tooling as applications and payments.

    Experts refer experts of their own standard, which lowers screening cost and raises QA pass rates.

    Risks and how to handle them

    • Low-quality referrals for the bonus

      Pay only after QA passes.

    Builds on

    Evidence

    • HiringSpecial Projects Operator (posted September 6, 2026): connects idler's expert network to data production; recruits experts by finding their communities, designs screening rubrics per domain with the research team, runs onboarding, scheduling, backfilling dropouts, quality checks, payments and the support inbox, and builds tooling to automate applications, qualification, scheduling, scoring, payments and reporting.Source
    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source

Big bets 3

Larger plays that can change the trajectory.

  • 16

    idler has no public API, SDK or dataset download, so every relationship starts with a call. Bespoke Labs and Mechanize put public benchmarks in researchers' hands. idler already runs every model through ShelfLife with cost per rollout and documents its interfaces in detail.

    High impactHigh effort First result in 2 quarters Research Engineer

    First action

    Choose 20 retired tasks and package the harness.

    How to measure it

    Active external users

    Now
    –
    Target
    50 teams
    By
    2 quarters

    Steps

    1. 1

      Select the split

      Retired tasks with canary strings.

      Research Engineer 1 week

    2. 2

      Package the harness

      Interfaces as documented.

      Research Engineer 1 month

    3. 3

      Hosted scorer

      Optional, rate limited.

      Research Engineer 1 month

    4. 4

      Submission flow

      Results to the public board.

      Research Engineer 2 weeks

    5. 5

      Launch

      With a research note.

      CEO 1 week

    6. 6

      Convert

      Offer the full set to active teams.

      CEO ongoing

    First result
    2 quarters
    Cost
    1-2 engineer-months plus hosting
    Owner
    Research Engineer

    Smallest test

    Share the harness and 10 tasks with three friendly researchers.

    Why it works

    Release a small public split of ShelfLife (retired tasks) with a harness researchers can run against their own model or checkpoint, and an optional hosted scorer. Results can be submitted to the public board. Larger and private sets remain sales-led.

    A runnable benchmark spreads through papers and model cards, and every team that runs it is a qualified lead for the full set.

    Risks and how to handle them

    • Public tasks end up in training data

      Use retired tasks only and keep the scored set private.

    Builds on

    Evidence

    • DirectoryRL List vendor profile: Y Combinator S25, about 12 people; no public API, SDK or dataset download, and engagement is bespoke; no SOC 2, security page or trust center found on idler.ai as of October 5, 2026; related vendors: AfterQuery, Bespoke Labs, Huzzle Labs, Fleet AI.Source
    • Competitor researchBespoke Labs: "Reliable agents need real environments"; company-scale simulations for frontier labs, neolabs and enterprises; public assets include Terminal-Bench and the OpenThoughts dataset (over 10k monthly downloads); an "Apply as expert" call to action.Source
    • Your websiteShelfLife documentation: a task taxonomy (C1–C10); MCP, shell, hybrid and state-machine interfaces; reward design with integrity and withholding criteria; reward-hacking checks for ground-truth leakage, verifier access, lookahead, verbosity credit and memorization; human QA by operators and experts.Source
  • 17

    ShelfLife's own premise is whether an agent can be trusted in a leadership role at a real company, and it reports cost per rollout alongside accuracy. That is exactly the question an enterprise asks before deploying an agent. AfterQuery and Bespoke Labs already court enterprises; idler's careers page lists enterprise as a target area.

    High impactHigh effort First result in 2 quarters CEO

    First action

    Pitch one mid-size operating company from the network on a paid pilot.

    How to measure it

    Enterprise pilots

    Now
    –
    Target
    2 paid pilots
    By
    3 quarters

    Steps

    1. 1

      Define the offer

      One workflow, six models, one report.

      CEO 2 weeks

    2. 2

      Price the pilot

      Fixed fee.

      CEO 1 week

    3. 3

      Find a pilot

      From the source-company pipeline.

      CEO 1 month

    4. 4

      Build the twin

      With their operators.

      Research Engineer 2 months

    5. 5

      Deliver the report

      Accuracy, integrity, cost.

      Research Engineer 2 weeks

    6. 6

      Decide

      Continue only if margins hold beside lab work.

      CEO 1 week

    First result
    2 quarters
    Cost
    A pilot's worth of research time
    Owner
    CEO

    Smallest test

    Offer one existing data partner a model-selection report on its own workflow.

    Why it works

    Offer enterprises a scoped digital twin of one workflow (built from their own de-identified data with their operators), scored across current models on accuracy, integrity and cost. Deliver a decision report and an ongoing re-test as models change.

    Enterprises pay for decision confidence, the method already exists, and each twin also becomes a new environment source (with permission).

    Risks and how to handle them

    • Distraction from lab demand idler already can't serve

      Cap at two pilots and choose ones that also yield lab environments.

    Builds on

    Evidence

    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
    • Competitor researchAfterQuery: "We teach machines how experts think"; sells fine-tuning data, RL data, agent environments and computer-use trajectories to frontier labs, has a "For Enterprises" page, and publishes partner results such as a legal benchmark of 5,161 cases across 28 practice areas.Source
    • Competitor researchBespoke Labs: "Reliable agents need real environments"; company-scale simulations for frontier labs, neolabs and enterprises; public assets include Terminal-Bench and the OpenThoughts dataset (over 10k monthly downloads); an "Apply as expert" call to action.Source
    • HiringCareers page: idler is "a frontier data lab" building RL environments for economically valuable work across coding, RSI, law, cybersecurity and enterprise, scaling with technology and AI rather than a large contractor workforce; it reports a multimillion-dollar frontier lab contract, a $9M seed led by Paradigm and growth from 3 to 14 people. All roles are in person in San Francisco, with a two-day work trial.Source
  • 18

    idler's homepage names "Economically Valuable Workflows" as a focus, and its collections already measure models on accounting, finance, legal work, retail operations, coding and security with integrity checks and cost. No one publishes a yearly, cross-domain measure of how much real work models can be trusted with.

    High impactHigh effort First result in 3 quarters CEO

    First action

    Agree the index metrics and which collections feed it.

    How to measure it

    Index citations

    Now
    –
    Target
    20 press and paper citations
    By
    1 quarter after launch

    Steps

    1. 1

      Define metrics

      Pass rate, integrity, cost per success.

      Research Engineer 2 weeks

    2. 2

      Run all collections

      On current models.

      Research Engineer 1 month

    3. 3

      Analyze

      Trends and surprises.

      Research Engineer 2 weeks

    4. 4

      Write

      Public report and customer cut.

      CEO 3 weeks

    5. 5

      Brief press

      Exclusive preview.

      CEO 1 week

    6. 6

      Launch

      Report, chart cards, research notes.

      CEO 1 week

    First result
    3 quarters
    Cost
    Inference spend plus a month of research time
    Owner
    CEO

    Smallest test

    Publish a one-page cross-collection snapshot first.

    Why it works

    Publish an annual report that combines all collections: pass rates by domain and skill, integrity failures, cost per successful task, and what changed over the year. Release headline numbers publicly and the detailed cut to customers.

    An annual index becomes the reference that press, investors and labs cite, and puts idler's category definition (real production work) at the centre.

    Risks and how to handle them

    • Revealing private set details

      Publish aggregates only.

    Builds on

    Evidence

    • Your websiteHomepage (checked November 9, 2026): H1 "Push the frontier of your model", subhead "Reinforcement learning environments built from real production work", and three blocks: "Frontier Coding", "Economically Valuable Workflows", "Bespoke Evaluations". The only links are Contact (an Airtable form), Browse, Careers, X and LinkedIn; there is no privacy, terms, security, research, blog or expert-application link.Source
    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
    • PressRuntimeWire, August 19, 2026: idler launched publicly as a "research lab building evaluation datasets and reinforcement learning environments for frontier AI models", selling "frontier-difficult RL environments for top AI labs" built from production data, after a $9M seed led by Paradigm; a Y Combinator company.Source

Wildcards 4

Unexpected experiments, each with a cheap way to test it.

  • 19

    idler hires through a two-day in-person work trial, and its Research Engineer builds systems that generate environments and check them for quality and fairness. The skill idler hires for (designing a fair, hard, unhackable task) can be tested in public, and the best entries become both candidates and environment ideas.

    Medium impactMedium effort First result in 1 quarter CEO

    First action

    Write the judging criteria from the ShelfLife documentation.

    How to measure it

    Challenge-sourced hires or experts

    Now
    –
    Target
    2 hires or 10 experts
    By
    2 quarters

    Steps

    1. 1

      Pick the dataset

      A small, de-identified slice.

      Research Engineer 1 week

    2. 2

      Write criteria

      Fairness, difficulty, reward-hacking resistance.

      Research Engineer 2 days

    3. 3

      Set prizes

      Cash plus fast-track.

      CEO 1 day

    4. 4

      Launch

      X, LinkedIn and university groups.

      CEO 1 week

    5. 5

      Judge

      With automated checks first.

      Research Engineer 1 week

    6. 6

      Follow up

      Invite top entrants to trials or expert work.

      Special Projects Operator 1 week

    First result
    1 quarter
    Cost
    Prize money plus judging time
    Owner
    CEO

    Smallest test

    Run a one-week mini challenge with one university group.

    Why it works

    Run a quarterly online challenge: design one environment from a provided de-identified dataset, judged on fairness, difficulty and resistance to reward hacking using idler's own criteria. Top entrants get fast-tracked to the work trial or paid expert work.

    Challenges reach engineers who never browse job boards, and judging uses criteria idler already documents.

    Risks and how to handle them

    • Judging load

      Automate the first pass with the existing QA systems.

    Evidence

    • HiringCareers page: idler is "a frontier data lab" building RL environments for economically valuable work across coding, RSI, law, cybersecurity and enterprise, scaling with technology and AI rather than a large contractor workforce; it reports a multimillion-dollar frontier lab contract, a $9M seed led by Paradigm and growth from 3 to 14 people. All roles are in person in San Francisco, with a two-day work trial.Source
    • HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source
    • Your websiteShelfLife documentation: a task taxonomy (C1–C10); MCP, shell, hybrid and state-machine interfaces; reward design with integrity and withholding criteria; reward-hacking checks for ground-truth leakage, verifier access, lookahead, verbosity credit and memorization; human QA by operators and experts.Source
  • 20

    Bespoke Labs' public benchmark and dataset spread through research use. idler's CorpLaw has 10 tasks no model has passed, and ShelfLife measures integrity and judgement that academic safety and agent researchers study. But both are request-only, so they don't appear in papers.

    Medium impactLow effort First result in 2 quarters Research Engineer

    First action

    Write a one-paragraph program and invite three academic groups.

    How to measure it

    Academic citations

    Now
    –
    Target
    5 papers
    By
    3 quarters

    Steps

    1. 1

      Define access

      Hosted scoring only.

      Research Engineer 1 week

    2. 2

      Write terms

      Citation and no redistribution.

      CEO 1 week

    3. 3

      Invite groups

      Agents, safety, legal NLP.

      CEO 2 weeks

    4. 4

      Support

      Office hours monthly.

      Research Engineer monthly

    5. 5

      List papers

      On each collection page.

      Research Engineer ongoing

    6. 6

      Review

      Inference spend vs citations.

      CEO quarterly

    First result
    2 quarters
    Cost
    Inference credits
    Owner
    Research Engineer

    Smallest test

    One group, one paper.

    Why it works

    Run an academic program: approved groups get hosted, scored evaluation access (not the raw data) and a standard citation. Publish a list of papers using idler benchmarks.

    Papers create lasting citations that lab researchers read, at the cost of some inference.

    Risks and how to handle them

    • Leakage through papers

      Hosted scoring; no task text released.

    Evidence

    • Competitor researchBespoke Labs: "Reliable agents need real environments"; company-scale simulations for frontier labs, neolabs and enterprises; public assets include Terminal-Bench and the OpenThoughts dataset (over 10k monthly downloads); an "Apply as expert" call to action.Source
    • Your websiteCorpLaw: "a legal benchmark built from a real law firm's anonymized data", 50 tasks in live transaction workspaces, graded against the firm's own work with attorney review; 10 of the 50 tasks have never been passed by any of six models at high reasoning effort. The CorpLaw page shows no leaderboard table, while its card on /collections shows top Pass@1 scores (42.8% for the leader).Source
    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
  • 21

    A lab's first question about a third-party eval is whether it is already in someone's training data. idler checks for memorization and ground-truth leakage, but that sits deep in the ShelfLife documentation; the collection pages that labs see first say nothing about how sets are kept clean or when tasks are retired.

    Medium impactLow effort First result in 2 weeks Research Engineer

    First action

    Write the panel text from the existing documentation.

    How to measure it

    Contamination objections

    Now
    –
    Target
    -50%
    By
    1 quarter

    Steps

    1. 1

      Write the policy

      Held-out, canaries, retirement.

      Research Engineer 1 day

    2. 2

      Add the panel

      On every collection page.

      Research Engineer half a day

    3. 3

      Date the checks

      Last memorization test per set.

      Research Engineer 1 hour

    4. 4

      Link to docs

      Full method.

      Research Engineer 1 hour

    5. 5

      Use in calls

      Send with the prep note.

      CEO ongoing

    6. 6

      Log objections

      Track in call notes.

      CEO monthly

    First result
    2 weeks
    Cost
    $0
    Owner
    Research Engineer

    Smallest test

    Add the panel to ShelfLife this week.

    Why it works

    Add a short "Kept clean" panel to each collection: held-out policy, canary strings, how memorization is tested, retirement rules and the date of the last check. Link to the full documentation.

    It answers the first objection before the call and differentiates against public benchmarks that labs already distrust.

    Risks and how to handle them

    • Promising more than can be guaranteed

      Describe checks, not guarantees.

    Evidence

    • Your websiteShelfLife documentation: a task taxonomy (C1–C10); MCP, shell, hybrid and state-machine interfaces; reward design with integrity and withholding criteria; reward-hacking checks for ground-truth leakage, verifier access, lookahead, verbosity credit and memorization; human QA by operators and experts.Source
    • Your websiteCollections page, "Built on real production data, measured against frontier models": ShelfLife (200 tasks), ShelfLife E-Sim (12 tasks) and CorpLaw (50 tasks, "Access Upon Request") show top scores on their cards; Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents are "Access Upon Request"; an eighth card, "Request a New Dataset", says "Tell us what to measure next." Every call to action is a 30-minute call booking or the Airtable interest form.Source
  • 22

    ShelfLife already shows cost per rollout next to accuracy and a cost-versus-performance frontier, and the spread is wide: from about $2 to $17 per rollout for very different pass rates. Most benchmarks show accuracy only. Cost per successful task is what a lab's product team and every enterprise buyer care about.

    Medium impactMedium effort First result in 1 month Research Engineer

    First action

    Compute cost per successful task from the existing ShelfLife results.

    How to measure it

    Cost-view visits

    Now
    –
    Target
    20% of collection traffic
    By
    2 quarters

    Steps

    1. 1

      Compute

      Cost per successful task by model.

      Research Engineer 1 day

    2. 2

      Design

      Frontier chart with filters.

      Web designer 1 week

    3. 3

      Build

      On each public collection.

      Research Engineer 1 week

    4. 4

      Combine

      Cross-collection page.

      Research Engineer 1 week

    5. 5

      Share

      Downloadable chart cards.

      CEO per release

    6. 6

      Measure

      Visits and requests from the view.

      Research Engineer monthly

    First result
    1 month
    Cost
    Design and engineering time
    Owner
    Research Engineer

    Smallest test

    Add cost per successful task as a column on ShelfLife.

    Why it works

    Add a cost-per-successful-task view to every public collection, with filters by skill category, and a combined page across collections. Update it with each re-run and let visitors download the chart.

    It gives a second audience (product and deployment teams) a reason to visit and share idler's boards, and it is hard for accuracy-only benchmarks to copy.

    Risks and how to handle them

    • Prices change often

      Date every cost figure.

    Builds on

    Evidence

    • Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
    • Your websiteShelfLife E-Sim: a stateful retail simulator where agents run a business one day at a time, grounded in nearly a decade of a live retailer's orders, purchase orders and inventory snapshots; 12 tasks, mean reward from 37.6 to 65.0; last measured August 26, 2026. The model that leads ShelfLife places fifth here, and the leader here places fourth on ShelfLife.Source

03

How Programmable can help

Where an idea in this playbook is something Programmable builds or runs, it's listed here with what we'd do. Everything else is yours to run.

ProgrammableInfo

Turn the research into a list you can act on

These ideas start with finding the right companies. Info finds them and the right person at each, researches every one and drafts the outreach, then asks before it sends.

  • Build this list in Info for idea #5 (opens programmable.info in a new tab)

New accounts: 100,000 free credits with code BETA100K

Find the companies (opens programmable.info in a new tab)
More example playbooks and reviews
ProgrammableDigital

Show idler’s product working

Turn screenshots or the website into an animated product demo in its own brand, for the homepage, launch posts and outreach in this playbook.

Make a demo (opens programmable.digital in a new tab)

More growth playbooks

Other companies we have researched, each read from what they are hiring for.

  • Mintive growth strategyInfrastructure for putting AI agents to work safely: an MCP gateway with role-based bundles, an LLM gateway, Mint Guard runtime…21 growth ideas
  • InLoop Robotics growth strategyInLoop builds and rents Loop V1, a two-armed robot system for warehouse work (box assembly, kitting, visual inspection, content…23 growth ideas
  • SkyWatch growth strategyA geospatial data platform from Kitchener, Canada: one catalog and price list for satellite, aerial, drone and elevation data from 700+…22 growth ideas
  • Evolve growth strategyEvolve is a tech-enabled vacation rental management company: it lists, prices and markets owners' homes on Airbnb, Vrbo, Booking.com and…22 growth ideas
  • Hera growth strategyMedicare-covered care management for older adults: each family gets a dedicated "Hero" (a social worker, nurse or therapist, supported by…21 growth ideas
  • BLUETTI growth strategyPortable power stations, expansion batteries, solar panels and home backup systems (Elite, Apex, Premium and EP series), plus ecosystem…22 growth ideas
  • Spotlight Media growth strategyA TikTok Shop-only Partner agency that runs shops, affiliates, creator retainers, paid media, LIVEs and a TikTok-to-Meta creator video…21 growth ideas
  • Klarent growth strategyAn agentic QA platform (formerly fore ai) whose AI agents discover, write, run and self-heal tests for web, iOS and Android apps from…21 growth ideas

See every playbook and website review

Independent research. Programmable wrote this playbook from public sources, as they stood on November 10, 2026, and cites them throughout. idler did not commission, review or endorse it, and Programmable is not affiliated with idler. Company and product names are trademarks of their owners, used only to identify them. The ideas are suggestions and opinion, not professional, legal or financial advice, and not statements about how idler runs its business.

Something wrong, or do you represent idler? Email [email protected] and we will correct it or take it down.

30/60/90 roadmap

What to focus on in each month, and the ideas that carry it.

  1. Days 1 to 306 ideas

    Bring the four private collection pages up to the ShelfLife standard, fix the CorpLaw leaderboard gap, add a sitemap and page descriptions, a qualifying request form, a security and data-handling page and one consistent pitch.

  2. Days 31 to 609 ideas

    Build supply and proof: expert application pages, a source-company page, regular re-measurement, research notes, the cyber capability map, refresh subscriptions, training-lift reports, the rankings story and referrals.

  3. Days 61 to 907 ideas

    Commit to the big bets (self-serve evaluation, enterprise digital twins, an annual index) and try the wildcards.

The research behind the plan

3 research areas, each summarised in a few lines. Open any of them for the full detail.

Customers

What customers say

Customers value environments built from real production work rather than synthetic tasks. The most cited problem: benchmarks that saturate or leak into training data.

  • Hard tasks: frontier models fail many of them
  • Synthetic environments that don't transfer to real deployments
  • In their words: “RL environments”

Market

Competitive landscape

4 competitors compared: AfterQuery, Bespoke Labs, Mechanize and Fleet AI.

  • Gap to exploit: Environments grounded in real companies' operating data
  • AfterQuery: Expert-data framing rather than real production data
  • Bespoke Labs: Broad product line competes for focus

Hiring signals

idler is hiring for supply, not sales.

  • Research Engineer: Scale environment generation and QA; product lines beyond coding. Ideas 1, 3, 7, 16.
  • Special Projects Operator: Recruit, screen and pay a growing expert network. Ideas 8, 15.
  • Offensive Security RL Researcher: A new cybersecurity line sold directly to lab researchers. Ideas 1, 4, 11.

From the team that wrote this playbook

All products
  • ProgrammableAgencyWorking AI systems in days, not quarters (opens programmable.agency in a new tab)
  • ProgrammableInfoAn agent that finds, researches and asks firstNew accounts get 100,000 free enrichment credits with code BETA100K (opens programmable.info in a new tab)
  • ProgrammableDigitalScreenshots in, an animated product demo out (opens programmable.digital in a new tab)
  • ProgrammableWatchKnow the day a supplier falls out of compliance (opens programmable.watch in a new tab)
  • ProgrammableTalkGet registrants for your events, and know who’s coming (opens programmable.talk in a new tab)
  • ProgrammableMediaAn AI note taker, interpreter and webinar host for your calls (opens programmable.media in a new tab)
  • ProgrammablePageText what you need, your customer gets a page that works (opens programmable.page in a new tab)
  • ProgrammableCommunityForums with labelled AI members, never empty (opens programmable.community in a new tab)
ProgrammableMarketing
ExamplesLeaderboardsProductsTermsPrivacyCookiesDPALegalContact

© 2026 AI-Powered Growth Intelligence