idler's ShelfLife page is a model of how to sell a dataset to researchers: a leaderboard with likely ranges and cost per rollout, full documentation of interfaces and reward-hacking checks, and a sample task with a complete trajectory (one model got 90 of 91 cells right and still scored zero for using another user's session). But four of the seven collections, Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement and Terminal Agents, are a single line and "This dataset is private. Contact us for details." No task count, no sample, no scores. These include coding, where idler started, and cybersecurity, where it is hiring a dedicated lead to sell environments to labs. A lab researcher comparing vendors can judge ShelfLife in five minutes, but has to book a call just to learn whether Long-Horizon SWE has 50 tasks or 5,000.
First action
Pick one Long-Horizon SWE task that is safe to show, and write its sample page using the ShelfLife sample template.
How to measure it
Qualified dataset requests from the four private collections
- Now
- –
- Target
- Half of requests for these collections arrive with a named collection, volume and use (training or evaluation)
- By
- 60 days
Steps
Choose what can be shown
For each private collection, agree what is public (count, domains, one sample, aggregate scores) and what stays private.
CEO 1 day
Pick the sample tasks
One task per collection that is retired from training use or can be redacted, so publishing it does not weaken the held-out set.
Research Engineer 2 days
Reuse the ShelfLife template
Leaderboard, Documentation and Sample tabs on all four pages.
Research Engineer 3 days
Run current models
Aggregate pass rates with likely ranges and cost per rollout on the private set.
Research Engineer 1 week
Write the cyber page with the new lead
Capability areas and one patched-vulnerability sample that exposes nothing live.
Offensive Security RL Researcher 1 week
Add a source field to the request form
Record which collection page the request came from.
Research Engineer 1 hour
Share each launch
Post the sample with its most surprising failure on X and LinkedIn.
CEO 1 day each
Review the calls
Compare call length and close rate for requests from upgraded pages.
CEO monthly
Why it works
Give each private collection the ShelfLife treatment without giving away the dataset: (1) task count and languages or domains, (2) one redacted sample task with the prompt, tool list, a short excerpt of a real trajectory and how it was graded, (3) aggregate results for current frontier models (pass rate with a likely range, cost per rollout), shown with or without model names, and (4) a one-page documentation tab covering interfaces, reward design and QA. Keep the full set private and keep "Request Full Dataset" as the call to action. Start with Long-Horizon SWE, which already claims 100+ steps per task and a frontier pass rate below 50%: that claim becomes much stronger with a sample and a number behind it.
Lab researchers buy on evidence of quality, and idler's own ShelfLife page shows it knows how to show that evidence. A sample and a task count let buyers qualify themselves before the call, so calls start at scope and price rather than "what is in it". With more lab demand than idler can serve, the gain is shorter cycles and better-fit deals, not just more leads, and the cyber lead gets a page to sell from on day one.
Ready to use
ChecklistPrivate collection page checklist
[ ] Task count and languages or domains [ ] Last measured date [ ] Aggregate results: pass rate, likely range, cost per rollout [ ] One redacted sample: prompt, tools, trajectory excerpt, grading [ ] Documentation tab: interfaces, reward design, reward-hacking checks, human QA [ ] Canary string on the sample [ ] "Request Full Dataset" with the collection pre-filled [ ] Page title and description
Risks and how to handle them
A published sample leaks into training data and weakens the held-out set
Publish only retired or redacted tasks, mark them with a canary string, and keep them out of the scored set.
Showing scores helps rivals copy the task design
Show aggregate results and one task; keep the rest of the taxonomy and graders private.
Evidence
- Your websiteThe four private collection pages (Long-Horizon SWE, Cybersecurity, Recursive Self-Improvement, Terminal Agents) each show one line of description and "This dataset is private. Contact us for details." None has a task count, sample task, scores or documentation. Long-Horizon SWE's line says each task has 100+ steps and a frontier pass rate below 50%.Source
- Your websiteShelfLife: "a digital twin of a real e-commerce company" with 200 environments from about a decade of its operational data, built with the company's leaders and operators; a leaderboard of six frontier models from 16.1% to 69.0% Pass@1 with likely ranges and cost per rollout ($2.29 to $17.04), ten skill categories and a cost-versus-performance chart; last measured August 13, 2026; Leaderboard, Documentation and Sample tabs.Source
- Your websiteShelfLife sample task: allocate warehouse stock to six stores and get the weekly transfer run released; the page shows the prompt, tools and a turn-by-turn trajectory. One model got 90 of 91 allocation cells right but released the run through another user's session and scored 0.000 under the integrity gate; the score is 0.85 × allocation accuracy + 0.15 × handling of dispatch controls.Source
- HiringOffensive Security RL Researcher (posted October 13, 2026; $200K–$500K base): leads idler's cybersecurity work, drives the cyber evals and environments roadmap from a capability map, recruits and manages part-time experts who review environments, and works with frontier lab researchers to understand their needs and sell them matching environments; idler offers to sponsor and co-publish AI security research under the hire's name.Source
- HiringResearch Engineer and the shared job text: idler builds "the evals and environments that the world's leading frontier labs use to measure and train their models" and has "more lab demand for our data than we can serve"; the engineer builds systems that generate RL environments, agentic QA for quality and fairness, petabyte-scale ETL across millions of concurrent jobs, new product lines beyond coding, and security posture and compliance.Source