SPINS has just launched Agentic Discovery on the promise that "AI agents can't recommend what they can't interpret". But its own robots.txt tells every crawler not to read three whole folders: "Disallow: /solutions/", "Disallow: /data/" and "Disallow: /audience/". They sit in a block meant for ad landing pages, yet the sitemap lists 16 pages in those folders as normal indexable pages: Liftoff, Destini, Amazon Growth, TriLens, Point-of-Sale Data, Product Intelligence, Consumer Panel Data and the three "Who We Serve" pages. Search engines and AI assistants that respect robots.txt can't read them, so when a brand asks an assistant "who has natural channel POS data" or "how do I get into Sprouts", SPINS' best answers are invisible. The new Sr. Director of Product Marketing is hired to own the agentic-commerce story; this one fix is where it starts.
First action
Edit robots.txt to remove the three folder rules and test them in Search Console.
How to measure it
Sitemap pages under /solutions/, /data/ and /audience/ indexed
- Now
- –
- Target
- 16 of 16
- By
- 30 days
Steps
Confirm the intent
List which ad landing pages the folder rules were meant to block.
Web and SEO Manager 2 hours
Rewrite the rules
Exact paths for ad pages; remove the three folder rules.
Web and SEO Manager 1 hour
Test
robots.txt tester for all 16 sitemap URLs.
Web and SEO Manager 1 hour
Request recrawl
Search Console and Bing Webmaster Tools.
Web and SEO Manager 1 hour
Baseline AI answers
Ten buyer questions in the main assistants, before and after.
Sr. Director, Product Marketing 1 day
Add a release check
Block any robots rule that matches a sitemap URL.
Senior Software Engineer 1 day
Measure
Indexed pages, impressions and demo requests from them.
Web and SEO Manager weekly
Why it works
Replace the three folder rules with exact rules for the ad landing pages they were meant to cover (or rely on the noindex tags those pages already carry), keep the sitemap as it is, and request recrawls in Google Search Console and Bing Webmaster Tools for the 16 pages. Then ask the main AI assistants the questions these pages answer, before and after, and save the results. Add a check to the release process so a robots.txt change can never again block a folder that appears in the sitemap.
It costs nothing, it puts the core product and data pages back into search and AI answers within weeks, and it lets SPINS show prospects it practises what Agentic Discovery preaches.
Ready to use
Checklistrobots.txt fix list
[ ] List the ad landing pages the folder rules were meant to cover [ ] Replace Disallow: /solutions/, /data/, /audience/ with exact paths [ ] Test all 16 sitemap URLs as allowed [ ] Request recrawl in Google and Bing [ ] Record ten AI assistant answers before and after [ ] Add a CI check: no robots rule may match a sitemap URL [ ] Review indexed count in 30 days
Risks and how to handle them
Ad landing pages become crawlable
They already carry noindex; add exact Disallow lines for any that must stay hidden.
Slow recrawl
Submit URLs individually and link them from fresh blog posts.
Evidence
- Your websiterobots.txt on www.spins.com (checked twice on October 23, 2026), in a block headed "Ad landing pages (noindex in Yoast - block crawling)", includes "Disallow: /solutions/", "Disallow: /data/" and "Disallow: /audience/". Those rules match every page under those folders, and the page sitemap lists 16 of them as normal indexable pages ("index, follow"), including /solutions/spins-liftoff/, /solutions/destini/, /solutions/amazon-growth/, /solutions/trilens-panel/, /data/pos-data/, /data/product-intelligence/, /data/consumer-panel-data/ and /audience/enterprise/, /audience/smb/ and /audience/retailers/ (the "Who We Serve" pages in the main menu).Source
- Your websiteAgentic Discovery page: "Win the AI search queries that drive your category"; "Your products are on the retail shelf. Do they show up on the AI shelf?"; "AI agents can't recommend what they can't interpret"; SPINS "is built on the Product Knowledge Graph" and works "at the product level across brand.com and retailer PDPs"; "Schedule a demo" and "Already a client? Try it for free". The press release (September 22, 2026) promises "Commercial-Grade AI Auditing" of how assistants and agents "interpret, summarize, cite, and recommend products" and "Revenue-Prioritized Optimization".Source
- HiringSr. Director, Product Marketing (Chicago hybrid or remote, posted August 28, 2026, $190,000-$210,000): "AI agents are starting to shape which products get discovered, recommended, and bought"; SPINS has "grown from a data and insights company into a broader CPG growth and innovation platform spanning commerce enablement, media, measurement, and AI-powered discovery". The role must "Lead the market narrative for how AI and agentic commerce are reshaping how brands get discovered", differentiate new AI products "from fast-moving startups" and established platforms "from legacy incumbents", build "product-led growth alongside enterprise sales", run "win/loss analysis and buyer research", and deliver "playbooks, competitor comparisons, competitive intelligence, demo scripts, and persona guides".Source
- Your websiteHomepage: "Decode shopper values. Accelerate growth."; Liftoff, "Want to get on shelf at Sprouts? Ready to expand into Kroger?"; customer quotes and videos from Carbone Fine Food, Local Hive Honey and Primal Kitchen; recent resources ("Launch Excellence", "Win the Agentic Shelf", "From Digital Shelf to Discovery Engine", October 2026); a monthly newsletter. The menu lists about 20 offers across Brand Solutions, eCommerce, Innovation, Commerce Enablement, Consumer Insights, Consulting, Product Experience Management (IX-ONE) and Shelf Intelligence (Datasembly, marked "NEW!"), with links out to datasembly.com and destini.co.Source