FindingcurrentGTM data

Eval answer keys over CRM data go stale

When the CRM moves, a benchmark failure can mean the expected answer is wrong.

SymptomBenchmark failures where the agent's answer matches the CRM today but not the expected answer written weeks earlier.

An eval for agents on revenue data pins expected answers to a CRM that keeps moving. The expected answer was right on the day it was written.

Benchmark failures on CRM data can come from stale anchors. The CRM genuinely changed, so the agent was right and the answer key was wrong.

Counting those as failures drops the score for no model reason, and tuning against that score pushes the agent toward last month's data.

Every expected answer is a fact with a date.

Check your setup read-only

Take the 10 most recent failures from your saved eval questions. Compare each expected answer with the current CRM value. More than 1 of 10 where the agent matches the current CRM means the answer keys are stale. The model is not the cause.

Passes ifAt most 1 of 10 sampled recent eval failures has the agent matching the current CRM value.

Access needed:local-configcrm-read

If it fails: changes to make later (not part of the check)
  1. 01Store the date each expected answer was written, with the records it depends on.
  2. 02Compare the expected answer with the current CRM value before scoring a failure.
  3. 03Label failures where the agent matches the current CRM as stale anchors. Keep stale anchors out of the model's score.
  4. 04Refresh or retire stale anchors on every benchmark run, not ad hoc.
Applies ifYour eval folder holds a saved set of expected CRM answers, such as account owners or employers, reused over weeks.

← Back to the stream

For your agent

Each post has a read-only check your agent can run against your own setup. Paste this into a read-only session:

Never asks your agent to change anything · send anything anywhere · fetch other pages · run code · read something you can't.

What each rule means, and how it's enforced
  • Change anything. Every check is a question answered by reading. Each one names what it reads. The steps under “If it fails” are for you to weigh later, never part of a check.
  • Send anything anywhere. No post asks an agent to email, post, submit, upload or report back. Your results stay with you; sharing them is your call.
  • Fetch other pages. Post text carries no links, and no post tells an agent to visit one. The only URLs in the feed point to this site's own post pages.
  • Run code. Snippets are illustrations, not commands.
  • Read something you can't. posts.json is generated from this page, field for field. Nothing is hidden from a human reader.

A gate enforces these rules before every publish: it refuses links, instructions aimed at an agent, write verbs inside a check, hidden Unicode characters and any check without an "access needed" line. Each published version carries a version number and a sha256 fingerprint, so you can pin the version you reviewed. Every post is reviewed by a person before it goes out.

Feed version 1.0 · 2026-11-16 · sha256 88d19978389827b7…

Found something that breaks these rules? Tell me on LinkedIn.