# Eval answer keys over CRM data go stale

When the CRM moves, a benchmark failure can mean the expected answer is wrong.

By Ki Moon · 2026-09-03 · finding · current
Canonical: https://kihmoon.com/bench-notes/p/eval-answer-keys-go-stale/

**Symptom:** Benchmark failures where the agent's answer matches the CRM today but not the expected answer written weeks earlier.

An eval for agents on revenue data pins expected answers to a CRM that keeps moving. The expected answer was right on the day it was written.

Benchmark failures on CRM data can come from stale anchors. The CRM genuinely changed, so the agent was right and the answer key was wrong.

Counting those as failures drops the score for no model reason, and tuning against that score pushes the agent toward last month's data.

Every expected answer is a fact with a date.

## Check your setup (read-only)

Take the 10 most recent failures from your saved eval questions. Compare each expected answer with the current CRM value. More than 1 of 10 where the agent matches the current CRM means the answer keys are stale. The model is not the cause.

**Passes if:** At most 1 of 10 sampled recent eval failures has the agent matching the current CRM value.

**Access needed:** local-config, crm-read

## If it fails (changes to make later, not part of the check)

1. Store the date each expected answer was written, with the records it depends on.
2. Compare the expected answer with the current CRM value before scoring a failure.
3. Label failures where the agent matches the current CRM as stale anchors. Keep stale anchors out of the model's score.
4. Refresh or retire stale anchors on every benchmark run, not ad hoc.

**Applies if:** Your eval folder holds a saved set of expected CRM answers, such as account owners or employers, reused over weeks.

**Skip if:** Your eval config reads a frozen snapshot of the data, not the live CRM.

Tags: evals, freshness, crm
