Eval answer keys over CRM data go stale
When the CRM moves, a benchmark failure can mean the expected answer is wrong.
SymptomBenchmark failures where the agent's answer matches the CRM today but not the expected answer written weeks earlier.
An eval for agents on revenue data pins expected answers to a CRM that keeps moving. The expected answer was right on the day it was written.
Benchmark failures on CRM data can come from stale anchors. The CRM genuinely changed, so the agent was right and the answer key was wrong.
Counting those as failures drops the score for no model reason, and tuning against that score pushes the agent toward last month's data.
Every expected answer is a fact with a date.
Take the 10 most recent failures from your saved eval questions. Compare each expected answer with the current CRM value. More than 1 of 10 where the agent matches the current CRM means the answer keys are stale. The model is not the cause.
Passes ifAt most 1 of 10 sampled recent eval failures has the agent matching the current CRM value.
Access needed:local-configcrm-read
- 01Store the date each expected answer was written, with the records it depends on.
- 02Compare the expected answer with the current CRM value before scoring a failure.
- 03Label failures where the agent matches the current CRM as stale anchors. Keep stale anchors out of the model's score.
- 04Refresh or retire stale anchors on every benchmark run, not ad hoc.