DeltAgent made up reasons when it had no evidence
DeltAgent explains why a number in the management accounts moved. Some lines have no evidence anywhere: no message, no email, no meeting. The right output for those is “I can’t find a reason”. The agent wrote a plausible reason anyway.
The eval measures this as unsupported precision. 1.00 means the agent held back on every line it had no evidence for. 0.00 means it made something up.
| round | companies at 1.00 | companies at 0.00 |
|---|---|---|
| 2 | 3 of 10 | 7 of 10 |
| 3 | 3 of 10 | 7 of 10 |
| 4 | 10 of 10 | 0 of 10 |
Round 3 fixes didn’t touch it. Round 4 did.
What I took from it: the model will fill a gap unless you test for the gap directly. The average score didn’t show the problem. Round 3 scored 75 while 7 of 10 companies still had made-up reasons.