I build AI-native financial services for health businesses.https://havryleshko.com/https://havryleshko.com/writing/deltagent-made-up-reasons/https://havryleshko.com/writing/deltagent-made-up-reasons/In two eval rounds the agent invented explanations for variances it had no evidence for. Round 4 fixed it.Sun, 04 Oct 2026 00:00:00 GMT<p>DeltAgent explains why a number in the management accounts moved. Some lines have no evidence anywhere: no message, no email, no meeting. The right output for those is “I can’t find a reason”. The agent wrote a plausible reason anyway.</p> <p>The eval measures this as unsupported precision. 1.00 means the agent held back on every line it had no evidence for. 0.00 means it made something up.</p> <table><thead><tr><th>round</th><th>companies at 1.00</th><th>companies at 0.00</th></tr></thead><tbody><tr><td>2</td><td>3 of 10</td><td>7 of 10</td></tr><tr><td>3</td><td>3 of 10</td><td>7 of 10</td></tr><tr><td>4</td><td>10 of 10</td><td>0 of 10</td></tr></tbody></table> <p>Round 3 fixes didn’t touch it. Round 4 did.</p> <p>What I took from it: the model will fill a gap unless you test for the gap directly. The average score didn’t show the problem. Round 3 scored 75 while 7 of 10 companies still had made-up reasons.</p>lessonhttps://havryleshko.com/writing/verified-knowledge-vs-model/https://havryleshko.com/writing/verified-knowledge-vs-model/In regulated work, does a small hand-checked source beat a better model recalling the rules on its own?Sun, 04 Oct 2026 00:00:00 GMT<p>A model can recall a rule from training and get it wrong without any sign that it’s wrong. In finance a wrong answer about a rule can cost money or break the law.</p> <p>fin-regbase is my test. Every page is checked by hand and cites a document and section.</p> <p>What I’ve measured: on 109 questions, search finds the right pages in the top 3 for 91. The full answer eval passed 69.7% on 13th of May, 2026.</p> <p>What I haven’t measured: the same questions answered by an agent with the wiki and without it. That comparison is the actual answer to this question, and I haven’t run it.</p>questionhttps://havryleshko.com/writing/human-boundary-finance/https://havryleshko.com/writing/human-boundary-finance/Which parts of a finance job an agent can do alone, and which still need a person.Sat, 03 Oct 2026 00:00:00 GMT<p>My guess right now: the agent does the work, a person checks it and signs off.</p> <p>I haven’t tested this on real work yet.</p> <p>The way I’ll measure it is minutes I spend per report on Sabiniana. If the minutes go down and the reports stay right, the agent is doing more of the job.</p> <p>What I have so far:</p> <ul> <li>DeltAgent’s eval average went from 49.3 to 92.5 out of 100 over 4 rounds. That’s on made-up companies, so it says little about real work.</li> <li>On Sabiniana every report goes through me before it’s sent. No reports have been sent yet.</li> </ul> <p>I’ll update this when Sabiniana has its first practices.</p>question