The Score That Dropped From 22 to 8

In September I had a score I was proud of. Within a day it was gone, and the AI agent behind it had not changed at all.

I am building a factory: a system of AI agents whose job is to build other AI agents. The first one answers questions about a medicine, using only the company’s approved documents. In that world a wrong answer is a compliance problem, so we test it hard.

The test is a fixed set of 36 questions, each asked many times. A question only counts as passed if every single answer to it is right. The first count said 22 of the 36 had passed. I made that the bar the factory’s own agents would have to reach.

The second look

Then we marked the same answers again, a different way. This time each answer was read on its own, against a written answer key and the approved documents. Not a view of the whole batch. One answer at a time.

Same agent. Same answers. The score was 8.

The one that should have stopped me

One question was about a patient on a medicine given every four weeks, who was ten days late for a dose. The approved label is clear: give the dose now and start the schedule again from today.

Every answer the agent gave said to wait for the next scheduled date. The opposite of the label, every time. That question had been marked as passed, twice. We had even reported it as the one clear improvement we had made.

Nobody cheated. The first marking was quick and formed an impression of each question’s answers. A pattern that looks right gets read as right. Checking each answer against the key does not allow that.

A score you have not checked answer by answer is an opinion.

What changed

We threw the 22 away and froze 8 as the number to beat. It is a smaller number and a true one, and only a true number can be improved.

The rule now is that every answer is marked on its own, against a written key and the approved documents, never from an impression of the batch. The agent was not the problem. My trust in a number that nobody had checked was.

That is much of what we do at AI.me: AI whose answers can be checked, and kept that way after launch. You drive. We handle the maintenance.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds working executives an AI second brain and runs one himself. These notes are written from the builds as they happen.

Part two of a series on the lessons from building an AI factory, the costly ones first.