The Grader That Changed Its Mind

I asked an AI judge to mark the same answer three times, with exactly the same instructions. On some answers, it did not give the same verdict every time.

I am building a factory: a system of AI agents whose job is to build other AI agents. The first one answers questions about a medicine, using only the company’s approved documents. Another AI marks every answer pass or fail. I call it the grader.

Everything the factory does next leans on that grader. Every score, every fix, every decision about whether an agent is ready. If the grader is shaky, everything built on it is shaky too. So when it marked two near-identical answers differently, I chose to measure it before spending a full test run on it.

The test

The test was simple. Take 40 answers. Have the grader mark each one three times. If the grader is reliable, the three verdicts match every time.

The lighter model I had been using changed its mind on about one answer in five. A stronger model, marking the same answers, changed its mind on one in twenty.

The wrong lesson

It is tempting to read that as a ranking: the stronger model wins, use it. That is only half of it.

When I looked at where even the stronger judge disagreed with itself, it was not mood. Each one was a rule I had decided and never written into the rulebook the judge reads. I had recorded my rulings in one place and not in the other. Where I had not told the judge what to do, it guessed.

Ten days later I measured again, on a new set of answers. It still wavered on a few. Every one was an answer where the agent declined, flagged a gap, or pointed the user elsewhere. No rule said whether a correct decline counts as a pass. I wrote one: a correct decline beats a wrong answer. Most of the wobbling stopped.

Before you trust a score, check that the judge agrees with itself.

What changed

We use the stronger judge, one verdict per answer. And every change to the judge is now checked against its own earlier verdicts before it is adopted.

A judge that changes its mind is not a judge. It is a coin with opinions. Mark the same answer twice, and see if you get the same verdict.

That is much of what we do at AI.me: AI whose answers can be checked, and kept that way after launch. You drive. We handle the maintenance.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds working executives an AI second brain and runs one himself. These notes are written from the builds as they happen.

Part three of a series on the lessons from building an AI factory, the costly ones first.