The Test It Had Never Seen

My AI’s checks scored 84% on the questions I had built them against. Then I gave them questions they had never seen, and they scored 59%.

I am building a factory: a system of AI agents whose job is to build other AI agents. The first one answers questions about a medicine, using only the company’s approved documents. Between that agent and the user sits a set of automatic checks. Each one blocks an answer that breaks a rule, like quoting the wrong source, or guessing where it should have declined.

To test the checks, a second AI writes tricky questions, the kind that sit right on the edge of a rule, along with the correct outcome for each. I run the checks, find the ones they get wrong, and fix them. Round after round, the score climbed. It reached 84%, and I felt good about it.

The test I could not cheat

Then I asked the second AI for a fresh batch. Different questions, same rules, written and sealed away where I could not see them or tune anything against them. I ran the same checks, unchanged.

What it means

It is tempting to say the checks were bad. They were not. They were good at the questions I could see. My reading is that every fix I made was a fix for a failure in front of me, so the checks slowly took the shape of that batch. A fresh batch was the first time anything pushed back.

A test you can see is a test you can pass.

There is a name for where this ends up: a hallucination box. A system so well fitted to its own tests that it can no longer see its own mistakes. It looks excellent from the inside.

What changed

Two things. One check that relied on a fixed pattern of words now uses a model that reads what the answer means. And every cycle, I compare the score on questions the checks were built against with the score on a fresh sealed set. If the fresh score keeps trailing, the box is forming.

The fresh score is the only number I trust. The other one is a rehearsal.

That is much of what we do at AI.me: AI that is tested on questions it has never seen, and kept honest after launch. You drive. We handle the maintenance.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds working executives an AI second brain and runs one himself. These notes are written from the builds as they happen.

Part four of a series on the lessons from building an AI factory, the costly ones first.