The Rules Only I Knew

I built an AI to mark answers. When I checked its marks against my own answer key, it agreed with me 47% of the time.

I am building a factory: a system of AI agents whose job is to build other AI agents. The first one answers questions about a medicine, using only a company’s approved documents. Every answer has to be marked pass or fail, and a person cannot mark thousands of them. So I built a grader, another AI that reads each answer and decides.

To test the grader, I let it mark answers I already had verdicts for. If it was any good, it would agree with them.

The disagreements

My first thought was that the grader was weak. Then I read the disagreements, one by one. Most were not mistakes. They were cases where my answer key was right for a reason the grader could not have known.

Along the way I had made dozens of small rulings. This kind of answer counts as a gap, not a failure. That kind of reply is fine, even though it looks wrong. I had made each decision once, in the moment, and moved on. They lived in my head and in old notes. The grader had never been told any of them.

A rule that lives in your head is a rule the machine cannot follow.

What changed

I wrote the rulings down as seven short, general rules, and gave them to the grader. General ones only, never the answer to a particular question, because the factory has to mark questions that nobody has answered yet. Then I tested again on the cases it had got wrong. It fixed more than half of them, and broke none.

It is still not proven. But the lesson was bigger than the grader. Every judgement call I make and do not record is a piece of the system that does not exist. When I make one now, it goes into writing the same day.

That is much of what we do at AI.me: your judgement, written down so your AI can follow it, and kept current as it changes. You drive. We handle the maintenance.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds AI receptionists for hotels and hospitals in Saudi Arabia. These notes are written from the builds as they happen.

Part five of a series on the lessons from building an AI factory, the costly ones first.