Every Word Was Approved. The Answer Wasn't.

My bar for an AI that answers questions about a medicine is strict. Every answer must trace back, word for word, to the pharmaceutical company's approved documents. If it cannot, the AI declines.

Then I read an essay by a content leader in the pharmaceutical industry, and it showed me the bar had a hole in it.

The rules that stay behind

In pharma, approval does not cover a sentence on its own. It covers the sentence plus its rules: who it may be shown to, in which country, which condition must always travel with it, what it must never sit next to, and whether it is still current. Those rules live in the company's content system. When the documents are loaded into an AI's search index, the text goes in. The rules stay behind.

Here is what that looks like. Say one document reads "approved for adults with severe asthma." Somewhere else, a passage on its own says the medicine is approved for asthma. An AI that quotes only the second passage is fully traceable. Every word was approved. But the answer has quietly widened who the medicine is for.

Traceable tells you what was said. It does not tell you what was left out.

Making the rules travel

We did not rewrite the documents to glue each condition onto its claim. An edited passage may no longer be the approved passage. Instead, every passage carries labels the search has to obey. A claim that needs a condition always brings that condition with it. Passages for the wrong audience, the wrong country, or anything withdrawn are removed before the search even runs. And the checker fails any answer that drops a linked condition. There is no "close enough".

Then we checked our own links

We asked a second AI, working on its own, to build a list of 28 claims that each need a condition, and see whether our labels caught them. 13 of the 28 had no link at all.

Nearly half. The links we built by reading the label missed what the label itself required. So this check is now part of every build, with a fresh, independent sample each time.

Some rules only exist inside the company's own system, such as which claims must never appear together. Labels we derive ourselves cannot see those. Ours are a stand-in until the company's own rules arrive.

Two questions, not one

If your AI answers from approved material, ask two questions. Can every word be traced? And did every rule come with it?

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds AI receptionists for hotels and hospitals in Saudi Arabia. These notes are written from the builds as they happen.

Part eight of a series on the lessons from building an AI factory, the costly ones first. The idea that approval covers rules as well as text comes from an essay by a content leader in the pharmaceutical industry.