Hippocratic AI Hit 99.38%. Designing for the Other 0.62%

Hippocratic AI had 6,234 licensed US clinicians test its healthcare AI across 307,038 calls. The accuracy score came back at 99.38%.

Hippocratic AI is a US company, valued at $3.5 billion in late 2025, that builds AI agents for patient phone calls such as follow-ups and scheduling. They do not diagnose.

That sounds like a perfect exam. Now do the sum. Out of every 10,000 calls, about 62 are still not scored correct. The company does not spell out exactly how it scores a call, so treat the figure as a guide, not a verdict.

I build AI agents for regulated work, so I read that number closely. It is the most honest picture I know of what the finish line looks like.

Hippocratic AI's climb is the achievement

Their published scores rose from 96.79% to 98.75% to 99.38% across three versions. A later paper reports 99.9% correct advice in one table and a lower 99.58% in a separate nationwide study. They publish both instead of rounding up.

The engineering behind it is serious: a main conversation model, more than thirty supervisor models checking its work, and a route to a human for calls that need one. Even so, the company's own article says that "five nines" reliability, 99.999%, remains aspirational. It names its core challenge as running a probabilistic system in messy real-world healthcare.

That is the point. These systems run on probability. Every answer is a very good guess, and a very good guess is sometimes wrong. A better model shrinks the gap. It does not close it.

Chasing 100% is a trap. Designing for the other 0.62% is the job.

What my own builds taught me

I am building a factory that produces AI agents for pharmaceutical medicine information, where every answer must trace back to approved documents. Four lessons from that work, each paid for with a failed test.

Take the words away from the AI. When my agent wrote its answers in its own words, the mistakes hid between the quotes: a fact added, a condition dropped. So I changed the design. The AI now only picks which approved passages to show, and code assembles the reply from word-for-word quotes. Correct answers on my test questions more than doubled. Then it read like a quote dump, so I let it write a short opening and a line or two applying the label to the person's situation, with every number checked in code.

Check the checker. The AI that marks my agent's answers changed its verdict on 1 answer in 5 when it marked the same ones three times. I wrote about that here.

Test on what it has never seen. My checks passed 84% of the material they were built against and 59% of fresh material. That story is here. Now a fix is accepted only if it helps on questions it was never tuned on.

Measure the safety net, too. I built gates that block wrong answers before a person sees them. In one test they caught 11 of 23 wrong answers and wrongly blocked 13 good ones out of 118. A net with holes is still worth having, but only if you know where the holes are.

What compliance-grade means

It does not mean the AI never misses. It means the system around it catches the miss before it reaches a person. The AI can only quote what is approved. Code checks what the AI adds. A human settles the disputes, including whether a question can be answered at all. Every fix is proven on questions it has never seen. Then you measure what is left over, say it out loud, and keep closing it.

That is what I take from Hippocratic AI's work. The score is not the story. The fact that they show you the leftover 0.62% is.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds AI receptionists for hotels and hospitals in Saudi Arabia. These notes are written from the builds as they happen.

AI.me is not affiliated with Hippocratic AI. Its figures are company-reported: its Polaris 3.0 page, its article on building AI for healthcare conversations and its January 2026 paper. Part of a series on the lessons from building an AI factory.