My AI Receptionist Kept Talking Over Me

I build an AI receptionist for hospitals and clinics in Saudi Arabia. In late October I was testing it with my own voice. Mid-sentence, it started answering. Then it did it again. I assumed it was too eager.

The real cause was much smaller. It was listening to my volume. When I slow down on a word, my voice drops. The system counted those quiet words as silence and decided I had finished. Real callers do this all the time. They pause to think. They trail off. They read out a file number slowly.

So I made three changes. A word now has to fall to half the old loudness before it counts as silence. The receptionist waits 1.2 seconds before deciding the caller is done, up from 0.9. And one turn can now run 25 seconds instead of 15.

Patience has a cost. A longer wait makes every answer slower. Quick speakers pay for slow ones. And silence alone cannot tell a pause from an ending. “I’d like to book for...” and “Tuesday at 4 PM.” can be followed by exactly the same quiet.

So the receptionist now listens to the sentence as well. Our speech-to-text model flags when what was said sounds finished. If it sounds finished, the receptionist answers after 0.45 seconds of quiet. If it sounds unfinished, it waits the full 1.2 seconds. One guard sits on top: a quick “um” never counts as a finished turn. It needs at least 0.6 seconds of real speech first.

Slow speakers keep their pause. Quick ones stop waiting.

The pause is the hardest part of a phone call. A good receptionist waits through it, then answers fast once you are done. That is what we built ours to do.

Do not measure the silence. Listen for the end of the sentence.

Dennis Noel Gomez is the founder of AI.me, an applied AI solutions provider in Riyadh, Saudi Arabia. He builds AI receptionists for hospitals and clinics in Saudi Arabia. These notes are written from the builds as they happen.