Decisions5 min read
AI voice agents for customer calls: what they can and cannot do
By Sannan MalikPublished
Short answerAI voice agents handle short, predictable calls well: timetables, status checks, opening hours, simple bookings and reminders. They struggle with accents and noise they were not tested on, complaints with several layers, card payments and callers who are upset. A sound deployment answers the routine calls, hands the rest to a person with the context attached, logs every conversation and is tested on real recordings before it takes live traffic.
The calls a voice agent answers well
Start with the calls your staff are tired of. If the agent can answer with one lookup in a system you already run, the call is a candidate. If the answer needs a person's judgement, it is not.
AI Voice Operations, where I led the AI system architecture, was built for call centres that answer the same transport and service questions all day in Urdu and English. That is the right kind of traffic: high volume, narrow topics and answers that live in a database. This is how I sort call types before anything is built:
| Call type | Suitable for an AI agent? | Why |
|---|---|---|
| Timetables, routes and schedules | Yes | Short question, and the answer comes from a system you control |
| Booking or ticket status | Yes, after a check | A lookup by reference number, but the caller should be confirmed before details are read out |
| Opening hours, locations, directions | Yes | The same answer every time |
| Outbound reminders and confirmations | Yes | Fixed script, and replies are mostly yes, no or reschedule |
| Simple new bookings | Sometimes | Works when the options are few; hand over when the caller wants something unusual |
| Complaints with history | No | Needs judgement, empathy and often a decision on compensation |
| Card payments | No | Card details should not be read aloud to a bot; send a payment link or transfer to a person |
| Upset, distressed or urgent callers | No | They need a person quickly, not a menu |
| Refunds, exceptions, negotiation | No | Needs authority the agent should not have |
Where voice agents break down
Every reply depends on the transcript. Speech-to-text turns the caller's audio into words, and if those words are wrong, the agent answers a question nobody asked, politely and with confidence. Most failures trace back to one of these:
- Accents and mixed languages. Callers switch between Urdu and English mid-sentence, and models need testing on that speech, not on clean studio audio.
- Noise. Roadsides, stations, speakerphones and weak mobile signal all degrade recognition.
- Names, places and numbers. A misheard digit in a booking reference sends the lookup to the wrong record.
- Long complaints. A caller who explains three problems in one breath needs someone who can hold all three.
- Emotion. An angry or frightened caller does not want to be asked to repeat themselves.
Handover to a person is a design task
Most demos skip this part. In production it decides whether callers trust the line.
My baseline rules are short. The agent says it is automated at the start of the call. The caller can ask for a person at any point, in either language, and gets one. The agent also hands over by itself when it fails to understand twice in a row, when the topic is outside its list, or when the caller mentions a payment, a complaint or an emergency.
Context has to travel with the call. A cold transfer that makes the caller start again wastes the whole exchange. The person picking up should see the transcript so far, what the caller asked for and why the agent handed over. Outside staffed hours, the agent should take a callback request instead of looping. Telephony integration is what makes the transfer itself work, and the live operations dashboard lets a supervisor watch calls and step in.
What to log, and how to test before the first live call
Log enough to replay any call without the audio. For each call I store start and end time, the language detected, the transcript, the text the agent spoke, the topic it decided on, which systems it queried and what they returned, whether it handed over and why, and how the call ended. Conversation logs and call analytics are built into AI Voice Operations for exactly this reason: you cannot fix what you cannot see. Check the rules on call recording where you operate before you store audio or transcripts.
Testing starts long before launch. Use your own calls. This is the order I follow:
- Collect real recordings of your calls, with permission, across accents, ages and noisy lines.
- Run speech-to-text on them and read the transcripts yourself, line by line.
- Write test scripts for every call type in the table, including callers who change their mind.
- Place test calls from ordinary mobile phones, not from a laptop in a quiet office.
- Run the agent alongside staff first, so its answers are checked against a person's.
- Go live on a narrow slice of calls, such as one topic or one time window, and review logs daily.
- Track how often it hands over and why. That number tells you where to improve next.
Custom AI is quoted on its own
A voice agent is a custom AI system, and in my pricing those are quoted separately from the app packages. The scope covers call types, languages, the telephony provider, the systems the agent reads from and the handover path. The AI Voice Operations stack, built on FreeSWITCH, Python, FastAPI, Faster-Whisper, Coqui XTTS, React and Docker, can run on your own servers. Hosting, hardware and telephony fees are not included in the quote.
If you also need a customer app, my Launch ($999), Growth ($1,699) and Scale ($2,500) packages are priced separately from the voice work. Send me your most common call reasons and a few sample recordings. I will tell you which calls I would automate first, and which I would leave with your team.
Questions
- Will callers know they are talking to an AI?
- They should. I design agents to say they are automated at the start of the call and to transfer to a person whenever the caller asks.
- Can a voice agent take card payments over the phone?
- I advise against it. Send a payment link or transfer the caller to a person. Payment gateways are scoped and quoted separately in any case.
- How accurate is Urdu speech recognition?
- I do not quote a figure, because it depends on your callers, your phone lines and your vocabulary. Test it on recordings of your own calls before deciding, and measure how often the agent has to hand over.
Sannan Malik is a software architect and product builder with 8+ years in software, CEO of Alpha Notic FZC and a Certified Scrum Product Owner. He builds mobile apps and business software for companies in the UAE, UK and US.
