Many of the improvements that make a voice agent feel more natural, reliable, and enjoyable to use come from engineering decisions that are easy to overlook during development but have a noticeable impact on the overall experience.
Design Tools Around Business Actions
Validate Retrieval Before Tuning Prompts
At Sigma Software, we explore these challenges together with Respeecher, a Ukrainian AI voice company and a portfolio startup of SID Venture Partners, co-founded by Sigma Software Group. Respeecher’s voice synthesis technology has been widely used in Hollywood productions including The Mandalorian and several Oscar-nominated films.
As part of this collaboration, our engineering teams regularly validate implementation approaches against real production projects, exchanging practical lessons while both companies continue building enterprise voice AI systems.
This article is based on a technical workshop organized jointly by Sigma Software and Respeecher, where Respeecher experts Nazar Vinnichuk and Ihor Isakov shared practical lessons from building production voice agents.
The following observations are shared from Respeecher’s engineering experience during the workshop.
Design Tools Around Business Actions
When we started building voice agents, we chose a generic approach. Why create multiple tools if the model can simply call a generic Make an HTTP Request function?
What we discovered pretty quickly was that flexibility came at a cost. The model had to reason about API mechanics instead of simply selecting the business action it needed. It had to figure out which endpoint to call, what headers and authentication to use, how to structure the request, and even which media type was expected. All of that consumed context.
We eventually replaced those generic tools with business-specific actions such as Create Support Ticket or Search Product. The model no longer had to think about HTTP requests. It simply chose the business action that matched the user’s intent.
We also changed what our tools returned. If a tool creates a support ticket, it should immediately return the ticket ID instead of forcing the model to call another function just to retrieve something it created a second earlier.
Another observation was that if two actions are almost always performed together, exposing them as separate tools only adds complexity and clutters context. In those cases, combining them into a single business operation reduces unnecessary reasoning.
Validate Retrieval Before Tuning Prompts
Whenever the agent gives a wrong answer, the first thing people usually do is start tweaking the prompt. Quite often, though, the problem isn’t the prompt—it’s that retrieval failed to provide the information the model needed. That’s why we start with a different question: Did retrieval actually return the information?
The easiest way to answer that question is to remove the language model from the equation altogether and inspect only the retrieved documents. If the information needed to answer the user’s question isn’t there, changing the prompt is unlikely to improve the result.
Prompt for Voice Conversations
Voice agents require a different approach to prompting. Unlike text-based assistants, they need prompts designed for spoken conversations.
We’ve found that shorter prompts generally work better than longer ones. We generally structure prompts into sections such as Role, Task, Context, Constraints, and Output Format.
Voice output has its own requirements.
Markdown, bullet lists, emojis, or code blocks may sound unnatural—or even be verbalized incorrectly—when passed directly to a text-to-speech model. We therefore prompt the model to produce plain conversational language instead of formatted text.
Response length also depends on the scenario. A support agent may need longer, more conversational replies, while a phone receptionist or IVR system should respond quickly and move the interaction forward. Ultimately, there is no single prompt that works equally well for every voice application.
Design Conversation Flow
Not every aspect of a voice conversation belongs in the prompt. Some aspects of a voice conversation are better handled outside the prompt.
Latency is a good example.
If a request takes several seconds because the agent is querying a knowledge base or waiting for another service, users often assume the system has stopped responding. We learned pretty quickly that even a short acknowledgment such as “Let me check that for you” makes the interaction feel much more natural while the agent continues working in the background.
Inactivity is another example.
Rather than waiting forever, the agent follows up naturally after a period of silence. If the user still doesn’t respond after several attempts, the conversation ends automatically.
Phone support introduced another challenge we didn’t expect.
Even when callers are asked a simple question like “Which department do you need?”, many immediately begin describing their entire problem. By the time the call reaches a human operator, they have to repeat exactly the same story.
To avoid that, the agent can interrupt politely after listening for a while and transfer the caller to a human operator instead of letting the conversation continue unnecessarily. This avoids asking users to repeat the same explanation when they reach a human operator.
Another small detail surprised us: adding subtle office background noise often makes phone conversations feel more natural by recreating the audio characteristics of a traditional phone call.
Conclusion
Looking across these examples, a common pattern emerges. Some of the most impactful engineering decisions aren’t driven by how the system is built, but by how the conversation unfolds.
Focusing on the conversation rather than the underlying implementation often leads to simpler solutions—and better voice experiences.
About Respeecher
Respeecher is a voice-cloning AI startup named 2025 Technology Pioneer by the World Economic Forum. Founded in 2018, it develops high-fidelity voice cloning technology used in film, gaming, animation, and healthcare. From de-aging Tom Hanks’s voice to refining actress Karla Sofía Gascón’s singing voice in Emilia Perez, Respeecher is a trusted player behind some of the most ambitious uses of voice AI in entertainment.
