Bringing Cresta AI Agent to Avaya Infinity: Inside a WebSocket Voice AI Integration

When a customer calls a contact center, some of the hardest milliseconds in the interaction fall between the moment the caller stops speaking and the moment the AI begins responding. Closing that gap requires a clean, fast path for audio to leave the phone system, reach an AI agent, and return while the call remains live and the customer is waiting.

Getting the timing right can make the difference between a frustrating wait and a seamless experience that builds immediate trust.

Avaya Infinity provides this path through a WebSocket media interface that allows an external service to stream call audio in both directions and exchange control messages in real time.

Cresta, an Avaya Infinity ecosystem partner, was among the first to build on the interface, connecting Cresta AI Agent to Infinity. 

In this post, we share what stood out to our engineering team, how we adapted our existing voice AI architecture, and what we learned while moving from a new protocol to a working contact center AI integration. 

What the Cresta and Avaya integration supports 

Working against Avaya Infinity’s WebSocket interface, the integration now supports a full voice AI workflow: 

• Real-time AI voice conversations. The caller talks to a Cresta AI Agent that listens, reasons, looks up answers, and speaks back. Even if the caller interrupts mid-sentence, the AI agent stops to listen, allowing for a natural flow. 
• Live-agent handoff. When the AI decides a person should take over, it routes the call to the right agent queue, maintaining all conversation context in the process.
• Direct call transfer. A direct transfer becomes the better move when the caller's request requires routing to a specialized department, a different queue, or an external service rather than the standard live-agent pool.
• Live transcript for the agent. When a person takes the call, the whole conversation the caller had with the AI agent is already on their screen. 
• AI-generated summary at handoff. The agent also gets a short, AI-written summary of the call, so there is no "can you repeat everything you just said?" moment.

What stood out about Avaya’s WebSocket media design 

Through our work with the Avaya team, we learned that the interface was designed to support more than a single bot integration. The initial use case was a virtual agent, but the broader goal was to provide a reusable platform connection for additional partners and services.

That design decision appears throughout the interface. We have connected Cresta AI Agent to several contact center platforms, so we have a clear sense of how these integrations typically work. Here is what stood out to us about the Avaya approach.

The platform calls you, not the other way around 

In many streaming setups, your service opens the connection, and the contact center responds. Avaya flips that. Its media system reaches out and opens the connection to you. 

That changes how you build: your server has to be ready to accept a flood of incoming connections, and you track each call by its session rather than by the connection itself, since connections can come and go. 

One connection supports two levels of use 

The interface is deliberately built to work at two levels. 

• If a service just needs the basics, the protocol hands over the raw media stream and gets out of the way: take the audio and route it however you like. 
• Build at a higher level, and every service is modeled from the same foundation on that same protocol, whether that's a virtual agent, automatic speech recognition, text-to-speech, transcription, agent assist, translation, or recording. 

That layered design is why setup happens in two steps. First, the two sides agree on the basics of the media stream, like the audio format and which streams are flowing. Second, you turn on the specific service you want. 

The shared foundation is the point

For the services Avaya builds natively, like automatic speech recognition, text-to-speech, and transcription, a partner's engine can stand in for Avaya's, and everything else running on Infinity keeps working as before. 

For capabilities Avaya doesn't build itself, that same modeled interface is the extension point. Our AI Agent is the first virtual agent of its kind on the platform, and it lands as a first-class service, not a bolt-on wired in around the edges. 

• A shared foundation still supports service-specific requirements. None of that means every service looks the same, and that's deliberate. Each service is more prescriptive on top of the common media layer, with a well-defined set of streams and events that unlock what makes that service valuable. Agent assist, for instance, carries audio per participant, which means handling multiple streams at once. 
• Conversation context flows back to Avaya Infinity. However a service uses the connection, the data moves the same way each time: up to Infinity, which decides what to do with it, whether that's placing a transcript on the agent's desktop at handoff or driving the next step of the call. And it isn't only audio that travels. The connection carries the conversation too, like the running transcript we hand to the agent's screen at handoff. 

One aspect that stood out to us was Avaya Infinity’s ability to carry the live conversation over the same connection. That capability became the focus of some of the most interesting engineering work. 
 

• Audio travels in tagged packets. Instead of sending a raw audio stream, Avaya Infinity wraps the audio in compact packets. Each packet starts with a small label that says which stream it belongs to, what order it goes in, and when it was captured. That lets several audio streams, such as two sides of a call, share one connection while staying clearly separated, and it gives both ends what they need to spot dropped audio and keep playback smooth. 
• Resilience is built into each session. Avaya Infinity treats each call session as something built to survive trouble. It has built-in ways to recover after a connection drops unexpectedly, regular check-in messages to confirm the link is still alive, and a clean, confirmed way to end a call. Sign-in is also simple to operate: each connection presents a signed token that's checked before the connection is even accepted, with no shared login state to keep in sync between systems. 

Put together, the design is generic where an integrator wants raw control, prescriptive where structure unlocks a service's value, and standardized enough that a partner integrates once and grows over time. Being first on the interface puts Cresta in a position to test whether that promise holds. 

How Cresta adapted its voice AI architecture 

Avaya's integrate-once philosophy has a mirror image on our side. 

Cresta's voice system keeps three things separate: 

• How audio moves on a given platform
• What the AI does with it
• How the call is controlled 

The AI pipeline handles speech-to-text, determines when the caller has finished a turn, processes the request through the AI Agent, and converts the response back into speech. The call-control layer handles functions such as transfer, hang-up, and handoff. 

Each new platform only needs a thin layer for moving audio, a translator for that platform's message format, and a handler for its call-control commands. Everything after that is shared. 

Building the Avaya Infinity integration came down to three jobs: teaching our translator to read and write Avaya’s audio packets, adding a handler for Avaya’s control messages, and connecting those components to our existing pipeline. 

The underlying plumbing had already been built. It carried us from a simple echo test to a live human handoff in six weeks, with one engineer dedicating 50% of their capacity to the work. 

Four engineering lessons from building a real-time voice AI integration

A spec tells you what's possible. Live calls tell you what's true. The interesting work was in the gap between them. 

1. Reply as soon as the platform is ready. 

Our first setup sequence held its confirmations back to keep things tidy, and it promptly froze. The platform waits for our reply before it sends the next step, so the fix was to confirm the session the instant it opens and then keep going. 

Two more things shaped how we handle setup:

• The platform sends setup messages together on purpose. Once batching is turned on, the message that opens the session and the message that starts the bot arrive in a single packet. That is by design, and it works in our favor: our side gets everything it needs to bring the call up in one exchange instead of two round trips, which keeps dead air off the front of the call. The practical effect is that one packet can carry more than one message, so we taught our reader to stop assuming one packet meant one message and to split them apart.
• We protect the caller’s opening words. The platform holds the caller's audio until we have acknowledged the setup, so with batching there is rarely anything waiting. We keep the buffer anyway as a safety margin, because whether the setup messages arrive together or one at a time is not something we control, and we never want to lose the opening words of a call.

2. Audio pacing affects responsiveness, not only sound.

Our first audio came out slow and choppy, but the cause wasn't the audio format. It was how finely we sliced the outgoing audio. When we sent it in tiny pieces, we relied on the system's internal timer hundreds of times a second, and its small timing wobbles turned into gaps in the agent's speech.

The obvious fix, bigger pieces, has a catch: the larger the piece already in flight, the longer it takes for the agent to stop talking when the caller jumps in. We started at the protocol's smallest slice, 20ms, hit performance problems on our side, and moved to the protocol maximum of 200ms, which made speech smooth and kept interruptions feeling quick. 

There's headroom to tune it back down for an even faster cut-off when the caller barges in, which is exactly why we kept chunk size as an adjustable knob rather than a hardcoded value. 

That range is also a quiet credit to Avaya Infinity’s design. Avaya Infinity’s protocol lets us select an operating point across the 20-millisecond to 200-millisecond range.  

3. The agent's desktop has its own transcript rules. 

Handing the live transcript to the agent at handoff surprised us a few times, because the screen has expectations of its own:

• The conversation appeared out of order. The screen arranges the conversation by each line's timestamp, and we were sending the lines all at once with none. Now we stamp each line and let the screen sort them into the order the call actually happened.
• Some of the transcript lines went missing. Even with the order fixed, sending the whole transcript in one quick burst caused some lines to drop. A bigger buffer wasn't the answer. We stopped sending everything at once and instead began sending each turn as soon as it was final, which spreads the transcript naturally across the real length of the call.
• The AI agent's replies arrived in pieces. The AI speaks a sentence a little at a time, and its reply grows as it is spoken, so a single answer could reach the screen as a handful of stray fragments. We now gather the pieces of one reply and stitch them into a single clean line before it goes out.

4. Shutdown is where transcripts can disappear. 

Sending each turn the moment it's final means a background worker keeps sending right up until the call ends. That sets up a race: the worker is still trying to send turns while the call is shutting down. 

The turn most at risk is the one you can least afford to lose. The last line before a handoff is rarely filler. It's usually either the AI Agent telling the caller they're being transferred, or the caller saying the very thing that triggered the transfer. Both are exactly the context the human agent picking up the call needs most.

Between our own testing and code review, we turned up three ways that last turn could go wrong:

• The false success. The AI finished its last line just after we'd shut the sender down. We logged the turn as sent, but it never actually reached Avaya. This was the dangerous one, because everything looked fine. The logs reported success, so no one would ever know a turn went missing.
• The visible drop. The AI finished its last line right as the sender was shutting down, so the turn was accepted and then discarded before it went out.
• The double start. A rarer internal condition could have started the sender twice and consumed unnecessary resources. We added a guard to prevent it. 

The fixes came in layers. First, we gave the worker an explicit "stopping" state, so a turn that arrives mid-shutdown is caught and handled on purpose instead of slipping through, which closes the first two scenarios. Second, we made every dropped turn show up loudly in the logs and be easy to search for, so if one ever does slip, we see it. 

We guarded against the double start even though it can't currently happen, because "can't happen today" has a habit of becoming "happened in production" after enough refactors. If a transcript can go missing, you want to hear about it from a dashboard, not from a customer.

Build controlled tolerance into a developing interface 

One broader lesson runs through all four: with a developing interface, build controlled tolerance into what the integration accepts. 

We built the integration to tolerate missing optional details, unexpected extra ones, and message types we don't handle yet, so a spec that's still changing can add things without knocking our calls offline. 

Being a good early adopter is mostly humility built into your defaults. 

What we are exploring next

The current integration gives a human agent access to the transcript and an AI-generated summary when a Cresta AI Agent hands off the call. 

We are now exploring with Avaya how that continuity could extend into the human portion of the interaction. One potential next step is connecting the transferred call with Cresta Agent Assist so the agent can receive answers, suggested next steps, and guidance in real time. 

We are also evaluating how the same underlying connection could support additional Cresta capabilities, including Real-Time Translation. Other areas under consideration include higher-quality audio, automatic reconnection, and expanded call controls. 

Each capability would require its own implementation and approval. The advantage is that the teams can build on an existing connection rather than begin again with an entirely separate integration. 

"Avaya has paved the way for a scalable and impactful deployment of AI by building a modern platform designed to work seamlessly with Cresta's solutions. We are incredibly excited about the future of the Infinity platform and the partnership between Cresta and Avaya." Dong Zhao, Tech Lead Manager, Voice AI Infra, Cresta

"As a leader in AI for Customer Experience with integrations across virtually every enterprise CCaaS platform, Cresta was an ideal early partner for Infinity's WebSocket media integration. They were with us throughout the design process and quick to provide meaningful feedback, rigorous in testing, and strong proof that the approach works for advanced voice AI."John Graybill, Director of Product Management, Avaya

Explore the Avaya Infinity partner ecosystem to learn more.

 

Acknowledgements: Cresta’s Dong Zhao, Nick Tomaselli, Nicky Budd-Thanos and Masoud Assali also contributed to this piece.