I need a voice API that allows for real-time interruption (low latency turn-taking).

Yes. If real-time interruption / barge-in and low-latency turn-taking are the core requirements, I’d shortlist these:

My recommendation

If you want the least engineering: go with ElevenLabs Agents. Its conversation system explicitly supports interruptions and configurable turn eagerness, including an eager mode for fast back-and-forth conversations.

If you want to build the infrastructure yourself: LiveKit Agents is particularly attractive. Its adaptive interruption handling distinguishes genuine barge-ins from things like "mm-hmm" or other brief acknowledgements, while keeping the interruption pipeline low latency.

If you want an integrated speech-to-speech model: OpenAI Realtime is worth evaluating. It is designed around persistent realtime sessions rather than stitching together separate STT → LLM → TTS calls.

One important distinction

For your use case, don't evaluate APIs based only on TTS latency. You want the entire interruption loop:

user starts talking → VAD detects speech → agent audio stops → current generation is cancelled → new user input is processed → new audio begins

ElevenLabs' Speech Engine, for example, explicitly cancels the in-flight LLM request when the user interrupts, rather than merely stopping playback.

If you tell me your stack (browser/mobile/phone), target latency (e.g. <300 ms), and whether you need your own LLM, I can narrow this down to the best 2–3 APIs and compare actual latency, pricing, WebSocket/WebRTC architecture, interruption behavior, and implementation complexity.

Building a low-latency voice agent with natural interruption handling (barge-in) requires an architecture that supports real-time streaming transport (ideally WebRTC or low-latency WebSockets) paired with intelligent turn-detection (so the system doesn't trigger an interrupt on a brief pause or a quiet filler word like "hmm" or "yeah").

To see how real-time interruption and multi-turn audio flows operate under the hood, watch this overview on handling interruptions:

32s Of the importance of interrupting properly the OpenAI Realtime ...1 year ago YouTube · WorkAdventure

The top industry APIs and frameworks for achieving this include:

1. OpenAI Realtime API
   • How it works: A stateful server-to-server or client-to-server WebSocket/WebRTC API that processes audio natively end-to-end without chaining separate STT, LLM, and TTS steps.
   • Interruption Handling: Built-in server-side Voice Activity Detection (VAD). When it detects user speech while outputting audio, it automatically halts generation, issues a truncation event (response.truncate ), and starts processing the user's new input.
   • Best for: Direct integration with GPT-4o real-time audio capabilities with minimal infrastructure overhead.
2. LiveKit Agents + Turn Detector v1
   • How it works: LiveKit provides open-source real-time infrastructure utilizing WebRTC, which drastically reduces audio packet loss jitter compared to raw WebSockets. Their agent framework features an advanced audio-based Turn Detector v1.
   • Interruption Handling: Instead of raw silence-based VAD (which cuts people off when they pause to think), LiveKit's turn detector analyzes acoustic cues, pitch, and phrasing to distinguish true barge-ins from backchannels or breathing.
   • Best for: Production-grade, highly resilient real-time audio routing where you want control over the underlying voice/LLM pipeline.
3. Inworld AI Realtime Voice API
   • How it works: Offers single-API pipeline coverage (STT, LLM, TTS) supporting both WebSocket and WebRTC transports with sub-100ms time-to-first-byte TTS.
   • Interruption Handling: Features semantic VAD and automatic barge-in via configuration flags (interrupt_response: true).
   • Best for: Developers looking for multi-model routing (220+ LLMs) with built-in semantic turn-taking.

Are you planning to connect clients via a web/mobile browser (WebRTC) or telephony/server-to-server (WebSockets) , and do you prefer a fully managed single-vendor model (like OpenAI) or an open framework (like LiveKit)?

If you need real-time interruption (barge-in), low latency turn-taking, and natural voice conversations, look for a streaming speech-to-speech API rather than a traditional STT → LLM → TTS pipeline.

Good options:

What I’d prioritize for your requirement

For human-like turn-taking, the API should support:

• Voice activity detection (VAD) — detect when the user starts speaking while the agent is talking.
• Interrupt/cancel current generation — stop audio output immediately.
• Audio truncation/context repair — remove unplayed speech so the model knows where it was interrupted.
• Low-latency streaming transport — WebRTC is usually preferable for browser/mobile voice.

Example architecture

Microphone
   ↓
Realtime voice connection (WebRTC/WebSocket)
   ↓
Speech-to-speech model
   ↓
Streaming audio response
   ↓
Speaker

User starts speaking:
   ↓
VAD detects interruption
   ↓
Cancel current response
   ↓
Continue conversation from interrupted state

If you are building a production voice agent

My shortlist would be:

1. OpenAI Realtime API — strongest fit if you want a single API handling audio input/output, interruption, and conversational state. It supports WebRTC/WebSocket/SIP and has configurable server VAD and semantic VAD with interruption controls.
2. Azure Voice Live API — good choice if you are already in the Azure ecosystem and need enterprise controls.
3. AssemblyAI Voice Agent API — worth considering if your main differentiator is speech intelligence and semantic interruption behavior.

If you tell me your target (browser app, mobile app, phone calls via SIP/Twilio, customer support agent, etc.), I can narrow this down to the best stack.

Building a voice application with real-time interruption (barge-in) and low-latency turn-taking requires a stack that handles Voice Activity Detection (VAD), semantic end-of-turn detection , and streaming transport (WebRTC or WebSockets) . Traditional chained pipelines (STT → LLM → TTS) introduce too much latency and clunky interruptions, which has driven the rise of native real-time audio models and specialized infrastructure.

The top choices for real-time interruptible voice APIs depend on whether you want an all-in-one model endpoint or a full-stack real-time orchestration framework:

1. OpenAI Realtime API

• How it works: Connects via WebSockets or WebRTC directly to an end-to-end multimodal audio model (gpt-4o-realtime).
• Interruption Handling: Built-in server-side VAD. When the user speaks while the model is outputting audio, the API triggers an input_audio_buffer.speech_started event, automatically cancels the ongoing audio generation response (response.cancel ), and truncates the unplayed audio buffer.
• Pros: Extremely low latency; no need to stitch together separate STT, LLM, and TTS vendors.
• Cons: VAD can sometimes trigger too aggressively on minor background noises or filler words ("hmm", "uh-huh" ), treating them as full interruptions.

Learn more about managing multi-turn conversations and interruptions with OpenAI's approach:

2. LiveKit (with LiveKit Agents & Turn Detector)

• How it works: LiveKit is an open-source real-time WebRTC infrastructure layer. Combined with livekit-agents , it lets you plug in any STT, LLM, and TTS providers you want while managing the media transport and edge network.
• Interruption Handling: Features Semantic Turn Detection and adaptive interruption handling. Instead of cutting the AI off the millisecond a sound is detected (like basic VAD which cuts off incorrectly >55% of the time), LiveKit’s model evaluates if the user's audio is a complete thought or just a cough, breath, or filler word ("mhm" ) before choosing to interrupt.
• Pros: Industry-leading reduction in false interruptions; robust WebRTC handling for poor network conditions; highly customizable component stack.
• Cons: More infrastructure and orchestration overhead compared to a single-endpoint API.

3. Inworld AI Realtime API

• How it works: A unified single API pipeline designed specifically for conversational voice agents, supporting WebSockets and WebRTC transports with shared event schemas.
• Interruption Handling: Features built-in semantic VAD (using Silero VAD and custom smart turn detectors) alongside an explicit interrupt_response: true parameter that manages barge-in recovery seamlessly.
• Pros: Offers sub-100ms time-to-first-byte TTS, routing across hundreds of LLMs, and OpenAI Realtime protocol compatibility if you want a drop-in migration path.
• Cons: Newer ecosystem relative to standalone WebRTC giants.

Key Technical Considerations for Low-Latency Barge-In

• Transport Layer: Prefer WebRTC over WebSockets if your client is a browser or mobile app. WebRTC handles jitter, packet loss, and adaptive bitrate much more smoothly for live audio streaming.
• Buffer Clearing: If you bridge to telephony (like Twilio) via WebSockets, your server must instantly fire a clear-buffer command ("event": "clear" ) the moment an interruption VAD event fires, otherwise the user will still hear another 200–500ms of queued audio playback from the AI.

To help narrow down the best fit, tell me:

• Are you building for a web/mobile browser or telephony (phone/SIP)?
• Do you want an all-in-one model API (like OpenAI) or the flexibility to mix-and-match STT/LLM/TTS providers (like LiveKit)?

Building a voice application with low-latency turn-taking and real-time interruption (barge-in) requires an architecture that supports full-duplex streaming and server-side Voice Activity Detection (VAD). When a user speaks over an AI agent, the system must instantly truncate audio playback, cancel ongoing generation, and register the new user input.

Here are the top production-ready Voice APIs and frameworks built specifically for this workflow:

1. OpenAI Realtime API

• How it works: Operates over WebSockets (or WebRTC), allowing you to stream audio directly to and from the model without stitching together separate STT, LLM, and TTS pipelines.
• Interruption Handling: Built-in server-side VAD detects when a user speaks while the model is outputting audio. It automatically sends truncation events, halts generation, and clears the unplayed audio buffer.
• Pros: Extremely low latency, minimal boilerplate, and native handling of conversational nuances.
• Cons: Can occasionally trigger false-positive interruptions on short acknowledgement/filler words ("hmm", "uh-huh") if the VAD sensitivity isn’t tightly tuned.
• Get started: Review the integration guides in the OpenAI Realtime Conversations Documentation.

For a walkthrough on managing conversational state and interruptions with this protocol, watch this hands-on guide:

2. LiveKit Agents (with Semantic Turn Detection)

• How it works: LiveKit provides an open-source WebRTC/WebSocket infrastructure framework tailored for real-time multimodal applications. Rather than relying purely on rudimentary audio energy VAD (which often cuts people off when they pause to breathe), LiveKit supports semantic turn detection.
• Interruption Handling: Evaluates the meaning of the user's speech via specialized lightweight models to understand if a thought is actually finished before letting the agent jump in or yield the floor. This dramatically cuts down on unwanted interruptions and false cutoffs.
• Pros: Full control over your STT/LLM/TTS cascade (or real-time endpoints), superior handling of awkward pauses, and production-grade WebRTC scaling.
• Cons: Requires hosting your own agent worker infrastructure rather than hitting a single monolithic cloud endpoint.
• Get started: Explore the architecture and plugins on the LiveKit Documentation or check out developer updates on the LiveKit YouTube Channel.

3. Inworld AI Realtime Voice API

• How it works: Provides an end-to-end voice pipeline (STT, LLM, TTS, and VAD) inside a single WebSocket or WebRTC session via the Inworld Router.
• Interruption Handling: Features automatic barge-in configuration (interrupt_response: true ) backed by a combination of Silero VAD and smart turn detectors to cleanly recover state when interrupted.
• Pros: Protocol-compatible with OpenAI’s Realtime structure for easy migration, low Time-To-First-Byte (TTFS <100ms) TTS, and access to over 220+ LLMs.
• Cons: Newer ecosystem relative to standalone telecom giants.
• Get started: Read the feature specs on the Inworld AI Resources Page.

Key Architecture Tip for Low Latency

Whichever API you choose, make sure your application layer immediately cuts off local audio playback client-side the moment local VAD registers speech—do not wait for a round-trip network acknowledgment from the server that an interruption has occurred, or the user will experience an unnatural echo/overlap.

If you can share whether you prefer a fully managed single-vendor API (like OpenAI) or an open infrastructure framework (like LiveKit) where you bring your own models, I can help provide a boilerplate code snippet for handling the WebSocket connection.

If real-time interruption / barge-in and low-latency turn-taking are your top requirements, I’d shortlist Deepgram Voice Agent API, OpenAI Realtime API, and Amazon Nova 2 Sonic.

My recommendation

1. Deepgram — I'd test this first if the primary goal is "the user can interrupt the agent and the agent immediately gets out of the way." Its Voice Agent API combines streaming STT, orchestration, TTS, and turn-taking in one WebSocket, with native barge-in handling.

2. OpenAI Realtime — excellent if you want a sophisticated conversational model behind the voice experience. Its Realtime API streams audio bidirectionally and automatically handles interruptions; OpenAI also supports semantic VAD for more natural decisions about when the user has finished speaking.

3. Nova 2 Sonic — worth considering if you're already heavily invested in AWS. It explicitly handles server-side barge-in and preserves conversation context when the user interrupts, although your client still needs to immediately clear queued audio.

One important distinction: don't evaluate these purely on "API latency." For natural conversation, what matters is time from the user starting to speak → interruption detected → currently playing audio stopped → new response begins. VAD configuration, audio buffering, network RTT, and client-side playback can easily dominate the experience.

If you tell me whether this is for a phone/telephony agent, browser voice chat, or an app, I can narrow this to the best 2–3 APIs and compare actual latency, pricing, WebRTC/WebSocket support, interruption behavior, and implementation complexity.

Building a real-time voice agent with smooth, low-latency interruption handling (often called "barge-in") requires moving away from traditional multi-step pipelines (STT → LLM → TTS stitched over standard telephony). [1]

Instead, you need architectures built on WebSockets, WebRTC , or native speech-to-speech models that support audio truncation and semantic turn detection. [1][2][3]

The top API choices and frameworks capable of handling low-latency turn-taking and real-time interruptions include:

1. OpenAI Realtime API

• How it works: It uses a native speech-to-speech WebSocket architecture. Audio streams directly in and out of the model without separate STT and TTS steps, drastically cutting down latency. [1][2]
• Interruption Handling: It has built-in Voice Activity Detection (VAD). When the user starts speaking while the model is outputting audio, the API sends a input_audio_buffer.speech_started event, allowing you to truncate the ongoing audio generation and switch contexts instantly. [1][2]
• Best for: Developers who want a direct, highly fluid model-level response without managing an orchestration stack. (Note: You may need to tune VAD sensitivity so it doesn't falsely trigger on filler words like "hmm" or minor coughs). [1][2]

2. LiveKit Agents & LiveKit Cloud

• How it works: LiveKit provides an open-source WebRTC/WebSockets framework (livekit-agents ) tailored for real-time audio/video.
• Interruption Handling: LiveKit features Adaptive Interruption Handling and Semantic Turn Detection . Rather than blindly cutting off the AI the microsecond a sound is detected (which causes awkward false cutoffs if a user coughs or says "yeah"), its semantic models evaluate whether the user's audio input constitutes a genuine interruption or completed thought.
• Best for: Maximum control, self-hosting flexibility, or applications needing ultra-low latency WebRTC transport layers. [1][2]

3. Vapi or Retell AI

• How it works: These are developer-focused voice orchestration platforms that sit on top of underlying STT (Deepgram/AssemblyAI), LLMs (OpenAI/Anthropic), and TTS (ElevenLabs/Deepgram) providers. [1]
• Interruption Handling: They handle the complex state management of cutting off the TTS audio stream and clearing out buffers the moment an interruption is flagged by underlying VAD engines, abstracting away the messy concurrency logic for telephony or web. [1]
• Best for: Rapidly deploying production-ready phone or web voice agents with minimal boilerplate. [1]

To see a breakdown of how these real-time platforms compare in latency and interruption handling, watch this overview:

2m I Tested The 6 Best Voice AI Platforms in 2026 (Here's ...2 months ago YouTube · Ashton Voss

Are you looking to build a web-based browser voice agent (via WebRTC/WebSockets) or deploy an AI phone bot (via Twilio/SIP), and do you prefer a fully managed API or an open-source framework like LiveKit?

Yes. If real-time interruption / barge-in and low-latency turn-taking are the core requirements, I’d shortlist these:

My recommendation: Deepgram or OpenAI

Deepgram Voice Agent API is particularly interesting for your requirement. It has a unified streaming interface covering STT, LLM, TTS, and turn-taking. Its runtime has native barge-in detection, so when the user starts talking, it can stop the agent rather than making you manually coordinate several streaming APIs.

OpenAI Realtime API is the other one I'd evaluate first. It streams audio in both directions over a persistent connection and supports automatic interruption handling. It's a good choice if you want a speech-to-speech model rather than assembling STT → LLM → TTS yourself.

What I'd look for technically

Don't just look for "streaming TTS." For genuinely natural turn-taking, you want:

• Streaming input + output
• Server-side or model-level speech detection
• Start-of-turn detection
• End-of-turn prediction, rather than waiting for a fixed silence timeout
• Barge-in that actually cancels generation
• Audio-buffer flushing, so already-generated speech doesn't continue playing after interruption
• Persistent WebSocket/WebRTC connection
• Ideally, speech detection integrated with the conversational model

For example, Deepgram's documented flow detects UserStartedSpeaking, stops the agent's turn, and then clears audio already buffered by Twilio. That's the important distinction between merely detecting an interruption and actually achieving a clean interruption.

If you're building a browser voice agent, I'd lean OpenAI Realtime. If you're building a phone/call-center agent, I'd strongly consider Deepgram Voice Agent + Twilio. Deepgram has a current reference architecture specifically for this, including barge-in.

If you tell me browser/mobile vs phone, and whether you want speech-to-speech vs your own LLM, I can narrow this down to the best 2–3 APIs and compare their actual latency, interruption behavior, pricing, and implementation complexity.

Yes. If real-time interruption / barge-in and low-latency turn-taking are the core requirements, I’d shortlist these:

My recommendation

1. OpenAI Realtime API — I'd start here if you want the simplest architecture. It streams audio bidirectionally and supports automatic interruption of the model's current response when speech starts. It supports WebRTC, WebSocket, and SIP, which is particularly useful if you're building either a browser/mobile voice experience or telephony.

2. Deepgram Voice Agent API — particularly compelling if barge-in quality is your primary concern. Its runtime handles STT, LLM orchestration, TTS, turn-taking, and interruption in one bidirectional stream. Deepgram specifically exposes StartOfTurn/barge-in behavior, so you don't have to build the interruption machinery yourself.

3. ElevenLabs Conversational AI — a strong option if you want exceptionally natural voices. It has explicit interruption handling and configurable turn eagerness (eager/normal/etc.), and its Conversational AI API handles the full realtime loop over WebSocket.

One important distinction

If by “voice API” you mean just STT/TTS, rather than a complete voice-agent API, I'd build the stack differently:

microphone → streaming STT + VAD → LLM → streaming TTS → speaker

For genuinely good interruption behavior, you need full-duplex audio + speech-start detection + immediate cancellation/flush of TTS audio. Merely using streaming TTS isn't enough. Deepgram's documentation, for example, describes explicitly flushing already-buffered audio when UserStartedSpeaking fires.

If you're optimizing specifically for <500 ms perceived response time, I'd lean OpenAI Realtime or Deepgram, and test them with your actual audio/network conditions rather than choosing based on advertised TTFB. A recent independent comparison found meaningful differences between providers in latency, missed turns, and unwanted interruptions.

If you tell me browser vs. phone/telephony, language(s), and whether you want to bring your own LLM, I can narrow this to the best 1–2 APIs and give you the architecture.

Building a low-latency, conversational voice agent with natural barge-in (interruption) capabilities requires an architecture that handles Voice Activity Detection (VAD) , server-side audio buffer clearing, and instant response cancellation.

The top options and architectures supporting real-time interruption include:

• OpenAI Realtime API : Operates over WebSockets (or WebRTC via integrations) and natively tracks conversational state. When VAD detects user speech while the model is responding, you send a response.cancel event via the socket to truncate the audio generation and clear the client/server playback buffers.
• LiveKit Agents Framework : An open-source, highly popular WebRTC/WebSocket orchestration layer designed specifically for real-time multimodal and voice agents. It pairs with STT/TTS providers and handles VAD and turn-taking natively so that interruptions cleanly cut off audio streams without dropping state.
• ElevenLabs Multi-Context WebSockets : Designed for responsive voice agents using expressive TTS. It introduces "contexts" over a single WebSocket connection, allowing you to instantly close an active speech context when a barge-in occurs and spin up a new response context without tearing down the socket.
• Twilio Conversation Relay : If you are building phone/telephony-based voice agents, Conversation Relay manages the low-level STT, TTS, and turn detection loop directly, passing clean text/audio transcripts to your chosen LLM with native handling for user interruptions.
• TEN Framework : An open-source real-time runtime framework built explicitly to handle complex voice agent interruptions, parallel tool calling, and mid-sentence cancellations.

Watch this walkthrough to see how real-time interruption and multi-turn audio flows are engineered using modern APIs:

20:52 How to Build an AI Voice Agent using OpenAI Realtime API ...1.6K views  · 1 year ago YouTube · Hashing | Learn Claude Code

To help narrow down the best choice, tell me:

• Are you building for web/mobile app (WebRTC/WebSockets) or telephony/phone calls (SIP/PSTN)?
• Do you want an all-in-one end-to-end model (like OpenAI Realtime) or a modular pipeline (separate STT, LLM, and expressive TTS like ElevenLabs)?