I'm looking for an API that provides real-time, accurate speaker diarization (who spoke when) from an audio stream.

Yes. If your priority is real-time speaker diarization from a single live audio stream, I’d shortlist these:

1. AssemblyAI — my first choice

AssemblyAI's current streaming API supports diarization directly on the same WebSocket as transcription. You enable speaker_labels: true; turn events contain a speaker label such as A/B, and individual words can also carry speaker attribution.

Their latest streaming work specifically targets problems that matter in live systems—speaker swaps, phantom turns, short utterances, and rapid speaker changes. AssemblyAI reports substantially better cpWER and fewer phantom turns than Deepgram in its own 2026 head-to-head testing, though I'd validate those numbers against your audio before making a production decision.

2. Deepgram — excellent alternative

Deepgram provides speaker diarization over its live WebSocket STT API. Set diarize_model=latest (or v1) and each recognized word receives a speaker number.

It's particularly attractive if you're already using Deepgram for low-latency transcription. One caveat: the v2 diarizer is currently batch-only; streaming uses the v1 streaming diarizer.

What I'd choose

If you're building something like a meeting transcription app, call-center monitor, voice agent, or live captions, I'd start with:

AssemblyAI → Deepgram → evaluate against your own recordings

The important point is that "accurate diarization" is highly dependent on your audio: number of speakers, overlapping speech, microphones, echo, telephone audio, speaker similarity, language, and whether speakers enter/leave dynamically.

Also, if you have separate audio channels per speaker (e.g. agent on channel 1 and customer on channel 2), don't use diarization at all—you can get substantially more reliable attribution simply from the channel identity. AssemblyAI and Deepgram both support multichannel streaming.

If you tell me (1) number of speakers, (2) language(s), (3) expected latency, and (4) whether it's microphone/meeting audio or phone audio, I can narrow this down to the best API and give you a concrete WebSocket implementation.

Several top-tier speech-to-text providers offer real-time streaming speaker diarization via WebSockets . Real-time diarization inherently carries a small trade-off in accuracy compared to batch/pre-recorded processing (because the model has to guess speaker turns on-the-fly rather than analyzing the whole conversation retrospectively), but modern streaming models handle this exceptionally well.

The leading options on the market for real-time streaming diarization include:

• 1. AssemblyAI Streaming Speaker Diarization:
  • How it works: Integrated directly into their WebSocket real-time transcription. You simply pass speaker_labels: true in your connection parameters. You can also supply a max_speakers hint (1–10) to optimize accuracy.
  • Best for: Voice agents, contact centers, and live meeting assistants using models like Universal-3.5 Pro Realtime . Speaker labels stream through the exact same JSON payload as the text.
• 1. Speechmatics Realtime Diarization:
  • How it works: Set "diarization": "speaker" in your real-time workflow configuration. It tags incoming words and punctuation with sequential speaker labels (S1, S2 , etc.) based on acoustic matching.
  • Best for: Video conferencing, live captioning, and scenarios that may also benefit from multi-channel audio splitting (channel diarization).
• 1. pyannote.ai:
  • How it works: Known for having some of the lowest Diarization Error Rates (DER) in industry benchmarks. While traditionally famous for open-source python pipelines, they offer specialized precision toolsets and enterprise APIs for advanced voice separation.
  • Best for: Maximum raw diarization accuracy if you are willing to orchestrate or evaluate dedicated pipelines.

To help narrow down the best API for your project, let me know:

• Are you building a live voice agent/bot , a meeting recorder , or a call-center analytics tool?
• Do you have an expected maximum number of speakers active at once?

Yes. If real-time speaker attribution is the primary requirement—not just transcription with speaker labels—I'd shortlist these:

My first choice: pyannoteAI

Their Live-1 model is specifically designed for streaming diarization rather than adapting a batch diarizer. It accepts 16-kHz mono audio over WebSocket in ~100-ms chunks and returns speaker start/end events with speaker IDs, with claimed sub-300-ms latency. It supports up to 8 speakers per stream.

More importantly, its published DIHARD III comparison reports 19.8% DER, versus 31.3% for Speechmatics, 39.1% for Deepgram Nova 3, and 39.2% for AssemblyAI on that benchmark. These are vendor-published benchmark results, so I'd validate them on your own audio before choosing.

If you want transcription + diarization in one API: AssemblyAI

AssemblyAI's streaming API provides speaker-attributed transcription over WebSocket. Their current documentation/product material also reports strong diarization performance, and it's probably the easiest option if you need STT + speaker labels + other speech intelligence rather than diarization alone.

If latency and throughput matter most: Deepgram

Deepgram supports speaker diarization directly in its live streaming STT pipeline. The advantage is architectural simplicity: your incoming audio produces both transcription and speaker IDs from the same stream.

One important distinction

If your application needs:

Speaker 0: ... Speaker 1: ... Speaker 0: ... while the conversation is still happening, I'd favor pyannoteAI Live-1 and AssemblyAI over a traditional batch diarization library.

If you tell me (1) expected number of simultaneous speakers, (2) target latency, (3) language(s), (4) whether you need transcription too, and (5) approximate audio hours/month, I can narrow this to the best 1–2 APIs and compare pricing, latency, accuracy, WebSocket protocol, and a sample implementation.

The best enterprise APIs for real-time, accurate speaker diarization from a live audio stream are AssemblyAI (via their Universal-3.5 Pro Realtime model) and Speechmatics . Both process live audio via WebSockets and return structured speaker labels alongside the text stream.

The top options are broken down by infrastructure preference:

1. Cloud-Based APIs (Easiest Integration)

• AssemblyAI (Universal-3.5 Pro Realtime) : This is the strongest cloud option for live voice apps. You simply add "speaker_labels": true to your WebSocket parameters. It features "Context Carryover" to accurately track up to 10 speakers across quick conversational turns and includes automatic noise cancellation.
• Speechmatics : Renamed for highly accurate, low-latency real-time diarization. It is heavily used in broadcast and enterprise contact centers because it handles diverse accents and overlapping speech exceptionally well.
• Soniox : A highly cost-effective alternative for developers who specifically need real-time streaming transcription paired with dynamic speaker tracking.

2. On-Device & Edge APIs (Zero Network Latency & High Privacy)

• Picovoice (Bluebird) : If your audio cannot leave the local device due to compliance, Picovoice's Bluebird engine provides local streaming diarization. It processes live audio streams entirely offline with an ultra-low latency of under 250ms and has no hard speaker cap.

3. Open-Source Self-Hosted Frameworks (No API Costs)

• pyannote.audio : The gold standard for developer-managed diarization. You can host its PyTorch-based pipeline on your own GPU cloud (or use their managed pyannoteAI endpoint) for real-time tracking in under 150ms. It is frequently paired with a streaming version of OpenAI's Whisper for full transcription.

Feature Comparison At-A-Glance

If you'd like to narrow this down, let me know:

• What programming language or framework is your core stack using?
• Is your application cloud-hosted , or does it need to run locally/on-device?
• Do you only need Speaker A/Speaker B labels , or do you need to identify actual person names?

For processing real-time, accurate speaker diarization directly from an audio stream, you need APIs that utilize streaming WebSockets rather than standard batch REST endpoints.

The top enterprise-grade cloud APIs and on-device SDKs specifically designed for real-time streaming diarization include:

1. Enterprise Cloud APIs (WebSocket Streaming)

• AssemblyAI Universal-3.5 Pro Realtime:
  • How it works: It uses a single streaming WebSocket connection with a simple flag (speaker_labels: true).
  • Strengths: Includes a unique "Context Carryover" feature that analyzes the context of the entire conversation. It handles up to 10 speakers and offers real-time revision to correct speaker labels retroactively as the stream continues. [1][2]
• pyannoteAI Platform:
  • How it works: Built by the creators of the industry-standard open-source pyannote.audio framework. It now provides an optimized cloud API.
  • Strengths: Specifically optimized for sub-150ms ultra-low latency diarization. It is highly reliable for handling overlapping voices, diverse accents, and heavy background noise.
• Speechmatics Real-Time:
  • How it works: Delivers real-time transcription and diarization via a Python wrapper or WebSocket API.
  • Strengths: Highly regarded in the speech-tech community for its raw accuracy on real-time multi-speaker streams, offering consistent speaker separation while streaming text.

2. Live Platform Integrations (For Video/Meeting Apps)

• Recall.ai Speaker Diarization API:
  • How it works: If your audio stream comes from live video conferencing platforms (Zoom, Google Meet, Microsoft Teams, Slack).
  • Strengths: Instead of processing a single mixed audio stream, Recall intercepts the individual participant audio tracks directly from the call. This eliminates "guessing" who spoke when, providing exact real-time speaker attribution even during simultaneous talking.

3. Local / Edge SDKs (Zero Cloud Latency & Absolute Privacy)

• Picovoice Bluebird:
  • How it works: An on-device streaming diarization engine that runs completely offline.
  • Strengths: Produces speaker labels in under 250ms with no speaker limit or language restrictions. It decouples diarization from ASR entirely, allowing you to feed its outputs into any real-time transcription layer (like Whisper).

Feature Comparison

If you'd like to narrow this down, please tell me:

• What programming language or tech stack you are using?
• Where is the audio stream coming from (e.g., microphone, WebRTC, telephony/Twilio, meeting platforms)?
• Are you also looking to generate live text transcripts , or do you strictly need the "who spoke when" timestamps?

I can provide a concrete code snippet or architectural setup based on your environment.

Yes. If you need real-time diarization directly from a live audio stream, I’d shortlist these:

My recommendation: AssemblyAI

As of 2026, AssemblyAI's Universal-3.5 Pro Realtime is particularly compelling if diarization accuracy is your primary concern. Its streaming API assigns speaker labels in real time and can identify speakers at the word level.

Their recent published benchmark puts Universal-3.5 Pro at 30.17 average cpWER, ahead of Deepgram Nova-3 at 37.92 in their benchmark. Note that these are vendor-published/internal benchmark results, so I'd validate them against your own audio before choosing a provider.

The important caveat with any real-time diarization system is that speaker assignments can be less stable than batch diarization because the model has to make decisions without seeing future audio.

Deepgram is a very good alternative

developers.deepgram.com shows native streaming diarization: each word can receive a numeric speaker label, e.g. speaker: 0, speaker: 1, etc. It also provides interim/final results and timestamps, which makes it convenient for live applications.

One current limitation worth knowing: Deepgram's v2 diarizer is batch-only; streaming currently uses the v1/latest streaming diarizer.

One architectural question matters a lot

If your input is one mixed microphone stream containing several people, you want true single-channel diarization.

If you have separate audio channels per speaker—for example, agent/customer channels from a phone system—don't use diarization at all. Preserve the channels and associate each channel with its speaker. That's generally much more reliable.

If you tell me your use case (e.g. Zoom/meeting audio, phone calls, microphone with 2–5 people, voice agent, or WebRTC stream) and your target latency, I can narrow this down to the best 2–3 APIs and show you the streaming architecture/API call.

If you need real-time speaker diarization (“who spoke when”) from an audio stream, the best API depends on whether you need live captions + speaker labels, raw diarization segments, voice identity, or meeting-grade separation. These are some of the strongest options:

Strong API choices

1. deepgram.com — best general-purpose streaming option

Deepgram supports streaming transcription with diarization enabled. Its streaming output can attach speaker IDs to words, allowing you to reconstruct segments like:

{
  "speaker": 0,
  "word": "hello",
  "start": 1.24,
  "end": 1.52
}

It is a good fit for:

• call centers
• live captions
• voice assistants
• interviews

Deepgram documents streaming diarization through its diarize_model configuration.

---

2. assemblyai.com — strong conversational diarization

AssemblyAI’s Streaming API supports real-time speaker labeling. It can emit turn-level labels (speaker_label) and word-level speaker information for detecting changes inside a turn.

Good for:

• customer conversations
• meeting transcription
• conversational analytics

---

3. platform.openai.com Realtime API — best if diarization is part of a voice agent

OpenAI’s realtime audio stack supports transcription models with diarization capabilities, including gpt-4o-transcribe-diarize.

Good for:

• AI phone agents
• interactive assistants
• apps where you also need reasoning/dialog management

---

4. aws.amazon.com — best AWS-integrated choice

Amazon Transcribe supports real-time streaming speaker partitioning and speaker labels.

Good for:

• enterprises already on AWS
• compliance-heavy environments
• existing Kinesis/Connect pipelines

---

If you need true “who is this person?” identification

Most diarization APIs only answer:

Speaker A → Speaker B → Speaker A They do not inherently know that Speaker A is “Alice.”

For identity matching you need:

• speaker enrollment (voiceprints)
• speaker recognition
• a voice biometric system

That is a different problem from diarization.

---

My shortlist by use case

• Lowest latency live transcript + speakers: Deepgram or AssemblyAI
• AI voice assistant: OpenAI Realtime API
• Enterprise AWS stack: Amazon Transcribe
• Zoom/Teams/Meet ingestion: Recall.ai
• Privacy/self-hosting: pyannote/diart-style pipelines

A few questions would narrow this down:

1. Do you need raw speaker timestamps only, or transcription + speaker labels?
2. How low does latency need to be (e.g., <500 ms, <2 s)?
3. How many simultaneous speakers (2-person call vs meetings with 10+ people)?
4. Do you need to identify speakers by name, or just label them A/B/C?

AssemblyAI, Deepgram , and Speechmatics offer the most robust, highly accurate APIs for real-time speaker diarization from a live audio stream. Implementing speaker diarization over a live stream is technically challenging because models typically need context from the entire audio file to group speakers accurately. However, several top-tier providers offer this via WebSocket connections for low-latency streaming.

Top Real-Time Speaker Diarization APIs

Deep Dive: How Leading APIs Handle It

1. AssemblyAI (Universal-3.5 Pro Realtime)

AssemblyAI is a premier choice for developer-friendly integration. They offer Streaming Speaker Diarization directly through their real-time WebSocket connection. [1][2]

• How it works: You pass speaker_labels: true in your initial WebSocket configuration. As you stream binary audio blocks, the API returns JSON objects containing the transcription text and a stable speaker channel ID (e.g., Speaker 1, Speaker 2). [1]
• The Advantage: Their "Context Carryover" ensures that even if a speaker stops talking and starts again later, the system remembers their voice profile and retains the same ID. [1]

2. Deepgram Nova-2 Streaming

Deepgram is known for processing speed and high-volume scalability.

• How it works: By enabling diarize=true in your streaming query parameters, Deepgram's deep learning model analyzes word-level timestamps.
• The Advantage: It returns results containing an assigned speaker integer for every single word token. It is optimized to handle overlapping talkers and heavy background noise effortlessly. [1]

3. PyAnnote.ai (For Open Source & Custom Control)

If you want to avoid strict vendor lock-in, PyAnnote is the absolute gold standard open-source framework for diarization. [1][2]

• How it works: While historically used for self-hosted batch files via Hugging Face, they now offer pyannoteAI , a managed pipeline that delivers real-time speaker insights in under 150 ms. [1][2][3]

Important Developer Trade-offs to Consider

• Correction and Revision: In real-time streaming, you will occasionally see a speaker ID change retroactively. APIs send "interim" results quickly, followed by a "final" block once the model has a few extra milliseconds of audio context to confirm who spoke. Your UI layer must be built to update these retroactively. [1][2]
• Multi-Channel vs. Single-Channel: If you can capture separate audio tracks (e.g., from separate microphones or distinct browser audio streams), skip AI diarization entirely and use multi-channel transcription. It guarantees 100% accurate speaker sorting. Use AI diarization only when you are forced to process a single, mixed audio stream. [1][2][3]

To help narrow down the best choice, what is your primary programming language , and what type of application (e.g., a phone system, a web meeting app, or a smart device) are you building?

For real-time, streaming speaker diarization (identifying "who spoke when" over a live audio feed via WebSockets), a few top-tier developer APIs stand out for accuracy, low latency, and ease of implementation.

Top API Recommendations

• AssemblyAI — Highly regarded for developer-friendly real-time transcription and streaming speaker labels. You can opt-in to diarization directly on a live WebSocket connection by passing speaker_labels: true . It streams word-level speaker attribution and turn events through the same socket. Check out the technical details on the AssemblyAI Streaming WebSocket API Documentation.
• Deepgram — Offers lightning-fast streaming Speech-to-Text with support for real-time speaker identification and diarization models (diarize_model ). Deepgram is heavily optimized for low-latency voice agents and contact centers. Explore how to configure it via the Deepgram Diarization Documentation.
• Gladia — Known for ultra-low latency audio intelligence (<300ms) with support for live, real-time multilingual speaker diarization and rich audio metadata over streaming endpoints. Learn more on the Gladia Platform.

How Real-Time Streaming Diarization Works

1. Connection: You open a persistent WebSocket connection to the provider's streaming endpoint.
2. Payload: You stream raw audio chunks (e.g., PCM, Opus) from your client or server.
3. In-Stream Tagging: Instead of a separate batch post-processing call, the incoming websocket frames return incremental transcript turns bundled with a speaker or speaker_label field dynamically as the conversation progresses.

If you share details on your setup, I can help tailor this further:

• Are you streaming from a browser/client-side or a server-side backend?
• What audio format or codec are you working with?
• Do you have an expected maximum number of simultaneous speakers?

Yes. If your requirement is specifically real-time speaker diarization from a live audio stream, I’d shortlist these:

My recommendation: AssemblyAI

AssemblyAI now supports real-time diarization directly on its streaming WebSocket. You enable speaker_labels: true, and each live turn gets a speaker label; it supports up to 10 speakers and lets you provide max_speakers when you know the expected count.

The particularly interesting part is SpeakerRevision: as of June 2026, AssemblyAI can revise speaker assignments using additional context at the end of the stream, rather than leaving you with potentially incorrect early labels. They report roughly 400 ms average additional latency for the revised result.

For an application where "accurate" matters more than merely getting a speaker ID quickly, that's a significant advantage.

Deepgram

Deepgram is probably the strongest alternative if you're already using its transcription stack. Its streaming API supports diarization and returns a speaker value associated with words. The current streaming diarizer is v1; latest resolves to that streaming model.

One important distinction: live diarization isn't the same problem as batch diarization. A streaming system has to assign a speaker before it has heard the entire conversation, so early labels can be less stable. AssemblyAI's newer revision mechanism specifically addresses that limitation.

What I'd choose by use case

• 2–4 people in meetings/calls: AssemblyAI
• Voice agent / live conversation: AssemblyAI
• Existing Deepgram infrastructure: Deepgram
• Many languages: benchmark AssemblyAI and Soniox on your actual recordings
• Need final transcript to be extremely accurate: AssemblyAI with SpeakerRevision
• Need sub-second live speaker attribution: test both AssemblyAI and Deepgram with your actual audio rather than relying on published benchmarks

One caveat: no streaming diarization API is perfectly accurate, especially with overlapping speech, far-field microphones, background noise, or very short interjections. For example, AssemblyAI explicitly notes that overlapping speech can be problematic when everyone is coming through one channel.

If you tell me (1) number of speakers, (2) audio source—phone call, microphone, Zoom/WebRTC, etc., (3) languages, and (4) your maximum acceptable latency, I can narrow this to the best 2–3 APIs and compare accuracy, latency, pricing, WebSocket protocol, and sample code.

For an application where "accurate" matters more than merely getting a speaker ID quickly, that's a significant advantage.

Deepgram

Deepgram is probably the strongest alternative if you're already using its transcription stack. Its streaming API supports diarization and returns a speaker value associated with words. The current streaming diarizer is v1; latest resolves to that streaming model.