Whisper Speaker Diarization — How to Add Speaker Labels to Whisper (2026)

Path 1 — WhisperX (Default Choice)

github.com/m-bain/whisperX · ~22.6k stars · MIT

Architecture: faster-whisper (CTranslate2 backend) + wav2vec2 forced alignment + pyannote diarization + VAD pre-pass. Batched inference, under 8GB VRAM for large-v2, ~70× realtime on a single GPU per their README.

Python (the canonical block every tutorial copies):

import whisperx

device = "cuda"
audio_file = "audio.mp3"
batch_size = 16

# 1-2. VAD + ASR via faster-whisper
model = whisperx.load_model("large-v2", device, compute_type="float16")
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)

# 3. Forced alignment for word-level timestamps
model_a, metadata = whisperx.load_align_model(
    language_code=result["language"], device=device)
result = whisperx.align(result["segments"], model_a, metadata,
                       audio, device, return_char_alignments=False)

# 4-5. Diarization + word→speaker assignment
diarize_model = whisperx.DiarizationPipeline(
    use_auth_token=HF_TOKEN, device=device)
diarize_segments = diarize_model(audio)
result = whisperx.assign_word_speakers(diarize_segments, result)

# result["segments"] now contains { start, end, text, speaker }

CLI:

whisperx audio.wav \
  --model large-v2 \
  --diarize \
  --highlight_words True \
  --hf_token <YOUR_HF_TOKEN>

Best for: the default choice. Fast batch processing, multilingual support, most actively maintained, biggest community.

Path 3 — Roll Your Own (Whisper + Raw pyannote)

If you want full control or are learning the pipeline, build from primitives. This is also the approach if you need to integrate with an existing custom ASR/diarization stack.

import whisper
from pyannote.audio import Pipeline

# 1-2. Transcribe with Whisper (word_timestamps=True is critical)
model = whisper.load_model("large-v3")
asr_result = model.transcribe("audio.wav", word_timestamps=True)

# 3-4. Diarize with pyannote
pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="YOUR_HF_TOKEN")
diarization = pipeline("audio.wav")

# 5. Align: map each word's midpoint timestamp to its speaker turn
for segment in asr_result["segments"]:
    for word in segment.get("words", []):
        midpoint = (word["start"] + word["end"]) / 2
        word["speaker"] = "UNKNOWN"
        for turn, _, speaker in diarization.itertracks(yield_label=True):
            if turn.start <= midpoint <= turn.end:
                word["speaker"] = speaker
                break

# Now each word has a speaker label

A good reference for a clean class-based version is the Scalastic three-class architecture (separate WhisperAudioTranscriber, PyannoteDiarizer, SpeakerAligner classes) — useful if you're building this into an existing codebase.

Best for: learning, custom pipelines, integrating diarization into an existing ASR system, or when you want to swap individual components (e.g., a custom alignment model).

OpenAI's gpt-4o-transcribe-diarize (2026)

The biggest recent change in the diarization landscape: OpenAI shipped a new model in 2026, gpt-4o-transcribe-diarize, that bundles ASR + diarization natively. Many existing tutorials (and older blog posts ranking on this query) don't mention it.

• Native diarization via /v1/audio/transcriptions with diarized_json response format
• Supports up to 4 known_speaker_references[] (2-10s audio clips) to map speaker IDs to known names
• Reported around $0.006/min at launch (verify on platform.openai.com)
• NOT available on the Realtime API yet — /v1/audio/transcriptions only

When to use it: simplest integration if you're already on OpenAI APIs, you don't need custom vocabulary or on-prem deployment, and basic speaker labeling is enough. When it's not enough: if you need consistent speaker identity across files (the "same person in episode 1 and episode 5" problem), custom-trained ASR, or features beyond diarization (summaries, meeting bot, multi-format export).

The GPT-4o Post-Processing Hack

From the OpenAI community thread on Whisper diarization — undocumented in any major tutorial: after running WhisperX or similar, send the rough transcript to GPT-4o and ask it to relabel speakers based on conversational cues. Works surprisingly well for content with clear conversational structure (interviews, podcasts where the host asks questions, customer-support calls).

prompt = f"""Below is a transcript with speakers labeled
SPEAKER_00, SPEAKER_01, etc. Based on conversational context
(questions, responses, name mentions, role indicators), relabel
the speakers with their likely roles or names if mentioned.

If you cannot confidently identify a speaker, keep the generic
SPEAKER_XX label rather than guessing.

Return the transcript with corrected speaker labels.

Transcript:
{transcript}
"""

# Then send to GPT-4o via the chat completions API

When it works: structured conversations (host + guest, interviewer + interviewee, agent + customer) where one speaker introduces themselves or asks distinctive questions.

When it doesn't: technical panel discussions, group meetings with multiple peers, audio without role-distinctive language. GPT-4o will sometimes hallucinate confident-sounding names — explicitly instruct it not to guess.

Production Deployment Realities

What every laptop tutorial skips. If you're going past hobby usage:

• • GPU cold starts — loading large-v3 + alignment + pyannote takes 30-90 seconds per cold start. Use warm pools or persistent workers. Don't spin up a fresh container per request.
• • Queueing — for multi-tenant or batch workloads, queue audio files. Don't try to run multiple diarizations in parallel on one GPU beyond batch_size limits — you'll OOM.
• • Memory management — keep models loaded across requests; loading per-request adds the cold-start hit every time. Watch out for memory growth in long-running workers.
• • Retries — pyannote pipelines occasionally hang. Implement timeouts + exponential backoff. The Hugging Face Forum has an active thread on this.
• • Dependency pinning — pin PyTorch, CUDA, cuDNN, CTranslate2, pyannote, faster-whisper versions in a Docker image. Update one at a time. Skip a major pyannote release and you'll spend a day re-pinning.
• • Speaker count hints — if you know N speakers, pass it. Auto-detect works but constrained always wins.
• • Multi-region / latency — if you serve users globally, plan GPU pools per region. A diarization job that takes 5 minutes round-trip from Sydney to a US-East GPU pool feels broken even when it's technically working.

Managed Services for Whisper + Diarization

Pricing changes — verify each before budgeting. As of mid-2026:

• • OpenAI gpt-4o-transcribe-diarize — native diarization on the OpenAI audio API. ~$0.006/min. Simplest if you're already on OpenAI; basic feature set.
• • AssemblyAI Universal-2 — strong all-around. Diarization is a small add-on fee. Good docs.
• • Deepgram Nova-3 — fast, has real-time streaming with diarization. Base ~$0.46/audio-hour PAYG plus diarization add-on.
• • Rev AI Turbo — cheapest mainstream API (~$0.10/audio-hour). Good for batch.
• • Speechmatics — strong accuracy reputation, on-prem option available, enterprise pricing.
• • VexaScribe — Whisper-based service with diarization, AI summaries, and meeting bot bundled. Best fit if you want speaker-labeled transcripts as part of a full transcription product rather than just a diarization API. From $0.30/audio-hour on the $2/month Starter plan.