Conversation Context for Voice Agents: What It Is and Why It Matters for Accuracy

Voice agents without conversation context guess blind on the calls that matter most: spelled-out numbers, one-word confirmations, and unfamiliar terms. Getting one wrong decides whether a call resolves in self-service or escalates to a live agent.

A 2024 dialogue-ASR system that combined contextual awareness with noise representation learning cut relative word error rate by 24% against the wav2vec2.0 baseline. Against the Whisper-large-v2 model, the same system cut relative WER 13%.

Those figures cover the combined system. One of its components is conversation context, which includes prior turns and expected response types. It can also include known entities your agent already holds when the caller speaks.

Short replies and spelled-out entities carry little acoustic signal: a one-word "yes," a single digit, and a spelled-out email address. Context resolves that ambiguity, but whether it reaches your transcription layer automatically or through plumbing you build yourself depends on your architecture.

This article covers where that ambiguity concentrates, how conversation context changes decoder predictions, and how to build and measure it in your own stack.

The evidence supports four practical conclusions:

• Ambiguity concentrates in short replies such as digits and in spelled-out entities, which carry minimal acoustic signal.
• Cascaded stacks hand STT services audio only by default; a unified session manages history for the reasoning layer while keyterms adapt STT.
• Flux keyterms can be updated as the conversation changes.
• Splitting B-WER from U-WER on your own calls proves conversation context helps.

Your choice determines whether you manage separate interfaces or one session with distinct history and adaptation controls. Compare the two on documented interface boundaries, pricing models, compliance options, and deployment control, and keep LLM history management separate from STT adaptation since those controls affect different layers.

The Deepgram unified API reduces service plumbing, while cascaded frameworks such as Pipecat or LiveKit Agents provide more component-level control.

Speech recognition guesses at the best match for what it hears, and that guesswork fails in predictable spots: short replies, spelled-out entities, and any turn where nothing resolves the ambiguity. That's a prediction problem at its core, one that concentrates in specific calls and compounds fast once it takes hold.

When two word sequences sound nearly identical, the acoustic evidence can't pick a winner. Decoding picks the sequence with the highest combined score from the acoustic model and the language-model prior, so when acoustic scores tie, the prior decides.

Homophones such as "wait" and "weight" produce the same signal, so the surrounding language tips the choice. Conversation context reshapes that prior with information the audio doesn't carry: what the agent just asked, and what kind of answer the dialogue expects.

Errors cluster in letter-by-letter spellings and short replies, including one-word confirmations or single digits. Alphanumeric strings can be difficult because confusable letters and digits arrive with no sentence structure to lean on.

Five9 reported that Nova-2 was 2 to 4 times more accurate than alternative STT options on alphanumeric inputs, including tracking numbers, account IDs, and member IDs.

Rare words fail at rates that would sink any production agent: 21.83% WER on LibriSpeech without contextual biasing. And in a multi-turn call, one error rarely stays contained. Say the agent misrecognizes an account number: it reads the number back wrong, the caller corrects it, and two more turns are gone before the task resumes.

The pipeline held the disambiguating information the whole time; the transcription layer never received it. If you've sat through that loop on a support call, you know how fast one bad digit turns into three wasted turns.

If the decoder knows the agent just asked a yes-or-no question, "yes" stops competing with acoustically similar noise. Passing prior turns and expected response types into decoding shifts probability toward the answer the dialogue predicts.

ASR decoding maximizes acoustic likelihood multiplied by a language-model prior, and contextual biasing intervenes at two different points in that computation. Shallow fusion adds a contextual score to each hypothesis during beam search, while deep biasing injects context into the model's internal representations before the output distribution is computed.

Named entities have the most room to fall, since a name the model has never seen is more likely to be misrecognized without help. In one measured case, a phonetic retrieval-based approach to LLM contextualization cut relative WER by 30.2% and named-entity error rates by 73.6%.

You can hand the decoder the previous turn directly. The PromptFormer model feeds prior-turn ASR output into a Conformer Transducer; that alone cut relative WER by 5.9% on multi-turn tests without degrading non-contextual cases.

Simply prompting the model with context doesn't reproduce this gain. The ProfASR-Bench benchmark found that lightweight textual prompts produced little to no change in average WER, even when the prompt contained the gold transcript. Context affects transcription only when the decoding process uses it; context confined to a voice agent prompt doesn't affect decoding.

For different ambiguity types, combine dialogue history with Keyterm Prompting. Pin up to 100 domain terms per request, capped at 500 tokens, on Deepgram's Nova-3 and Flux speech-to-text models.

Nova-2 and older models use the earlier Keywords feature instead. Keyterms are plain text, with no weights or intensifiers, and on Flux you can replace the list mid-stream as the call shifts topics.

Keyterms and dialogue history solve different problems here. A keyterm list asserts which terms are acoustically plausible; dialogue history helps establish which one the caller means right now. Pin your product names and jargon with keyterms, and let dialogue history resolve the rest.

Choose a managed session for less service plumbing, or thread turns by hand between three separate services.

Expect to thread transcription context yourself in most cascaded stacks. Frameworks route conversation history to the LLM by default, not to the STT layer. That interface boundary leaves the component making the hardest prediction, resolving ambiguous audio, with the least information.

One provider exception exists: a real-time STT service can accept prior assistant turns as context-carryover input, on by default. Outside that exception, additional transcription context in this cascaded setup has to go through your chosen STT service's own controls, such as assembling and pushing a lexical biasing list yourself.

One WebSocket session carries STT, LLM orchestration, and TTS in Deepgram's Voice Agent API. Prior turns go into the LLM reasoning layer through the agent.context.messages configuration at session start, while STT-side adaptation runs separately through keyterms in the listen configuration.

Separately, Flux TTS isn't integrated with the Voice Agent API, but it keeps conversational state across turns automatically. Its Mamba state-space backbone holds a fixed-size memory of its own prior acoustic generations, without transformer attention's quadratic cost, and excludes user audio, user text, and LLM reasoning.

No configuration is needed; the memory persists by default. When your Flux TTS Interrupt includes playback_offset, the response reports response fields named text_spoken and text_remaining. You can use those fields to reconcile LLM history with what the caller heard.

When a caller barges in mid-response, what your pipeline records can diverge from what the caller heard. Some systems log the full reply as committed even though playback cut off early.

For example, a system might commit "the pool opens" to history while the caller actually hears "the pool opens at six," with the extra words leaking into the next assistant turn. The general fix is to truncate committed history to match what the caller actually heard, not what the model generated.

If a vendor demo transcript is your only evidence, you have none for your own call mix; recordings from your traffic give you some. Build that test set once and the context-on versus context-off comparison becomes repeatable.

Production recordings from production calls give you a representative benchmark when they cover hard acoustic conditions as well as the clean and medium tiers. A small clean sample can understate production error rates because hard calls drive the real failures. Use human-verified verbatim transcripts as ground truth. Never substitute another STT vendor's output.

Use the B-WER and U-WER split from LibriSpeech biasing work: measure B-WER on the words context supplied and U-WER on everything else. Improve B-WER without degrading U-WER.

Over-boosting is a documented failure: one default boosting approach cut B-WER while pushing U-WER from 16.81 to 21.39, and overall WER rose with it. A follow-up n-gram-aware fix recovered all three metrics at once: 10.83 WER, 10.44 U-WER, 13.42 B-WER.

If overall WER improves but latency or U-WER slips, the variant loses. Hamming's A/B testing guide formalizes this with one change per variant. A variant only wins if the primary metric improves while latency, safety, escalation, tool behavior, and repeat-contact rates all stay within the bounds set in advance.

Replay the same recorded audio through context-on and context-off configurations before touching live traffic. In production, monitor WER weekly and investigate drift greater than 2% from baseline.

A decoder that knows the previous turn ranks different words as probable. It rescues short replies such as digits and spelled-out entities. Choose whether to thread prior turns between services or use a unified session with separate LLM history and STT keyterms.

Before committing to either approach, run these checks against your current stack (an hour here is cheaper than debugging it in a support ticket):

• Trace whether conversation history can reach your STT layer at all, or only the LLM.
• Replay a week of recorded calls through context-on and context-off configurations.
• Interrupt the agent mid-sentence, then diff the committed history against the audio the caller heard.
• Watch B-WER and U-WER separately so gains on target terms can't hide regressions elsewhere.

Test your hardest calls: the spelled-out emails and one-word answers your current stack fumbles without conversation context. Deepgram's Voice Agent API handles that context in one session, from LLM history to STT keyterms, so you're not stitching it together yourself.

Create a free account and use your $200 free credits on those calls.

In Deepgram's Voice Agent API, conversation history is on by default; set settings.flags.history to false to turn it off.

Whisper's effective prompt is capped at 224 tokens, so long histories need trimming before injection.

Deepgram agent sessions close after 2 hours, with a warning 5 minutes ahead, so long calls need prior turns replayed into agent.context on restart.

On Flux, a Configure message replaces the entire keyterm list rather than merging it, and already-transcribed audio isn't reprocessed.

Test significance with paired methods such as MAPSSWE in NIST SCTK, since the same utterances appear in both arms. Apply identical text normalization to hypothesis and reference before scoring.