The cost of context.
In voice systems, latency is not a metric. It is the experience. A few hundred milliseconds is the difference between a conversation and a wait, and a wait is the moment a caller decides the thing on the other end is not quite human.
For a long time, every turn of a voice call carried the same weight. The full set of instructions. The full set of edge cases. Every fallback we had ever written. As the conversation grew, the prompt grew with it. By the time a caller reached the middle of a flow, the model was reading pages of text that had nothing to do with the next thing it had to say.
We did not notice for a long time because it worked. Accuracy was fine. But every turn added time, and in a voice interface, time is something people can feel.
We were asking the wrong question.
The first instinct when latency creeps up is to ask: how do we make this prompt faster? Smaller models. Faster infrastructure. Heavier caching.
We tried the usual levers. The returns were real but modest.
The question we should have been asking was quieter — what does the model actually need right now? Most of the time, the answer was surprisingly little.
From context dump to intent-aware context.
Every voice turn now passes through a thin prediction layer before the model is ever called. It reads where the conversation is, anticipates the caller's likely next intent, and hands the model only the slice of context that intent actually needs — the same answer, from a request a fraction of the size.
Full context injection
All history + all instructions
Intent prediction layer
Predict the next user intent
Focus on the most likely next step
Smart prompt
Only what's needed for this turn
Every rule, every example, every past turn — shipped on every single call. Defensive. Comprehensive. Slow.
This rule. This example. The last two turns. Built fresh for this one moment — and nothing beyond it.
How a single turn is shaped.
Between the caller speaking and the model replying, four small things happen in the space of a blink. None of them involve summarising anything or throwing anything away — the layer simply decides, for this one turn, what the model needs and what it doesn't.
-
01
Locate the moment.
Every call is made up of a handful of recognisable moments — a greeting, an availability check, a scheduling step, a handoff, a close. Before anything else, the layer figures out which one we're in. The moment is a small, finite label; it tells the rest of the layer which rules are even relevant right now.
-
02
Read the caller.
The caller's last utterance is compared against a small set of intents that are plausible from the current moment — not every intent imaginable, only the ones that make sense here. A greeting moment might have three reasonable next intents; a scheduling moment has a different three. The layer ranks them and picks the most likely one.
-
03
Assemble a focused bundle.
For the winning intent, the layer puts together a tight packet: the specific instructions that govern that intent, the last two turns of conversation, one retrieved fact if the intent needs one, and a single relevant example. Everything else — other rules, other examples, earlier turns, fallbacks for intents we didn't pick — is left out.
-
04
Ship that, and nothing else.
The model sees the focused bundle and nothing else. No unused fallbacks, no unrelated examples, no stale history. When the caller speaks again, the whole sequence runs fresh for the new turn — new moment, new intent, new bundle.
The important thing to notice is what we are not doing. We are not compressing the prompt. We are not summarising history into a smaller blob. We are not dropping information we might later need. We are deciding, turn by turn, that most of the prompt was never going to matter for this particular reply — and simply not sending it.
See it run.
A scripted replay of a single call. Watch the prompt swap as the conversation moves from one moment to the next. Lines with a lightning mark play back from cache the instant that moment begins. Turns that need the model — intent routing, anything dynamic — wait on it, and return in under half a second because the prompt is tiny.
Agent lines marked are cached — fixed lines in the flow, played back the moment the node is entered. Turns that need the model (intent routing, dynamic replies) are the only ones that wait on it.
Smart Prompting.
So we built a thin layer — nothing fancy. It sits between the conversation and the model, and it does three small, deliberate things.
Know where the conversation is.
Every voice call moves through a handful of recognisable moments — a greeting, a confirmation, a clarification, a handoff. The layer tracks which moment you are in.
Guess what comes next.
Given that moment and what the caller just said, there is usually a small set of plausible next turns. Most instructions in the full prompt only matter for one or two of them.
Send only what is relevant.
Everything else stays out of the request. The model sees a prompt tailored to the moment, not a scroll of every rule we have ever written.
This is not compression. Nothing is summarised, nothing is lost. It is closer to a librarian who hands you the one page you need, instead of the whole shelf.
Prediction, not optimisation.
This is the part that surprised us most.
The bottleneck was never prompt size. Prompt size was the symptom. The bottleneck was that the prompt was trying to answer every possible question the model might have — a defensive, all-at-once design — when in practice the model only ever had one question at a time.
Reframed that way, it stopped being an optimisation problem. It became a prediction problem: which slice of this prompt does the model need, given what we know about the call so far?
Good predictions produce small prompts. Small prompts produce fast replies. Fast replies keep the conversation feeling human.
What changed.
Roughly fivefold smaller prompts. Roughly fivefold lower latency. No model change, no new infrastructure, no caching tricks.
The surprising part was not the number. It was how obvious the fix looks in hindsight — and how completely invisible it was until we changed the question.
Simple in hindsight. Not obvious in practice.