Does adding the agent's own audio
as context really help?

The base model is a standard encoder–decoder ASR system. In standard inference, only the user's audio enters the encoder. In fusion mode, we concatenate the agent's audio with the user's audio before encoding:

The mechanism is clear and the intuition is sound. But does it hold on real data? To find out, acoustic fusion was evaluated on 282 call-centre conversation pairs — each pair consisting of an agent utterance followed by a user response. The agent audio was prepended with a fixed 1000 ms silence gap and zero blank penalty, the simplest possible configuration, to see what raw context injection does before any parameter tuning.

Every sample was decoded twice — user audio alone, then fused — and the combined CER + WER delta determined which of three outcomes occurred.

Aggregate CER numbers tell you fusion failed — they don't tell you how. To find out, every decode pass was instrumented at the joint-network level: logits captured at each emitting step, encoder hidden states recorded at every token-emitting encoder frame, and emitting frames matched across the user-only and fusion decodes. Three sample classes emerged across 282 samples.

The finding is precise: fusion does not fail by hallucinating wrong tokens. It fails by silencing correct ones — the decoder's hidden state, primed by the assistant audio, raises blank preference at the exact positions that were already emitting correctly. The token identity is clear; the decoder just chooses not to say it.

For each sample, encoder frames that produced a non-blank token in the user-only decode were matched against the fusion decode (±2 frame tolerance). The gross new rate — fusion emissions with no user counterpart — is nearly identical across all three classes (~10%). Degradation is explained entirely by suppression, not hallucination.

All features below are measured only at joint-network steps where the model chose to emit a non-blank token. Δ = fusion − user. Note that entropy_top20 barely changes for degraded (−0.051): the model already knew which token to emit — blank is overriding a confident, correct decision. dur_entropy is identical across classes, confirming the contamination targets blank preference exclusively, not generalised decoder confusion.

If the encoder were causing blank suppression, degraded samples would show the largest shift at selected frames. They show the smallest. The suppression comes from the prediction network's recurrent hidden state — accumulated during assistant audio processing — not from the per-frame encoder representation. Encoder shift magnitude is inversely correlated with suppression rate.

The blank penalty acts on the logit. But there is a completely independent knob that acts on the decoder's state: the silence gap inserted between agent and user audio. The decoder is a recurrent system — it carries memory of every frame it has processed. When agent audio ends and user audio immediately begins, that memory is still active. The decoder is not ready to hear a new speaker.

Silence changes this. A sufficiently long gap looks acoustically like the natural pause between two utterances. The decoder's hidden state drains, its agent-mode prior fades, and blank_p returns to a level where user-speech tokens can compete. With the right silence duration and zero blank penalty, suppression decreases on its own — no logit manipulation required.

Grid search finds the best parameters for a given sample — but it cannot generalise. A more useful capability would be to predict the right blank penalty and silence duration directly from features of the input: the audio's acoustic properties, the agent's duration, the encoder's hidden-state statistics. The idea is to treat the fusion system as a learnable function — extract features from what we can observe before decoding, and train a lightweight model that maps those features to the parameter pair that maximises transcription quality. If the feature set is sufficiently discriminative, the model could replace the grid search entirely, selecting parameters per-sample at inference time with no additional decoding passes.

Seven candidate features were engineered from the audio and encoder state. Each was evaluated for its ability to separate strong positive samples from hard negatives — measured as the mean difference between the two tiers (Diff: strong positive vs. hard negative). Features with a near-zero difference carry no signal for the predictor.

Sample combinations were categorised into tiers based on win_rate — the fraction of 293 parameter combos that beat no-fusion — and best_combined_delta, the maximum improvement achieved by any combo. The tiers define how learnable each sample's optimal parameters are:

Acoustic fusion carries a genuine signal. The agent's prior question, encoded alongside the user's response, does shift the decoder's output toward contextually appropriate tokens. But the mechanism is fragile — blank suppression means the decoder cannot hear the user at all unless two knobs are tuned in tandem. When both are tuned jointly via grid search, the approach delivers up to +5.30 points on the combined CER+WER metric; when either knob is fixed unilaterally, performance collapses by as much as −12.84 points.

The run analysis also tells a more nuanced story. Fusion is not failing uniformly — in the best runs, over 100 samples improved while fewer than 45 degraded. The aggregate metric suffers because degradation on a subset drowns out the gains. The signal is real; the problem is the bleed.

The harder open question is whether the right parameters can be predicted without a grid search. Seven candidate features — spanning audio duration ratios, speaking rate, encoder hidden-state statistics, entropy, and cosine similarity — were evaluated for discriminative power. None showed more than a weak signal; five of the seven sat near zero. The implication is direct: no lightweight model trained on these features can reliably replace the grid search at this stage. Per-sample parameter selection at inference time is not yet tractable from observable inputs alone.