Testing conversational repair in voice AI agents - Roark

Every voice agent mishears people. The question that decides whether your agent keeps the call is what happens in the next 800 milliseconds. Does it ask for a repeat with a generic "I didn't catch that," parrot back the wrong value with confidence, or thread a specific clarification that gets the conversation moving again? That behavior has a name in the linguistics literature: conversational repair. It is the connective tissue of a working call, and it is the layer teams ship with the least testing.

Repair is not a feature you turn on. It is a set of behaviors that emerge from your prompts, your ASR settings, your no-input timeouts, and the way your model handles low-confidence signals. When any of those drift, repair breaks in ways your happy-path evals never catch. This post is about how to test repair the same way you test tool calls: with deliberate scenarios, scored metrics, and regressions that fail your build.

What conversational repair actually is

The academic frame is useful here. Repair is the interactional effort by which participants suspend the ongoing talk to address potential trouble, categorized by who spots the problem and who fixes it. In a voice call, the two patterns you'll see constantly are:

• Agent-initiated repair. The agent detects ambiguity or a low-confidence signal and asks a targeted question ("You said Baker Street, one four two, is that right?"). Google's own guidance recommends the "You mean X?" structure for exactly this reason: it gives the user something specific to confirm or correct, rather than starting the slot over.
• Caller-initiated repair. The caller notices the agent has the wrong value and corrects it mid-flow ("no, one four two, not one forty-two"). Your agent has to accept the correction, roll back the slot, and not restart the whole intent.

The third case, and the one that quietly ruins retention, is the no-input branch: the caller says nothing at all. Dialogflow's own recommendation is a max of three No-Match/No-Input events per page before you escalate to a human. Very few production agents actually count and cap this. They just loop.

The four repair failures teams ship

Before we get to what to test, name the failures. These are the ones we see over and over on production calls:

1. The "didn't catch that" loop. Same generic reprompt three times in a row, then a hang-up or an unhelpful transfer. Google's guidance is explicit: avoid asking obvious and redundant questions like "I didn't catch that" or "I don't understand." Yet it's the default fallback in half the agents in the wild.
2. The confident confabulator. The ASR hypothesis was garbage, but the LLM smoothed it into something plausible and moved on. The caller gets a booking they didn't ask for, or a policy quoted for the wrong product. This is the failure mode that costs the most, because the call sounds fine on the recording.
3. The over-checker. The agent confirms every slot even when confidence is high, turning a two-minute call into a four-minute one. Confirmations are cheap on their own and expensive in aggregate. Callers get impatient and interrupt, which triggers barge-in edge cases you haven't tested either.
4. The parroter. The agent asks the caller to repeat the whole utterance verbatim ("Can you say that again?") instead of narrowing to the ambiguous span. On phone lines, if it didn't parse the first time, it usually won't parse the second time either. You need to narrow, not restart.

Each of these fails silently in offline transcript-only evaluation. You need audio, and you need scenarios that produce the trouble in the first place.

The full-duplex twist

If you're on a cascaded pipeline (STT into LLM into TTS), your repair loop is bounded by turn detection. The model waits until the caller finishes speaking, then reasons about whether it heard something ambiguous, then responds.

Full-duplex models change the shape of the problem. GPT-Live and the newer gpt-realtime family continuously process incoming audio and can listen while speaking, without a separate turn detector controlling the live audio path. That means the caller can start correcting mid-response ("no, one four two, not one forty..."), and the agent needs to actually stop, reroute, and pick up the correction without losing the slot it was building.

The gpt-realtime announcement highlights the exact behaviors this unlocks: reading disclaimer scripts word-for-word, repeating back alphanumerics, and switching languages mid-sentence. Every one of those is a repair-adjacent behavior. Every one of them is worth testing explicitly, because "the model can do it" is not the same as "your prompt makes it do it reliably on a jittery cellular line."

Where repair fires inside a turn

What to test, concretely

A repair test suite has four scenario families. Every family has a pass condition that is measurable from audio, not just from the final tool call.

1. ASR-corruption scenarios

You want deliberately hard inputs, and you want the agent's first response scored on whether it repaired vs. confabulated.

• Mumbled utterances at the boundary of the wake slot.
• Low-SNR calls (traffic, wind, a dishwasher). Roark supports background-noise environments as first-class persona attributes so you can bake this into the caller, not into your test harness.
• Accents outside your prompt's default. If your agent serves a US phone number but 20% of your callers are second-language English speakers, test that mix. The audio-native metrics you score against, pronunciation, pace, interruptions, are the ones that break first on accented input.
• Code-switching mid-utterance ("my address is one forty-two, calle Ocho").

Pass condition: agent asks a narrowed clarifying question ("You said Baker, is the number one four two or one forty-two?") within N seconds. Score against your repair-quality metric, not just "did it eventually get the slot."

2. Ambiguity scenarios

The ASR is fine; the meaning isn't.

• Homophones ("Two, to, too" in an order-quantity slot).
• Partial names in a directory ("I want to talk to Sam"). There are three Sams.
• Pronouns without antecedents ("book the other one").
• Numbers spoken naturally that could parse two ways ("fifteen fifty" as $15.50 or 3:50 PM).

Pass condition: the agent surfaces the ambiguity instead of picking. "You mean X?" structure, one round, resolved.

3. No-input scenarios

Silence tests are the ones every team says they'll add and never does.

• Full silence after the prompt (initial-silence timeout).
• Silence mid-utterance, then completion (end-of-speech timeout). Tools like SignalWire expose this as a per-turn parameter because it needs to change based on what you just asked.
• Silence after a long agent explanation (the caller is still reading their screen).
• Silence that turns out to be call-quality (packet loss), not human hesitation.

Pass condition: the reprompt is different from the original prompt, escalates specificity, and the loop is bounded. Three strikes and you escalate.

4. Correction scenarios

The caller notices the mistake before the agent does.

• Correction in the same turn ("one four two, sorry, one four two").
• Correction one turn later ("wait, that's wrong").
• Correction after a confirmation ("no, not that").

Pass condition: the correction is accepted, the previously filled slot is invalidated, and the agent does not restart the whole intent.

Repair scenario suite pre-launch

The metrics that matter

You can't score repair with pass/fail alone. A useful repair-quality bar looks like this across a batch of calls:

Every one of these needs the audio, not just the transcript. A transcript-only eval will happily mark a confabulation as a pass, because the tool call succeeded and the words on the page look reasonable.

How this fits into a testing workflow

The workflow that actually works looks like this:

1. Pre-launch simulation. Run a repair suite that dials your agent over real telephony with personas configured to produce the four scenario families above. Roark's simulations dial your agent over PSTN or WebRTC and drive personas with configured voices, accents, pace, emotional register, and background noise, so a "mumble in a moving car" is a persona attribute, not a hand-cut audio file.
2. Score every live call against the same metric suite. Once you're in production, every call gets scored against the repair metrics you defined pre-launch. When the false-confirmation rate ticks up after a model swap, you see it in the dashboard before the CSAT complaints show up.
3. Turn real repair failures into regression tests. Production call replay captures the failure, then re-runs it against the updated agent so you can prove the fix without waiting for it to happen again. Your best test cases live in your call logs.
4. Gate CI on the repair suite. Roark's simulation runs are triggerable over HTTP, which means you can fail a deploy on a regression in narrowed-reprompt rate the same way you'd fail on a broken tool call.

Repair-loop call with the failure filed

Anti-patterns to strip from your prompt today

Even without a testing platform, three prompt changes clean up the majority of repair failures we see:

• Replace generic "I didn't catch that" with a slot-specific reprompt that echoes what you did hear. "I heard the street name Baker, but I'm not sure of the number. Was it one four two?"
• Bound the loop in the prompt itself. Two narrowed clarifications, then transfer, and say so in the system prompt so the model doesn't invent its own escalation path.
• Never confirm high-confidence slots by default. Confirm only when the model's own confidence is low, or when the slot is high-stakes (money, patient identity, address). Confirmations are a tax; make sure you're only paying it when you have to.

None of these fixes work if you can't measure them. Repair is one of the few voice-agent behaviors where "we tightened up the prompt" produces confident regressions if you don't have the audio in a loop with the tests. Build the suite, run it before launch, keep scoring it after.

Voice agents fail loudly on the happy path and quietly on repair. The teams that ship reliably are the ones that treat repair as its own testable surface, not a side effect of the prompt.

Written by

Daniel Gauci Mizzi · Co-founder & CTO @ Roark

Building Roark — the quality platform that simulates, monitors, and auto-improves voice and chat agents.