Turn-taking and barge-in

Timing is where voice agents fail. Not pronunciation — timing. The model answers the half-sentence you abandoned, or waits through a silence you meant as an invitation, or keeps talking over you for two seconds after you cut in. This page is the contract for that. Some of it is the server’s job, some of it is yours, and the parts that go wrong go wrong in ways you can reproduce.

How the server decides you stopped

Inbound audio is resampled to 16 kHz and fed to a voice activity detector in 512-sample frames. The detector emits two events, and both cross the socket:

input_audio_buffer.speech_started — speech began. If a reply is playing, this is a barge-in.
input_audio_buffer.speech_stopped — the silence after your speech got long enough to count as the end of a turn. The reply’s clock starts here.

The detector reads arrival timing, not a length field. Frames that arrive in a burst look like one continuous stretch of speech with no pauses in it.

Pace your sends at the rate the audio was recorded — about 20 ms of 16 kHz mono PCM16 per message. Pushing a file into the socket at once does not make the answer come sooner; it makes endpointing wrong, and the failure looks like a model that cannot tell when you have finished.

Knobs you can turn

All four live under session.audio.input.turn_detection and take effect on the next frame:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": {
        "turn_detection": {
          "type": "server_vad",
          "threshold": 0.6,
          "silence_duration_ms": 64,
          "interrupt_response": true,
          "create_response": true
        }
      }
    }
  }
}

Field	What it changes
`threshold`	Confidence needed to call a frame speech, 0 to 1. Raise it in a noisy room; lower it for a quiet talker.
`silence_duration_ms`	How much silence ends a turn. Raise it for people who pause mid-thought; lower it for a snappier agent that will interrupt more.
`interrupt_response`	`false` lets the reply finish while the user talks over it. Their speech is still transcribed. This is what an interpreter needs, because being a beat behind the speaker is its steady state.
`create_response`	`false` transcribes turns without answering them — dictation with live captions. An explicit `response.create` still works.

type is required. turn_detection is a discriminated union, and both members declare type as mandatory. A block without it fails validation, and the whole session.update is rejected with it — instructions and voice included. What comes back is one error event with type unknown_or_invalid_event naming session.update and not the offending field, so a client that does not render errors sees a session that quietly kept its old configuration.Use "server_vad". threshold and silence_duration_ms are its fields; the "semantic_vad" member accepts eagerness instead, and this backend does not act on it. prefix_padding_ms and idle_timeout_ms validate and are also ignored here. The detector is the server’s own either way, and reads only threshold, silence_duration_ms and interrupt_response — so the choice of member costs nothing but the field names you get.A translation session is the one you did not configure: the gateway injects a semantic_vad block on every frame that names no turn detection, for the interrupt_response: false inside it.

Values you set are per session. The serving pipeline is pooled and reused, so the server restores its own defaults when your session ends; the next caller never inherits your barge-in sensitivity.

A pause is not always the end of a turn

An end-of-turn decision made from silence alone is wrong often enough to be noticeable, so the server does not commit to it immediately. A turn that ends softly stays reopenable for about a second. If you start speaking again inside that window and nothing has answered yet, the same turn is reopened and re-transcribed with the continuation attached, rather than becoming a second turn that contradicts the first. If no reply has committed the turn at all — the model is still thinking — the window stretches to about seven seconds, because the conversation has not moved on. A turn stops accumulating after 30 seconds of audio, at which point a new one starts; without that cap a slow first token would make a still-talking speaker re-transcribe a buffer that grows with every sentence. You do not configure this, and you cannot observe the reopen directly. What you see is the effect: conversation.item.input_audio_transcription.completed arriving with the whole thought instead of the first half of it.

Barge-in

When the user speaks over a reply, the server cancels rather than finishing. In order:

input_audio_buffer.speech_started is sent.
Generation is cancelled — the language model and the synthesizer both abort mid-token, and audio already queued for you is dropped.
The turn closes with response.done, response.status: "cancelled" and status_details.reason: "turn_detected".
The pipeline is already transcribing the new utterance.

Your client owes it one thing:

ws.addEventListener("message", (message) => {
  const event = JSON.parse(message.data);
  if (event.type === "input_audio_buffer.speech_started") {
    player.stop();     // stop now
    player.clear();    // and throw away what is buffered
  }
});

Draining the buffer you already have is the most common cause of an agent that feels slow despite fast server numbers. The server cancels in milliseconds; an audio element that keeps talking for another two seconds is answering a question the person has moved on from. Barge-in is a client responsibility as much as a server one.

response.cancel does the same thing on your own initiative, closing the turn with reason: "client_cancelled".

conversation.item.truncate is accepted and acknowledged with conversation.item.truncated, but the server keeps no per-item audio history to trim, so it changes nothing on this backend. Barge-in is already handled by the path above. Send it if your client library does; do not rely on it to correct the model’s context.

What still goes wrong

Three failures are worth knowing before you hear one and conclude the model is bad.

A thinking pause read as the end of a turn

Somebody says “so the invoice number is…” and stops to look it up. The reopen window covers a second of that, and silence_duration_ms covers more if you raise it, but a five-second hunt through a drawer will be answered. If your users routinely pause to fetch information, raise silence_duration_ms and tell the model in instructions to wait rather than to fill silence.

An interruption during the first token

Cancellation is cooperative: a handler already blocked inside a network call is closed rather than polled, but the very first moments of a turn are the tightest race in the pipeline. Occasionally a syllable of the cancelled reply reaches you. Dropping playback on speech_started hides it; draining your buffer does not.

Recovery after a cut-off turn

A cancelled turn is still a turn: it is in the conversation as the partial thing the model said. If your agent tends to restart its whole answer after an interruption, that is a prompt problem, not a protocol one — say in instructions that it should answer the new question rather than resume the sentence it was cut off in.

Measure it instead of listening for it

Every session records what actually happened, so you can find a turn-taking bug in data rather than by ear. Per organization, the aggregate carries the distribution and the barge-in count. This one is always recorded, whatever your deployment does with audio:

curl "https://api.eesi.ai/v1/organizations/turn-metrics/summary?days=7" \
  -H "Authorization: Bearer $EESI_API_KEY"

from eesi_sdk import EESIClient

with EESIClient(api_key="sk-eesi-...") as client:
    report = client.get_organization_turn_metrics_summary(days=7)
    print(report.summary.turn_count, report.summary.eou_to_first_audio_p50_ms)
    print(report.summary.barge_in_count)

import { EESIClient } from "@eesi/sdk";

const client = new EESIClient({ apiKey: process.env.EESI_API_KEY });
const report = await client.getOrganizationTurnMetricsSummary({ days: 7 });
// Response models keep the wire's snake_case field names.
console.log(report.summary.eou_to_first_audio_p95_ms, report.summary.barge_in_count);

A null percentile means nothing was measured in that window, never zero: a turn contributes only when the reply actually produced audio, so a greeting the agent opened with carries no number. A rising barge_in_count with a stable latency curve usually means the agent is too verbose rather than too slow — see Latency. Per turn, where the deployment retains session transcripts, each assistant entry in a session’s transcript carries its own ttfb_ms and an interrupted: true flag, so you can read which turn was cut off and what the model had been asked to say in it:

curl "https://api.eesi.ai/v1/speech/sessions/$SESSION_ID" \
  -H "Authorization: Bearer $EESI_API_KEY"

An environment that keeps no session audio keeps no transcript either, and the transcript field comes back empty. The organization aggregate above is unaffected. Where a recording does exist it tells you the rest: the assistant track is cut back to the wall-clock point playback reached, so what you hear is what the caller heard, interruption included.

Latency

Where the seconds go, and which of them you control.

Duet

Two live sessions, which is how you exercise all of this on purpose.

Realtime events

The event names above, in both directions.

Sessions

Transcripts, per-turn timing and recordings.

Start here

Realtime interaction

Speech

Platform

Turn-taking and barge-in

How the server decides you stopped

Knobs you can turn

A pause is not always the end of a turn

Barge-in

What still goes wrong

Measure it instead of listening for it

Next

Latency

Duet

Realtime events

Sessions

​How the server decides you stopped

​Knobs you can turn

​A pause is not always the end of a turn

​Barge-in

​What still goes wrong

​Measure it instead of listening for it

​Next

Latency

Duet

Realtime events

Sessions

How the server decides you stopped

Knobs you can turn

A pause is not always the end of a turn

Barge-in

What still goes wrong

Measure it instead of listening for it

Next