Latency

In a live conversation the number that matters is time to first audio: the gap between someone finishing their sentence and hearing the first sound back. Tokens per second and total generation time are invisible to the person on the call. This page is about which parts of that gap you control. Several of the obvious levers do nothing, and the one that works is not an infrastructure change.

Where the time goes

Stage	What happens	Share of the budget
End-of-turn detection	The detector waits out the pause and decides you stopped	Small, and mostly the server’s
Understanding	The model hears the turn and starts deciding what to say	Small
Synthesis	The first sentence of the reply is turned into audio	Most of it
Network	Frames cross to your client	Your distance to the region

Synthesis dominates, for a structural reason worth understanding before you tune anything else.

Synthesis happens per sentence

The model’s output is split at sentence boundaries, and each complete sentence is synthesized as one piece while the earlier ones play. Nothing is heard until the first sentence has been generated and rendered. Two things follow, and they are the whole page:

The length of the first sentence decides time to first audio.
The length of the whole reply decides how long the turn takes to finish, and therefore how long somebody has to wait to speak again.

Both are set from the prompt:

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "Answer in one or two sentences. Never read more than three items aloud. If a full answer would take longer than about fifteen seconds to say, give the short answer and offer to go deeper."
  }
}

That instruction is worth more than any infrastructure change available to you. Two sentences instead of two paragraphs is the difference between a conversation and a lecture, and it happens to also be the difference between a fast turn and a slow one. Three rules that follow from it: Ask for brevity with a number. “Be concise” is advice. “One or two sentences” is a rule the model can apply. Do not let the model read structure aloud. Bulleted lists, headings and numbered steps are long by construction. If your agent has a list to convey, have it say the first item and ask whether to continue. Push long content out of the voice channel. “I have sent that to your email” finishes in a second. Reading a policy paragraph does not.

Skip synthesis when nobody is listening

If a turn’s output is going to a screen rather than a speaker, ask for text and the pipeline does not synthesize at all:

{ "type": "response.create", "response": { "output_modalities": ["text"] } }

Set on the session it applies to every turn; set per response it applies to one. This is the largest single reduction available, because it removes the stage that dominates the budget. It is the reason a typed translation lands so much faster than a spoken one.

Pace your audio in real time

Send about 20 ms of 16 kHz mono PCM16 per message, at the rate it was recorded. The detector reads arrival timing to find the end of a turn, so pushing a whole file into the socket at once does not make the answer come sooner. It makes endpointing wrong: everything looks like one continuous burst of speech with no pauses in it. A microphone feeding the socket in real time is the case the detector is tuned for. Endpointing itself is a small, adjustable part of the budget: lower silence_duration_ms and the agent answers sooner and interrupts more; raise it and it waits. See Turn-taking.

Cut playback on `speech_started`

When someone interrupts, generation cancels server-side and queued audio is dropped. Your client has to do the same thing locally:

ws.addEventListener("message", (message) => {
  const event = JSON.parse(message.data);
  if (event.type === "input_audio_buffer.speech_started") {
    player.stop();
    player.clear();
  }
});

Draining a buffer you already hold is the most common cause of an agent that feels laggy despite good server numbers: the server cancelled in 50 ms and your audio element kept talking for another two seconds.

What does not help

Tempting	Why the number does not move
Sending audio faster than real time	Breaks end-of-turn detection. The reply starts later, not sooner.
Smaller audio chunks	20 ms is already far below every other term in the budget.
Pre-warming with an empty `response.create`	Produces a real turn the user hears.
Sending 24 kHz and hoping for better audio	The input is resampled to the pipeline rate regardless, and an undeclared rate is decoded wrong. See Live.
Holding a socket open between conversations	Costs money and a pipeline slot without moving latency at all. Open one when a conversation starts, close it when the conversation ends.

Measure it, do not estimate it

Every turn is timed from input_audio_buffer.speech_stopped to the first audio frame you could hear, and stored. Use that instead of a stopwatch. The organization aggregate below is recorded on every deployment, whatever it does with session audio.

# The distribution across your organization, last seven days.
curl "https://api.eesi.ai/v1/organizations/turn-metrics/summary?days=7" \
  -H "Authorization: Bearer $EESI_API_KEY"

from eesi_sdk import EESIClient

with EESIClient(api_key="sk-eesi-...") as client:
    summary = client.get_organization_turn_metrics_summary(days=7).summary
    print("turns", summary.turn_count)
    print("p50 ms", summary.eou_to_first_audio_p50_ms)
    print("p95 ms", summary.eou_to_first_audio_p95_ms)

import { EESIClient } from "@eesi/sdk";

const client = new EESIClient({ apiKey: process.env.EESI_API_KEY });
const { summary } = await client.getOrganizationTurnMetricsSummary({ days: 7 });
// Response models keep the wire's snake_case field names.
console.log(summary.turn_count, summary.eou_to_first_audio_p50_ms);

A null percentile means nothing was measured in that window, never zero — a turn contributes only when the reply produced audio, so a greeting the agent opened with carries no number. To find the bad conversation rather than the bad week, list sessions: each carries turns and median_reply_ms without shipping you the transcript. Open one and every assistant entry has its own ttfb_ms, so you can see which turn was slow and read what the model was asked to say in it.

curl "https://api.eesi.ai/v1/speech/sessions?limit=50" \
  -H "Authorization: Bearer $EESI_API_KEY"

Both of those come from the stored transcript, so they are present only where the deployment retains one. An environment that keeps no session audio keeps no transcript, and median_reply_ms comes back null — fall back to the aggregate. Nine times out of ten a slow session is a verbose one, and the fix is in instructions.

Batch synthesis is a different problem

POST /v1/audio/speech supports incremental phrase synthesis with nur-tts-v1 when response_format: "pcm" and stream_format: "sse" are combined. This lets playback start before the whole request finishes. File formats such as MP3 and WAV still wait for complete synthesis before their encoded bytes stream. For bulk generation, split long text into paragraph-sized pieces and write to disk as the response arrives. Limit concurrency when sharing GPU capacity with Live sessions. See Stream speech to a file.

Turn-taking and barge-in

The detector’s knobs, and how to tell a timing bug from a slow one.

Live

The protocol, and the sample-rate trap that presents as a bad model.

Pricing and limits

What an open socket costs while nobody is talking into it.

Sessions

Per-session timing, transcripts and recordings.

Start here

Realtime interaction

Speech

Platform

Where the time goes

Synthesis happens per sentence

Skip synthesis when nobody is listening

Pace your audio in real time

Cut playback on `speech_started`

What does not help

Measure it, do not estimate it

Batch synthesis is a different problem

Next

Turn-taking and barge-in

Live

Pricing and limits

Sessions

​Where the time goes

​Synthesis happens per sentence

​Skip synthesis when nobody is listening

​Pace your audio in real time

​Cut playback on speech_started

​What does not help

​Measure it, do not estimate it

​Batch synthesis is a different problem

​Next

Turn-taking and barge-in

Live

Pricing and limits

Sessions

Where the time goes

Synthesis happens per sentence

Skip synthesis when nobody is listening

Pace your audio in real time

Cut playback on `speech_started`

What does not help

Measure it, do not estimate it

Batch synthesis is a different problem

Next