Realtime events

Everything on a live socket is a JSON text frame with a type. This page is the reference for both directions. If you are writing a client from scratch, read it end to end; the two warnings near the bottom are the ones that cost production incidents.

The handshake

Four frames open every socket, in this order:

eesi.synthetic_audio
eesi.recording_disclosure
eesi.session
session.created

The first three are the gateway’s own, described below. It sends all three before it relays anything from the model, on both realtime endpoints and on every connection to them — a resumed session and a translation session included. Read until you see the type you want. A client that takes frame one as session.created reads a synthetic-audio marker instead, and the field it wanted is not there. session.created is the model’s opening frame, and it tells you what this end supports instead of making you probe for it.

{
  "type": "session.created",
  "event_id": "event_…",
  "eesi_protocol_version": "1",
  "capabilities": {
    "audio_input_rates": [16000, 24000],
    "audio_formats": ["pcm16", "g711_ulaw", "g711_alaw"],
    "events": { "resume": false, "ping": false, "recording_disclosure": true }
  },
  "session": { "type": "realtime", "id": "sess_…", "model": "nur-realtime-v1" }
}

Field	What to do with it
`eesi_protocol_version`	The dialect version. Absent means a gateway that predates versioning — treat that as version 0, never as an empty string.
`capabilities.audio_input_rates`	PCM16 rates the server accepts. The first entry is the default when you omit `audio.input.format`. Compare it against your capture rate before sending a frame.
`capabilities.audio_formats`	`pcm16` (`audio/pcm`), `g711_ulaw` (`audio/pcmu`, 8 kHz), `g711_alaw` (`audio/pcma`, 8 kHz).
`capabilities.events`	Optional events the speech server emits or honours. `recording_disclosure` is true in version 1. `resume` is false here because resumption belongs to the gateway, not the model server — the gateway advertises it separately on `eesi.session`. `ping` is reserved.

Both extra fields are additive. An OpenAI-compatible client that ignores them is still correct; an EESI client reads them rather than guessing.

What you send

Event	What it does
`session.update`	Deep-merges configuration: instructions, voice, speed, turn detection, audio formats, tools, `output_modalities`. Acked with `session.updated`.
`input_audio_buffer.append`	One base64 audio chunk. At most two seconds of audio per event; oversized or non-base64 payloads answer `invalid_audio`.
`input_audio_buffer.commit`	Closes the buffer by hand. Committing an empty buffer answers `input_audio_buffer_commit_empty`. Not needed when turn detection is on.
`conversation.item.create`	Injects an `input_text` message or a `function_call_output`. Does not start a turn. Up to 32 items may queue behind an active response; overflow answers `conversation_item_limit_reached`.
`conversation.item.truncate`	Trims an item’s audio to what the listener actually heard. Acked with `conversation.item.truncated`.
`response.create`	Starts a turn. Accepts per-response `instructions`, `tool_choice` and `output_modalities` overrides.
`response.cancel`	Cancels the response in flight and re-enables listening. Closes it with `status: "cancelled"`, `reason: "client_cancelled"`.
`eesi.keepalive`	Gateway-only. Says a client is still there so a quiet session is not reclaimed. Never reaches the model. See Connections.

Anything else answers an error with type unknown_or_invalid_event. That includes eesi.keepalive on a gateway that does not advertise it — check the eesi.session greeting first.

What arrives

Event	Meaning
`session.created`	The handshake above. The model’s first frame, and the socket’s fourth.
`session.updated`	Your `session.update` was accepted. Some clients gate readiness on this.
`input_audio_buffer.speech_started`	The server heard speech start. Stop playback here.
`input_audio_buffer.speech_stopped`	End of the user’s speech segment. The clock on the reply starts here.
`conversation.item.input_audio_transcription.delta`	What the user has said so far — cumulative, see below.
`conversation.item.input_audio_transcription.completed`	Final transcript of the user’s turn.
`conversation.item.input_audio_transcription.failed`	Transcription failed for that item; an `error` follows.
`conversation.item.created`	An injected item was accepted. No generation follows.
`conversation.item.truncated`	Acknowledges a `conversation.item.truncate`. See Turn-taking for what it does and does not change.
`response.created`	A turn opened. Carries the effective `response.audio.output.format`.
`response.output_audio.delta`	Assistant audio, base64, in the negotiated format.
`response.output_audio.done`	The turn’s audio is finished.
`response.output_audio_transcript.delta`	One flushed sentence of the reply, as text.
`response.output_audio_transcript.done`	The same sentence, terminated — one per segment, see below.
`response.output_text.delta` / `.done`	The text-only path, used when `output_modalities` excludes audio. The `.done` arrives once at close and only on a completed response.
`response.function_call_arguments.done`	A tool call: `call_id`, `name`, JSON `arguments`.
`response.done`	Turn complete. `response.status` is `cancelled` when barge-in cut it short, with `status_details.reason` of `turn_detected` or `client_cancelled`.
`error`	Something failed. Read `error.message`; `error.code` (or `error.type`) classifies it.

Three more are the gateway’s own. They open the socket, in the order below, before any model frame reaches you, and are namespaced eesi. so an OpenAI-compatible client can ignore all three:

Event	Meaning
`eesi.synthetic_audio`	Everything this session emits is model-generated. Carries the same marker record as the `X-Synthetic-Audio` header. See Synthetic audio marking.
`eesi.recording_disclosure`	Whether this session is recorded, and `disclosure.text` to show the person before they speak. See Compliance.
`eesi.session`	The resume handle, the keepalive contract, and — mid-call — the frame that says a limit ended the session. See Connections.

The two events that read backwards

Both of these fail silently. You get a plausible transcript that is wrong.conversation.item.input_audio_transcription.delta is cumulative. Each delta carries the whole utterance so far, so it replaces the open entry. Append them and you get “hello hello there hello there world”.response.output_audio_transcript.done fires per segment, not per turn. The pipeline splits the reply at sentence boundaries and synthesizes each sentence separately, emitting a .delta and a matching .done for every one. Keep only the last .done and every multi-segment reply is truncated to its last sentence. Accumulate the deltas; treat done as a terminator. A reply ends at response.done, not at a .done.

The recorded event streams in spec/realtime/v1/ pin this behaviour, and every EESI client — the console, the iPhone app, the Mac engines, the gateway’s own recorder — replays that corpus. There is one reducer behind all of them, @eesi/realtime, but it is an internal package: it is not published, so write your own decoder against the rules above rather than reaching for it.

Legacy spellings

Older OpenAI variants spell four events without output_:

Legacy	Canonical
`response.audio.delta`	`response.output_audio.delta`
`response.audio.done`	`response.output_audio.done`
`response.audio_transcript.delta`	`response.output_audio_transcript.delta`
`response.audio_transcript.done`	`response.output_audio_transcript.done`

This server emits only the canonical names. If your client also talks to a backend that uses the legacy ones, fold both onto one name at the edge of your decoder — handling one is how a client goes silent against exactly one backend.

A worked turn

What one exchange looks like on the wire, with a barge-in at the end:

← eesi.synthetic_audio                              the socket opens
← eesi.recording_disclosure
← eesi.session
← session.created
→ session.update                                    (voice, instructions)
← session.updated
→ input_audio_buffer.append   × N                   (20 ms each, paced)
← input_audio_buffer.speech_started
← conversation.item.input_audio_transcription.delta "what's"
← conversation.item.input_audio_transcription.delta "what's the weather"
← input_audio_buffer.speech_stopped
← conversation.item.input_audio_transcription.completed "What's the weather?"
← response.created
← response.output_audio_transcript.delta "It's sunny."
← response.output_audio_transcript.done  "It's sunny."
← response.output_audio.delta × N
← response.output_audio_transcript.delta " And mild."
← response.output_audio_transcript.done  " And mild."
← response.output_audio.done
← response.done                                     status: completed
← input_audio_buffer.speech_started                 the user cuts in
← response.done                                     status: cancelled, reason: turn_detected

Each flushed sentence produces a delta and a done carrying the same bytes, so a client that concatenates deltas and one that replaces on done see the same text — as long as the second one accumulates across segments. The reply is “It’s sunny. And mild.” Keeping only the last done gives you “And mild.”

Turn-taking and barge-in

What decides speech_stopped, and what a cancelled turn owes your client.

Connections that survive

The eesi.session frames in full, and every close code.

Errors

Every failure this API produces, and which are worth a retry.

OpenAI compatibility

What an existing realtime client has to branch on.

Start here

Realtime interaction

Speech

Platform

Realtime events

The handshake

What you send

What arrives

The two events that read backwards

Legacy spellings

A worked turn

Next

Turn-taking and barge-in

Connections that survive

Errors

OpenAI compatibility

​The handshake

​What you send

​What arrives

​The two events that read backwards

​Legacy spellings

​A worked turn

​Next

Turn-taking and barge-in

Connections that survive

Errors

OpenAI compatibility

The handshake

What you send

What arrives

The two events that read backwards

Legacy spellings

A worked turn

Next