Live

Audio in, audio out, one model. There is no transcribe-then-think-then-speak round trip, and end-of-turn detection, barge-in and cancellation happen on the server rather than in your client. WS /v1/realtime speaks the OpenAI Realtime protocol — session.update, input_audio_buffer.append, response.create, response.output_audio.delta — so existing realtime clients work against it. In the console this is Studio → Nur Live.

Connect

The model rides the query string; the credential rides whichever channel your runtime can use.

# pip install "websockets>=14"
import asyncio, json, websockets

URL = "wss://api.eesi.ai/v1/realtime?model=nur-realtime-v1"

async def main():
    async with websockets.connect(
        URL, additional_headers={"Authorization": "Bearer sk-eesi-..."}
    ) as ws:
        # Three eesi.* declarations open the socket ahead of the model's
        # first frame, so read until the type you want — never frame one.
        async for raw in ws:
            event = json.loads(raw)
            if event["type"] == "session.created":
                print(event["capabilities"]["audio_input_rates"])
                break

asyncio.run(main())

// npm install ws — a server-side socket can set the header.
import WebSocket from "ws";

const ws = new WebSocket("wss://api.eesi.ai/v1/realtime?model=nur-realtime-v1", {
  headers: { Authorization: `Bearer ${process.env.EESI_API_KEY}` },
});

ws.on("message", (raw) => {
  const event = JSON.parse(raw.toString());
  if (event.type === "session.created") {
    console.log(event.capabilities.audio_input_rates);
  }
});

# A browser cannot set a header, so it opens with a ticket instead.
# Mint it on your server; the response is {"ticket": "rt_...", "expires_in": 30}.
curl -X POST https://api.eesi.ai/v1/realtime/ticket \
  -H "Authorization: Bearer $EESI_API_KEY"

In the browser, pass that ticket as token:

const ws = new WebSocket(
  `wss://api.eesi.ai/v1/realtime?model=nur-realtime-v1&token=${ticket}`,
);

The parameter name is a contract, not a preference. A ticket under any other name is not a credential, and the handshake is refused with an HTTP 403 that carries no reason — see Connections for the credential channels and how that failure reaches your client. Optional query parameters: timezone (an IANA zone, so the model knows what “tomorrow” means), source (which surface opened the session, recorded on the row), paired_session_id and pair_id for Duet, and resume with resume_token for reconnecting. The socket opens with three gateway declarations — eesi.synthetic_audio, eesi.recording_disclosure, eesi.session — before session.created arrives from the model. See Realtime events for what each one carries.

Configure the session

Send one session.update after session.created. The gateway composes your instructions against the platform prompt and resolves the voice to its stored reference clip before the frame reaches the model.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "instructions": "You are warm and concise. Answer in one or two sentences.",
    "audio": {
      "output": { "voice": "ev_abc123", "speed": 1.0 }
    }
  }
}

Field	Notes
`type`	Must be `"realtime"`. The gateway fills it in if you omit it, because the upstream validator rejects the whole frame without it.
`instructions`	Yours, appended after the platform prompt. Up to 64 KiB.
`audio.output.voice`	A `voice_id` from Voices. Read below.
`audio.output.speed`	Speaking rate, clamped to 0.5–2.0. Out-of-range values are clamped, not rejected.
`audio.input.turn_detection`	Requires `"type": "server_vad"`. `threshold` and `silence_duration_ms` tune endpointing; `interrupt_response` and `create_response` change what a turn does. See Turn-taking.
`output_modalities`	`["text"]` skips synthesis entirely for every turn. Omit it, or include `"audio"`, for speech.
`tools`	Function tools, answered with `response.function_call_arguments.done`.

Updates merge recursively, so a later frame that names only a voice keeps your instructions. An explicit null clears a field.

Never send "voice": null. With no speaker conditioning the synthesis model samples a new voice on every utterance, so the session changes speaker between sentences. The gateway pins the platform default when you name no voice at all — the failure is only reachable by explicitly clearing one. The voice belongs in session.audio.output.voice; the serving backend reads nowhere else.

Sample rates, and the two ways to break a session without noticing

Both defaults are 16 kHz mono PCM16, and neither the gateway nor the server checks what you actually put on the wire.

Do not write rate: 16000. The audio format is a discriminated union whose PCM member admits only rate: 24000, so { "type": "audio/pcm", "rate": 16000 } fails validation — and the entire session.update is rejected with it, voice and instructions included, while the socket stays open and healthy. What you get back is one error event with type unknown_or_invalid_event naming session.update rather than the field, so render errors or you will not notice.An untagged { "rate": 16000 } neither fails nor corrupts anything: it resolves to the mu-law member of the union with type left null, and the decoder dispatches on type, so it falls through to PCM at the rate you named. Your audio is decoded correctly, by an accident of the fall-through rather than by a rule — and the same fall-through accepts rates the tagged form rejects, { "rate": 8000 } included. Do not build on it.To get 16 kHz, omit the format. To send or receive 24 kHz, declare it: { "type": "audio/pcm", "rate": 24000 }. Those are the only two shapes the schema means you to write.

Nothing on the wire validates the rate you actually send. The server resamples from the rate you declared, so undeclared 24 kHz input is stretched to 1.5× its length and transcription accuracy collapses; playing 16 kHz output through a 24 kHz context runs it 1.5× fast and high-pitched. Both present as a bad model rather than as a configuration error.Read session.created.capabilities.audio_input_rates and compare it against your own capture rate before you send a frame. The first entry is the default.

{
  "type": "session.update",
  "session": {
    "type": "realtime",
    "audio": {
      "input": { "format": { "type": "audio/pcm" } },
      "output": { "format": { "type": "audio/pcm", "rate": 24000 } }
    }
  }
}

response.created and response.done report the encoder’s effective format in response.audio.output.format, so a client that reads it never has to guess. G.711 is accepted too, always at 8 kHz — but mind the two spellings. A format object takes { "type": "audio/pcmu" } or { "type": "audio/pcma" }. capabilities.audio_formats lists the same two codecs under OpenAI’s legacy names, g711_ulaw and g711_alaw, which are for reading, not for sending: put one of them in format and the union matches nothing, so the whole session.update is rejected exactly as rate: 16000 is.

Send audio

16 kHz mono PCM16, base64-encoded, in about 20 ms per message (640 bytes), paced at the rate it was recorded. Server-side VAD reads arrival timing to find the end of a turn, so pushing a file in at once produces bad endpointing rather than a faster answer. A single event may not carry more than two seconds of audio.

{ "type": "input_audio_buffer.append", "audio": "<base64 PCM16>" }

You can also type into the same session:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [{ "type": "input_text", "text": "Where is the train station?" }]
  }
}

Text has no voice activity detection behind it, so adding an item does not start a turn. Follow it with {"type": "response.create"}. Audio needs no such nudge — turn detection fires the response when you stop speaking. That is the one asymmetry between the two paths.

Play the reply

import base64, json

async def play(ws, player):
    caption = ""
    async for raw in ws:
        event = json.loads(raw)
        if event["type"] == "response.output_audio.delta":
            player.write(base64.b64decode(event["delta"]))
        elif event["type"] == "input_audio_buffer.speech_started":
            player.stop()          # barge-in: drop what is queued
            player.clear()
        elif event["type"] == "response.output_audio_transcript.delta":
            caption += event["delta"]

Stopping playback on input_audio_buffer.speech_started is not optional. The server cancels generation in milliseconds; if your buffer keeps talking for another two seconds, the agent feels slow no matter what the server numbers say. Turn-taking and barge-in covers the whole contract. Two events read the opposite way to how they look, and both fail silently: conversation.item.input_audio_transcription.delta is cumulative, and response.output_audio_transcript.done fires per segment. Read Realtime events before you write transcript handling.

Language

The model is multilingual and code-switches mid-sentence without configuration. Pin it to one language in instructions if you need that. To interpret rather than converse — the same words in another language and nothing else — use Live translation. Instructions alone will not do it: a conversational session carries a persona that tells the model to introduce itself and to translate only when asked, and that persona arrives first.

What this costs and where it stops

Metered per minute of wall clock from the moment the upstream pipeline connects, at the same rate as synthesis and transcription.
Three hours of wall clock per session, and five minutes with no frame from your client. See Connections.
Inbound is bounded at 256 KB/s sustained with a 2 MiB burst, and one frame may not exceed 1 MiB. A client that exceeds the sustained rate is closed 1008.
The balance and the free-tier day are re-checked once a minute, so a limit can end an established call — with a frame that tells you why, before the close.

Realtime events

Every event in both directions, and the capability handshake.

Turn-taking and barge-in

How the server decides you stopped, and what to do when someone interrupts.

Connections that survive

Tickets, keepalives, close codes, resume.

Build a voice agent in a browser

Working end-to-end code, reconnect included.

Start here

Realtime interaction

Speech

Platform

Connect

Configure the session

Sample rates, and the two ways to break a session without noticing

Send audio

Play the reply

Language

What this costs and where it stops

Next

Realtime events

Turn-taking and barge-in

Connections that survive

Build a voice agent in a browser

​Connect

​Configure the session

​Sample rates, and the two ways to break a session without noticing

​Send audio

​Play the reply

​Language

​What this costs and where it stops

​Next

Realtime events

Turn-taking and barge-in

Connections that survive

Build a voice agent in a browser

Connect

Configure the session

Sample rates, and the two ways to break a session without noticing

Send audio

Play the reply

Language

What this costs and where it stops

Next