Audio intelligence

Audio intelligence is transcription with the speakers, the timing and the acoustics already in it. One pass over a recording returns the transcript, anonymous speaker labels, segment and word timestamps, per-word confidence, and an audibility reading that says whether a turn was on mic, off mic or too faint to trust. There is no separate diarization stage, no forced aligner to bolt on, and no chunk-and-normalize pipeline of your own to maintain. It rides the endpoint you already know. Name nur-stt-v1 and ask for verbose_json.

curl https://api.eesi.ai/v1/audio/transcriptions \
  -H "Authorization: Bearer $EESI_API_KEY" \
  -F file=@interview.mp3 \
  -F model=nur-stt-v1 \
  -F response_format=verbose_json

from eesi_sdk import EESIClient

# A long-form decode outlives the SDK's 60-second default read timeout.
with EESIClient(api_key="sk-eesi-...", timeout=1800) as client:
    with open("interview.mp3", "rb") as audio:
        result = client.create_transcription(
            file=("interview.mp3", audio, "audio/mpeg"),
            model="nur-stt-v1",
            response_format="verbose_json",
        )

for segment in result["segments"]:
    print(f"[{segment['start']:6.2f}] {segment['speaker']}: {segment['text']}")

import { readFile } from "node:fs/promises";
import { EESIClient } from "@eesi/sdk";

// Raise both budgets. Nothing arrives until the decode is done, so the wait
// is bounded by connectTimeoutMs (10 s by default), not by readTimeoutMs.
const client = new EESIClient({
  apiKey: process.env.EESI_API_KEY,
  connectTimeoutMs: 1_800_000,
  readTimeoutMs: 1_800_000,
});

const bytes = await readFile("interview.mp3");
const result = (await client.createTranscription({
  file: new Blob([bytes], { type: "audio/mpeg" }),
  model: "nur-stt-v1",
  responseFormat: "verbose_json",
})) as { segments: { start: number; speaker: string; text: string }[] };

for (const segment of result.segments) {
  console.log(segment.start.toFixed(2), segment.speaker, segment.text);
}

There is one transcription id, and it is the diarizing model: nur-stt-v1 returns speakers for a voice note and for an hour of meeting alike. Ask for verbose_json and they are there — no second id to opt into.

The payload

{
  "task": "transcribe",
  "language": "en",
  "language_probability": 1.0,
  "duration": 9.19,
  "noise_floor_db": -54.9,
  "speech_seconds": 8.99,
  "silence_seconds": 0.2,
  "text": "The property has been fully renovated... Can you move a little to the left, please?",
  "segments": [
    {
      "id": 0, "start": 0.07, "end": 3.61, "speaker": "S01",
      "text": "The property has been fully renovated with new hardwood floors throughout.",
      "pace_wpm": 186, "energy_db": -13.5,
      "level_db": -13.5, "snr_db": 41.4, "audibility": "on_mic", "confidence": 0.962
    },
    {
      "id": 1, "start": 3.65, "end": 5.55, "speaker": "S02",
      "text": "Can you move a little to the left, please?",
      "pace_wpm": 284, "energy_db": -19.8,
      "level_db": -51.6, "snr_db": 3.3, "audibility": "faint", "confidence": 0.943
    }
  ],
  "words": [
    { "word": "The", "start": 0.02, "end": 0.08, "confidence": 0.979,
      "segment_id": 0, "speaker": "S01", "level_db": -16.8, "snr_db": 38.1, "audibility": "on_mic" },
    { "word": "property", "start": 0.16, "end": 0.58, "confidence": 0.984,
      "segment_id": 0, "speaker": "S01", "level_db": -13.7, "snr_db": 41.2, "audibility": "on_mic" }
  ],
  "gaps": [],
  "disfluencies": { "filler": 0, "repetition": 0, "false_start": 0 }
}

Speaker labels (S01, S02, …) are consistent inside one file and anonymous. They identify voices, not people, and they reset between files — the first voice heard in an upload is S01 in that upload and nowhere else.

Words

words is a flat, OpenAI-shaped array. Each entry carries segment_id and speaker, so you can regroup it into turns without a second pass.

Field	Meaning
`start`, `end`	Word boundaries in seconds, from CTC forced alignment against the decoded transcript, clamped inside the turn the word belongs to.
`confidence`	How strongly the audio supports that word where it was aligned, 0 to 1. `null` when the word could not be aligned — a numeral, a symbol — and its timing is then interpolated between its neighbours.
`filler`, `disfluency`	`filler` marks the standard set (`uh`, `um`, `mm-hmm`). `disfluency` is `filler`, `repetition` (the same word twice) or `false_start` (an immediate restart). Discourse markers like “like” and “right” are deliberately not flagged; by lexicon they fire on ordinary speech.
`level_db`, `snr_db`, `audibility`	Acoustics, below.

Word timing is on by default. Send timestamp_granularities[]=segment to skip the alignment pass, which is worth a few seconds of GPU on a long file. When alignment was asked for and could not run, the payload says so with words_unavailable: true rather than omitting the key — an absent key is indistinguishable from an older backend, and that ambiguity once hid a GPU out-of-memory behind a transcript that otherwise looked perfect.

confidence is acoustic support, not probability of correctness. Timing and confidence come from force-aligning the finished transcript back to the audio, so the number says how well the audio matches the word where it was placed — not what the recognizer believed. The two come apart on hard audio: on a noise-mixed clip the model transcribed every word correctly while this score collapsed. Read it as “the audio is difficult here, listen to it”, not “this word is probably wrong”, and measure it on your own material before you gate anything on it.

Audibility: which lines you can quote

Faint off-mic speech — a question from behind the camera, crew direction bleeding into a take — sits 30 dB or more under the dialogue. Whole-file loudness normalization cannot recover it, because integrated loudness is dominated by the loud material. This service normalizes adaptively in short frames instead, which does recover it: an utterance 35 dB down transcribes word-perfect and with the right speaker. That creates a second problem. After normalization every level looks the same, so the transcript alone cannot tell you which lines were actually on mic. The level is therefore measured on the pre-normalization signal, against the file’s own noise floor.

Field	Meaning
`noise_floor_db`	The file’s room tone in dBFS. Top level.
`level_db`	Speech level of this word or turn before normalization, dBFS.
`snr_db`	`level_db` minus the noise floor — how far the speech stands above the room.
`audibility`	`on_mic` at 25 dB SNR or more, `off_mic` from 12 to 25 dB, `faint` below 12 dB.
`energy_db`	Delivery energy after normalization. It ranks loud against flat delivery inside a take. It is not an audibility signal.

In the payload above the presenter reads 41.4 dB SNR (on_mic) and the off-camera request reads 3.3 dB (faint). Both are fully transcribed, and afterwards you can tell them apart — which is what lets you quote one and not the other.

Silence, pace, language

Field	Meaning
`gaps`	Every silence of 0.5 s or longer, including the lead-in and the tail, as `{start, end, duration}`. Computed over merged turns, so overlapping speakers cannot hide a gap.
`speech_seconds`, `silence_seconds`	The same split, as two totals.
`pace_wpm`	Words per minute for the turn.
`language`, `language_probability`	Identified from the transcript when you do not pass `language`. A supplied `language` is echoed untouched.
`disfluencies`	Whole-file counts of `filler`, `repetition` and `false_start`.
`truncated`	Present and `true` when the model’s output was cut off, so the final turn’s `end` is an estimate rather than a measurement.

A silent upload — b-roll, room tone — is answered by a voice-activity gate before the GPU is occupied: "text": "", empty segments, and "no_speech": true.

Sound events

Alongside the transcript the whole file is swept with an audio tagger and returned as scene: what sound occurs when, for the entire duration, rather than one dominant label per asset.

{
  "scene": [
    { "start": 0.0, "end": 42.0, "label": "speech", "raw_label": "Speech", "score": 0.71 },
    { "start": 0.0, "end": 42.0, "label": "music", "raw_label": "Music", "score": 0.28, "role": "bed" },
    { "start": 42.0, "end": 58.0, "label": "music", "raw_label": "Theme music", "score": 0.66, "role": "featured" }
  ],
  "scene_summary": { "speech": 0.72, "music": 0.41, "dominant": "speech" }
}

label is one of music, speech, crowd, ambience, room_tone, sfx; raw_label is the strongest underlying class in that region; role applies to music only, bed when dialogue runs across it and featured otherwise. Events overlap on purpose. Dialogue over a music bed in a room with traffic outside is three simultaneous answers, and a timeline reporting one winner per moment throws two of them away — a speech clip mixed with broadband noise came back as 100% ambience, the speech erased. So each bucket gets its own run of events, and music and speech covering the same seconds is the normal case. Shares in scene_summary overlap for the same reason and sum above 1.

Keeping a run

POST /v1/audio/transcriptions is a pure relay: it returns the result and keeps nothing. POST /v1/speech/transcriptions is the library — one upload that both transcribes and files the run, so it has a row, its audio, and a place to come back to.

# Transcribe and keep. Streams SSE by default; the X-Session-Id response
# header names the row it created.
curl https://api.eesi.ai/v1/speech/transcriptions \
  -H "Authorization: Bearer $EESI_API_KEY" \
  -F file=@interview.mp3 \
  -F model=nur-stt-v1

# The library.
curl "https://api.eesi.ai/v1/speech/transcriptions?limit=20" \
  -H "Authorization: Bearer $EESI_API_KEY"

# One run, with its full result and any speaker names you set.
curl "https://api.eesi.ai/v1/speech/transcriptions/$SESSION_ID" \
  -H "Authorization: Bearer $EESI_API_KEY"

# Playback: a short-lived signed URL, not the bytes.
curl "https://api.eesi.ai/v1/speech/transcriptions/$SESSION_ID/audio" \
  -H "Authorization: Bearer $EESI_API_KEY"

# Erase the audio and the row.
curl -X DELETE "https://api.eesi.ai/v1/speech/transcriptions/$SESSION_ID" \
  -H "Authorization: Bearer $EESI_API_KEY"

One upload does both, because letting the browser transcribe and then post the result back would send a 90-minute file over the wire twice. stream defaults to true here; pass stream=false for a single JSON body.

Naming the speakers

S01 is not a name. Operator names are stored beside the model’s own labels, never over them, so re-running or re-reading the result never loses either:

curl -X PATCH "https://api.eesi.ai/v1/speech/transcriptions/$SESSION_ID/speakers" \
  -H "Authorization: Bearer $EESI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"names": {"S01": "Dana", "S02": "Camera operator"}}'

The detail read returns speaker_names next to result, and it is yours to render. Deleting a run erases the uploaded audio first and the row second. If storage refuses, nothing is deleted and the delete can be retried — the other order would strand an audio object that nothing references and nothing can find again.

When the library fails

/v1/speech/* is not one of the OpenAI-shaped routes, so it answers in two envelopes rather than one. The gates it shares with /v1/audio/transcriptions keep the OpenAI envelope; everything the route raises itself uses FastAPI’s plain {"detail": …}. A bad form field here is a 422, not the 400 you get one endpoint over.

Status	Body	What happened
404	`{"detail": "Transcription not found"}`	No run with that id in your organization. A run belonging to another organization reads the same way.
404	`{"detail": "This transcription was saved without its audio"}`	The row is there and has no stored audio, so `/audio` has nothing to sign.
413	OpenAI envelope, `content_size_limit_exceeded`	The upload is over the 1 GB gateway cap.
422	`{"detail": [...]}`	A missing or malformed form field — no `model`, say.
500	`{"detail": …}`	The audio is filed under a storage backend this deployment cannot read, or a URL could not be signed. Neither is fixable from the request; quote the session id.
502	`{"detail": "Failed to delete the audio"}`	Storage refused the erase, so the row was kept and nothing was deleted. Retry.

Creating a run applies the same gates as the OpenAI-compatible route, and those keep the OpenAI envelope: rate_limit_exceeded, free_tier_daily_limit and insufficient_quota at 429, plus model_not_found at 404 and model_unavailable at 503. A client here needs to read both shapes.

Limits


Duration	No minute count is enforced. The serving pool’s context window binds first — 32,768 tokens shared between the audio and a transcript that costs roughly 15 tokens per second of speech — and the service’s audio handling is sized for files up to about 90 minutes. Split longer recordings at silence boundaries, and read `truncated` on what comes back.
Upload	1 GB at the gateway, 256 MB at the deployed backend. The smaller number is the one you will meet.
Decode budget	Up to 30 minutes of upstream read time, which is what lets a long file and a cold start both finish.
Diarization quality	Speaker separation needs enough continuous speech per voice. On a file of two-second fragments it will sometimes split one person in two.
Price	$0.001 per minute of audio, the same rate as plain transcription.

Speaker labels are per file. Two recordings of the same meeting do not agree about who S01 is, so map labels to people once per run and store the mapping with the run rather than inferring it again.

Transcribe a meeting

The recipe end to end, including saving and naming.

Speech to text

Plain transcription, and the request fields you share with it.

Sessions

The other kind of record: live conversations.

Compliance

Recording and consent duties that come with keeping audio.

Start here

Realtime interaction

Speech

Platform

Audio intelligence

The payload

Words

Audibility: which lines you can quote

Silence, pace, language

Sound events

Keeping a run

Naming the speakers

When the library fails

Limits

Next

Transcribe a meeting

Speech to text

Sessions

Compliance

​The payload

​Words

​Audibility: which lines you can quote

​Silence, pace, language

​Sound events

​Keeping a run

​Naming the speakers

​When the library fails

​Limits

​Next

Transcribe a meeting

Speech to text

Sessions

Compliance

The payload

Words

Audibility: which lines you can quote

Silence, pace, language

Sound events

Keeping a run

Naming the speakers

When the library fails

Limits

Next