Audio intelligence turns a recording into something you can search, cut and report on. Upload an interview, podcast, meeting, lecture or speech and you get:
  • The transcript by speaker, with the timing of every word and how clearly each one came through.
  • How every line was delivered: a label (highlight, hesitant, flat, weak take), the emotion, how confident and engaged the speaker sounds, their pace compared with their own usual pace, the words they stressed, and the evidence behind each call.
  • The audience’s reactions, such as laughter and applause, with timings.
  • Key moments: each speaker’s strongest takes, emotional peaks, hesitant or flat stretches, audio you should not use, and long pauses to tighten.
  • Speaker profiles and a written summary of the whole recording.
  • The sound under the voice: room, music and noise, and whether it competes with the voice.
  • On request (not available yet): split the audio into dialogue, music and effects, pull out one person’s voice, isolate any sound you describe, or find every place a sound happens.
  • Projects: the same person keeps one identity, and one name, across every recording in a project.

​
How it works

1

Upload

POST /v1/speech/transcriptions transcribes the recording by speaker and saves it, with its audio, to your organization’s library.
2

The analysis runs by itself

Once the transcript is saved, the delivery analysis starts in the background. For a long recording it usually takes a few minutes. Poll GET /v1/speech/transcriptions/{session_id}/intelligence until status is complete.
3

Use it

Read the analysis, jump to its moments, name the speakers, split stems or search for sounds. Everything stays with the recording until you delete it.

​
Quickstart

​
1. Upload a recording

The response is the transcript. This is a real one, trimmed to its first two lines: a 1933 fireside chat by Franklin D. Roosevelt, with a piano rag playing underneath.
The music under the voice is why these lines read off_mic even though Roosevelt is close to his microphone: the voice is only about 12 dB above everything else. "intelligence": {"status": "queued"} means the analysis has started.
Leave out stream, which defaults to true, to get the transcript line by line as Server-Sent Events. session.created carries the session_id, each transcript.segment is one finished line, and transcript.text.done carries the full result.

​
2. Wait for the analysis

While the analysis runs, stage and progress show how far it has got: The document fills in as it goes: measurements appear first, then each line’s delivery. You can show results before the analysis finishes.

​
3. Read the analysis

This is a real analysis of John F. Kennedy’s 1962 address at Rice University, which runs 17 minutes 40 seconds. It is trimmed to one example of each item:
How to read it:
  • Line 44, “…for knowledge and progress is determined and cannot be deterred.”, is one of Kennedy’s standout takes: heard as assured and a strong take, labeled great with a score of 0.87. He stresses “determined”, “cannot” and “deterred”. The listening also heard it as rushed, but the measured pace is 6% slower than his usual, so no rushed flag is raised.
  • The applause from 8:39 to 8:48 is the crowd answering “We choose to go to the moon”.
  • The moments list a soundbite to use (“We mean to be a part of it.”, with applause after it), a stretch recorded off mic that you should not use, and a pause to tighten.
  • The speaker rates 0.75 for confidence and 0.67 for engagement across the speech, and twelve of his lines stand out.
  • timings.total is how long the analysis took: about two minutes for this recording.

​
What the analysis tells you

​
Where each signal comes from

Every flag, stressed word and audio issue says in source what backs it: Pace, pitch and level are compared with each speaker’s own usual delivery (pace_vs_speaker, pitch_vs_speaker_st, energy_vs_speaker_db), so “rushed” means fast for that person. A heard “rushed” or “slow” counts only when the measured pace leans the same way. Confidence describes how sure the speaker sounds, not how sure we are of the transcript.

​
Each line

Each entry in segments matches a transcript line by id: The listener rates each line in words, and each word is stored as a number from 0 to 1, so the ratings are steps rather than a continuous score: Most everyday speech is steady, engaged and usable. The top step is kept for a line that clearly stands out.

​
Delivery labels

A line with a problem flag (a hesitation, a pace that is off for the speaker, trailing off, mumbling, a false start) is never great, however it was rated, and a hesitant or flat line scores at most 0.6. Slow, deliberate lines with pauses in them read as good: that is an orator’s delivery, not hesitation. A line that applause covers for most of its length isn’t graded: its measurements would describe the room, not the voice. It has delivery.label and delivery.score of null, delivery.not_graded set to "under_applause", and under lists the applause over it. Every line with words also carries speech_start and speech_end: where the voice starts and stops inside the line, which is where to cut.

​
Flags

delivery.flags explains the label. Each flag has a type and a source, and measured flags carry their evidence, such as "31% faster than this speaker's usual pace".

​
Audio issues

quality.issues lists what’s wrong with the audio, each with a severity from 0 to 1: clipping, distortion, wind, plosives, hum, echo, background_noise, off_mic, and overlap (someone else speaking at the same time). Plosives are counted only on words that start with “p” or “b”. Pops, hum, wind, echo and overlap are fixed in the mix or the cut, so they are listed on the line but never make it unusable: only clipping, distortion, noise that buries the voice, and off-mic speech do. What is true of the whole recording is said once, in conditions, rather than on every line: mains hum through the recording, or a voice that is off mic throughout because it was taken from a PA or a room microphone. A line is marked off_mic only when it is quieter than the rest of the recording.

​
Reactions and events

events is a timeline of what happened. Events aren’t tied to one line: Reactions list in sources every way they were detected: A reaction is reported only when two detectors agree, when the acoustic tagger alone is sure, or when the words say so. One detector’s word is never enough: searching alone (located) and listening alone (heard) both find laughter in ordinary talk. The one exception is an ovation: in a recording where another detector has heard the audience applaud, a long, sure stretch of applause found by searching counts.

​
Key moments

moments is an editor’s read of the recording. Each moment has an id, a reason you can show as it is, and the segment_ids it covers:

​
Speakers and summary

speakers profiles each voice across the whole recording:
  • talk time and number of turns
  • average confidence and engagement
  • their emotions, and how their lines were labeled
  • their reactions
  • a trajectory of how their delivery moved from start to end
  • a one-line summary with timestamps
summary.overview is a short written account of the recording, and summary.watch_outs lists at most three things to check before you use it, each with its clearest times. The summary names voices by their labels (S01); show your own names in their place.

​
Music and background

music lists the music in the recording: when it plays, its instruments and style, and how loud it is compared with the voice (vs_dialogue_db). Music under dialogue has role: "bed". ambience_profile describes the room or place the recording sounds like, over time.

​
Quiet speech, recovered

recovered lists lines picked out from under noise or applause and read again: quiet asides, or a question from behind the camera. They sit beside the transcript and are never merged into it, so you decide what to use.

​
The transcript

The upload response carries the transcript, and so does result on GET /v1/speech/transcriptions/{session_id}.
A word’s confidence is how strongly the audio supports the word, not the chance that it’s right. On noisy audio it drops even when the word is correct. Read a low value as “listen to this bit again”, and check it against your own recordings before you filter on it. In the Rice University address, Kennedy greets “President Pitzer” and the transcript reads “Pritzker”, with a confidence of 0.098: exactly the word to check.
Speaker labels (S01, S02, …) stay the same within one recording but start again in the next one. To keep a person’s identity across recordings, use a project.

​
Your recordings

Everything you upload is kept in your organization’s library until you delete it.

​
Retries and your own ids

A pipeline retries. Send an Idempotency-Key with a batch upload (stream=false), and a retry with the same key returns the recording the first request made, with Idempotent-Replayed: true, instead of transcribing and billing a second time. Keys hold for 24 hours.
external_id is your own id for the asset, such as a clip id in your asset manager, and metadata is a JSON object of your own. Both come back with the recording and in the library. GET /v1/speech/transcriptions?external_id=RICE62 finds every upload of that asset; the id isn’t unique, so a re-delivery is a new recording with the same id. A batch response also says what it billed, in billed_seconds. The library lists each recording with a summary:
Names you set are kept beside the speaker labels and never replace them, so nothing is lost if you rename someone. For a recording in a project, name the person in the project instead, and the name shows in every recording.

​
Timecode

Times in every response are seconds from the start of the file. To work in your editing system’s timecode, give the recording a timebase: its frame rate, whether it counts in drop frame, the timecode of its first frame, and a tape or reel name. The studio then shows timecode, and marks land on the same frames your editing system uses.
  • frame_rate is one of 23.976, 24, 25, 29.97, 30, 50, 59.94 and 60. Drop frame applies only at 29.97 and 59.94.
  • Leave start_timecode out when the file has no timecode; it then counts from 00:00:00:00.
  • The recording returns its timebase as media, or media: null if none was set.
  • In points round down to the frame they fall in, and out points round up, as in every editing system. With the timebase above, the laughter at 274.12–274.68 seconds in JFK’s Rice address runs from 01:04:34;03 to 01:04:34;21.

​
Markers for your editing system

Export the analysis as markers your editing system opens, at the recording’s timecode. Set the timebase first (see Timecode); without one, times count at 23.976 from 00:00:00:00.
Part of the Resolve file for JFK’s Rice address, at 29.97 drop frame from 01:00:00;00:
  • Highlights are green and reactions cyan. Takes to review are yellow, and unusable audio red; they’re included only with include=needs_attention. Pauses and hesitations never become markers.
  • The same ovation reported twice becomes one marker.
  • Pass frame_rate, drop_frame and start_timecode to write one export in a different timebase, for example to match a sequence that starts at 10:00:00:00.
  • Exports are available once the analysis is complete; before that the request returns 409.

​
Captions

srt and vtt build captions from the word timings, to broadcast limits by default: two lines of at most 42 characters, up to 7 seconds on screen. Each cue leaves the screen at most half a second after its last word, and a new speaker starts a new cue. preset=social gives one line of at most 32 characters and up to 3 seconds. Captions need only the transcript, not the analysis, and their times count from the start of the file.
The response’s X-Caption-QC header says what to check first, for example cues=196; over_limit=0; faster_than_17cps=14; low_confidence_words=86.

​
Split into stems

Not available yet: this request answers 503 for now. The rest of the analysis is unaffected.
Separate a saved recording into stems. They are kept with the recording and deleted with it.
You can ask for all three in one request. It returns at once with 202 and the stems it queued:
Then poll until each stem is complete:
  • Stems are 48 kHz FLAC files. Download links last an hour. List the stems again to get fresh links.
  • quality rates the separation from 1 (poor) to 5 (clean).
  • Dialogue, music and effects add back up to the original, so you can remix.
  • Asking for a stem again replaces the earlier one.

​
Dub into another language

Not available yet: this request answers 503 for now. The rest of the analysis is unaffected.
A dub speaks every line of a saved recording again in another language, in each speaker’s own voice, where their mouth moves. A video comes back with its picture and the dub on it.
terms is optional: names and terms spelled as they should be. The transcript listens for them and the script keeps them. It returns at once with 202 and the dub’s id (ar-en). The dub then:
  1. separates the dialogue from the music and effects, which are kept as they are (these are the recording’s dialogue, music and effects stems);
  2. reads the whole transcript for its people, places and terms, listens to the recording again for those names, repairs what speech recognition misheard, and adapts each line to the seconds the speaker’s mouth moves;
  3. finds each speaker’s cleanest continuous 10 to 30 seconds, isolates it as that one person, and clones the voice from it;
  4. voices each line in that voice, rewrites a line that cannot fit, voices it at the length of the original line, and reads it back to check it is clear;
  5. mixes the new voices onto the original music and effects.
Poll GET /v1/speech/transcriptions/$SESSION_ID/dubs until the dub is complete. Each dub lists its lines: what speech recognition heard (asr), what was said (source), the dub line (text), where the mouth moves (mouth), and how the voiced line fits (slot, voiced, speed, fit, back_text). sync measures the whole dub on both voices: coverage is the share of the original speech the dub covers, spill the share of the dub spoken where the original is silent, and onset_ms the median gap between each line’s original and dubbed start. video_url, audio_url and voice_url (the new dialogue alone) are links that last an hour. To correct a line, send what was said, the dub line, or both, then voice the corrections. Only those lines change:
  • A dub clones the speakers’ voices, so consent_attested must say you have the rights to the recording and its voices.
  • Dubbing the same language again replaces that dub. Deleting the recording deletes its dubs and their voice references.
  • Dubs take recordings up to 10 minutes.
  • For a video, the mouth is then redrawn to the new voice (lipsync on the dub: progress by chunks, and lipsync.url when it is done). Frames without a clear face, such as wide shots, are left as they were filmed. To redraw again after corrections, POST .../dubs/ar-en/lipsync; voicing corrections starts it on its own.

​
Find a sound

Not available yet: this request answers 503 for now. The rest of the analysis is unaffected.
Describe a sound and get every place it happens:
The search runs in the background. List the results:
This is the result for “applause” in the Rice University address, trimmed to three of its seventeen spans:
Loud, common sounds such as applause, music, a door or a phone are found reliably. For rarer sounds, check the score: in a large, echoing hall, “microphone feedback” can match the echo of the room.

​
Projects: one person, one identity

Speaker labels start again in every recording. File recordings in a project and each person keeps one identity across all of them. Roosevelt is the same person in a 1933 fireside chat and a 1940 one, and a name you set once shows in every recording.
Each recording’s speaker_map, in its analysis and in its full detail, links its own labels to the project’s people. matched: false means the voice is new to the project:
Name a person once, for the whole project:
To manage projects:
Matching uses a voice sample from each recording, and the API never returns it. Deleting a recording removes what it contributed; deleting a project removes all of it. Matching works best with a few seconds of clear speech per person. A voice that doesn’t clearly match anyone becomes a new person in the project rather than a guess.

​
Just the transcript

If you only need the transcript and don’t want to keep the recording, use the OpenAI-compatible endpoint. It returns the same speakers, word timings, audibility and scene, and saves nothing:
The delivery analysis, stems, sound search and projects need a saved recording, so they are only available through /v1/speech/transcriptions. See Speech to text for this endpoint’s options.

​
In the studio

Everything on this page is also in the Audio intelligence studio. Drop in a recording to:
  • read the transcript by speaker, in minutes and seconds or in timecode
  • select words in the transcript to mark in and out, snapped to the quiet around them, and copy the take with its timecode
  • export markers for Media Composer or DaVinci Resolve, and captions as SRT or WebVTT, from the Export menu
  • use the keys you already know, in a Premiere Pro or Media Composer layout (J, K, L, I, O, X or T, and more; press ? for the list)
  • filter to highlights (the soundbites), lines that need attention, or reactions
  • open any line to see whether it’s a take to use and why, with the measurements one click away
  • jump through the key moments
  • split stems and play them against the original
  • search for sounds
  • name the people in a project

​
Errors

The upload shares its limits with POST /v1/audio/transcriptions, and those errors use the OpenAI error format: 413, 429, model_not_found (404) and model_unavailable (503). Everything else uses {"detail": …}.

​
Limits

​
Next

Speech to text

Plain transcription through the OpenAI-compatible endpoint.

API reference

Every audio intelligence endpoint, with request and response schemas.