ElevenLabs Realtime

Realtime models are ElevenLabs models built for low-latency voice conversation. There are two ways to use them:

• An ElevenAgents agent (an agent ID, the dashboard, or an elevenlabs agents project with agent_configs/): set the agent's Voice mode to Realtime with conversation_config.realtime_model, and ElevenLabs manages its LLM, TTS model, and ASR settings. See Run an agent on a Realtime model.
• A browser app built on OpenAI's Realtime SDK: keep the SDK and point it at the OpenAI-compatible ElevenLabs Realtime API, wss://api.elevenlabs.io/v1/convai/realtime. See Migrate an app from OpenAI Realtime.

When the user is starting from scratch and it's unclear which they want, ask. An agent brings a managed system prompt, tools, knowledge base, widget, and telephony; the Realtime API is the quicker path for code that already speaks OpenAI's realtime events. For agent features beyond the voice mode (prompt, tools, workflows, widget), use the agents skill.

Models

Start with Mini. Choose Max when the work involves long or complex instructions or multi-step tool calls and Mini's responses aren't good enough, or when an app being migrated was on a larger OpenAI model for reasoning-heavy work. Agents add compliance and data residency constraints on the choice; see Choose the model.

Run an agent on a Realtime model

Every voice agent has a Voice mode. In Custom mode the agent's LLM, TTS model, and ASR settings are chosen individually. In Realtime mode the agent runs on a Realtime model and ElevenLabs manages those settings. The mode is one field: conversation_config.realtime_model set to a model ID means Realtime, null means Custom.

The system prompt, first message, voice, languages, temperature, tools, knowledge base, and conversation flow settings (turn eagerness, timeouts, interruptions) work the same in both modes.

Before you start

• These steps call the ElevenLabs API. The CLI and SDKs read ELEVENLABS_API_KEY from the environment; if it isn't set, use the setup-api-key skill. Never ask the user to paste the key into the conversation.
• Realtime models are rolling out gradually. If the API rejects realtime_model with Realtime models are not available for this workspace., the workspace doesn't have access yet. Stop and tell the user; no configuration change fixes it.

Step 1: Check the agent can use Realtime

For an existing agent, fetch its configuration first (elevenlabs agents pull --agent <agent-id>, or agents.get in the SDK). Keep the agent in Custom mode, and tell the user why, if any of these apply:

A workflow override can be removed and a transfer partner can be switched to Realtime too, but only with the user's agreement: both change how other parts of their setup behave.

Step 2: Choose the model

Start with eleven_realtime_v1_mini. Choose eleven_realtime_v1_max when the agent follows long or complex instructions or relies on multi-step tool calls and Mini isn't good enough, or when the agent must comply with Zero Retention Mode or HIPAA. Then check availability:

Zero Retention Mode applies whether it's enforced on the workspace or enabled on the agent. Ask the user when the workspace's data residency or compliance settings aren't clear from the project. If neither model is available for the agent's environment, keep it in Custom mode.

Max also needs agent.prompt.max_tokens of at least 1024, or -1 for no limit.

Step 3: Set the model

On an existing agent, change only realtime_model; leave the rest of the configuration alone.

With the CLI, pull the agent, set the field in its file under agent_configs/, and push:

elevenlabs agents pull --agent <agent-id>

{
  "conversation_config": {
    "realtime_model": "eleven_realtime_v1_mini"
  }
}

elevenlabs agents push --agent <agent-id>

With the SDKs:

from elevenlabs import ElevenLabs

elevenlabs = ElevenLabs()

elevenlabs.conversational_ai.agents.update(
    agent_id="<agent-id>",
    conversation_config={
        "realtime_model": "eleven_realtime_v1_mini",
    },
)

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

await elevenlabs.conversationalAi.agents.update("<agent-id>", {
  conversationConfig: {
    realtimeModel: "eleven_realtime_v1_mini",
  },
});

For a new agent, put realtime_model (realtimeModel in TypeScript) in the conversation_config passed to agents.create, alongside the prompt, first message, and voice. Don't pick an LLM or TTS model for it; a Realtime agent ignores them.

To turn Realtime off, set realtime_model to null (None in Python). The agent goes back to the LLM, TTS model, and ASR settings stored on it.

Managed settings

While a Realtime model is set, the agent ignores the stored values of these fields in conversation_config. They come back into use when the agent switches to Custom, so don't clear them, and don't edit them to tune a Realtime agent:

agent.prompt.max_tokens keeps applying. Temperature does too, but the dashboard hides it in Realtime mode, so change it through the CLI or API. If the Realtime model errors or doesn't respond in time, the agent falls back to its backup LLMs (LLM cascading); the Realtime model sets how long it waits.

Step 4: Publish and verify

The Step 1 rules are checked when the agent is published, not when a draft is saved: a draft that breaks one saves, then publishing fails with an error naming the setting to change.

1. Fetch the agent and confirm conversation_config.realtime_model holds the chosen model. If the push, update, or the user's publish in the dashboard fails, match the error below.
2. Find every agent that transfers into this one and confirm it's in Realtime mode too. A Custom-mode agent's transfer into a Realtime agent is only caught when that agent is next published; until then the transfer is rejected mid-conversation and the caller stays with the current agent.
3. Have the user hold a test conversation.

Errors

Pricing

A Realtime agent is billed the standard ElevenAgents per-minute rate plus its input, cached input, and output tokens at the Realtime model's rates; Max costs more per token than Mini. Point the user to https://elevenlabs.io/pricing rather than quoting rates.

Migrate an app from OpenAI Realtime

ElevenLabs Realtime speaks the same wire protocol as OpenAI's GA Realtime API: the same client and server events over WebSocket. An app built on OpenAI's Realtime SDK keeps that SDK; the migration changes where credentials are minted, which URL the transport connects to, and the model, voice, and session configuration.

The one structural change: ElevenLabs Realtime is WebSocket-only. In a browser, RealtimeSession defaults to WebRTC, which wires the microphone and speaker for you. After switching to the WebSocket transport, the app has to capture and play audio itself.

This covers apps where the browser opens the realtime connection. It doesn't cover apps where a server opens the connection and streams audio to its clients over its own protocol, such as a bespoke WebSocket relay or a telephony bridge. Connecting to ElevenLabs Realtime from a server should work, but this skill has no steps for it. Tell the user so, and suggest they open an issue or a pull request at https://github.com/elevenlabs/skills describing their setup, rather than guessing at the server-side migration.

Step 0: Inventory the app

Find every place the app touches OpenAI Realtime before editing anything:

grep -rnE "api\.openai\.com/v1/realtime|client_secrets|realtime/sessions|@openai/agents-realtime|@openai/agents/realtime|gpt-realtime|gpt-4o(-mini)?-realtime|OpenAIRealtimeWebRTC|OpenAIRealtimeWebSocket|RealtimeSession|OPENAI_API_KEY" --include='*.{ts,tsx,js,jsx,mjs,py}' .

Classify what you find, because each part migrates differently:

• Credential minting: a server route that calls /v1/realtime/client_secrets (GA) or /v1/realtime/sessions (beta). Migrate in Step 1.
• Browser session: RealtimeSession, OpenAIRealtimeWebRTC, OpenAIRealtimeWebSocket, or a hand-rolled WebSocket/RTCPeerConnection. Migrate in Steps 2 to 4.
• Server-side realtime connections: a Node or Python process that opens the realtime socket itself, for example a relay that streams audio to clients over its own WebSocket protocol, a telephony bridge, openai Python's client.realtime.connect(), or a RealtimeSession running in Node with a full API key. Not covered; see the scope above. If the app also has a browser session, migrate only that part, and only with the user's agreement.
• Features beyond a voice conversation: function tools, handoffs, guardrails, MCP servers, image input, SIP. ElevenLabs documents the voice conversation flow only. Leave this code in place, list it for the user as unverified against ElevenLabs, and have them test it.

OPENAI_API_KEY often serves other OpenAI calls too. Remove it only if nothing else uses it.

Step 1: Mint the client secret from ElevenLabs

The browser never sees the ElevenLabs API key. A server route mints a short-lived client secret and returns only its value.

The request body keeps the GA shape, { session, expires_after }. The session config sent here is the authoritative initial session for that credential. The model string selects the agent behavior directly; there is no agent_id.

Mint with the ElevenLabs SDK (npm install @elevenlabs/elevenlabs-js or pip install elevenlabs), which sends the xi-api-key header for you:

import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

const clientSecret =
  await elevenlabs.conversationalAi.realtime.createClientSecret({
    session: {
      type: "realtime",
      model: "eleven_realtime_v1_mini",
      instructions:
        "You are a helpful, friendly voice assistant. Keep replies concise.",
      outputModalities: ["audio"],
      audio: {
        input: {
          format: { type: "audio/pcm", rate: 24000 },
          turnDetection: { type: "semantic_vad", eagerness: "medium" },
        },
        output: {
          format: { type: "audio/pcm" },
          voice: "brian",
        },
      },
    },
    expiresAfter: { anchor: "created_at", seconds: 600 },
  });

import os

from elevenlabs import (
    CreateClientSecretRequest,
    ExpiresAfter,
    OpenAiAudioConfigInput,
    OpenAiAudioInputConfigInput,
    OpenAiAudioInputFormat,
    OpenAiAudioOutputConfigInput,
    OpenAiAudioOutputFormat,
    OpenAiSessionConfigInput,
    OpenAiTurnDetection,
)
from elevenlabs.client import ElevenLabs

elevenlabs = ElevenLabs(api_key=os.environ["ELEVENLABS_API_KEY"])

client_secret = elevenlabs.conversational_ai.realtime.create_client_secret(
    request=CreateClientSecretRequest(
        session=OpenAiSessionConfigInput(
            type="realtime",
            model="eleven_realtime_v1_mini",
            instructions="You are a helpful, friendly voice assistant. Keep replies concise.",
            output_modalities=["audio"],
            audio=OpenAiAudioConfigInput(
                input=OpenAiAudioInputConfigInput(
                    format=OpenAiAudioInputFormat(type="audio/pcm", rate=24000),
                    turn_detection=OpenAiTurnDetection(type="semantic_vad", eagerness="medium"),
                ),
                output=OpenAiAudioOutputConfigInput(
                    format=OpenAiAudioOutputFormat(type="audio/pcm"),
                    voice="brian",
                ),
            ),
        ),
        expires_after=ExpiresAfter(anchor="created_at", seconds=600),
    ),
)

Return only clientSecret.value (client_secret.value in Python) to the browser.

Keep the route's existing framework, path, and response contract; only the upstream call changes. If the app minted with the beta /v1/realtime/sessions endpoint, its frontend reads client_secret.value. Either update the frontend to read the top-level value, or have the route return { client_secret: { value } } so the frontend stays untouched.

Step 2: Convert the session config to the GA shape

Send the GA session shape. Apps written against OpenAI's beta API use flat fields (modalities, voice, input_audio_format, turn_detection, …) that need converting, and ElevenLabs rejects the deprecated modalities field outright. This matters most for the mint request body and any hand-rolled session.update: the SDK's config option converts deprecated camelCase fields to the GA wire shape for you, though moving it to the GA form too removes the deprecation warnings.

Read Session config mapping for the full beta-to-GA field table, voice handling, and the camelCase form the SDK's config option uses.

Step 3: Point the browser session at ElevenLabs

Replace the transport with OpenAIRealtimeWebSocket aimed at the ElevenLabs URL, set the model on the session, and pass the minted value as the API key:

import {
  OpenAIRealtimeWebSocket,
  RealtimeAgent,
  RealtimeSession,
} from "@openai/agents-realtime";

const { value: clientSecret } = await fetch("/api/token").then((res) =>
  res.json(),
);

const agent = new RealtimeAgent({
  name: "Assistant",
  instructions:
    "You are a helpful, friendly voice assistant. Keep replies concise.",
});

const transport = new OpenAIRealtimeWebSocket({
  url: "wss://api.elevenlabs.io/v1/convai/realtime",
});

const session = new RealtimeSession(agent, {
  transport,
  model: "eleven_realtime_v1_mini",
  config: {
    outputModalities: ["audio"],
    audio: { output: { voice: "brian" } },
  },
});

await session.connect({ apiKey: clientSecret });

Keep the app's existing RealtimeAgent name, instructions, and event handlers. Drop WebRTC-only options (audioElement, mediaStream, changePeerConnection) and any transport: "webrtc" setting; they have no WebSocket equivalent.

Step 4: Wire up audio (when the app was on WebRTC)

If the app previously used WebRTC (an explicit OpenAIRealtimeWebRTC, transport: "webrtc", or no transport at all in a browser), it now has to:

1. Capture microphone audio as PCM16 at 24 kHz and stream it with session.sendAudio(...).
2. Play PCM16 chunks from the session's audio event.
3. Stop playback on audio_interrupted. Without this, the assistant keeps talking over the user after a barge-in, because already-buffered audio keeps playing.
4. Release the microphone and close the session on teardown.

Read Browser audio over WebSocket for a working implementation with wavtools, the helper OpenAI uses in its own WebSocket examples, and its pitfalls.

Apps already on OpenAIRealtimeWebSocket have this wiring; check the sample rate is 24 kHz and keep it.

Step 5: Verify

1. Find where the server loads OPENAI_API_KEY from (.env, .env.local, deployment secrets) and ask the user to store ELEVENLABS_API_KEY there. Don't ask for the key in the conversation or write it into a file yourself. Add an empty ELEVENLABS_API_KEY= to .env.example if the project has one.

2. Once the user confirms the key is in place, run the app, start a conversation, and speak.

3. Confirm the session.created transport event reports the eleven_realtime_* model and the chosen voice:

   session.on("transport_event", (event) => {
     if (event.type === "session.created") console.log(event);
   });

4. Interrupt the assistant mid-sentence and confirm playback stops.

5. Tell the user which features from Step 0 remain unverified (tools, handoffs, guardrails, …).

Troubleshooting

References

• Session config mapping
• Browser audio over WebSocket