Realtime API

The Realtime API enables real-time audio/voice agent interactions with event-driven architecture. This allows building voice assistants, phone systems, and other real-time conversational AI applications.

Overview

The Realtime API provides:

• Real-time audio streaming
• Voice activity detection
• Turn-based conversation management
• Function calling in real-time
• Integration with telephony systems (Twilio, etc.)

Capabilities

Realtime Agent

Agent configured for real-time audio interactions.

class RealtimeAgent:
    """
    Agent for real-time audio interactions.

    Extends AgentBase with real-time specific configuration
    for voice conversations.
    """

Usage example:

from agents.realtime import RealtimeAgent

agent = RealtimeAgent(
    name="Voice Assistant",
    instructions="You are a helpful voice assistant."
)

Realtime Runner

Runner for executing realtime agents.

class RealtimeRunner:
    """Runner for realtime agents."""

    @classmethod
    async def run(
        starting_agent: RealtimeAgent,
        *,
        session: RealtimeSession,
        context = None,
        hooks = None,
        config = None
    ):
        """
        Run realtime agent.

        Parameters:
        - starting_agent: Realtime agent to run
        - session: Realtime session for audio
        - context: Optional context object
        - hooks: Lifecycle hooks
        - config: Configuration options

        Returns:
        - Realtime result
        """

Usage example:

from agents.realtime import RealtimeAgent, RealtimeRunner, RealtimeSession

agent = RealtimeAgent(
    name="Voice Assistant",
    instructions="Help users via voice."
)

session = RealtimeSession(...)  # Configure session

result = await RealtimeRunner.run(
    agent,
    session=session
)

Realtime Session

Session for managing real-time audio interactions.

class RealtimeSession:
    """
    Session for realtime interactions.

    Manages audio streaming, turn detection,
    and real-time events.
    """

Realtime Events

Event types emitted during real-time execution (specific event types depend on the implementation).

Integration Examples

CLI Voice Assistant

Basic command-line voice assistant:

# See examples/realtime/cli/demo.py in the repository
from agents.realtime import RealtimeAgent, RealtimeRunner

agent = RealtimeAgent(
    name="CLI Assistant",
    instructions="You are a voice assistant."
)

# Run with audio I/O
result = await RealtimeRunner.run(agent, session=cli_session)

Twilio Phone Integration

Integration with Twilio for phone-based agents:

# See examples/realtime/twilio/server.py in the repository
from agents.realtime import RealtimeAgent

phone_agent = RealtimeAgent(
    name="Phone Assistant",
    instructions="You are a phone assistant. Be concise."
)

# Integrate with Twilio websocket
# See repository examples for complete implementation

Web Application

Real-time voice in web applications:

# See examples/realtime/app/server.py in the repository
from agents.realtime import RealtimeAgent

web_agent = RealtimeAgent(
    name="Web Assistant",
    instructions="You are a web-based voice assistant."
)

# Connect to WebSocket for browser audio
# See repository examples for complete implementation

Features

Voice Activity Detection

Automatic detection of when user is speaking.

Turn Management

Automatic turn-taking between user and agent.

Function Calling

Real-time function/tool calling during conversation.

Audio Streaming

Bidirectional audio streaming for low-latency interactions.

Best Practices

1. Concise Responses: Voice responses should be shorter than text
2. Turn Detection: Configure appropriate VAD sensitivity
3. Error Handling: Handle audio connection drops gracefully
4. Latency: Minimize tool execution time for real-time feel
5. Instructions: Provide voice-specific instructions to agents
6. Testing: Test with real audio I/O, not just simulated
7. Bandwidth: Consider network bandwidth for audio quality
8. Interruptions: Handle user interruptions appropriately

Examples Location

Complete working examples are available in the repository:

• examples/realtime/cli/demo.py - CLI voice assistant
• examples/realtime/app/server.py - Web application integration
• examples/realtime/twilio/server.py - Twilio phone integration
• examples/realtime/twilio_sip/server.py - Twilio SIP integration

Refer to these examples for complete implementation details.

Note

The Realtime API is a specialized feature for voice applications. Most use cases should use the standard Agent and Runner classes. Use Realtime API when you specifically need:

• Real-time audio interaction
• Voice-based assistants
• Phone system integration
• Low-latency conversational AI

For complete API reference and implementation details, refer to the source code and examples in the repository.