0
# Model Providers
1

2
Model providers enable the SDK to work with different LLM backends. The SDK includes built-in support for OpenAI models and can integrate with 100+ LLMs through the LiteLLM provider. Custom providers can be implemented for proprietary or specialized models.
3

4
## Capabilities
5

6
### Model Interface
7

8
Base interface for LLM implementations.
9

10
```python { .api }
11
class Model:
12
    """Base interface for LLM."""
13

14
    async def get_response(
15
        input: list[TResponseInputItem],
16
        instructions: str | None,
17
        tools: list[Tool],
18
        model_settings: ModelSettings,
19
        response_id: str | None,
20
        tracing: ModelTracing
21
    ) -> ModelResponse:
22
        """
23
        Get response from model.
24

25
        Parameters:
26
        - input: Input items
27
        - instructions: System instructions
28
        - tools: Available tools
29
        - model_settings: Model configuration
30
        - response_id: Response ID for continuation
31
        - tracing: Tracing configuration
32

33
        Returns:
34
        - ModelResponse: Model response with outputs and usage
35
        """
36

37
    async def stream_response(
38
        input: list[TResponseInputItem],
39
        instructions: str | None,
40
        tools: list[Tool],
41
        model_settings: ModelSettings,
42
        response_id: str | None,
43
        tracing: ModelTracing
44
    ) -> AsyncIterator[TResponseStreamEvent]:
45
        """
46
        Stream response from model.
47

48
        Parameters:
49
        - Same as get_response
50

51
        Returns:
52
        - AsyncIterator[TResponseStreamEvent]: Stream of response events
53
        """
54
```
55

56
### Model Provider Interface
57

58
Base interface for model providers.
59

60
```python { .api }
61
class ModelProvider:
62
    """Base interface for model provider."""
63

64
    def get_model(model_name: str) -> Model:
65
        """
66
        Get model by name.
67

68
        Parameters:
69
        - model_name: Model identifier (e.g., "gpt-4o", "claude-3-5-sonnet")
70

71
        Returns:
72
        - Model: Model instance
73

74
        Raises:
75
        - ValueError: If model not supported
76
        """
77
```
78

79
### OpenAI Provider
80

81
Provider for OpenAI models using official API.
82

83
```python { .api }
84
class OpenAIProvider(ModelProvider):
85
    """
86
    OpenAI model provider.
87

88
    Supports OpenAI models through both Responses API
89
    and Chat Completions API.
90
    """
91

92
    def get_model(model_name: str) -> Model:
93
        """
94
        Get OpenAI model by name.
95

96
        Parameters:
97
        - model_name: OpenAI model name
98

99
        Returns:
100
        - Model: OpenAI model instance
101
        """
102
```
103

104
Usage example:
105

106
```python
107
from agents import Agent, Runner, RunConfig, OpenAIProvider
108

109
provider = OpenAIProvider()
110
config = RunConfig(
111
    model="gpt-4o",
112
    model_provider=provider
113
)
114

115
agent = Agent(name="Assistant")
116
result = Runner.run_sync(agent, "Hello", run_config=config)
117
```
118

119
### Multi Provider
120

121
Provider that automatically routes to appropriate backend based on model name.
122

123
```python { .api }
124
class MultiProvider(ModelProvider):
125
    """
126
    Multi-provider model lookup.
127

128
    Automatically selects appropriate provider based on
129
    model name patterns. Default provider for RunConfig.
130

131
    Supported:
132
    - OpenAI models (gpt-*, o1-*, etc.)
133
    - Anthropic models (claude-*)
134
    - Other providers via LiteLLM
135
    """
136

137
    def get_model(model_name: str) -> Model:
138
        """
139
        Get model from appropriate provider.
140

141
        Parameters:
142
        - model_name: Model identifier
143

144
        Returns:
145
        - Model: Model instance from appropriate provider
146
        """
147
```
148

149
Usage example:
150

151
```python
152
from agents import Agent, Runner, RunConfig, MultiProvider
153

154
# MultiProvider is default, routes automatically
155
config = RunConfig(model="gpt-4o")  # Uses OpenAI
156
agent = Agent(name="Assistant")
157
result = Runner.run_sync(agent, "Hello", run_config=config)
158

159
# Can also specify explicitly
160
provider = MultiProvider()
161
config = RunConfig(model="claude-3-5-sonnet-20241022", model_provider=provider)
162
```
163

164
### OpenAI Chat Completions Model
165

166
Implementation using OpenAI Chat Completions API.
167

168
```python { .api }
169
class OpenAIChatCompletionsModel(Model):
170
    """
171
    OpenAI Chat Completions API model.
172

173
    Uses the chat.completions endpoint for text generation.
174
    """
175

176
    async def get_response(...) -> ModelResponse:
177
        """Get response from Chat Completions API."""
178

179
    async def stream_response(...) -> AsyncIterator[TResponseStreamEvent]:
180
        """Stream response from Chat Completions API."""
181
```
182

183
### OpenAI Responses Model
184

185
Implementation using OpenAI Responses API (newer API).
186

187
```python { .api }
188
class OpenAIResponsesModel(Model):
189
    """
190
    OpenAI Responses API model.
191

192
    Uses the responses endpoint for advanced features
193
    like response chaining and conversation management.
194
    """
195

196
    async def get_response(...) -> ModelResponse:
197
        """Get response from Responses API."""
198

199
    async def stream_response(...) -> AsyncIterator[TResponseStreamEvent]:
200
        """Stream response from Responses API."""
201
```
202

203
### Model Tracing Configuration
204

205
Enum for configuring model tracing behavior.
206

207
```python { .api }
208
class ModelTracing(Enum):
209
    """
210
    Tracing configuration for models.
211

212
    Values:
213
    - DISABLED: No tracing
214
    - ENABLED: Full tracing with data
215
    - ENABLED_WITHOUT_DATA: Tracing without sensitive data
216
    """
217

218
    DISABLED = "disabled"
219
    ENABLED = "enabled"
220
    ENABLED_WITHOUT_DATA = "enabled_without_data"
221

222
    def is_disabled() -> bool:
223
        """
224
        Check if tracing is disabled.
225

226
        Returns:
227
        - bool: True if disabled
228
        """
229

230
    def include_data() -> bool:
231
        """
232
        Check if should include data in traces.
233

234
        Returns:
235
        - bool: True if data should be included
236
        """
237
```
238

239
## LiteLLM Provider
240

241
Provider for accessing 100+ LLMs through LiteLLM unified interface.
242

243
```python { .api }
244
# In agents.extensions.models
245
class LiteLLMProvider(ModelProvider):
246
    """
247
    LiteLLM model provider.
248

249
    Provides access to 100+ LLMs including:
250
    - Anthropic (Claude)
251
    - Google (Gemini, PaLM)
252
    - Cohere
253
    - Mistral
254
    - Together AI
255
    - Replicate
256
    - Azure OpenAI
257
    - AWS Bedrock
258
    - And many more
259

260
    Requires: pip install litellm
261
    """
262

263
    def __init__(self, **kwargs):
264
        """
265
        Initialize LiteLLM provider.
266

267
        Parameters:
268
        - **kwargs: Configuration passed to LiteLLM
269
        """
270

271
    def get_model(model_name: str) -> Model:
272
        """
273
        Get model from LiteLLM.
274

275
        Parameters:
276
        - model_name: Model identifier (e.g., "claude-3-5-sonnet-20241022")
277

278
        Returns:
279
        - Model: LiteLLM model instance
280
        """
281
```
282

283
Usage example:
284

285
```python
286
from agents import Agent, Runner, RunConfig
287
from agents.extensions.models import LiteLLMProvider
288

289
# Use Claude via LiteLLM
290
provider = LiteLLMProvider()
291
config = RunConfig(
292
    model="claude-3-5-sonnet-20241022",
293
    model_provider=provider
294
)
295

296
agent = Agent(
297
    name="Assistant",
298
    instructions="You are helpful."
299
)
300

301
result = Runner.run_sync(agent, "Hello", run_config=config)
302

303
# Use Gemini
304
config = RunConfig(
305
    model="gemini/gemini-1.5-pro",
306
    model_provider=provider
307
)
308

309
# Use AWS Bedrock
310
config = RunConfig(
311
    model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
312
    model_provider=provider
313
)
314
```
315

316
## Configuration
317

318
### Global OpenAI Configuration
319

320
Configure OpenAI client globally for all agents.
321

322
```python
323
from agents import set_default_openai_key, set_default_openai_client, set_default_openai_api
324
from openai import AsyncOpenAI
325

326
# Set API key
327
set_default_openai_key("sk-...", use_for_tracing=True)
328

329
# Set custom client
330
client = AsyncOpenAI(
331
    api_key="sk-...",
332
    base_url="https://api.openai.com/v1"
333
)
334
set_default_openai_client(client, use_for_tracing=True)
335

336
# Set default API mode
337
set_default_openai_api("responses")  # or "chat_completions"
338
```
339

340
### Per-Agent Model Configuration
341

342
```python
343
from agents import Agent, ModelSettings
344

345
agent = Agent(
346
    name="Assistant",
347
    model="gpt-4o",  # Specify model
348
    model_settings=ModelSettings(
349
        temperature=0.7,
350
        max_tokens=1000
351
    )
352
)
353
```
354

355
### Per-Run Model Configuration
356

357
```python
358
from agents import RunConfig, ModelSettings
359

360
config = RunConfig(
361
    model="gpt-4o-mini",  # Override agent model
362
    model_settings=ModelSettings(
363
        temperature=0.0,
364
        top_p=0.9
365
    )
366
)
367

368
result = Runner.run_sync(agent, "Hello", run_config=config)
369
```
370

371
## Custom Model Provider
372

373
Implement a custom provider for proprietary or specialized models:
374

375
```python
376
from agents import Model, ModelProvider, ModelResponse, ModelTracing
377
from agents.models.interface import TResponseInputItem, TResponseStreamEvent
378

379
class MyCustomModel(Model):
380
    """Custom model implementation."""
381

382
    def __init__(self, model_name: str):
383
        self.model_name = model_name
384

385
    async def get_response(
386
        self,
387
        input: list[TResponseInputItem],
388
        instructions: str | None,
389
        tools: list,
390
        model_settings,
391
        response_id: str | None,
392
        tracing: ModelTracing
393
    ) -> ModelResponse:
394
        # Call your model API
395
        response = await my_model_api.call(
396
            messages=input,
397
            system=instructions,
398
            temperature=model_settings.temperature
399
        )
400

401
        # Convert to ModelResponse
402
        return ModelResponse(
403
            output=[...],  # Convert response
404
            usage=Usage(...),  # Usage stats
405
            response_id=response.id
406
        )
407

408
    async def stream_response(
409
        self,
410
        input,
411
        instructions,
412
        tools,
413
        model_settings,
414
        response_id,
415
        tracing
416
    ):
417
        # Stream from your model
418
        async for chunk in my_model_api.stream(...):
419
            yield convert_chunk(chunk)
420

421
class MyModelProvider(ModelProvider):
422
    """Custom model provider."""
423

424
    def get_model(self, model_name: str) -> Model:
425
        if model_name.startswith("my-model-"):
426
            return MyCustomModel(model_name)
427
        raise ValueError(f"Unsupported model: {model_name}")
428

429
# Use custom provider
430
from agents import RunConfig
431

432
provider = MyModelProvider()
433
config = RunConfig(
434
    model="my-model-v1",
435
    model_provider=provider
436
)
437

438
result = Runner.run_sync(agent, "Hello", run_config=config)
439
```
440

441
## Model Selection Strategies
442

443
### Agent-Level Model
444

445
```python
446
# Each agent uses specific model
447
research_agent = Agent(
448
    name="Researcher",
449
    model="gpt-4o",  # More capable
450
    instructions="Research deeply."
451
)
452

453
summary_agent = Agent(
454
    name="Summarizer",
455
    model="gpt-4o-mini",  # Faster, cheaper
456
    instructions="Summarize briefly."
457
)
458
```
459

460
### Dynamic Model Selection
461

462
```python
463
def select_model(input: str) -> str:
464
    """Select model based on input complexity."""
465
    if len(input) > 1000 or "complex" in input.lower():
466
        return "gpt-4o"
467
    return "gpt-4o-mini"
468

469
# Use in run config
470
model = select_model(user_input)
471
config = RunConfig(model=model)
472
result = Runner.run_sync(agent, user_input, run_config=config)
473
```
474

475
### Fallback Strategy
476

477
```python
478
async def run_with_fallback(agent, input):
479
    """Try primary model, fallback to secondary."""
480
    try:
481
        config = RunConfig(model="gpt-4o")
482
        return await Runner.run(agent, input, run_config=config)
483
    except Exception as e:
484
        print(f"Primary model failed: {e}, trying fallback")
485
        config = RunConfig(model="gpt-4o-mini")
486
        return await Runner.run(agent, input, run_config=config)
487
```
488

489
## Best Practices
490

491
1. **Use MultiProvider**: Default provider handles most use cases automatically
492
2. **Model Selection**: Choose appropriate model for task complexity
493
3. **Cost Optimization**: Use smaller models for simple tasks
494
4. **Error Handling**: Implement fallbacks for model failures
495
5. **Testing**: Test with multiple providers to ensure compatibility
496
6. **API Keys**: Securely manage API keys, use environment variables
497
7. **Rate Limits**: Implement retry logic for rate limit errors
498
8. **Monitoring**: Track model usage and costs
499
9. **Caching**: Use prompt caching where available to reduce costs
500
10. **Provider Features**: Leverage provider-specific features when needed
501