0
# Portkey AI
1

2
Python client library for the Portkey API - Control Panel for AI Apps. Built on top of the OpenAI SDK, allowing seamless integration of Portkey's advanced features including AI Gateway capabilities (unified API signature, automated fallbacks & retries, load balancing, semantic caching, virtual keys), and comprehensive observability features (logging, request tracing, custom metadata, feedback collection, analytics).
3

4
## Package Information
5

6
- **Package Name**: portkey-ai
7
- **Language**: Python
8
- **Installation**: `pip install portkey-ai`
9

10
## Core Imports
11

12
```python
13
from portkey_ai import Portkey, AsyncPortkey
14
```
15

16
For OpenAI compatibility:
17

18
```python
19
from portkey_ai import openai
20
```
21

22
For specific components:
23

24
```python
25
from portkey_ai import (
26
    Config, 
27
    LLMOptions, 
28
    Modes,
29
    ProviderTypes,
30
    createHeaders
31
)
32
```
33

34
## Basic Usage
35

36
```python
37
from portkey_ai import Portkey
38

39
# Initialize client with API key and virtual key
40
portkey = Portkey(
41
    api_key="PORTKEY_API_KEY", 
42
    virtual_key="VIRTUAL_KEY"
43
)
44

45
# Make a chat completion request
46
chat_completion = portkey.chat.completions.create(
47
    messages=[{"role": "user", "content": "Say this is a test"}],
48
    model="gpt-4"
49
)
50

51
print(chat_completion)
52
```
53

54
### Async Usage
55

56
```python
57
import asyncio
58
from portkey_ai import AsyncPortkey
59

60
async def main():
61
    portkey = AsyncPortkey(
62
        api_key="PORTKEY_API_KEY",
63
        virtual_key="VIRTUAL_KEY"
64
    )
65
    
66
    chat_completion = await portkey.chat.completions.create(
67
        messages=[{"role": "user", "content": "Say this is a test"}],
68
        model="gpt-4"
69
    )
70
    
71
    print(chat_completion)
72

73
asyncio.run(main())
74
```
75

76
## Architecture
77

78
Portkey's architecture centers around two main client classes that provide comprehensive AI API management:
79

80
- **Portkey/AsyncPortkey**: Main client classes providing unified interface to 40+ AI providers
81
- **API Resources**: Organized into functional areas (chat, completions, images, audio, etc.)
82
- **Configuration System**: Flexible config management for providers, fallbacks, retries, and routing
83
- **Observability Layer**: Built-in logging, tracing, analytics, and feedback collection
84
- **Security Layer**: Virtual key management and secure credential handling
85

86
The dual client pattern ensures both synchronous and asynchronous support across all operations, while maintaining full OpenAI SDK compatibility.
87

88
## Capabilities
89

90
### Core Client & Configuration
91

92
Primary client classes and configuration utilities for initializing and managing Portkey connections with comprehensive provider support and advanced routing capabilities.
93

94
```python { .api }
95
class Portkey:
96
    def __init__(
97
        self,
98
        *,
99
        api_key: Optional[str] = None,
100
        base_url: Optional[str] = None,
101
        virtual_key: Optional[str] = None,
102
        config: Optional[Union[Mapping, str]] = None,
103
        provider: Optional[str] = None,
104
        trace_id: Optional[str] = None,
105
        metadata: Union[Optional[dict[str, str]], str] = None,
106
        **kwargs
107
    ) -> None: ...
108

109
class AsyncPortkey:
110
    def __init__(self, **kwargs) -> None: ...
111

112
def createHeaders(**kwargs) -> dict: ...
113
```
114

115
[Core Client & Configuration](./core-client.md)
116

117
### Chat Completions
118

119
OpenAI-compatible chat completion API with support for all major providers, streaming, function calling, and advanced Portkey features like fallbacks and load balancing.
120

121
```python { .api }
122
class ChatCompletion:
123
    completions: Completions
124

125
class Completions:
126
    def create(
127
        self,
128
        *,
129
        messages: Iterable[dict],
130
        model: Optional[str] = "portkey-default",
131
        **kwargs
132
    ) -> Union[ChatCompletions, Iterator[ChatCompletionChunk]]: ...
133
    
134
    messages: ChatCompletionsMessages
135

136
class AsyncChatCompletion:
137
    completions: AsyncCompletions
138
```
139

140
[Chat Completions](./chat-completions.md)
141

142
### Text Completions
143

144
Legacy text completion API for non-chat models, supporting streaming and all OpenAI-compatible parameters with Portkey enhancements.
145

146
```python { .api }
147
class Completion:
148
    def create(
149
        self,
150
        *,
151
        prompt: Union[str, List[str]],
152
        model: str,
153
        **kwargs
154
    ) -> Union[TextCompletion, Iterator[TextCompletionChunk]]: ...
155

156
class AsyncCompletion:
157
    async def create(
158
        self,
159
        *,
160
        prompt: Union[str, List[str]],
161
        model: str,
162
        **kwargs
163
    ) -> Union[TextCompletion, AsyncIterator[TextCompletionChunk]]: ...
164
```
165

166
[Text Completions](./text-completions.md)
167

168
### Prompt Management
169

170
Advanced prompt management system with templating, rendering, versioning, and prompt execution capabilities.
171

172
```python { .api }
173
class Prompts:
174
    def render(self, **kwargs): ...
175
    def completions(self, **kwargs): ...
176

177
class AsyncPrompts:
178
    async def render(self, **kwargs): ...
179
    async def completions(self, **kwargs): ...
180

181
class Generations:
182
    def create(self, **kwargs): ...
183

184
class AsyncGenerations:
185
    async def create(self, **kwargs): ...
186
```
187

188
[Prompt Management](./prompt-management.md)
189

190
### Embeddings
191

192
Text embedding generation for semantic similarity, search, and machine learning applications with support for multiple providers.
193

194
```python { .api }
195
class Embeddings:
196
    def create(
197
        self,
198
        *,
199
        input: str,
200
        model: Optional[str] = "portkey-default",
201
        dimensions: Union[int, NotGiven] = NOT_GIVEN,
202
        encoding_format: Union[str, NotGiven] = NOT_GIVEN,
203
        user: Union[str, NotGiven] = NOT_GIVEN,
204
        **kwargs
205
    ) -> CreateEmbeddingResponse: ...
206

207
class AsyncEmbeddings:
208
    async def create(self, **kwargs) -> CreateEmbeddingResponse: ...
209
```
210

211
[Embeddings](./embeddings.md)
212

213
### Models
214

215
Model management and information retrieval for accessing available AI models across different providers.
216

217
```python { .api }
218
class Models:
219
    def list(self, **kwargs) -> ModelList: ...
220
    def retrieve(
221
        self,
222
        model: str,
223
        *,
224
        timeout: Union[float, NotGiven] = NOT_GIVEN,
225
        **kwargs
226
    ) -> Model: ...
227
    def delete(
228
        self,
229
        model: str,
230
        *,
231
        timeout: Union[float, NotGiven] = NOT_GIVEN,
232
        **kwargs
233
    ) -> ModelDeleted: ...
234

235
class AsyncModels:
236
    async def list(self, **kwargs) -> ModelList: ...
237
    async def retrieve(self, model: str, **kwargs) -> Model: ...
238
    async def delete(self, model: str, **kwargs) -> ModelDeleted: ...
239
```
240

241
[Models](./models.md)
242

243
### Multimodal APIs
244

245
Image generation, audio processing (speech-to-text, text-to-speech, translation), and content moderation capabilities with OpenAI compatibility.
246

247
```python { .api }
248
class Images:
249
    def generate(self, **kwargs): ...
250
    def create_variation(self, **kwargs): ...
251
    def edit(self, **kwargs): ...
252

253
class Audio:
254
    transcriptions: Transcriptions
255
    translations: Translations
256
    speech: Speech
257

258
class Moderations:
259
    def create(self, **kwargs): ...
260
```
261

262
[Multimodal APIs](./multimodal-apis.md)
263

264
### File Management
265

266
File upload, management, and processing capabilities including support for assistants, fine-tuning, and batch operations.
267

268
```python { .api }
269
class MainFiles:
270
    def create(self, **kwargs): ...
271
    def list(self, **kwargs): ...
272
    def retrieve(self, **kwargs): ...
273
    def delete(self, **kwargs): ...
274

275
class Uploads:
276
    def create(self, **kwargs): ...
277
    parts: Parts
278
```
279

280
[File Management](./file-management.md)
281

282
### Assistants & Threads
283

284
OpenAI Assistants API implementation with thread management, message handling, and tool execution support.
285

286
```python { .api }
287
class Assistants:
288
    def create(self, **kwargs): ...
289
    def list(self, **kwargs): ...
290
    def retrieve(self, **kwargs): ...
291
    def update(self, **kwargs): ...
292
    def delete(self, **kwargs): ...
293

294
class Threads:
295
    def create(self, **kwargs): ...
296
    def retrieve(self, **kwargs): ...
297
    def update(self, **kwargs): ...
298
    def delete(self, **kwargs): ...
299
    messages: Messages
300
    runs: Runs
301
```
302

303
[Assistants & Threads](./assistants-threads.md)
304

305
### Vector Stores
306

307
Vector database operations for retrieval-augmented generation (RAG) applications with file management and batch processing.
308

309
```python { .api }
310
class VectorStores:
311
    def create(self, **kwargs): ...
312
    def list(self, **kwargs): ...
313
    def retrieve(self, **kwargs): ...
314
    def update(self, **kwargs): ...
315
    def delete(self, **kwargs): ...
316
    files: VectorFiles
317
    file_batches: VectorFileBatches
318
```
319

320
[Vector Stores](./vector-stores.md)
321

322
### Fine-Tuning
323

324
Model fine-tuning capabilities with job management, checkpoint handling, and training monitoring.
325

326
```python { .api }
327
class FineTuning:
328
    jobs: Jobs
329
    checkpoints: Checkpoints
330

331
class Jobs:
332
    def create(self, **kwargs): ...
333
    def list(self, **kwargs): ...
334
    def retrieve(self, **kwargs): ...
335
    def cancel(self, **kwargs): ...
336
```
337

338
[Fine-Tuning](./fine-tuning.md)
339

340
### Batch Processing
341

342
Batch API for processing large volumes of requests efficiently with cost optimization and result management.
343

344
```python { .api }
345
class Batches:
346
    def create(self, **kwargs): ...
347
    def list(self, **kwargs): ...
348
    def retrieve(self, **kwargs): ...
349
    def cancel(self, **kwargs): ...
350
```
351

352
[Batch Processing](./batch-processing.md)
353

354
### Administration & User Management
355

356
Administrative functions for managing users, workspaces, invitations, and organizational settings.
357

358
```python { .api }
359
class Admin:
360
    users: Users
361
    invites: Invites
362
    workspaces: Workspaces
363

364
class Users:
365
    def create(self, **kwargs): ...
366
    def list(self, **kwargs): ...
367
    def retrieve(self, **kwargs): ...
368
    def update(self, **kwargs): ...
369
    def delete(self, **kwargs): ...
370
```
371

372
[Administration & User Management](./administration.md)
373

374
### Configuration Management
375

376
Advanced configuration system for managing provider settings, routing rules, fallback strategies, and load balancing configurations.
377

378
```python { .api }
379
class Configs:
380
    def create(self, **kwargs): ...
381
    def list(self, **kwargs): ...
382
    def retrieve(self, **kwargs): ...
383
    def update(self, **kwargs): ...
384
    def delete(self, **kwargs): ...
385
```
386

387
[Configuration Management](./configuration-management.md)
388

389
### API Key & Virtual Key Management
390

391
Secure API key and virtual key management for credential protection and access control.
392

393
```python { .api }
394
class ApiKeys:
395
    def create(self, **kwargs): ...
396
    def list(self, **kwargs): ...
397
    def retrieve(self, **kwargs): ...
398
    def update(self, **kwargs): ...
399
    def delete(self, **kwargs): ...
400

401
class VirtualKeys:
402
    def create(self, **kwargs): ...
403
    def list(self, **kwargs): ...
404
    def retrieve(self, **kwargs): ...
405
    def update(self, **kwargs): ...
406
    def delete(self, **kwargs): ...
407
```
408

409
[API Key & Virtual Key Management](./key-management.md)
410

411
### Observability & Analytics
412

413
Comprehensive logging, request tracing, analytics, and monitoring capabilities with custom metadata and performance metrics.
414

415
```python { .api }
416
class Logs:
417
    def list(self, **kwargs): ...
418
    def retrieve(self, **kwargs): ...
419

420
class Responses:
421
    def list(self, **kwargs): ...
422
    def retrieve(self, **kwargs): ...
423
    input_items: InputItems
424
    output_items: OutputItems
425

426
class Labels:
427
    def create(self, **kwargs): ...
428
    def list(self, **kwargs): ...
429
    def update(self, **kwargs): ...
430
    def delete(self, **kwargs): ...
431
```
432

433
[Observability & Analytics](./observability-analytics.md)
434

435
### Feedback & Collections
436

437
User feedback collection and data organization capabilities for improving AI applications and gathering insights.
438

439
```python { .api }
440
class Feedback:
441
    def create(self, **kwargs): ...
442
    def list(self, **kwargs): ...
443
    def retrieve(self, **kwargs): ...
444

445
class Collections:
446
    def create(self, **kwargs): ...
447
    def list(self, **kwargs): ...
448
    def retrieve(self, **kwargs): ...
449
    def update(self, **kwargs): ...
450
    def delete(self, **kwargs): ...
451
```
452

453
[Feedback & Collections](./feedback-collections.md)
454

455
### Evaluation & Testing
456

457
AI evaluation framework with automated grading, test execution, and performance measurement capabilities.
458

459
```python { .api }
460
class Evals:
461
    runs: EvalsRuns
462

463
class EvalsRuns:
464
    def create(self, **kwargs): ...
465
    def list(self, **kwargs): ...
466
    def retrieve(self, **kwargs): ...
467

468
class Alpha:
469
    graders: Graders
470

471
class Graders:
472
    def create(self, **kwargs): ...
473
    def list(self, **kwargs): ...
474
```
475

476
[Evaluation & Testing](./evaluation-testing.md)
477

478
### Container & Content Management
479

480
Container-based deployment and content management capabilities for scalable AI applications.
481

482
```python { .api }
483
class Containers:
484
    def create(self, **kwargs): ...
485
    def list(self, **kwargs): ...
486
    def retrieve(self, **kwargs): ...
487
    def update(self, **kwargs): ...
488
    def delete(self, **kwargs): ...
489
    files: ContainersFiles
490

491
class Content:
492
    def create(self, **kwargs): ...
493
    def list(self, **kwargs): ...
494
    def retrieve(self, **kwargs): ...
495
```
496

497
[Container & Content Management](./container-content.md)
498

499
### Large File Uploads
500

501
Multi-part upload system for large files with chunked upload support and integrity verification.
502

503
```python { .api }
504
class Uploads:
505
    def create(
506
        self,
507
        *,
508
        bytes: int,
509
        filename: str,
510
        mime_type: str,
511
        purpose: Any,
512
        **kwargs
513
    ) -> Upload: ...
514
    
515
    def upload_file_chunked(
516
        self,
517
        *,
518
        file: Union[os.PathLike[str], bytes],
519
        mime_type: str,
520
        purpose: Any,
521
        **kwargs
522
    ) -> Any: ...
523
    
524
    def complete(
525
        self,
526
        upload_id: str,
527
        *,
528
        part_ids: List[str],
529
        **kwargs
530
    ) -> Upload: ...
531
    
532
    parts: Parts
533

534
class AsyncUploads:
535
    async def create(self, **kwargs) -> Upload: ...
536
    async def upload_file_chunked(self, **kwargs) -> Any: ...
537
    async def complete(self, upload_id: str, **kwargs) -> Upload: ...
538
    parts: AsyncParts
539
```
540

541
[Large File Uploads](./uploads.md)
542

543
### Beta Realtime API
544

545
Real-time audio and WebSocket-based AI interactions for building conversational applications with low-latency voice communication.
546

547
```python { .api }
548
class BetaRealtime:
549
    def connect(
550
        self,
551
        *,
552
        model: str,
553
        websocket_connection_options: WebsocketConnectionOptions = {},
554
        **kwargs
555
    ) -> RealtimeConnectionManager: ...
556
    
557
    sessions: BetaSessions
558

559
class BetaSessions:
560
    def create(
561
        self,
562
        *,
563
        model: Any = "portkey-default",
564
        modalities: Union[List[Any], NotGiven] = NOT_GIVEN,
565
        voice: Union[Any, NotGiven] = NOT_GIVEN,
566
        instructions: Union[str, NotGiven] = NOT_GIVEN,
567
        **kwargs
568
    ) -> SessionCreateResponse: ...
569

570
class AsyncBetaRealtime:
571
    def connect(self, **kwargs) -> AsyncRealtimeConnectionManager: ...
572
    sessions: AsyncBetaSessions
573
```
574

575
[Beta Realtime API](./beta-realtime.md)
576

577
### Provider & Integration Management
578

579
Management of AI providers and third-party integrations with workspace-level and model-level configuration.
580

581
```python { .api }
582
class Providers:
583
    def list(self, **kwargs): ...
584
    def retrieve(self, **kwargs): ...
585

586
class Integrations:
587
    workspaces: IntegrationsWorkspaces
588
    models: IntegrationsModels
589

590
class IntegrationsWorkspaces:
591
    def create(self, **kwargs): ...
592
    def list(self, **kwargs): ...
593
    def retrieve(self, **kwargs): ...
594
```
595

596
[Provider & Integration Management](./provider-integration.md)
597

598
### Framework Integrations
599

600
Pre-built integrations with popular AI frameworks including LangChain and LlamaIndex callback handlers.
601

602
```python { .api }
603
from portkey_ai.langchain import PortkeyLangchainCallbackHandler
604
from portkey_ai.llamaindex import PortkeyLlamaCallbackHandler
605
```
606

607
[Framework Integrations](./framework-integrations.md)
608

609
## Types
610

611
### Core Configuration Types
612

613
```python { .api }
614
class Config(TypedDict):
615
    """Configuration object for Portkey requests"""
616
    ...
617

618
class LLMOptions(TypedDict):
619
    """LLM-specific options and parameters"""
620
    ...
621

622
class Modes(Enum):
623
    """Available Portkey modes"""
624
    SINGLE = "single"
625
    FALLBACK = "fallback"
626
    LOADBALANCE = "loadbalance"
627
    AB_TEST = "ab_test"
628

629
class ProviderTypes(Enum):
630
    """Supported AI providers"""
631
    OPENAI = "openai"
632
    ANTHROPIC = "anthropic"
633
    AZURE_OPENAI = "azure-openai"
634
    COHERE = "cohere"
635
    HUGGINGFACE = "huggingface"
636
    # ... and 35+ more providers
637

638
class RetrySettings(TypedDict):
639
    """Retry configuration settings"""
640
    attempts: int
641
    on_status_codes: List[int]
642
```
643

644
### Response Types
645

646
```python { .api }
647
class PortkeyResponse:
648
    """Base response class for all Portkey operations"""
649
    ...
650

651
class ChatCompletions:
652
    """Chat completion response"""
653
    id: str
654
    object: str
655
    created: int
656
    model: str
657
    choices: List[dict]
658
    usage: dict
659

660
class ChatCompletionChunk:
661
    """Streaming chat completion chunk"""
662
    id: str
663
    object: str
664
    created: int
665
    model: str
666
    choices: List[dict]
667

668
class TextCompletion:
669
    """Text completion response"""
670
    id: str
671
    object: str
672
    created: int
673
    model: str
674
    choices: List[dict]
675
    usage: dict
676

677
class Message:
678
    """Message object for chat completions"""
679
    role: str
680
    content: str
681
```
682

683
### Exception Types
684

685
```python { .api }
686
class APIStatusError(Exception):
687
    """Base exception for API status errors"""
688
    ...
689

690
class BadRequestError(APIStatusError):
691
    """400 Bad Request error"""
692
    ...
693

694
class AuthenticationError(APIStatusError):
695
    """401 Authentication error"""
696
    ...
697

698
class PermissionDeniedError(APIStatusError):
699
    """403 Permission denied error"""
700
    ...
701

702
class NotFoundError(APIStatusError):
703
    """404 Not found error"""
704
    ...
705

706
class RateLimitError(APIStatusError):
707
    """429 Rate limit error"""
708
    ...
709

710
class InternalServerError(APIStatusError):
711
    """500 Internal server error"""
712
    ...
713
```