0
# LLM Integration
1
2
Unified interface for various language models with support for completion, chat, and function calling APIs across multiple providers.
3
4
## Capabilities
5
6
### Base LLM Interface
7
8
Core LLM abstraction providing unified interface across providers.
9
10
```python { .api }
11
class LLM:
12
"""
13
Base class for Language Model integrations.
14
15
Args:
16
model: Model name or identifier
17
temperature: Sampling temperature (0.0 to 1.0)
18
max_tokens: Maximum tokens to generate
19
additional_kwargs: Provider-specific arguments
20
callback_manager: Callback manager for events
21
**kwargs: Additional arguments
22
"""
23
def __init__(
24
self,
25
model=None,
26
temperature=0.1,
27
max_tokens=None,
28
additional_kwargs=None,
29
callback_manager=None,
30
**kwargs
31
): ...
32
33
def complete(self, prompt, **kwargs):
34
"""
35
Complete a text prompt.
36
37
Args:
38
prompt: Text prompt to complete
39
**kwargs: Additional completion arguments
40
41
Returns:
42
CompletionResponse: Generated completion
43
"""
44
45
def chat(self, messages, **kwargs):
46
"""
47
Generate chat response from message history.
48
49
Args:
50
messages: List of ChatMessage objects
51
**kwargs: Additional chat arguments
52
53
Returns:
54
ChatResponse: Generated chat response
55
"""
56
57
def stream_complete(self, prompt, **kwargs):
58
"""Stream completion response."""
59
60
def stream_chat(self, messages, **kwargs):
61
"""Stream chat response."""
62
63
async def acomplete(self, prompt, **kwargs):
64
"""Async completion."""
65
66
async def achat(self, messages, **kwargs):
67
"""Async chat."""
68
```
69
70
### OpenAI Integration
71
72
OpenAI language models including GPT-3.5, GPT-4, and function calling support.
73
74
```python { .api }
75
class OpenAI(LLM):
76
"""
77
OpenAI language model integration.
78
79
Args:
80
model: OpenAI model name (e.g., "gpt-3.5-turbo", "gpt-4")
81
temperature: Sampling temperature
82
max_tokens: Maximum tokens to generate
83
api_key: OpenAI API key
84
api_base: Custom API base URL
85
api_version: API version
86
**kwargs: Additional OpenAI arguments
87
"""
88
def __init__(
89
self,
90
model="gpt-3.5-turbo",
91
temperature=0.1,
92
max_tokens=None,
93
api_key=None,
94
api_base=None,
95
api_version=None,
96
**kwargs
97
): ...
98
99
def complete(self, prompt, **kwargs):
100
"""OpenAI completion."""
101
102
def chat(self, messages, **kwargs):
103
"""OpenAI chat completion."""
104
105
def get_tool_calls_from_response(self, response, error_on_no_tool_call=True):
106
"""
107
Extract tool calls from OpenAI response.
108
109
Args:
110
response: OpenAI API response
111
error_on_no_tool_call: Whether to error if no tool calls found
112
113
Returns:
114
List: List of tool call objects
115
"""
116
```
117
118
### Chat Messages
119
120
Message representations for chat-based LLM interactions.
121
122
```python { .api }
123
class ChatMessage:
124
"""
125
Chat message representation.
126
127
Args:
128
role: Message role (user, assistant, system, tool)
129
content: Message content
130
additional_kwargs: Additional message metadata
131
"""
132
def __init__(self, role, content=None, additional_kwargs=None): ...
133
134
@property
135
def role(self):
136
"""Message role."""
137
138
@property
139
def content(self):
140
"""Message content."""
141
142
class MessageRole:
143
"""Message role constants."""
144
USER = "user"
145
ASSISTANT = "assistant"
146
SYSTEM = "system"
147
TOOL = "tool"
148
FUNCTION = "function" # Deprecated, use TOOL
149
```
150
151
### LLM Response Types
152
153
Response objects for LLM completions and chat.
154
155
```python { .api }
156
class CompletionResponse:
157
"""
158
Response from completion API.
159
160
Args:
161
text: Generated text
162
additional_kwargs: Additional response metadata
163
raw: Raw response from provider
164
"""
165
def __init__(self, text=None, additional_kwargs=None, raw=None): ...
166
167
@property
168
def text(self):
169
"""Generated text."""
170
171
class ChatResponse:
172
"""
173
Response from chat API.
174
175
Args:
176
message: Generated message
177
additional_kwargs: Additional response metadata
178
raw: Raw response from provider
179
"""
180
def __init__(self, message=None, additional_kwargs=None, raw=None): ...
181
182
@property
183
def message(self):
184
"""Generated chat message."""
185
```
186
187
### Function Calling
188
189
Support for structured function calling with LLMs.
190
191
```python { .api }
192
class FunctionCallingLLM(LLM):
193
"""Base class for LLMs with function calling support."""
194
195
def predict_and_call(self, tools, user_msg, **kwargs):
196
"""
197
Predict and execute function calls.
198
199
Args:
200
tools: List of available tools/functions
201
user_msg: User message
202
**kwargs: Additional arguments
203
204
Returns:
205
AgentChatResponse: Response with function call results
206
"""
207
208
def get_tool_calls_from_response(self, response, error_on_no_tool_call=True):
209
"""Extract tool calls from response."""
210
211
class ToolSelection:
212
"""
213
Tool selection for function calling.
214
215
Args:
216
tool_id: Tool identifier
217
tool_name: Tool name
218
tool_kwargs: Tool arguments
219
"""
220
def __init__(self, tool_id, tool_name, tool_kwargs=None): ...
221
```
222
223
### Custom LLM Implementation
224
225
Base classes for implementing custom LLM integrations.
226
227
```python { .api }
228
class CustomLLM(LLM):
229
"""
230
Base class for custom LLM implementations.
231
232
Subclasses should implement:
233
- _complete: Core completion logic
234
- _chat: Core chat logic (optional)
235
- metadata: Model metadata
236
"""
237
238
@property
239
def metadata(self):
240
"""
241
Model metadata including context window and model name.
242
243
Returns:
244
LLMMetadata: Model metadata object
245
"""
246
247
def _complete(self, prompt, **kwargs):
248
"""
249
Implementation-specific completion logic.
250
251
Args:
252
prompt: Text prompt
253
**kwargs: Additional arguments
254
255
Returns:
256
CompletionResponse: Generated completion
257
"""
258
raise NotImplementedError
259
260
def _chat(self, messages, **kwargs):
261
"""Implementation-specific chat logic."""
262
```
263
264
### Multi-Modal LLM Support
265
266
Support for LLMs that can process images and other media.
267
268
```python { .api }
269
class MultiModalLLM(LLM):
270
"""Base class for multi-modal LLMs."""
271
272
def complete(self, prompt, image_documents=None, **kwargs):
273
"""
274
Complete with optional image inputs.
275
276
Args:
277
prompt: Text prompt
278
image_documents: List of image documents
279
**kwargs: Additional arguments
280
281
Returns:
282
CompletionResponse: Generated completion
283
"""
284
285
def chat(self, messages, **kwargs):
286
"""Chat with multi-modal message support."""
287
```
288
289
## LLM Utilities
290
291
```python { .api }
292
class LLMMetadata:
293
"""
294
Metadata about an LLM model.
295
296
Args:
297
context_window: Maximum context window size
298
num_output: Maximum output tokens
299
is_chat_model: Whether model supports chat API
300
is_function_calling_model: Whether model supports function calling
301
model_name: Model identifier
302
"""
303
def __init__(
304
self,
305
context_window=None,
306
num_output=None,
307
is_chat_model=False,
308
is_function_calling_model=False,
309
model_name=None,
310
): ...
311
312
def resolve_llm(llm=None):
313
"""
314
Resolve LLM instance from various inputs.
315
316
Args:
317
llm: LLM instance, string identifier, or None
318
319
Returns:
320
LLM: Resolved LLM instance
321
"""
322
```
323
324
## Integration Examples
325
326
### Using Different Providers
327
328
```python
329
# OpenAI
330
from llama_index.llms.openai import OpenAI
331
llm = OpenAI(model="gpt-4", temperature=0.1)
332
333
# Anthropic
334
from llama_index.llms.anthropic import Anthropic
335
llm = Anthropic(model="claude-3-sonnet")
336
337
# Local models via Ollama
338
from llama_index.llms.ollama import Ollama
339
llm = Ollama(model="llama2")
340
341
# Azure OpenAI
342
from llama_index.llms.azure_openai import AzureOpenAI
343
llm = AzureOpenAI(
344
model="gpt-35-turbo",
345
deployment_name="my-deployment",
346
api_key="...",
347
azure_endpoint="https://my-resource.openai.azure.com/"
348
)
349
```