0
# llama-cpp-python
1
2
Python bindings for the llama.cpp library providing high-performance large language model inference with comprehensive APIs for text completion, chat, embeddings, and multimodal processing. Offers both high-level Python interfaces and low-level C bindings with OpenAI-compatible endpoints.
3
4
## Package Information
5
6
- **Package Name**: llama-cpp-python
7
- **Package Type**: PyPI
8
- **Language**: Python
9
- **Installation**: `pip install llama-cpp-python`
10
11
## Core Imports
12
13
```python
14
import llama_cpp
15
```
16
17
Common high-level imports:
18
19
```python
20
from llama_cpp import Llama, LlamaGrammar, LlamaCache
21
```
22
23
OpenAI-compatible types:
24
25
```python
26
from llama_cpp.llama_types import (
27
CreateCompletionResponse,
28
CreateChatCompletionResponse,
29
CreateEmbeddingResponse
30
)
31
```
32
33
## Basic Usage
34
35
```python
36
from llama_cpp import Llama
37
38
# Initialize model
39
llm = Llama(
40
model_path="./models/llama-model.gguf",
41
n_ctx=2048, # Context window
42
n_threads=8, # CPU threads
43
)
44
45
# Generate text completion
46
output = llm.create_completion(
47
prompt="The capital of France is",
48
max_tokens=32,
49
temperature=0.7,
50
top_p=0.9,
51
)
52
print(output['choices'][0]['text'])
53
54
# Create chat completion
55
messages = [
56
{"role": "system", "content": "You are a helpful assistant."},
57
{"role": "user", "content": "Hello! How are you?"}
58
]
59
60
response = llm.create_chat_completion(
61
messages=messages,
62
max_tokens=100,
63
temperature=0.7,
64
)
65
print(response['choices'][0]['message']['content'])
66
67
# Generate embeddings
68
embeddings = llm.create_embedding(
69
input=["Hello world", "Python is great"],
70
)
71
print(embeddings['data'][0]['embedding'][:5]) # First 5 dimensions
72
```
73
74
## Architecture
75
76
The llama-cpp-python package provides multiple layers of abstraction:
77
78
- **High-level API**: The `Llama` class offers convenient methods for common operations with sensible defaults
79
- **Low-level bindings**: Direct access to llama.cpp C functions through ctypes for maximum control
80
- **OpenAI compatibility**: Drop-in replacement for OpenAI API endpoints with identical response formats
81
- **Extensible components**: Modular caching, tokenization, grammar, and formatting systems
82
83
Key design patterns:
84
- **Lazy loading**: Models and contexts are loaded only when needed
85
- **Memory management**: Automatic cleanup and manual control options
86
- **Hardware optimization**: CPU, CUDA, and Metal acceleration support
87
- **Format flexibility**: Support for various model formats (GGUF, GGML) and quantization levels
88
89
## Capabilities
90
91
### Core Model and Inference
92
93
High-level model loading, text generation, and inference operations including completion, sampling, state management, and performance optimization.
94
95
```python { .api }
96
class Llama:
97
def __init__(self, model_path: str, **kwargs): ...
98
def create_completion(self, prompt: str, **kwargs) -> CreateCompletionResponse: ...
99
def create_chat_completion(self, messages: List[dict], **kwargs) -> CreateChatCompletionResponse: ...
100
def create_embedding(self, input: Union[str, List[str]], **kwargs) -> CreateEmbeddingResponse: ...
101
def tokenize(self, text: str, add_bos: bool = True, special: bool = False) -> List[int]: ...
102
def detokenize(self, tokens: List[int], decode: bool = True) -> str: ...
103
```
104
105
[Core Model and Inference](./llama-model.md)
106
107
### Chat Completions and Formatting
108
109
OpenAI-compatible chat completions with extensive formatting options, role-based conversations, function calling, and custom message templates for different model types.
110
111
```python { .api }
112
def get_chat_completion_handler(chat_format: str) -> LlamaChatCompletionHandler: ...
113
def register_chat_completion_handler(chat_format: str, chat_handler: LlamaChatCompletionHandler): ...
114
115
class Jinja2ChatFormatter:
116
def __init__(self, template: str, **kwargs): ...
117
def format_messages(self, messages: List[dict]) -> ChatFormatterResponse: ...
118
```
119
120
[Chat Completions and Formatting](./chat-completion.md)
121
122
### Tokenization
123
124
Native llama.cpp tokenization and HuggingFace tokenizer integration with support for different vocabulary types, encoding/decoding, and model-specific preprocessing.
125
126
```python { .api }
127
class LlamaTokenizer:
128
def tokenize(self, text: str, add_bos: bool = True, special: bool = False) -> List[int]: ...
129
def detokenize(self, tokens: List[int], decode: bool = True) -> str: ...
130
@classmethod
131
def from_ggml_file(cls, path: str) -> "LlamaTokenizer": ...
132
133
class LlamaHFTokenizer:
134
@classmethod
135
def from_pretrained(cls, pretrained_model_name_or_path: str) -> "LlamaHFTokenizer": ...
136
```
137
138
[Tokenization](./tokenization.md)
139
140
### Caching
141
142
Memory and disk-based caching systems for model states, context, and computed results to improve inference performance and enable state persistence.
143
144
```python { .api }
145
class LlamaRAMCache:
146
def __init__(self, capacity_bytes: int = 2 << 30): ...
147
148
class LlamaDiskCache:
149
def __init__(self, cache_dir: str = ".cache/llama_cpp"): ...
150
```
151
152
[Caching](./caching.md)
153
154
### Grammar and Structured Generation
155
156
Constrained text generation using formal grammars (GBNF), JSON Schema validation, and built-in templates for structured outputs like JSON, code, and domain-specific formats.
157
158
```python { .api }
159
class LlamaGrammar:
160
@classmethod
161
def from_string(cls, grammar_str: str, verbose: bool = True) -> "LlamaGrammar": ...
162
@classmethod
163
def from_json_schema(cls, schema: dict, verbose: bool = True) -> "LlamaGrammar": ...
164
165
def json_schema_to_gbnf(schema: dict, **kwargs) -> str: ...
166
```
167
168
[Grammar and Structured Generation](./grammar.md)
169
170
### Vision and Multimodal
171
172
LLaVA vision model integration for processing images alongside text, supporting various image formats and multimodal conversation flows.
173
174
```python { .api }
175
def llava_image_embed_make_with_filename(ctx_clip, image_path: str): ...
176
def llava_image_embed_make_with_bytes(ctx_clip, image_bytes: bytes, image_bytes_length: int): ...
177
def llava_validate_embed_size(n_embd: int, n_image_embd: int) -> bool: ...
178
```
179
180
[Vision and Multimodal](./vision.md)
181
182
### Server Components
183
184
FastAPI-based web server with OpenAI-compatible endpoints, settings management, and multi-model configuration support for production deployments.
185
186
```python { .api }
187
class ModelSettings:
188
model: str
189
n_ctx: int = 2048
190
temperature: float = 0.7
191
192
class ServerSettings:
193
host: str = "127.0.0.1"
194
port: int = 8000
195
interrupt_requests: bool = True
196
```
197
198
[Server Components](./server.md)
199
200
### Low-Level API
201
202
Direct access to llama.cpp C functions through ctypes bindings, providing maximum control over model loading, context management, and backend operations.
203
204
```python { .api }
205
def llama_model_load_from_file(path_model: bytes, params) -> llama_model_p: ...
206
def llama_new_context_with_model(model: llama_model_p, params) -> llama_context_p: ...
207
def llama_backend_init() -> None: ...
208
def llama_backend_free() -> None: ...
209
```
210
211
[Low-Level API](./low-level.md)
212
213
## Types
214
215
```python { .api }
216
# Core response types
217
CreateCompletionResponse = TypedDict('CreateCompletionResponse', {
218
'id': str,
219
'object': str,
220
'created': int,
221
'model': str,
222
'choices': List[CompletionChoice],
223
'usage': CompletionUsage,
224
})
225
226
CreateChatCompletionResponse = TypedDict('CreateChatCompletionResponse', {
227
'id': str,
228
'object': str,
229
'created': int,
230
'model': str,
231
'choices': List[ChatCompletionResponseChoice],
232
'usage': CompletionUsage,
233
})
234
235
CreateEmbeddingResponse = TypedDict('CreateEmbeddingResponse', {
236
'object': str,
237
'data': List[Embedding],
238
'model': str,
239
'usage': EmbeddingUsage,
240
})
241
242
# Message types for chat
243
ChatCompletionRequestMessage = TypedDict('ChatCompletionRequestMessage', {
244
'role': str,
245
'content': Optional[str],
246
})
247
248
ChatCompletionRequestSystemMessage = TypedDict('ChatCompletionRequestSystemMessage', {
249
'role': Literal['system'],
250
'content': str,
251
})
252
253
ChatCompletionRequestUserMessage = TypedDict('ChatCompletionRequestUserMessage', {
254
'role': Literal['user'],
255
'content': str,
256
})
257
258
ChatCompletionRequestAssistantMessage = TypedDict('ChatCompletionRequestAssistantMessage', {
259
'role': Literal['assistant'],
260
'content': Optional[str],
261
})
262
263
# JSON serializable type
264
JsonType = Union[None, int, float, str, bool, List['JsonType'], Dict[str, 'JsonType']]
265
```