0
# Transformers
1
2
State-of-the-art Machine Learning library for JAX, PyTorch and TensorFlow. Transformers provides a unified API for working with over 350 pre-trained models across natural language processing, computer vision, audio, and multimodal tasks. The library democratizes access to cutting-edge AI models with simple, efficient interfaces for both inference and training.
3
4
## Package Information
5
6
- **Package Name**: transformers
7
- **Language**: Python
8
- **Installation**: `pip install transformers`
9
10
## Core Imports
11
12
```python
13
import transformers
14
```
15
16
Common patterns for specific functionality:
17
18
```python
19
# High-level Pipeline API (recommended for most use cases)
20
from transformers import pipeline
21
22
# Auto classes for automatic model/tokenizer selection
23
from transformers import AutoModel, AutoTokenizer, AutoConfig
24
25
# Specific model classes
26
from transformers import BertModel, BertTokenizer
27
from transformers import GPT2LMHeadModel, GPT2Tokenizer
28
29
# Training utilities
30
from transformers import Trainer, TrainingArguments
31
32
# Feature extraction for audio/vision
33
from transformers import AutoFeatureExtractor, AutoImageProcessor
34
```
35
36
## Basic Usage
37
38
### Quick Start with Pipelines
39
40
```python
41
from transformers import pipeline
42
43
# Text classification
44
classifier = pipeline("text-classification")
45
results = classifier("I love using transformers!")
46
47
# Question answering
48
qa_pipeline = pipeline("question-answering")
49
answer = qa_pipeline(
50
question="What is transformers?",
51
context="Transformers is a library for natural language processing."
52
)
53
54
# Text generation
55
generator = pipeline("text-generation", model="gpt2")
56
output = generator("The future of AI is", max_length=50, num_return_sequences=1)
57
58
# Image classification
59
image_classifier = pipeline("image-classification")
60
results = image_classifier("path/to/image.jpg")
61
```
62
63
### Working with Models Directly
64
65
```python
66
from transformers import AutoModel, AutoTokenizer
67
68
# Load model and tokenizer
69
model_name = "bert-base-uncased"
70
model = AutoModel.from_pretrained(model_name)
71
tokenizer = AutoTokenizer.from_pretrained(model_name)
72
73
# Encode text
74
text = "Hello, world!"
75
inputs = tokenizer(text, return_tensors="pt")
76
77
# Forward pass
78
outputs = model(**inputs)
79
last_hidden_states = outputs.last_hidden_state
80
```
81
82
### Training a Model
83
84
```python
85
from transformers import Trainer, TrainingArguments
86
from transformers import AutoModelForSequenceClassification, AutoTokenizer
87
88
# Load model for fine-tuning
89
model = AutoModelForSequenceClassification.from_pretrained(
90
"bert-base-uncased",
91
num_labels=2
92
)
93
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
94
95
# Configure training
96
training_args = TrainingArguments(
97
output_dir="./results",
98
num_train_epochs=3,
99
per_device_train_batch_size=16,
100
per_device_eval_batch_size=64,
101
warmup_steps=500,
102
weight_decay=0.01,
103
logging_dir="./logs",
104
)
105
106
# Initialize trainer
107
trainer = Trainer(
108
model=model,
109
args=training_args,
110
train_dataset=train_dataset,
111
eval_dataset=eval_dataset,
112
)
113
114
# Start training
115
trainer.train()
116
```
117
118
## Architecture
119
120
The transformers library is built around several key architectural components:
121
122
- **Auto Classes**: Automatically select the correct model, tokenizer, or configuration based on a model name or path
123
- **Model Classes**: Implement specific architectures (BERT, GPT, T5, etc.) with consistent APIs across frameworks
124
- **Tokenizers**: Convert text to tokens and back, handling different tokenization strategies and vocabularies
125
- **Pipelines**: High-level abstraction providing simple interfaces for common ML tasks
126
- **Trainer**: Comprehensive training framework with built-in optimization, logging, and evaluation
127
- **Hub Integration**: Seamless downloading, caching, and sharing of models via Hugging Face Hub
128
129
This design enables transformers to serve as the foundational layer for the AI/ML ecosystem, providing consistent interfaces across 350+ model architectures while maintaining compatibility with PyTorch, TensorFlow, and JAX.
130
131
## Capabilities
132
133
### High-Level Pipeline API
134
135
Simple, task-oriented interface for common ML operations. Pipelines abstract away model selection, preprocessing, and postprocessing, providing immediate access to state-of-the-art capabilities.
136
137
```python { .api }
138
def pipeline(
139
task: str = None,
140
model: str = None,
141
tokenizer: str = None,
142
**kwargs
143
) -> Pipeline
144
```
145
146
[Pipelines](./pipelines.md)
147
148
### Model Management
149
150
Automatic model selection and loading with support for 350+ architectures. Auto classes intelligently choose the correct implementation based on model names or configurations.
151
152
```python { .api }
153
class AutoModel:
154
@classmethod
155
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PreTrainedModel
156
157
class AutoTokenizer:
158
@classmethod
159
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PreTrainedTokenizer
160
161
class AutoConfig:
162
@classmethod
163
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PretrainedConfig
164
```
165
166
[Models](./models.md)
167
168
### Training and Fine-tuning
169
170
Comprehensive training framework with built-in optimization, distributed training support, and extensive customization options.
171
172
```python { .api }
173
class Trainer:
174
def __init__(
175
self,
176
model: PreTrainedModel,
177
args: TrainingArguments,
178
train_dataset = None,
179
eval_dataset = None,
180
**kwargs
181
)
182
183
def train(self) -> None
184
def evaluate(self) -> Dict[str, float]
185
def predict(self, test_dataset) -> PredictionOutput
186
187
class TrainingArguments:
188
def __init__(
189
self,
190
output_dir: str,
191
num_train_epochs: float = 3.0,
192
per_device_train_batch_size: int = 8,
193
learning_rate: float = 5e-5,
194
**kwargs
195
)
196
```
197
198
[Training](./training.md)
199
200
### Text Generation
201
202
Advanced text generation capabilities with multiple decoding strategies, fine-grained control over output, and support for conversational AI.
203
204
```python { .api }
205
class GenerationMixin:
206
def generate(
207
self,
208
inputs = None,
209
max_length: int = None,
210
num_beams: int = 1,
211
temperature: float = 1.0,
212
do_sample: bool = False,
213
**kwargs
214
) -> torch.Tensor
215
216
class GenerationConfig:
217
def __init__(
218
self,
219
max_length: int = 20,
220
num_beams: int = 1,
221
temperature: float = 1.0,
222
**kwargs
223
)
224
```
225
226
[Generation](./generation.md)
227
228
### Tokenization
229
230
Comprehensive tokenization with support for 100+ different tokenizers, handling subword tokenization, special tokens, and efficient batch processing.
231
232
```python { .api }
233
class PreTrainedTokenizer:
234
def encode(
235
self,
236
text: str,
237
add_special_tokens: bool = True,
238
**kwargs
239
) -> List[int]
240
241
def decode(
242
self,
243
token_ids: List[int],
244
skip_special_tokens: bool = False
245
) -> str
246
247
def __call__(
248
self,
249
text,
250
return_tensors: str = None,
251
padding: bool = False,
252
truncation: bool = False,
253
**kwargs
254
) -> BatchEncoding
255
```
256
257
[Tokenization](./tokenization.md)
258
259
### Feature Extraction
260
261
Audio and image preprocessing capabilities for multimodal models, providing consistent interfaces for different modalities.
262
263
```python { .api }
264
class AutoFeatureExtractor:
265
@classmethod
266
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
267
268
class AutoImageProcessor:
269
@classmethod
270
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
271
```
272
273
[Feature Extraction](./feature-extraction.md)
274
275
### Model Optimization
276
277
Advanced optimization techniques including quantization, mixed precision training, and hardware acceleration for efficient inference and training.
278
279
```python { .api }
280
class BitsAndBytesConfig:
281
def __init__(
282
self,
283
load_in_8bit: bool = False,
284
load_in_4bit: bool = False,
285
bnb_4bit_compute_dtype = None,
286
**kwargs
287
)
288
```
289
290
[Optimization](./optimization.md)
291
292
## Types
293
294
Core type definitions used throughout the library:
295
296
```python { .api }
297
class PreTrainedModel:
298
"""Base class for all model implementations."""
299
def forward(self, **kwargs)
300
def save_pretrained(self, save_directory: str, **kwargs)
301
@classmethod
302
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
303
304
class PretrainedConfig:
305
"""Base configuration class for all models."""
306
def save_pretrained(self, save_directory: str, **kwargs)
307
@classmethod
308
def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
309
310
class BatchEncoding:
311
"""Container for tokenizer outputs with tensor conversion capabilities."""
312
input_ids: List[List[int]]
313
attention_mask: List[List[int]]
314
def to(self, device: str) -> 'BatchEncoding'
315
316
class Pipeline:
317
"""Base class for all pipeline implementations."""
318
def __call__(self, inputs, **kwargs)
319
def save_pretrained(self, save_directory: str, **kwargs)
320
321
class ModelOutput:
322
"""Base class for all model outputs."""
323
last_hidden_state: torch.Tensor
324
hidden_states: Tuple[torch.Tensor]
325
attentions: Tuple[torch.Tensor]
326
```