0
# Transformers
1

2
State-of-the-art Machine Learning library for JAX, PyTorch and TensorFlow. Transformers provides a unified API for working with over 350 pre-trained models across natural language processing, computer vision, audio, and multimodal tasks. The library democratizes access to cutting-edge AI models with simple, efficient interfaces for both inference and training.
3

4
## Package Information
5

6
- **Package Name**: transformers
7
- **Language**: Python
8
- **Installation**: `pip install transformers`
9

10
## Core Imports
11

12
```python
13
import transformers
14
```
15

16
Common patterns for specific functionality:
17

18
```python
19
# High-level Pipeline API (recommended for most use cases)
20
from transformers import pipeline
21

22
# Auto classes for automatic model/tokenizer selection
23
from transformers import AutoModel, AutoTokenizer, AutoConfig
24

25
# Specific model classes
26
from transformers import BertModel, BertTokenizer
27
from transformers import GPT2LMHeadModel, GPT2Tokenizer
28

29
# Training utilities
30
from transformers import Trainer, TrainingArguments
31

32
# Feature extraction for audio/vision
33
from transformers import AutoFeatureExtractor, AutoImageProcessor
34
```
35

36
## Basic Usage
37

38
### Quick Start with Pipelines
39

40
```python
41
from transformers import pipeline
42

43
# Text classification
44
classifier = pipeline("text-classification")
45
results = classifier("I love using transformers!")
46

47
# Question answering
48
qa_pipeline = pipeline("question-answering")
49
answer = qa_pipeline(
50
    question="What is transformers?",
51
    context="Transformers is a library for natural language processing."
52
)
53

54
# Text generation
55
generator = pipeline("text-generation", model="gpt2")
56
output = generator("The future of AI is", max_length=50, num_return_sequences=1)
57

58
# Image classification
59
image_classifier = pipeline("image-classification")
60
results = image_classifier("path/to/image.jpg")
61
```
62

63
### Working with Models Directly
64

65
```python
66
from transformers import AutoModel, AutoTokenizer
67

68
# Load model and tokenizer
69
model_name = "bert-base-uncased"
70
model = AutoModel.from_pretrained(model_name)
71
tokenizer = AutoTokenizer.from_pretrained(model_name)
72

73
# Encode text
74
text = "Hello, world!"
75
inputs = tokenizer(text, return_tensors="pt")
76

77
# Forward pass
78
outputs = model(**inputs)
79
last_hidden_states = outputs.last_hidden_state
80
```
81

82
### Training a Model
83

84
```python
85
from transformers import Trainer, TrainingArguments
86
from transformers import AutoModelForSequenceClassification, AutoTokenizer
87

88
# Load model for fine-tuning
89
model = AutoModelForSequenceClassification.from_pretrained(
90
    "bert-base-uncased", 
91
    num_labels=2
92
)
93
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
94

95
# Configure training
96
training_args = TrainingArguments(
97
    output_dir="./results",
98
    num_train_epochs=3,
99
    per_device_train_batch_size=16,
100
    per_device_eval_batch_size=64,
101
    warmup_steps=500,
102
    weight_decay=0.01,
103
    logging_dir="./logs",
104
)
105

106
# Initialize trainer
107
trainer = Trainer(
108
    model=model,
109
    args=training_args,
110
    train_dataset=train_dataset,
111
    eval_dataset=eval_dataset,
112
)
113

114
# Start training
115
trainer.train()
116
```
117

118
## Architecture
119

120
The transformers library is built around several key architectural components:
121

122
- **Auto Classes**: Automatically select the correct model, tokenizer, or configuration based on a model name or path
123
- **Model Classes**: Implement specific architectures (BERT, GPT, T5, etc.) with consistent APIs across frameworks
124
- **Tokenizers**: Convert text to tokens and back, handling different tokenization strategies and vocabularies
125
- **Pipelines**: High-level abstraction providing simple interfaces for common ML tasks
126
- **Trainer**: Comprehensive training framework with built-in optimization, logging, and evaluation
127
- **Hub Integration**: Seamless downloading, caching, and sharing of models via Hugging Face Hub
128

129
This design enables transformers to serve as the foundational layer for the AI/ML ecosystem, providing consistent interfaces across 350+ model architectures while maintaining compatibility with PyTorch, TensorFlow, and JAX.
130

131
## Capabilities
132

133
### High-Level Pipeline API
134

135
Simple, task-oriented interface for common ML operations. Pipelines abstract away model selection, preprocessing, and postprocessing, providing immediate access to state-of-the-art capabilities.
136

137
```python { .api }
138
def pipeline(
139
    task: str = None,
140
    model: str = None,
141
    tokenizer: str = None,
142
    **kwargs
143
) -> Pipeline
144
```
145

146
[Pipelines](./pipelines.md)
147

148
### Model Management
149

150
Automatic model selection and loading with support for 350+ architectures. Auto classes intelligently choose the correct implementation based on model names or configurations.
151

152
```python { .api }
153
class AutoModel:
154
    @classmethod
155
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PreTrainedModel
156

157
class AutoTokenizer:
158
    @classmethod 
159
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PreTrainedTokenizer
160

161
class AutoConfig:
162
    @classmethod
163
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs) -> PretrainedConfig
164
```
165

166
[Models](./models.md)
167

168
### Training and Fine-tuning
169

170
Comprehensive training framework with built-in optimization, distributed training support, and extensive customization options.
171

172
```python { .api }
173
class Trainer:
174
    def __init__(
175
        self,
176
        model: PreTrainedModel,
177
        args: TrainingArguments,
178
        train_dataset = None,
179
        eval_dataset = None,
180
        **kwargs
181
    )
182
    
183
    def train(self) -> None
184
    def evaluate(self) -> Dict[str, float]
185
    def predict(self, test_dataset) -> PredictionOutput
186

187
class TrainingArguments:
188
    def __init__(
189
        self,
190
        output_dir: str,
191
        num_train_epochs: float = 3.0,
192
        per_device_train_batch_size: int = 8,
193
        learning_rate: float = 5e-5,
194
        **kwargs
195
    )
196
```
197

198
[Training](./training.md)
199

200
### Text Generation
201

202
Advanced text generation capabilities with multiple decoding strategies, fine-grained control over output, and support for conversational AI.
203

204
```python { .api }
205
class GenerationMixin:
206
    def generate(
207
        self,
208
        inputs = None,
209
        max_length: int = None,
210
        num_beams: int = 1,
211
        temperature: float = 1.0,
212
        do_sample: bool = False,
213
        **kwargs
214
    ) -> torch.Tensor
215

216
class GenerationConfig:
217
    def __init__(
218
        self,
219
        max_length: int = 20,
220
        num_beams: int = 1,
221
        temperature: float = 1.0,
222
        **kwargs
223
    )
224
```
225

226
[Generation](./generation.md)
227

228
### Tokenization
229

230
Comprehensive tokenization with support for 100+ different tokenizers, handling subword tokenization, special tokens, and efficient batch processing.
231

232
```python { .api }
233
class PreTrainedTokenizer:
234
    def encode(
235
        self, 
236
        text: str,
237
        add_special_tokens: bool = True,
238
        **kwargs
239
    ) -> List[int]
240
    
241
    def decode(
242
        self,
243
        token_ids: List[int],
244
        skip_special_tokens: bool = False
245
    ) -> str
246
    
247
    def __call__(
248
        self,
249
        text,
250
        return_tensors: str = None,
251
        padding: bool = False,
252
        truncation: bool = False,
253
        **kwargs
254
    ) -> BatchEncoding
255
```
256

257
[Tokenization](./tokenization.md)
258

259
### Feature Extraction
260

261
Audio and image preprocessing capabilities for multimodal models, providing consistent interfaces for different modalities.
262

263
```python { .api }
264
class AutoFeatureExtractor:
265
    @classmethod
266
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
267

268
class AutoImageProcessor:
269
    @classmethod
270
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
271
```
272

273
[Feature Extraction](./feature-extraction.md)
274

275
### Model Optimization
276

277
Advanced optimization techniques including quantization, mixed precision training, and hardware acceleration for efficient inference and training.
278

279
```python { .api }
280
class BitsAndBytesConfig:
281
    def __init__(
282
        self,
283
        load_in_8bit: bool = False,
284
        load_in_4bit: bool = False,
285
        bnb_4bit_compute_dtype = None,
286
        **kwargs
287
    )
288
```
289

290
[Optimization](./optimization.md)
291

292
## Types
293

294
Core type definitions used throughout the library:
295

296
```python { .api }
297
class PreTrainedModel:
298
    """Base class for all model implementations."""
299
    def forward(self, **kwargs)
300
    def save_pretrained(self, save_directory: str, **kwargs)
301
    @classmethod
302
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
303

304
class PretrainedConfig:
305
    """Base configuration class for all models."""
306
    def save_pretrained(self, save_directory: str, **kwargs)
307
    @classmethod
308
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs)
309

310
class BatchEncoding:
311
    """Container for tokenizer outputs with tensor conversion capabilities."""
312
    input_ids: List[List[int]]
313
    attention_mask: List[List[int]]
314
    def to(self, device: str) -> 'BatchEncoding'
315

316
class Pipeline:
317
    """Base class for all pipeline implementations."""
318
    def __call__(self, inputs, **kwargs)
319
    def save_pretrained(self, save_directory: str, **kwargs)
320

321
class ModelOutput:
322
    """Base class for all model outputs."""
323
    last_hidden_state: torch.Tensor
324
    hidden_states: Tuple[torch.Tensor]
325
    attentions: Tuple[torch.Tensor]
326
```