0
# PEFT (Parameter-Efficient Fine-Tuning)
1

2
A comprehensive library for efficiently adapting large pretrained models using state-of-the-art parameter-efficient fine-tuning methods. PEFT enables fine-tuning of only a small number of model parameters instead of all parameters, significantly reducing computational and storage costs while achieving performance comparable to fully fine-tuned models.
3

4
## Package Information
5

6
- **Package Name**: peft
7
- **Language**: Python
8
- **Installation**: `pip install peft`
9
- **Requirements**: Python >=3.9.0
10

11
## Core Imports
12

13
```python
14
import peft
15
```
16

17
Common imports for PEFT models:
18

19
```python
20
from peft import get_peft_model, PeftConfig, PeftModel
21
```
22

23
Specific PEFT method imports:
24

25
```python
26
from peft import LoraConfig, AdaLoraConfig, IA3Config
27
```
28

29
Auto model imports:
30

31
```python
32
from peft import AutoPeftModel, AutoPeftModelForCausalLM
33
```
34

35
## Basic Usage
36

37
```python
38
import torch
39
from transformers import AutoModelForCausalLM, AutoTokenizer
40
from peft import get_peft_model, LoraConfig
41

42
# Load base model and tokenizer
43
model_name = "microsoft/DialoGPT-medium"
44
model = AutoModelForCausalLM.from_pretrained(model_name)
45
tokenizer = AutoTokenizer.from_pretrained(model_name)
46

47
# Create PEFT configuration
48
peft_config = LoraConfig(
49
    task_type="CAUSAL_LM",
50
    inference_mode=False,
51
    r=8,
52
    lora_alpha=32,
53
    lora_dropout=0.1
54
)
55

56
# Get PEFT model
57
peft_model = get_peft_model(model, peft_config)
58

59
# Print trainable parameters
60
peft_model.print_trainable_parameters()
61

62
# Train the model (with your training loop)
63
# ...
64

65
# Save the PEFT adapter
66
peft_model.save_pretrained("./peft-adapter")
67

68
# Load PEFT model later
69
from peft import AutoPeftModelForCausalLM
70
peft_model = AutoPeftModelForCausalLM.from_pretrained("./peft-adapter")
71
```
72

73
## Architecture
74

75
PEFT follows a modular architecture that separates configuration, model wrapping, and method-specific implementations:
76

77
- **Core Models**: `PeftModel` and task-specific variants wrap base transformers models
78
- **Configurations**: Each PEFT method has a dedicated config class (e.g., `LoraConfig`)
79
- **Tuners**: Method-specific implementations in the `tuners` module
80
- **Auto Classes**: Automatic model selection based on task type and saved configurations
81
- **Mixed Models**: Support for combining multiple PEFT methods
82

83
This design allows seamless integration with Hugging Face Transformers while providing extensibility for custom PEFT methods.
84

85
## Capabilities
86

87
### Core Model Functions
88

89
Primary functions for creating and managing PEFT models, including the main entry point and core model classes.
90

91
```python { .api }
92
def get_peft_model(
93
    model: PreTrainedModel,
94
    peft_config: PeftConfig,
95
    adapter_name: str = "default",
96
    mixed: bool = False,
97
    autocast_adapter_dtype: bool = True,
98
    revision: Optional[str] = None,
99
    low_cpu_mem_usage: bool = False
100
) -> PeftModel | PeftMixedModel: ...
101
```
102

103
[Core Models](./core-models.md)
104

105
### LoRA Methods
106

107
Low-Rank Adaptation methods including standard LoRA, AdaLoRA, and variants like LoHa, LoKr, and specialized configurations.
108

109
```python { .api }
110
class LoraConfig(PeftConfig):
111
    def __init__(
112
        self,
113
        r: int = 8,
114
        lora_alpha: int = 8,
115
        target_modules: Optional[Union[List[str], str]] = None,
116
        lora_dropout: float = 0.0,
117
        bias: str = "none",
118
        task_type: str = None,
119
        **kwargs
120
    ): ...
121

122
class AdaLoraConfig(PeftConfig):
123
    def __init__(
124
        self,
125
        target_r: int = 8,
126
        init_r: int = 12,
127
        tinit: int = 0,
128
        tfinal: int = 0,
129
        deltaT: int = 1,
130
        **kwargs
131
    ): ...
132
```
133

134
[LoRA Methods](./lora-methods.md)
135

136
### Prompt Learning Methods
137

138
Prompt-based tuning methods including Prompt Tuning, P-tuning, Prefix Tuning, and Multitask Prompt Tuning.
139

140
```python { .api }
141
class PromptTuningConfig(PromptLearningConfig):
142
    def __init__(
143
        self,
144
        num_virtual_tokens: int,
145
        prompt_tuning_init: PromptTuningInit = PromptTuningInit.RANDOM,
146
        prompt_tuning_init_text: Optional[str] = None,
147
        **kwargs
148
    ): ...
149

150
class PrefixTuningConfig(PromptLearningConfig):
151
    def __init__(
152
        self,
153
        num_virtual_tokens: int,
154
        encoder_hidden_size: int,
155
        prefix_projection: bool = False,
156
        **kwargs
157
    ): ...
158
```
159

160
[Prompt Learning](./prompt-learning.md)
161

162
### Advanced Methods
163

164
Specialized PEFT methods including orthogonal fine-tuning (OFT, BOFT), structured methods (IA3, Vera), and experimental approaches.
165

166
```python { .api }
167
class OFTConfig(PeftConfig):
168
    def __init__(
169
        self,
170
        r: int = 8,
171
        module_dropout: float = 0.0,
172
        target_modules: Optional[Union[List[str], str]] = None,
173
        **kwargs
174
    ): ...
175

176
class IA3Config(PeftConfig):
177
    def __init__(
178
        self,
179
        target_modules: Optional[Union[List[str], str]] = None,
180
        feedforward_modules: Optional[Union[List[str], str]] = None,
181
        **kwargs
182
    ): ...
183
```
184

185
[Advanced Methods](./advanced-methods.md)
186

187
### Auto Classes
188

189
Automatic model loading and task-specific PEFT model classes for different NLP tasks.
190

191
```python { .api }
192
class AutoPeftModel:
193
    @classmethod
194
    def from_pretrained(
195
        cls,
196
        pretrained_model_name_or_path,
197
        adapter_name: str = "default",
198
        is_trainable: bool = False,
199
        config: Optional[PeftConfig] = None,
200
        revision: Optional[str] = None,
201
        **kwargs
202
    ): ...
203

204
class AutoPeftModelForCausalLM(AutoPeftModel): ...
205
class AutoPeftModelForSequenceClassification(AutoPeftModel): ...
206
class AutoPeftModelForFeatureExtraction(AutoPeftModel): ...
207
class AutoPeftModelForQuestionAnswering(AutoPeftModel): ...
208
class AutoPeftModelForSeq2SeqLM(AutoPeftModel): ...
209
class AutoPeftModelForTokenClassification(AutoPeftModel): ...
210
```
211

212
[Auto Classes](./auto-classes.md)
213

214
### Utilities and State Management
215

216
Functions for managing model state, loading/saving adapters, and preparing models for training.
217

218
```python { .api }
219
def get_peft_model_state_dict(
220
    model,
221
    state_dict: Optional[dict] = None,
222
    adapter_name: str = "default"
223
) -> dict: ...
224

225
def set_peft_model_state_dict(
226
    model,
227
    peft_model_state_dict: dict,
228
    adapter_name: str = "default"
229
): ...
230

231
def load_peft_weights(model_id: str, device: Optional[str] = None) -> dict: ...
232

233
def prepare_model_for_kbit_training(
234
    model,
235
    use_gradient_checkpointing: bool = True,
236
    gradient_checkpointing_kwargs: Optional[dict] = None
237
): ...
238

239
def get_peft_config(config_dict: dict) -> PeftConfig: ...
240

241
def inject_adapter_in_model(
242
    peft_config: PeftConfig,
243
    model,
244
    adapter_name: str = "default"
245
): ...
246

247
def replace_lora_weights_loftq(
248
    model,
249
    loftq_config: LoftQConfig,
250
    adapter_name: str = "default"
251
): ...
252

253
def register_peft_method(
254
    name: str,
255
    config_cls,
256
    model_cls,
257
    prefix: Optional[str] = None,
258
    is_mixed_compatible: bool = False
259
) -> None: ...
260
```
261

262
[Utilities](./utilities.md)
263

264
## Types and Enums
265

266
Core types and enumerations used throughout the PEFT library.
267

268
```python { .api }
269
class PeftType(str, Enum):
270
    PROMPT_TUNING = "PROMPT_TUNING"
271
    MULTITASK_PROMPT_TUNING = "MULTITASK_PROMPT_TUNING"
272
    P_TUNING = "P_TUNING"
273
    PREFIX_TUNING = "PREFIX_TUNING"
274
    LORA = "LORA"
275
    ADALORA = "ADALORA"
276
    BOFT = "BOFT"
277
    ADAPTION_PROMPT = "ADAPTION_PROMPT"
278
    IA3 = "IA3"
279
    LOHA = "LOHA"
280
    LOKR = "LOKR"
281
    OFT = "OFT"
282
    POLY = "POLY"
283
    LN_TUNING = "LN_TUNING"
284
    VERA = "VERA"
285
    FOURIERFT = "FOURIERFT"
286
    XLORA = "XLORA"
287
    HRA = "HRA"
288
    VBLORA = "VBLORA"
289
    CPT = "CPT"
290
    BONE = "BONE"
291
    MISS = "MISS"
292
    RANDLORA = "RANDLORA"
293
    TRAINABLE_TOKENS = "TRAINABLE_TOKENS"
294
    SHIRA = "SHIRA"
295
    C3A = "C3A"
296

297
class TaskType(str, Enum):
298
    SEQ_CLS = "SEQ_CLS"
299
    SEQ_2_SEQ_LM = "SEQ_2_SEQ_LM"
300
    CAUSAL_LM = "CAUSAL_LM"
301
    TOKEN_CLS = "TOKEN_CLS"
302
    QUESTION_ANS = "QUESTION_ANS"
303
    FEATURE_EXTRACTION = "FEATURE_EXTRACTION"
304

305
class PeftConfig:
306
    """Base configuration class for all PEFT methods."""
307
    def __init__(
308
        self,
309
        peft_type: Optional[PeftType] = None,
310
        task_type: Optional[TaskType] = None,
311
        inference_mode: bool = False,
312
        r: int = 8,
313
        target_modules: Optional[Union[List[str], str]] = None,
314
        modules_to_save: Optional[List[str]] = None,
315
        **kwargs
316
    ): ...
317
```