0
# Advanced Methods
1

2
Specialized parameter-efficient fine-tuning methods that use orthogonal transformations, structured adaptations, and experimental approaches. These methods often provide unique advantages for specific architectures or training scenarios.
3

4
## Capabilities
5

6
### Orthogonal Fine-Tuning (OFT)
7

8
Maintains hyperspherical energy by applying orthogonal transformations to preserve geometric properties of pretrained features.
9

10
```python { .api }
11
class OFTConfig(PeftConfig):
12
    """Configuration for OFT (Orthogonal Fine-tuning)."""
13
    def __init__(
14
        self,
15
        r: int = 8,
16
        module_dropout: float = 0.0,
17
        target_modules: Optional[Union[List[str], str]] = None,
18
        init_weights: bool = True,
19
        layers_to_transform: Optional[Union[List[int], int]] = None,
20
        layers_pattern: Optional[str] = None,
21
        modules_to_save: Optional[List[str]] = None,
22
        coft: bool = False,
23
        eps: float = 6e-5,
24
        block_share: bool = False,
25
        **kwargs
26
    ):
27
        """
28
        Args:
29
            r: OFT rank (constraint dimension)
30
            module_dropout: Module dropout probability
31
            target_modules: Names of modules to apply OFT to
32
            init_weights: Whether to initialize weights
33
            layers_to_transform: Layers to apply OFT to
34
            layers_pattern: Pattern for layer names
35
            modules_to_save: Modules to set as trainable and save
36
            coft: Whether to use constrained OFT
37
            eps: Epsilon for numerical stability
38
            block_share: Whether to share blocks across layers
39
        """
40

41
class OFTModel:
42
    """OFT model implementation."""
43
    def __init__(self, model, config: OFTConfig, adapter_name: str): ...
44
```
45

46
### Butterfly OFT (BOFT)
47

48
Extends OFT with butterfly factorization for improved parameter efficiency and structured transformations.
49

50
```python { .api }
51
class BOFTConfig(PeftConfig):
52
    """Configuration for BOFT (Butterfly OFT)."""
53
    def __init__(
54
        self,
55
        boft_block_size: int = 4,
56
        boft_block_num: int = 0,
57
        boft_n_butterfly_factor: int = 1,
58
        target_modules: Optional[Union[List[str], str]] = None,
59
        boft_dropout: float = 0.0,
60
        fan_in_fan_out: bool = False,
61
        init_weights: bool = True,
62
        layers_to_transform: Optional[Union[List[int], int]] = None,
63
        layers_pattern: Optional[str] = None,
64
        modules_to_save: Optional[List[str]] = None,
65
        **kwargs
66
    ):
67
        """
68
        Args:
69
            boft_block_size: Size of butterfly blocks
70
            boft_block_num: Number of butterfly blocks
71
            boft_n_butterfly_factor: Number of butterfly factors
72
            target_modules: Names of modules to apply BOFT to
73
            boft_dropout: BOFT dropout probability
74
            fan_in_fan_out: Whether layer stores weights as (fan_in, fan_out)
75
            init_weights: Whether to initialize weights
76
            layers_to_transform: Layers to apply BOFT to
77
            layers_pattern: Pattern for layer names
78
            modules_to_save: Modules to set as trainable and save
79
        """
80

81
class BOFTModel:
82
    """BOFT model implementation."""
83
    def __init__(self, model, config: BOFTConfig, adapter_name: str): ...
84
```
85

86
### IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations)
87

88
Learns vectors that rescale inner activations, providing a lightweight adaptation mechanism.
89

90
```python { .api }
91
class IA3Config(PeftConfig):
92
    """Configuration for IA3."""
93
    def __init__(
94
        self,
95
        target_modules: Optional[Union[List[str], str]] = None,
96
        feedforward_modules: Optional[Union[List[str], str]] = None,
97
        fan_in_fan_out: bool = False,
98
        modules_to_save: Optional[List[str]] = None,
99
        init_ia3_weights: bool = True,
100
        **kwargs
101
    ): ...
102

103
class IA3Model:
104
    """IA3 model implementation."""
105
    def __init__(self, model, config: IA3Config, adapter_name: str): ...
106
```
107

108
### Vera (Vector-based Random Matrix Adaptation)
109

110
Adaptation method using vector-based random matrix decomposition for parameter-efficient fine-tuning.
111

112
```python { .api }
113
class VeraConfig(PeftConfig):
114
    """Configuration for Vera."""
115
    def __init__(
116
        self,
117
        r: int = 256,
118
        target_modules: Optional[Union[List[str], str]] = None,
119
        projection_prng_key: int = 0,
120
        vera_dropout: float = 0.0,
121
        d_initial: float = 0.1,
122
        **kwargs
123
    ): ...
124

125
class VeraModel:
126
    """Vera model implementation."""
127
    def __init__(self, model, config: VeraConfig, adapter_name: str): ...
128
```
129

130
### Poly (Polynomial Adaptation)
131

132
Polynomial-based adaptation method for efficient fine-tuning.
133

134
```python { .api }
135
class PolyConfig(PeftConfig):
136
    """Configuration for Poly."""
137
    def __init__(
138
        self,
139
        r: int = 8,
140
        target_modules: Optional[Union[List[str], str]] = None,
141
        modules_to_save: Optional[List[str]] = None,
142
        poly_type: str = "poly",
143
        n_tasks: int = 1,
144
        n_skills: int = 4,
145
        n_splits: int = 1,
146
        **kwargs
147
    ): ...
148

149
class PolyModel:
150
    """Poly model implementation."""
151
    def __init__(self, model, config: PolyConfig, adapter_name: str): ...
152
```
153

154
### LayerNorm Tuning
155

156
Fine-tuning method that targets LayerNorm parameters for efficient adaptation.
157

158
```python { .api }
159
class LNTuningConfig(PeftConfig):
160
    """Configuration for LayerNorm Tuning."""
161
    def __init__(
162
        self,
163
        target_modules: Optional[Union[List[str], str]] = None,
164
        modules_to_save: Optional[List[str]] = None,
165
        **kwargs
166
    ): ...
167

168
class LNTuningModel:
169
    """LNTuning model implementation."""
170
    def __init__(self, model, config: LNTuningConfig, adapter_name: str): ...
171
```
172

173
### FourierFT
174

175
Fourier Transform-based parameter efficient fine-tuning method.
176

177
```python { .api }
178
class FourierFTConfig(PeftConfig):
179
    """Configuration for FourierFT."""
180
    def __init__(
181
        self,
182
        n_frequency: int = 1000,
183
        scaling: float = 300.0,
184
        target_modules: Optional[Union[List[str], str]] = None,
185
        modules_to_save: Optional[List[str]] = None,
186
        **kwargs
187
    ): ...
188

189
class FourierFTModel:
190
    """FourierFT model implementation."""
191
    def __init__(self, model, config: FourierFTConfig, adapter_name: str): ...
192
```
193

194
### XLoRA (Mixture of LoRA Experts)
195

196
Mixture of LoRA experts with learned gating mechanisms for multi-task adaptation.
197

198
```python { .api }
199
class XLoraConfig(LoraConfig):
200
    """Configuration for XLoRA."""
201
    def __init__(
202
        self,
203
        hidden_size: int,
204
        xlora_depth: int = 1,
205
        xlora_size: int = 2048,
206
        enable_softmax: bool = True,
207
        softmax_temperature: float = 1.0,
208
        layernorm_type: str = "rms",
209
        use_bias: bool = False,
210
        xlora_dropout_p: float = 0.2,
211
        **kwargs
212
    ): ...
213

214
class XLoraModel:
215
    """XLoRA model implementation."""
216
    def __init__(self, model, config: XLoraConfig, adapter_name: str): ...
217
```
218

219
### BONE (Block-wise One-shot Neural Architecture Evolution)
220

221
Block-wise neural architecture evolution method for efficient model adaptation.
222

223
```python { .api }
224
class BoneConfig(PeftConfig):
225
    """Configuration for BONE."""
226
    def __init__(
227
        self,
228
        r: int = 8,
229
        target_modules: Optional[Union[List[str], str]] = None,
230
        modules_to_save: Optional[List[str]] = None,
231
        **kwargs
232
    ): ...
233

234
class BoneModel:
235
    """BONE model implementation."""
236
    def __init__(self, model, config: BoneConfig, adapter_name: str): ...
237
```
238

239
### VB-LoRA (Variational Bayesian LoRA)
240

241
Variational Bayesian extension of LoRA for uncertainty quantification.
242

243
```python { .api }
244
class VBLoRAConfig(LoraConfig):
245
    """Configuration for VB-LoRA."""
246
    def __init__(
247
        self,
248
        vb_r: int = 8,
249
        vb_pi: float = 0.5,
250
        vb_beta: float = 1.0,
251
        **kwargs
252
    ): ...
253

254
class VBLoRAModel:
255
    """VB-LoRA model implementation."""
256
    def __init__(self, model, config: VBLoRAConfig, adapter_name: str): ...
257
```
258

259
### Additional Advanced Methods
260

261
The following methods are also available in PEFT for specialized use cases:
262

263
```python { .api }
264
# HRA (High-Rank Adaptation)
265
class HRAConfig(PeftConfig): ...
266
class HRAModel: ...
267

268
# MISS (Mixture of Integrated Sparse Structures)  
269
class MissConfig(PeftConfig): ...
270
class MissModel: ...
271

272
# RandLoRA (Random LoRA)
273
class RandLoraConfig(PeftConfig): ...
274
class RandLoraModel: ...
275

276
# SHIRA (Shared Representations Adaptation)
277
class ShiraConfig(PeftConfig): ...
278
class ShiraModel: ...
279

280
# C3A (Cascaded Cross-domain Continual Adaptation)
281
class C3AConfig(PeftConfig): ...
282
class C3AModel: ...
283

284
# CPT (Continuous Prompt Tuning)
285
class CPTConfig(PeftConfig): ...
286

287
# AdaptionPrompt (Adaption Prompt)
288
class AdaptionPromptConfig(PeftConfig): ...
289
class AdaptionPromptModel: ...
290

291
# Trainable Tokens
292
class TrainableTokensConfig(PeftConfig): ...
293
class TrainableTokensModel: ...
294
```
295

296
## Usage Examples
297

298
### OFT for Preserving Feature Geometry
299

300
```python
301
from transformers import AutoModelForCausalLM
302
from peft import get_peft_model, OFTConfig
303

304
model = AutoModelForCausalLM.from_pretrained("gpt2")
305

306
oft_config = OFTConfig(
307
    r=8,
308
    target_modules=["c_attn", "c_proj"],
309
    module_dropout=0.05,
310
    task_type="CAUSAL_LM"
311
)
312

313
peft_model = get_peft_model(model, oft_config)
314
```
315

316
### IA3 for Lightweight Adaptation
317

318
```python
319
from peft import IA3Config
320

321
ia3_config = IA3Config(
322
    target_modules=["k_proj", "v_proj", "down_proj"],
323
    feedforward_modules=["down_proj"],
324
    task_type="CAUSAL_LM"
325
)
326

327
peft_model = get_peft_model(model, ia3_config)
328
```
329

330
### Vera with Random Projections
331

332
```python
333
from peft import VeraConfig
334

335
vera_config = VeraConfig(
336
    r=256,
337
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
338
    projection_prng_key=42,
339
    vera_dropout=0.1,
340
    d_initial=0.1,
341
    task_type="CAUSAL_LM"
342
)
343

344
peft_model = get_peft_model(model, vera_config)
345
```
346

347
### Layer Normalization Tuning
348

349
```python
350
from peft import LNTuningConfig
351

352
ln_config = LNTuningConfig(
353
    target_modules=["input_layernorm", "post_attention_layernorm"],
354
    task_type="CAUSAL_LM"
355
)
356

357
peft_model = get_peft_model(model, ln_config)
358
```
359

360
### X-LoRA Mixture of Experts
361

362
```python
363
from peft import XLoraConfig
364

365
xlora_config = XLoraConfig(
366
    hidden_size=768,
367
    xlora_size=64,
368
    xlora_depth=2,
369
    layernorm_type="rms",
370
    xlora_dropout_p=0.1,
371
    task_type="CAUSAL_LM"
372
)
373

374
# Load multiple LoRA adapters first, then apply X-LoRA
375
peft_model = get_peft_model(model, xlora_config)
376
```