0
# Prompt Learning Methods
1
2
Prompt-based parameter-efficient fine-tuning methods that learn soft prompts or prefix tokens to guide model behavior without modifying the base model weights. These methods are particularly effective for few-shot learning and task adaptation.
3
4
## Capabilities
5
6
### Prompt Tuning
7
8
Learns soft prompt tokens that are prepended to input sequences, allowing task-specific adaptation through prompt optimization.
9
10
```python { .api }
11
class PromptTuningConfig(PromptLearningConfig):
12
"""Configuration for Prompt Tuning."""
13
def __init__(
14
self,
15
num_virtual_tokens: int,
16
prompt_tuning_init: PromptTuningInit = PromptTuningInit.RANDOM,
17
prompt_tuning_init_text: Optional[str] = None,
18
prompt_tuning_init_state_dict_path: Optional[str] = None,
19
num_transformer_submodules: Optional[int] = None,
20
num_attention_heads: Optional[int] = None,
21
num_layers: Optional[int] = None,
22
token_dim: Optional[int] = None,
23
num_text_tokens: Optional[int] = None,
24
**kwargs
25
):
26
"""
27
Args:
28
num_virtual_tokens: Number of virtual tokens
29
prompt_tuning_init: Initialization method for prompt tuning
30
prompt_tuning_init_text: Text to initialize prompt (if using TEXT init)
31
prompt_tuning_init_state_dict_path: Path to state dict for initialization
32
num_transformer_submodules: Number of transformer submodules
33
num_attention_heads: Number of attention heads
34
num_layers: Number of transformer layers
35
token_dim: Token embedding dimension
36
num_text_tokens: Number of text tokens for initialization
37
"""
38
39
class PromptTuningInit(str, Enum):
40
"""Initialization methods for prompt tuning."""
41
RANDOM = "RANDOM"
42
TEXT = "TEXT"
43
44
class PromptEmbedding(torch.nn.Module):
45
"""Embedding layer for learnable prompt tokens."""
46
def __init__(self, config: PromptTuningConfig, word_embeddings): ...
47
48
def forward(self, indices): ...
49
```
50
51
### P-Tuning v2
52
53
Learns continuous prompt representations using a prompt encoder, providing more flexibility than standard prompt tuning.
54
55
```python { .api }
56
class PromptEncoderConfig(PromptLearningConfig):
57
"""Configuration for P-Tuning v2."""
58
def __init__(
59
self,
60
num_virtual_tokens: int,
61
encoder_reparameterization_type: PromptEncoderReparameterizationType = PromptEncoderReparameterizationType.MLP,
62
encoder_hidden_size: Optional[int] = None,
63
encoder_num_layers: int = 2,
64
encoder_dropout: float = 0.0,
65
**kwargs
66
):
67
"""
68
Args:
69
num_virtual_tokens: Number of virtual tokens
70
encoder_reparameterization_type: Type of encoder architecture
71
encoder_hidden_size: Hidden size of prompt encoder
72
encoder_num_layers: Number of layers in prompt encoder
73
encoder_dropout: Dropout probability in prompt encoder
74
"""
75
76
class PromptEncoderReparameterizationType(str, Enum):
77
"""Reparameterization types for prompt encoder."""
78
MLP = "MLP"
79
LSTM = "LSTM"
80
81
class PromptEncoder(torch.nn.Module):
82
"""Prompt encoder for P-Tuning v2."""
83
def __init__(self, config: PromptEncoderConfig): ...
84
85
def forward(self, indices): ...
86
```
87
88
### Prefix Tuning
89
90
Learns prefix tokens that are prepended to key and value vectors in attention layers, optimizing continuous task-specific vectors.
91
92
```python { .api }
93
class PrefixTuningConfig(PromptLearningConfig):
94
"""Configuration for Prefix Tuning."""
95
def __init__(
96
self,
97
num_virtual_tokens: int,
98
encoder_hidden_size: int,
99
prefix_projection: bool = False,
100
**kwargs
101
):
102
"""
103
Args:
104
num_virtual_tokens: Number of virtual tokens
105
encoder_hidden_size: Hidden size of the encoder
106
prefix_projection: Whether to project prefix parameters
107
"""
108
109
class PrefixEncoder(torch.nn.Module):
110
"""Prefix encoder for generating prefix embeddings."""
111
def __init__(self, config: PrefixTuningConfig): ...
112
113
def forward(self, prefix: torch.Tensor): ...
114
```
115
116
### Multitask Prompt Tuning
117
118
Extends prompt tuning to handle multiple tasks simultaneously with shared and task-specific prompt components.
119
120
```python { .api }
121
class MultitaskPromptTuningConfig(PromptLearningConfig):
122
"""Configuration for Multitask Prompt Tuning."""
123
def __init__(
124
self,
125
num_virtual_tokens: int,
126
prompt_tuning_init: MultitaskPromptTuningInit = MultitaskPromptTuningInit.RANDOM,
127
prompt_tuning_init_text: Optional[str] = None,
128
num_tasks: int = 1,
129
**kwargs
130
):
131
"""
132
Args:
133
num_virtual_tokens: Number of virtual tokens per task
134
prompt_tuning_init: Initialization method
135
prompt_tuning_init_text: Text for initialization
136
num_tasks: Number of tasks to support
137
"""
138
139
class MultitaskPromptTuningInit(str, Enum):
140
"""Initialization methods for multitask prompt tuning."""
141
RANDOM = "RANDOM"
142
TEXT = "TEXT"
143
144
class MultitaskPromptEmbedding(torch.nn.Module):
145
"""Embedding layer for multitask prompt tuning."""
146
def __init__(self, config: MultitaskPromptTuningConfig, word_embeddings): ...
147
148
def forward(self, indices, task_ids): ...
149
```
150
151
### Adaption Prompt
152
153
Specialized prompt learning method designed for vision-language models and multimodal tasks.
154
155
```python { .api }
156
class AdaptionPromptConfig(PeftConfig):
157
"""Configuration for Adaption Prompt."""
158
def __init__(
159
self,
160
target_modules: Optional[Union[List[str], str]] = None,
161
adapter_len: int = 10,
162
adapter_layers: int = 30,
163
**kwargs
164
):
165
"""
166
Args:
167
target_modules: Names of modules to apply adaption prompt to
168
adapter_len: Length of adapter prompts
169
adapter_layers: Number of layers to apply adapters to
170
"""
171
172
class AdaptionPromptModel:
173
"""Adaption prompt model implementation."""
174
def __init__(self, model, config: AdaptionPromptConfig, adapter_name: str): ...
175
```
176
177
### Base Prompt Learning Configuration
178
179
Base configuration class inherited by all prompt learning methods.
180
181
```python { .api }
182
class PromptLearningConfig(PeftConfig):
183
"""Base configuration for prompt learning methods."""
184
def __init__(
185
self,
186
num_virtual_tokens: int,
187
token_dim: Optional[int] = None,
188
num_transformer_submodules: Optional[int] = None,
189
num_attention_heads: Optional[int] = None,
190
num_layers: Optional[int] = None,
191
encoder_reparameterization_type: Optional[str] = None,
192
encoder_hidden_size: Optional[int] = None,
193
**kwargs
194
):
195
"""
196
Args:
197
num_virtual_tokens: Number of virtual tokens
198
token_dim: Dimension of token embeddings
199
num_transformer_submodules: Number of transformer submodules
200
num_attention_heads: Number of attention heads
201
num_layers: Number of layers
202
encoder_reparameterization_type: Type of encoder reparameterization
203
encoder_hidden_size: Hidden size of encoder
204
"""
205
206
@property
207
def is_prompt_learning(self) -> bool:
208
"""Returns True for prompt learning configs."""
209
return True
210
```
211
212
## Usage Examples
213
214
### Basic Prompt Tuning
215
216
```python
217
from transformers import AutoModelForCausalLM
218
from peft import get_peft_model, PromptTuningConfig, TaskType
219
220
model = AutoModelForCausalLM.from_pretrained("gpt2")
221
222
prompt_config = PromptTuningConfig(
223
task_type=TaskType.CAUSAL_LM,
224
num_virtual_tokens=10,
225
prompt_tuning_init="RANDOM"
226
)
227
228
peft_model = get_peft_model(model, prompt_config)
229
```
230
231
### Prompt Tuning with Text Initialization
232
233
```python
234
from peft import PromptTuningConfig, PromptTuningInit
235
236
prompt_config = PromptTuningConfig(
237
task_type=TaskType.CAUSAL_LM,
238
num_virtual_tokens=10,
239
prompt_tuning_init=PromptTuningInit.TEXT,
240
prompt_tuning_init_text="The following is a helpful response:"
241
)
242
243
peft_model = get_peft_model(model, prompt_config)
244
```
245
246
### P-Tuning v2 with MLP Encoder
247
248
```python
249
from peft import PromptEncoderConfig, PromptEncoderReparameterizationType
250
251
p_tuning_config = PromptEncoderConfig(
252
task_type=TaskType.CAUSAL_LM,
253
num_virtual_tokens=20,
254
encoder_reparameterization_type=PromptEncoderReparameterizationType.MLP,
255
encoder_hidden_size=128,
256
encoder_num_layers=2,
257
encoder_dropout=0.1
258
)
259
260
peft_model = get_peft_model(model, p_tuning_config)
261
```
262
263
### Prefix Tuning
264
265
```python
266
from peft import PrefixTuningConfig
267
268
prefix_config = PrefixTuningConfig(
269
task_type=TaskType.CAUSAL_LM,
270
num_virtual_tokens=30,
271
encoder_hidden_size=512,
272
prefix_projection=True
273
)
274
275
peft_model = get_peft_model(model, prefix_config)
276
```
277
278
### Multitask Prompt Tuning
279
280
```python
281
from peft import MultitaskPromptTuningConfig
282
283
multitask_config = MultitaskPromptTuningConfig(
284
task_type=TaskType.CAUSAL_LM,
285
num_virtual_tokens=10,
286
num_tasks=3,
287
prompt_tuning_init="RANDOM"
288
)
289
290
peft_model = get_peft_model(model, multitask_config)
291
292
# During inference, specify task_id
293
outputs = peft_model(input_ids=input_ids, task_ids=task_ids)
294
```
295
296
### Adaption Prompt for Vision Models
297
298
```python
299
from transformers import CLIPModel
300
from peft import AdaptionPromptConfig
301
302
clip_model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
303
304
adaption_config = AdaptionPromptConfig(
305
target_modules=["visual.transformer.resblocks"],
306
adapter_len=10,
307
adapter_layers=12
308
)
309
310
peft_model = get_peft_model(clip_model, adaption_config)
311
```