or run

npx @tessl/cli init
Log in

Version

Tile

Overview

Evals

Files

docs

advanced-methods.mdauto-classes.mdcore-models.mdindex.mdlora-methods.mdprompt-learning.mdutilities.md
tile.json

tessl/pypi-peft

State-of-the-art Parameter-Efficient Fine-Tuning (PEFT) methods for efficiently adapting large pretrained models

Workspace
tessl
Visibility
Public
Created
Last updated
Describes
pypipkg:pypi/peft@0.17.x

To install, run

npx @tessl/cli install tessl/pypi-peft@0.17.0

index.mddocs/

PEFT (Parameter-Efficient Fine-Tuning)

A comprehensive library for efficiently adapting large pretrained models using state-of-the-art parameter-efficient fine-tuning methods. PEFT enables fine-tuning of only a small number of model parameters instead of all parameters, significantly reducing computational and storage costs while achieving performance comparable to fully fine-tuned models.

Package Information

  • Package Name: peft
  • Language: Python
  • Installation: pip install peft
  • Requirements: Python >=3.9.0

Core Imports

import peft

Common imports for PEFT models:

from peft import get_peft_model, PeftConfig, PeftModel

Specific PEFT method imports:

from peft import LoraConfig, AdaLoraConfig, IA3Config

Auto model imports:

from peft import AutoPeftModel, AutoPeftModelForCausalLM

Basic Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig

# Load base model and tokenizer
model_name = "microsoft/DialoGPT-medium"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create PEFT configuration
peft_config = LoraConfig(
    task_type="CAUSAL_LM",
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1
)

# Get PEFT model
peft_model = get_peft_model(model, peft_config)

# Print trainable parameters
peft_model.print_trainable_parameters()

# Train the model (with your training loop)
# ...

# Save the PEFT adapter
peft_model.save_pretrained("./peft-adapter")

# Load PEFT model later
from peft import AutoPeftModelForCausalLM
peft_model = AutoPeftModelForCausalLM.from_pretrained("./peft-adapter")

Architecture

PEFT follows a modular architecture that separates configuration, model wrapping, and method-specific implementations:

  • Core Models: PeftModel and task-specific variants wrap base transformers models
  • Configurations: Each PEFT method has a dedicated config class (e.g., LoraConfig)
  • Tuners: Method-specific implementations in the tuners module
  • Auto Classes: Automatic model selection based on task type and saved configurations
  • Mixed Models: Support for combining multiple PEFT methods

This design allows seamless integration with Hugging Face Transformers while providing extensibility for custom PEFT methods.

Capabilities

Core Model Functions

Primary functions for creating and managing PEFT models, including the main entry point and core model classes.

def get_peft_model(
    model: PreTrainedModel,
    peft_config: PeftConfig,
    adapter_name: str = "default",
    mixed: bool = False,
    autocast_adapter_dtype: bool = True,
    revision: Optional[str] = None,
    low_cpu_mem_usage: bool = False
) -> PeftModel | PeftMixedModel: ...

Core Models

LoRA Methods

Low-Rank Adaptation methods including standard LoRA, AdaLoRA, and variants like LoHa, LoKr, and specialized configurations.

class LoraConfig(PeftConfig):
    def __init__(
        self,
        r: int = 8,
        lora_alpha: int = 8,
        target_modules: Optional[Union[List[str], str]] = None,
        lora_dropout: float = 0.0,
        bias: str = "none",
        task_type: str = None,
        **kwargs
    ): ...

class AdaLoraConfig(PeftConfig):
    def __init__(
        self,
        target_r: int = 8,
        init_r: int = 12,
        tinit: int = 0,
        tfinal: int = 0,
        deltaT: int = 1,
        **kwargs
    ): ...

LoRA Methods

Prompt Learning Methods

Prompt-based tuning methods including Prompt Tuning, P-tuning, Prefix Tuning, and Multitask Prompt Tuning.

class PromptTuningConfig(PromptLearningConfig):
    def __init__(
        self,
        num_virtual_tokens: int,
        prompt_tuning_init: PromptTuningInit = PromptTuningInit.RANDOM,
        prompt_tuning_init_text: Optional[str] = None,
        **kwargs
    ): ...

class PrefixTuningConfig(PromptLearningConfig):
    def __init__(
        self,
        num_virtual_tokens: int,
        encoder_hidden_size: int,
        prefix_projection: bool = False,
        **kwargs
    ): ...

Prompt Learning

Advanced Methods

Specialized PEFT methods including orthogonal fine-tuning (OFT, BOFT), structured methods (IA3, Vera), and experimental approaches.

class OFTConfig(PeftConfig):
    def __init__(
        self,
        r: int = 8,
        module_dropout: float = 0.0,
        target_modules: Optional[Union[List[str], str]] = None,
        **kwargs
    ): ...

class IA3Config(PeftConfig):
    def __init__(
        self,
        target_modules: Optional[Union[List[str], str]] = None,
        feedforward_modules: Optional[Union[List[str], str]] = None,
        **kwargs
    ): ...

Advanced Methods

Auto Classes

Automatic model loading and task-specific PEFT model classes for different NLP tasks.

class AutoPeftModel:
    @classmethod
    def from_pretrained(
        cls,
        pretrained_model_name_or_path,
        adapter_name: str = "default",
        is_trainable: bool = False,
        config: Optional[PeftConfig] = None,
        revision: Optional[str] = None,
        **kwargs
    ): ...

class AutoPeftModelForCausalLM(AutoPeftModel): ...
class AutoPeftModelForSequenceClassification(AutoPeftModel): ...
class AutoPeftModelForFeatureExtraction(AutoPeftModel): ...
class AutoPeftModelForQuestionAnswering(AutoPeftModel): ...
class AutoPeftModelForSeq2SeqLM(AutoPeftModel): ...
class AutoPeftModelForTokenClassification(AutoPeftModel): ...

Auto Classes

Utilities and State Management

Functions for managing model state, loading/saving adapters, and preparing models for training.

def get_peft_model_state_dict(
    model,
    state_dict: Optional[dict] = None,
    adapter_name: str = "default"
) -> dict: ...

def set_peft_model_state_dict(
    model,
    peft_model_state_dict: dict,
    adapter_name: str = "default"
): ...

def load_peft_weights(model_id: str, device: Optional[str] = None) -> dict: ...

def prepare_model_for_kbit_training(
    model,
    use_gradient_checkpointing: bool = True,
    gradient_checkpointing_kwargs: Optional[dict] = None
): ...

def get_peft_config(config_dict: dict) -> PeftConfig: ...

def inject_adapter_in_model(
    peft_config: PeftConfig,
    model,
    adapter_name: str = "default"
): ...

def replace_lora_weights_loftq(
    model,
    loftq_config: LoftQConfig,
    adapter_name: str = "default"
): ...

def register_peft_method(
    name: str,
    config_cls,
    model_cls,
    prefix: Optional[str] = None,
    is_mixed_compatible: bool = False
) -> None: ...

Utilities

Types and Enums

Core types and enumerations used throughout the PEFT library.

class PeftType(str, Enum):
    PROMPT_TUNING = "PROMPT_TUNING"
    MULTITASK_PROMPT_TUNING = "MULTITASK_PROMPT_TUNING"
    P_TUNING = "P_TUNING"
    PREFIX_TUNING = "PREFIX_TUNING"
    LORA = "LORA"
    ADALORA = "ADALORA"
    BOFT = "BOFT"
    ADAPTION_PROMPT = "ADAPTION_PROMPT"
    IA3 = "IA3"
    LOHA = "LOHA"
    LOKR = "LOKR"
    OFT = "OFT"
    POLY = "POLY"
    LN_TUNING = "LN_TUNING"
    VERA = "VERA"
    FOURIERFT = "FOURIERFT"
    XLORA = "XLORA"
    HRA = "HRA"
    VBLORA = "VBLORA"
    CPT = "CPT"
    BONE = "BONE"
    MISS = "MISS"
    RANDLORA = "RANDLORA"
    TRAINABLE_TOKENS = "TRAINABLE_TOKENS"
    SHIRA = "SHIRA"
    C3A = "C3A"

class TaskType(str, Enum):
    SEQ_CLS = "SEQ_CLS"
    SEQ_2_SEQ_LM = "SEQ_2_SEQ_LM"
    CAUSAL_LM = "CAUSAL_LM"
    TOKEN_CLS = "TOKEN_CLS"
    QUESTION_ANS = "QUESTION_ANS"
    FEATURE_EXTRACTION = "FEATURE_EXTRACTION"

class PeftConfig:
    """Base configuration class for all PEFT methods."""
    def __init__(
        self,
        peft_type: Optional[PeftType] = None,
        task_type: Optional[TaskType] = None,
        inference_mode: bool = False,
        r: int = 8,
        target_modules: Optional[Union[List[str], str]] = None,
        modules_to_save: Optional[List[str]] = None,
        **kwargs
    ): ...