Port Model To SD.Next And Diffusers

Read the task, identify the model architecture and artifact layout, choose the integration path that requires the least amount of new code while adhering to SD.Next patterns, implement the loader and pipeline wiring, and validate the result.

When To Use

• The user wants to add a new model family to SD.Next
• A standalone inference script needs to become a Diffusers-style pipeline
• A raw checkpoint or safetensors repo needs an SD.Next loader
• A model already exists in Diffusers but is not yet wired into SD.Next
• A custom architecture needs a repo-local pipelines/<model> package and loader

Guidance

• Consult .github/instructions/core.instructions.md for relevant core runtime and model porting guidance before proceeding.

Core Rule

Prefer the smallest correct integration path.

• If upstream Diffusers already supports the model cleanly, reuse the upstream pipeline and only add SD.Next wiring.
• If the model requires custom architecture or sampler behavior, add a repo-local pipeline package under pipelines/<model>.
• If the model ships as a raw checkpoint instead of a Diffusers repo, load or remap weights explicitly instead of pretending it is a standard Diffusers layout.

Handle Gated Models

Before collecting inputs: If the model repository is gated (HTTP 403, access agreement required, or waiting list):

1. Check secrets.json in the workspace root for a huggingface_token field
2. If token exists, use it to authenticate and attempt access
3. If token missing, invalid, or access still denied, abort the port work immediately and report:
   • Model name and repo link
   • Specific access barrier (gated, closed, waiting list, license agreement)
   • Required steps for user to gain access or provide token
   • Do not proceed with implementation

Inputs To Collect First

Before editing anything, determine these facts:

• Model task: text-to-image, img2img, inpaint, editing, video, multimodal, or other
• Artifact format: Diffusers repo, local folder, single-file safetensors, ckpt, GGUF, or custom layout
• Core components: transformer or UNet, VAE or VQ model, text encoder, tokenizer or processor, scheduler, image encoder, adapters
• Whether output is latent-space or direct pixel-space
• Whether the model family has one fixed architecture or multiple variants
• Whether prompt encoding follows a normal tokenizer path or a custom chat/template path
• Whether there is an existing in-repo analogue to copy structurally

If any of these remain unclear after reading the repo and source artifacts, ask concise clarifying questions before implementing.

Mandatory Category Question

Before implementing model-reference updates, explicitly ask the user which category the model belongs to:

• base
• cloud
• quant
• distilled
• nunchaku
• community

Do not guess this category. Use the user answer to decide which reference JSON file(s) to update.

Mandatory Pipeline Question

Before implementing a pipeline, explicitly ask the user if the model already has an upstream Diffusers pipeline that can be reused. If the desired model implementation does not already exist in upstream Diffusers, ask for the reference implementation's code path. A repository URL is acceptable when the code path is not yet available locally. Do not begin a custom pipeline port until the reference implementation is identified.

Custom Pipeline Handoff

When no usable upstream Diffusers implementation exists:

1. Collect the reference implementation code path, destination path in this repository, and target pipeline class name required by port-pipeline. Ask for any missing values before proceeding.
2. Invoke the port-pipeline skill with the model task, artifact layout, components, source code path, destination path, target class name, and relevant findings already gathered. Do not ask the user to repeat known details.
3. Let port-pipeline implement and validate the custom Diffusers pipeline, then return control to this port-model workflow. Continue the remaining SD.Next model integration and validation here.
4. If port-pipeline was invoked from this port-model run, its final handoff must return to this run rather than invoke port-model again.

Repo Files To Check

Start by reading the task description, then inspect the closest matching implementations.

• .github/copilot-instructions.md
• .github/instructions/core.instructions.md
• pipelines/generic.py
• pipelines/model_*.py files similar to the target model
• modules/sd_models.py
• modules/sd_detect.py
• modules/modeldata.py

Useful examples by pattern:

• Existing upstream Diffusers loader: pipelines/model_chroma.py, pipelines/model_z_image.py
• Custom in-repo pipeline package: pipelines/f_lite/
• Shared Qwen loader pattern: pipelines/model_z_image.py, pipelines/model_flux2_klein.py

Integration Decision Tree

Use this quick order before diving into detailed path requirements:

1. If an upstream Diffusers pipeline already covers the model, choose path 1.
2. If upstream support is insufficient but the model can be expressed as a Diffusers-style custom package, choose path 2.
3. If artifacts are raw checkpoints or single-file weights without a usable Diffusers layout, choose path 3.

1. Upstream Diffusers Support Exists

Use this path when the model already has a usable Diffusers pipeline and component classes.

Implement:

• pipelines/model_<name>.py
• modules/sd_models.py dispatch branch in load_diffuser_force
• modules/sd_detect.py filename autodetect branch if appropriate
• modules/modeldata.py model type detection branch

Every base-model family loader, whether upstream or custom, must be dispatched from an explicit detected-model-type branch in modules/sd_models.py:load_diffuser_force. Do not rely on load_diffuser_folder or another generic fallback to trigger a family loader.

Reuse:

• generic.load_transformer(...)
• generic.load_text_encoder(...)
• Existing processor or tokenizer loading patterns
• sd_hijack_te.init_hijack(pipe) and sd_hijack_vae.init_hijack(pipe) where relevant

2. Custom Pipeline Needed

Use this path when the model architecture, sampler, or prompt encoding is not available upstream.

Implement:

• pipelines/<model>/__init__.py
• pipelines/<model>/model.py
• pipelines/<model>/pipeline.py
• pipelines/model_<name>.py
• SD.Next registration in modules/sd_models.py, modules/sd_detect.py, and modules/modeldata.py

Model module responsibilities:

• Architecture classes
• Config handling and defaults
• Weight remapping or checkpoint conversion helpers
• Raw checkpoint loading helpers if needed

Pipeline module responsibilities:

• DiffusionPipeline subclass
• __init__
• from_pretrained
• encode_prompt or equivalent prompt preparation
• __call__
• Output dataclass
• Optional callback handling and output conversion

If custom pipeline is provided by user, check it for accuracy and completness but do not assume it is perfect. Make necessary adjustments to fit SD.Next patterns and validate the result. Fix all relative imports to be absolute and compatible with SD.Next repo structure, make sure that all imports are resolvable and make sure it passes ruff checks.

3. Raw Checkpoint Or Single-File Weights

Use this path when the model source is not a normal Diffusers repository.

Requirements:

• Resolve checkpoint path from local file, local directory, or Hub repo
• Load state dict directly
• Remap keys when training format differs from inference format
• Infer config from tensor shapes only if the model family truly varies and no config file exists
• Raise explicit errors for ambiguous or incomplete layouts

Do not fake a from_pretrained implementation that silently assumes missing subfolders exist.

Required SD.Next Touchpoints

Most new model families need all of these:

• pipelines/model_<name>.py Purpose: SD.Next loader entry point
• modules/sd_models.py Purpose: route detected model type to the correct loader
• modules/sd_detect.py Purpose: detect model family from filename or repo name
• modules/modeldata.py Purpose: classify loaded pipeline instance back into SD.Next model type

Add only what the model actually needs.

Check these conditional runtime registries and update them when the new model type needs the behavior they control:

• modules/sd_samplers_common.py:flow_models when the model needs scheduler configs with use_flow_sigmas to use flow sigmas.
• modules/vae/model_map.py:vae_map when SD.Next VAE handling needs to map the model type to a VAE family.

Reference catalog touchpoints are also required for model ports intended to appear in SD.Next model references.

• data/reference-base.json for base
• data/reference-cloud.json for cloud
• data/reference-quantized.json for quant
• data/reference-distilled.json for distilled
• data/reference-nunchaku.json for nunchaku
• data/reference-community.json for community

If the model belongs to multiple categories, update each corresponding data/reference*.json file.

When adding a new model in the user-confirmed base category, also update both wiki model pages:

1. Add the model family to wiki/Model-Support.md, using the existing linked-name and variant style.
2. Add one or more rows to wiki/Models.md for the new base model variants, following the existing table columns and formatting.
3. Use official source links and verified architecture, parameter, encoder, size, license, and release-date details. Leave unavailable details blank or clearly unspecified; do not infer them.
4. Check both pages for an existing entry first and update it instead of adding a duplicate.

These wiki updates apply to new base models, not models added only to cloud, quant, distilled, nunchaku, or community categories.

Possible extra integration points:

• diffusers.pipelines.auto_pipeline.AUTO_*_PIPELINES_MAPPING when task switching matters
• pipe.task_args when SD.Next needs default runtime kwargs such as output_type
• VAE hijack, TE hijack, or task-specific processors

Loader Conventions

In pipelines/model_<name>.py:

• Use sd_models.path_to_repo(checkpoint_info)
• Call sd_models.hf_auth_check(checkpoint_info)
• Use model_quant.get_dit_args(...) for load args
• Respect devices.dtype
• Log meaningful loader details
• When using generic.load_transformer(...), define a TransformerSpec for the model and pass it through the native_spec= argument. Put the spec in the model package or another appropriate shared module, and define the model's checkpoint prefixes, converter, and sibling handling as needed; do not omit the spec or rely on the generic single-file fallback.
• Leave allow_quant at its loader default for generic.load_transformer(...) and generic.load_text_encoder(...) unless a verified model-specific constraint requires disabling or otherwise overriding quantization.
• Explicitly preload only the transformer and text encoder through generic.load_transformer(...) and generic.load_text_encoder(...). Let the pipeline's from_pretrained(...) load its processor, tokenizer, and other declared components unless a verified model-specific requirement calls for separate loading.
• Reuse generic.load_text_encoder(...) when possible
• Set pipe.task_args = {'output_type': 'np'} when the pipeline should default to numpy output for SD.Next
• Clean up temporary references and call devices.torch_gc(...)

Do not hardcode assumptions about CUDA-only execution, local paths, or one-off environment state.

Pipeline Conventions

When building a custom pipeline:

• Inherit from diffusers.DiffusionPipeline
• Register modules with DiffusionPipeline.register_modules(...)
• Keep prompt encoding and output conversion inside the pipeline
• Support prompt, negative_prompt, generator, output_type, and return_dict when the task is text-to-image-like
• Expose only parameters that are part of the model’s actual sampling surface
• Preserve direct pixel-space output if the model does not use a VAE
• Use a VAE only if the model genuinely outputs latents

Do not add generic Stable Diffusion arguments that the model does not support.

Validation Checklist

After implementation, validate in this order:

1. Syntax or bytecode compilation for new files
2. Focused linting on touched files
3. Import-level smoke test for new modules when safe
4. Loader-path validation without doing a full generation if the model is too large
5. One real load and generation pass if feasible

Always report what was validated and what was not.

Reference Asset Update Requirements

When the user asks to add or port a model for references, also perform these steps:

1. Update the correct data/reference*.json file(s) based on the user-confirmed category.
2. Create a placeholder zero-byte thumbnail file in models/Reference for the new model.

Notes:

• In this repo, the folder is models/Reference (capital R).
• Use a deterministic filename that matches the model entry naming convention used in the target reference JSON.
• If a real thumbnail already exists, do not overwrite it with a zero-byte file.

CHANGELOG Update Requirement

When porting a new model or model family to SD.Next, also update CHANGELOG.md:

1. Locate the Models section under the most recent date heading (e.g., ## Update for 2026-04-10)
2. Add a new bullet entry at the top of the Models list with:
   • Model name as a markdown link to the HF repo or official source
   • Short one-line description (1-2 sentences max)
   • Key details in italics on separate lines:
     • Unique architecture features, parameter counts, or efficiency claims
     • Variant information (e.g., Normal, Edit, Lite for distilled versions)
     • Notable text encoder or technology notes

Example format:

- [BRIA FIBO](https://huggingface.co/briaai/FIBO) 8B parameter text-to-image model using Flow Matching  
    includes *Normal*, *Edit*, and *Lite* (distilled) variants  
    features lightweight SmolLM3-3B text encoder with efficient inference

Common Failure Modes

• Model type is added to sd_models.py but not sd_detect.py
• Loader exists but modules/modeldata.py still classifies the pipeline incorrectly
• A custom pipeline is named in a way that collides with a broader existing branch such as Chroma
• torch_dtype or other loader args are passed twice
• The code assumes a Diffusers repo layout when the source is really a single checkpoint file
• Negative prompt handling does not match prompt batch size
• Output is postprocessed as latents even though the model is pixel-space
• Custom config inference uses fragile defaults without clear failure paths

Output Expectations

When using this skill, the final implementation should usually include:

• A clear integration path choice
• The minimal set of new files required
• SD.Next routing updates
• Targeted validation results
• Any remaining runtime risks called out explicitly

Example Request Shapes

• "Port this standalone inference script into an SD.Next Diffusers pipeline"
• "Add support for this Hugging Face model repo to SD.Next"
• "Wire this upstream Diffusers pipeline into SD.Next autodetect and loading"
• "Convert this single-file checkpoint model into a custom Diffusers pipeline for SD.Next"