Add a New AI Model to Kiln

Branch check first: if the request involves a provider Kiln does not support yet, start at Adding a Net-New Provider. That workflow spans core, server, UI, tests, and tooling, and is gated on a client release — the model-entry steps below are only its final, gated piece.

For a new model on an already-supported provider, integrating it into libs/core/kiln_ai/adapters/ml_model_list.py requires:

1. ModelName enum – add an enum member
2. built_in_models list – add a KilnModel(...) entry with providers
3. ModelFamily enum – only if the vendor is brand-new

After code changes, run paid integration tests, then draft a Discord post.

Global Rules

These apply throughout the entire workflow.

• Slug verification: NEVER guess or infer model slugs from naming patterns. Every model_id must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.
• Date awareness: These models are often released very recently. Web search for current info before assuming you know the details.

Phase 1 – Model Discovery (only when asked to find new/missing models)

If the user asks you to find new models, do NOT just web search "new AI models this week" — that only surfaces major releases. Instead, systematically check each family against both the LiteLLM catalog and models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.

1. Read the ModelFamily and ModelName enums to know what we already have.

2. Query both catalogs for each family (run in parallel where possible):

   LiteLLM catalog — filters out mirror providers to avoid duplicates:

   curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != "openrouter" and .provider != "bedrock" and .provider != "bedrock_converse" and .provider != "vertex_ai-anthropic_models" and .provider != "azure") | .id] | unique | .[]'

   models.dev — search all model IDs across all providers:

   curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"

   For details on a specific provider+model: curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

3. Search terms (one query per term): claude, gpt, o1, o3, o4 (OpenAI reasoning), gemini, llama, deepseek, qwen, qwq, mistral, grok, kimi, glm, minimax, hunyuan, ernie, phi, gemma, seed, step, pangu

4. Union and cross-reference results from both catalogs against ModelName. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). Skip pure coding models (e.g. codestral, deepseek-coder, qwen-coder).

5. Run targeted web searches per family to catch very fresh releases not yet in either catalog:

   • "[family] new model [current year]"
   • "[family] release [current month] [current year]"

6. Present findings as a summary. Let the user decide which to add.

Phase 1B – Lagging-Provider Backfill Check (every run)

Some providers — Fireworks AI, Together AI, SiliconFlow, Cloudflare — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now and miss the window to backfill recently-added models whose provider support has since grown.

Run this check on every invocation of the skill, regardless of whether you're in discovery mode or adding a specific model.

1. Pull the 10 most recently added models from git history. List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:

   git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20

2. For the model you're adding (if any) AND each of those 10 models, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the Lagging Providers Reference. Do NOT trust models.dev / LiteLLM as the final word for these providers.

3. If a lagging provider now supports a recently-added model that isn't yet in its KilnModel entry, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.

4. Run the Cloudflare thinking-level checker (see Cloudflare thinking levels). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.

Phase 2 – Gather Context

1. Read the predecessor model in ml_model_list.py (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.

2. Query the LiteLLM catalog for the new model. This is the primary slug source since Kiln uses LiteLLM. See the Slug Lookup Reference for query syntax and all verified sources.

3. Get the OpenRouter slug via:

   • curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"
   • Fallback: WebSearch for openrouter [model name] model id

4. Get the direct-provider slug (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the Slug Lookup Reference for provider-specific URLs.

5. Identify quirks — check the Provider Quirks Reference for the relevant provider, and web search for any new quirks:

   • Structured output mode (JSON schema vs function calling)?
   • Reasoning model (needs reasoning_capable, parsers, OpenRouter options)?
   • Vision/multimodal support? Which MIME types?
   • Provider-specific flags (temp_top_p_exclusive, etc.)?
   • Rate limit concerns (max_parallel_requests)?

6. Determine thinking levels — does the model support configurable reasoning effort? See Thinking Levels Reference for the full lookup chain. Key quick checks:

   • Check the vendor model page (e.g. OpenAI model pages say "Reasoning.effort supports: X, Y, Z")
   • Check OpenRouter supported_parameters — if reasoning is absent, skip thinking levels
   • R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts

Phase 3 – Code Changes

All changes go in libs/core/kiln_ai/adapters/ml_model_list.py.

3a. ModelName enum

• snake_case: claude_opus_4_6 = "claude_opus_4_6"
• Place before predecessor (newer first within group). If the vendor is brand-new there is no predecessor — start a new group at the end of the enum
• Follow existing grouping (all claude together, all gpt together, etc.)

3b. KilnModel entry in built_in_models

• Place per the ordering rules in 3c — this placement is user-visible, get it right
• Copy predecessor's structure and modify: name, friendly_name, model_id per provider, flags
• friendly_name must follow the existing naming pattern of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use "Claude {version} Sonnet" (e.g. "Claude 4.5 Sonnet"), not "Claude Sonnet {version}". Do NOT use the vendor's marketing name if it differs from Kiln's established convention.

Provider model_id formats:

Every single model_id must be verified from an authoritative source. No exceptions.

Setting flags — use catalog data + predecessor as dual signals:

The LiteLLM catalog and models.dev responses include capability flags (supports_vision, supports_function_calling, supports_reasoning, etc.). Use these as the primary signal for what to enable on the new model:

• If the catalog says supports_vision: true → enable supports_vision, multimodal_capable, and vision MIME types (see 2c)
• If the catalog says supports_function_calling: true → use StructuredOutputMode.json_schema (or function_calling depending on provider norms — check predecessor)
• If the catalog says supports_reasoning: true → the model can reason, but do NOT reflexively set reasoning_capable=True — default to reasoning_capable=False (see Reasoning Capable Default). Still add available_thinking_levels if it supports effort levels, and check parser/formatter flags.

Then cross-check against the predecessor. The predecessor tells you how Kiln configures a similar model (which structured_output_mode, which provider-specific flags, etc.). The catalog tells you what the model can do. Use both:

• Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.
• Predecessor has temp_top_p_exclusive but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.
• Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.

Common flags:

• structured_output_mode – how the model handles JSON output
• suggested_for_evals / suggested_for_data_gen – see zero-sum rule below. If the entry also sets available_thinking_levels, its default_thinking_level must be a reasoning level — see Judge Models Must Reason by Default
• multimodal_capable / supports_vision / supports_doc_extraction – see multimodal rules below
• reasoning_capable – for thinking/reasoning models. Default new models to reasoning_capable=False unless the model always emits its reasoning (see Reasoning Capable Default)
• temp_top_p_exclusive – Anthropic models that can't have both temp and top_p
• parser / formatter – for models needing special parsing (e.g. R1-style thinking)

2c. Multimodal capabilities

If the model supports non-text inputs, configure:

• multimodal_capable=True and supports_doc_extraction=True if it supports any MIME types
• supports_vision=True if it supports images
• multimodal_requires_pdf_as_image=True if vision-capable but no native PDF support (also add KilnMimeType.PDF to MIME list). Always set this on OpenRouter providers — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.
• Always include KilnMimeType.TXT and KilnMimeType.MD on any multimodal_capable model

Strategy: start broad, narrow based on test failures. Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.

Full MIME superset (Gemini uses all):

# documents
KilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD
# images
KilnMimeType.JPG, KilnMimeType.PNG
# audio
KilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG
# video
KilnMimeType.MP4, KilnMimeType.MOV

3c. Ordering — the list IS the UI

The order of built_in_models is exactly the order users see: model dropdowns group by provider and list each provider's models in list order, and the model library page lists models in list order. A misplaced entry ships a scrambled dropdown to every client via the remote config. Rules:

1. Families are contiguous. Every model of a family sits in one block. Never append a new family member at the bottom of the file or after another family — that strands it (past bugs: Mistral Small 4/3 ended up inside the Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).

2. Within a family, versions run newest → oldest, top to bottom. 4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the family block. A new model of an existing version goes inside that version's group — NOT at the top of the family and NOT below older versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7 landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).

3. Within a version, big → small. Always.

   • Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite; Large > Medium > Small.
   • Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and VL Thinking blocks) stay intact, ordered descending internally.
   • Tier-grouped families follow the same rule for their tier blocks: Claude runs Fable, then Opus > Sonnet > Haiku, versions descending inside each tier. (It once ran Haiku-first, and several families ran sizes ascending — those were bugs, not conventions. Do not preserve an ascending run because it's "what the family already does.")

4. A net-new family's block goes at the END of built_in_models (this is the existing convention — the newest niche vendors sit at the bottom of the list). The "place before predecessor" rule only applies within an existing family; a new family has no predecessor.

5. Verify after editing — print the order you just shipped and eyeball the affected family:

   uv run python -c "
   from kiln_ai.adapters.ml_model_list import built_in_models
   for m in built_in_models: print(m.family, '|', m.friendly_name)"

3d. suggested_for_evals / suggested_for_data_gen

Only set these if the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).

Zero-sum rule: When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. Ask the user to confirm the swap before making changes.

3e. ModelFamily enum (only if needed)

Only add a new family if the vendor is completely new.

3f. Thinking Levels (available_thinking_levels / default_thinking_level)

If the model supports configurable reasoning effort (not just on/off), add available_thinking_levels and default_thinking_level to each provider entry. See Thinking Levels Reference for the full lookup chain and existing constants.

Quick rules:

• Reuse an existing _THINKING_LEVELS constant if the levels match exactly
• Create a new constant only if levels differ; name it {MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS
• default_thinking_level must be one of the values in available_thinking_levels
• If the entry sets suggested_for_evals=True, default_thinking_level must be "medium" (or "low" if the model has no medium level) — never "none". See Judge Models Must Reason by Default

Phase 4 – Run Tests

Tests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.

Vertex AI authentication: Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run gcloud auth application-default login before trying. These failures are auth issues, not model config problems.

-k filter syntax: Always use bracket notation for model+provider filtering, never and:

• Good: -k "test_name[glm_5-fireworks_ai]" or -k "glm_5"
• Bad: -k "glm_5 and fireworks" — and is a pytest keyword expression that can match wrong tests

4.0 — If the test env can't build (blocked git dependency)

Kiln's core lib pins together to a git fork (libs/core/pyproject.toml: together = { git = "https://github.com/scosman/together-python" }). In a sandboxed environment whose GitHub access is scoped to kiln-ai/kiln only (e.g. Claude Code Web), uv sync fails to fetch that fork with a 403 from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only kiln-ai/kiln resolves.

Workaround to build the venv for testing (the together fork isn't exercised by model-integration tests, which route through LiteLLM):

1. Do NOT use uv sync --no-sources — it strips the workspace workspace = true sources too and makes kiln-root → kiln-ai unsatisfiable.
2. Instead, temporarily comment out ONLY the together = { git = ... } line in libs/core/pyproject.toml, then run uv sync (this resolves together from PyPI).
3. Run the tests.
4. Revert the workaround with git checkout -- libs/core/pyproject.toml uv.lock — this restores the commented-out together git-fork pin in libs/core/pyproject.toml (NOT the repo-root pyproject.toml, which the workaround never touches) and reverts uv.lock if it changed. Only ml_model_list.py (and any intended test-file edits) should remain modified.

Note: the PyPI together may pull slightly different transitive deps (e.g. a newer starlette), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your -k filters to the model files and ignore those.

4a. Parallel testing + API keys

Parallel testing is already on. There is no pytest.ini — pytest config lives in [tool.pytest.ini_options] in the root pyproject.toml, and addopts = "-n auto" is active. No edit and no revert are needed. (Only override to -n 8 if a provider rate-limits you.)

Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings. conftest.py has an autouse use_temp_settings_dir fixture that points Config.settings_path at a temp dir, so ~/.kiln_ai/settings.yaml is deliberately ignored during tests. A key the user added through the app's provider page will not be seen — the test fails with "Attempted to use X without an API key set", which looks like a config bug but isn't.

Bridge the key from the user's settings into the test environment without ever printing it:

export FEATHERLESS_AI_API_KEY="$(uv run python -c \
  "from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)" 2>/dev/null | tail -1)"

Use the provider's env_var name from libs/core/kiln_ai/utils/config.py. Some providers need more than one value: Cloudflare needs both CLOUDFLARE_API_KEY and CLOUDFLARE_ACCOUNT_ID (and optionally CLOUDFLARE_AI_GATEWAY_ID). To check which keys are available before running, print booleans only — never the values:

uv run python -c "from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))"

Many paid tests also carry the ollama marker, so --runpaid alone silently skips them. Always pass --runpaid --ollama together, and use -rs to see skip reasons when a test you expected to run reports as skipped.

4b. Smoke test — verify slug works

Run a single test+provider combo first:

uv run pytest --runpaid --ollama -k "test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]"

If it fails, fix the slug/config before proceeding. Use --collect-only to find exact parameter IDs if unsure.

4c. Full test suite

uv run pytest --runpaid --ollama -k "MODEL_ENUM" -v 2>&1 | grep -E "PASSED|FAILED|ERROR|short test|=====|collected"

If tests fail — debug one at a time:

1. Pick ONE failing test, run it with -v for full output
2. Fix the config
3. Re-run that single test to verify
4. Only re-run the full suite once the single test passes

Anthropic API key gotcha: if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as KILN_ANTHROPIC_API_KEY instead of ANTHROPIC_API_KEY (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't export it globally:

ANTHROPIC_API_KEY="$KILN_ANTHROPIC_API_KEY" uv run pytest --runpaid ...

4d. Extraction tests (if supports_doc_extraction=True)

Tests are in libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py.

# See what will run:
uv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM

# Run them:
uv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k "MODEL_ENUM"

If a provider rejects a data type (400 error), remove that KilnMimeType and re-run.

4e. Confirm failures are actually yours

Before treating a failure as a problem with your change, check whether the same test already fails for an existing provider of that model. Several assertions are provider-independent and fail regardless.

Known example: test_structured_input_cot_prompt_builder asserts len(trace) == 5 unconditionally, which is incompatible with any provider setting reasoning_capable=True (that selects the single-call strategy, producing 3 messages). It fails for gpt_oss_120b on fireworks_ai on a clean tree.

uv run pytest --runpaid --ollama -q "path::test_name[MODEL-OTHER_PROVIDER]"

If it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.

4f. Test output format

Collect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:

• ✅ for passed tests
• ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason
• ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason
• List every test using the full pytest parametrize ID, grouped by provider
• Include extraction tests (Phase 4d) if they were run

Phase 5 – Create Pull Request

5.0 — Important context about Claude Code Web's stop hook

This skill is often run via Claude Code Web (Slack connector). That environment has a non-user-configurable stop hook which, at end of session, will:

• Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
• Instruct the agent to commit and push any local work before stopping
• Explicitly tell the agent NOT to create a PR unless the user asked for one

The problems this causes:

1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned add-model/* branches on the remote.
2. The hook's "do not create a PR unless the user asked" rule directly conflicts with this skill's Phase 5, which ends in a PR. Running this skill is the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.

The user's desires, in priority order:

1. Ask before you push. If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
2. No abandoned branches. Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
3. If the user says to abandon: revert your local changes (git restore / git clean the specific files you touched) and delete any branch you created (git checkout main && git branch -D add-model/MODEL_NAME) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.
4. On a successful run, push and open the PR as described in 5a/5b. Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.

5.1 — Gate before pushing

Do NOT commit, push, or create a branch if any of the following are true:

• Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
• The smoke test (4b) failed and wasn't resolved
• Any step in Phases 2–4 was skipped or incomplete
• You are unsure whether a ⚠️ flake is actually a real failure

If any of the above apply, stop and ask the user what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.

After all tests pass, commit the changes and open a PR against main.

5a. Commit and push

1. Create a new branch named add-model/MODEL_NAME (e.g. add-model/glm-5-1)
2. Stage only the changed files (typically just ml_model_list.py)
3. Commit with a concise message (e.g. "Add GLM 5.1 to model list (together_ai, siliconflow_cn)")
4. Push the branch

5b. Create the PR

Use gh pr create against main. Follow Rule 0, Rule 1 and Step 3 of the open-pr skill (.agents/skills/open-pr/SKILL.md) for the human header, the CLA and the title:

• The title is WIP: chore: followed by the model-list subject, for example WIP: chore: add GLM 5.1 to model list. Use chore, because adding a model is maintenance work, not a new feature. Only a human removes the WIP: prefix.
• Copy everything above # Agentic PR Summary from .github/pull_request_template.md with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the ## Contributor License Agreement section only when the PR author's GitHub username is scosman, sfierro, leonardmq, tawnymanticore, or chiang-daniel. Keep it, unsigned, for any other author.

Replace the `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` placeholder under # Agentic PR Summary with the test results, in this exact format:

## What does this PR do?

 Test Results

[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]

[Model Name] ([provider]):
- [N] passed, [N] skipped[, [N] failed]
- [Any notable failures or flakes]

[Repeat for each model+provider combo]

---
[Model Name] ([provider]):
✅ test_data_gen_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]
✅ test_all_built_in_models_llm_as_judge[model_enum-provider]
✅ test_all_built_in_models_structured_output[model_enum-provider]
✅ test_all_built_in_models_structured_input[model_enum-provider]
✅ test_structured_output_cot_prompt_builder[model_enum-provider]
✅ test_all_models_providers_plaintext[model_enum-provider]
✅ test_cot_prompt_builder[model_enum-provider]
⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason
❌ test_name[model_enum-provider] — brief reason

[Repeat for each model+provider combo]

Rules for the PR body:

• Every test that ran must appear in the per-test dump, using the full pytest parametrize ID
• Group tests by [Model Name] ([provider]): headers
• The summary section at the top gives a quick pass/skip/fail count per model+provider
• The detailed section below the --- inside the summary (not the ---- above # Agentic PR Summary) lists every individual test result
• Use ⚠️ for content quality flakes (not real failures), ❌ for real errors

Checklist

• ModelName enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)
• KilnModel entry added to built_in_models per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)
• Printed the resulting list order and eyeballed the affected family (3c verification command)
• friendly_name matches the naming pattern of sibling models in the same family
• If the provider is net-new: followed Adding a Net-New Provider, including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)
• ModelFamily enum updated (only if new family)
• All provider slugs verified from authoritative sources
• Flags inherited from predecessor and adjusted for quirks
• reasoning_capable defaulted to False for adaptive-reasoning models (only True for always-emits-reasoning models — see Reasoning Capable Default)
• Thinking levels configured if model supports reasoning effort (see Thinking Levels Reference)
• Cloudflare and OpenRouter: check_thinking_levels.py reports no ❌ finding (see Cloudflare thinking levels and OpenRouter (general))
• Every suggested_for_evals entry with thinking levels defaults to "medium" (or "low"), never "none" (see Judge Models Must Reason by Default)
• Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)
• Zero-sum applied if model is suggested for evals/data gen
• RAG config templates updated if the new model replaces one used in app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts
• API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)
• Smoke test passed
• Full test suite passed
• Failures cross-checked against an existing provider before being called regressions (see 4e)
• PR created against main with a WIP: title, the untouched human header, and test results in the Agentic PR Summary

Reasoning Capable Default

Default newly-added models to reasoning_capable=False, even when the catalog reports supports_reasoning: true.

Most recent models use adaptive reasoning and sometimes return no reasoning at all. Kiln raises RuntimeError("Reasoning is required for this model, but no reasoning was returned.") whenever a provider has reasoning_capable=True and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.

reasoning_capable is orthogonal to available_thinking_levels — you keep the important behavior with it set to False:

• Thinking-level / effort selection still works (every GPT-5.x entry sets available_thinking_levels with reasoning_capable unset).
• Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).
• The test_thinking_level_reasoning_content paid test still runs and still asserts reasoning is present — it is gated on available_thinking_levels, not reasoning_capable. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.

What you give up with reasoning_capable=False:

• The conditional "reasoning is present in intermediate_outputs" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).
• When a user attaches a chain-of-thought prompt, the model uses the two-call two_message_cot strategy instead of the single-call native single_turn_r1_thinking.

Keep reasoning_capable=True only for models that always emit reasoning in a native <think> format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.

Narrower alternative: if a model reliably reasons but you only hit the error on structured output, set reasoning_optional_for_structured_output=True (requires reasoning_capable=True) instead of disabling reasoning entirely.

Adding a Net-New Provider

Adding a provider Kiln has never supported is a bigger job than adding a model, and it has a hard sequencing constraint. Follow this section end to end.

The release-gating rule (do NOT skip)

The remote config is generated from main's built_in_models and reaches all existing clients immediately — but provider support (name map, connect flow, API key handling) only reaches users through a client app release. If model entries for a brand-new provider land on main before a client release with the provider plumbing is live, every deployed client shows the raw provider ID (e.g. featherless_ai) in the model library and offers models nobody can connect to. This happened with Featherless in Aug 2026 and the entries had to be rolled back.

Sequence it in two PRs:

1. PR 1 — provider plumbing only. The ModelProviderName enum member and everything else in the checklist below EXCEPT the model catalog. Merge whenever ready.
2. PR 2 — model catalog. ALL ml_model_list.py changes: ModelName members, ModelFamily (if the vendor is new), and the built_in_models entries. Open it, but merge only after a client release containing PR 1 is live. (Only built_in_models is published via the remote config, but the enums belong in the same PR as the entries they exist for.)

Touchpoint checklist (from the Featherless integration, #1618)

First, identify the provider's authentication model — the checklist below describes the common single-API-key pattern, but not every provider fits it. Existing variants to crib from: Bedrock stores an access key + secret pair, Fireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project ID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base URL with no credentials. Follow the closest existing analog's plumbing through config.py, provider_warnings, provider_api.py, and the connect page — the credential fields, validation call, and UI steps all change with the auth model.

libs/core:

• ModelProviderName enum — libs/core/kiln_ai/datamodel/datamodel_enums.py
• Credential storage — libs/core/kiln_ai/utils/config.py (for API-key providers: a new key with env_var; otherwise whatever fields the auth model needs)
• LiteLLM provider mapping — libs/core/kiln_ai/utils/litellm.py
• libs/core/kiln_ai/adapters/provider_tools.py — three spots: the provider_name_from_id match (friendly name; pyright flags a missed case), provider_warnings (missing-credential message, required_config_keys), and the adapter config (credential / base URL / headers plumbing)
• Model catalog — libs/core/kiln_ai/adapters/ml_model_list.py: ModelName members, ModelFamily if needed, and the built_in_models entries (PR 2 only, see gating rule)

Desktop server (app/desktop/studio_server/provider_api.py):

• connect_<provider> credential-validation endpoint (find a cheap authenticated call; see the Featherless connect function for a pattern when the provider has no authenticated GET to ping)
• Disconnect handling (clear the stored credentials)
• Tests in test_provider_api.py

Web UI (app/web_ui):

• src/lib/stores.ts — provider_name_map entry (friendly name)
• src/lib/ui/provider_image.ts + SVG in static/images/ — icon must match the monochrome convention: bare glyph, currentColor, no background tile, viewBox cropped to the artwork
• Connect page — src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte: provider card (name, description, and the auth flow — api_key_steps / api_key_fields for key providers, or the custom flow the auth model needs) plus connected-status wiring from settings keys
• src/lib/api_schema.d.ts — regenerate with make schema (needs the server running on :8757)

Tests & tooling:

• libs/core/kiln_ai/adapters/test_provider_tools.py, test_adapter_registry.py, model_adapters/test_litellm_adapter.py — extend the per-provider parametrized cases
• .agents/scripts/provider_utils.py — add the provider's model-catalog endpoint so agent tooling can enumerate its models
• Thinking levels — find out how the provider takes reasoning effort, and where it publishes the values each model accepts. Add a quirks section that names that source and says thinking levels are required wherever it lists values. If the provider silently remaps unsupported values, add the provider to check_thinking_levels.py, because a paid test cannot see a remap.
• model_adapters/test_thinking_level_paid.py — add the provider to the provider list of test_thinking_level_reasoning_content, so each of its thinking levels gets a paid test case

Known gap: the generated Copilot API client (app/desktop/studio_server/api_client/) mirrors the remote Copilot service's schema — it can't learn the new provider until that service updates. Note it in the PR rather than hand-editing generated code.

Friendly names everywhere

The raw enum value must never be user-visible. When you add the provider, verify a friendly name exists in both name maps (python provider_name_from_id and web provider_name_map) — pyright and typescript respectively force these when the enum gains a member, which is why the plumbing PR must not skip them.

Provider Quirks Reference

Anthropic

• Newer models (Opus 4.1+, Sonnet 4.5+) need temp_top_p_exclusive=True
• Opus 4.5+ uses json_schema; older Opus uses function_calling
• Extended thinking models: anthropic_extended_thinking=True + reasoning_capable=True

OpenAI

• Most GPT models use json_schema for structured output
• GPT-5.x models support available_thinking_levels — see Thinking Levels Reference
• Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort
• o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)

Google/Gemini

• gemini_reasoning_enabled=True for reasoning-capable models
• Gemini 3.x models support available_thinking_levels — see Thinking Levels Reference
• Rich multimodal support (audio, video, images, documents)

DeepSeek

• R1 models: parser=ModelParserID.r1_thinking + reasoning_capable=True
• V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN
• Some need r1_openrouter_options=True + require_openrouter_reasoning=True

OpenRouter (general)

• Slugs: vendor/model-name
• Never add a :free slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as :free, wait for a paid slug.
• Reasoning models: may need require_openrouter_reasoning=True
• Some models: openrouter_skip_required_parameters=True
• Logprobs: logprobs_openrouter_options=True if supported
• Always multimodal_requires_pdf_as_image=True (OpenRouter's PDF routing breaks LiteLLM)
• Thinking levels: offer only values in the model's reasoning.supported_efforts (see Thinking Levels Reference). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no reasoning.supported_efforts, OpenRouter turns the effort into a thinking-token budget for Claude, so CLAUDE_OPENROUTER_THINKING_LEVELS (with Minimal) fits only those Claude models. check_thinking_levels.py checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.

Fireworks AI and Together AI: thinking levels

Neither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes xhigh, max and minimal on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.

Featherless AI

Serverless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:

• json_instructions is the only usable structured output mode. LiteLLM's featherless_ai provider rejects response_format outright (UnsupportedParamsError), so json_schema, json_mode, and json_instruction_and_object are all unavailable. Routing it as a custom openai provider bypasses that gate, but was tested and is not reliable: GLM 5.2 accepts json_schema and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.
• Gated models return HTTP 403 model_gated_needs_oauth and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. Always filter is_gated before adding. Note all 20 official meta-llama/* repos are gated, so no Llama variant is usable.
• No cost reporting. Featherless models aren't in LiteLLM's price map (only two legacy Qwerky entries, on main too — not a version issue), and Featherless doesn't return cost in the usage object. Runs record tokens with cost: null.
• Not in models.dev or the LiteLLM catalog, so their /v1/models endpoint is the only authoritative source. See Lagging Providers.
• Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.

TypeSafe AI

Serves TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:

• Adapter-backed, not LiteLLM-backed. Every provider entry must set adapter=ModelAdapterId.jev; JevAdapter calls POST /v1/systemone directly. lite_llm_core_config_for_provider and get_litellm_provider_info raise for typesafe on purpose, so the LiteLLM parametrized tests exclude it.
• Structured output only. No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set supports_structured_output=True, supports_data_gen=False, supports_function_calling=False, and structured_output_mode=StructuredOutputMode.json_schema.
• No logprobs. supports_logprobs=False. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.
• Not in models.dev or the LiteLLM catalog, and GET /v1/models cannot confirm a model_id. It lists aliases (jev-latest, jev-preview), not the pinned versions Kiln registers, and it returns {"models": [{"name", ...}]} rather than the OpenAI {"data": [{"id", ...}]} shape, so the shared fetch_openai_compat path cannot read it either. That is why typesafe is in SKIP_PROVIDERS in .agents/scripts/provider_utils.py rather than in PROVIDER_CONFIG. The authoritative check for a pinned id is a POST /v1/systemone round trip that the API accepts — that is how jev-1.13.0 was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)

Cloudflare (Workers AI)

Runs through LiteLLM's generic openai/ route against Cloudflare's OpenAI-compatible endpoint (/accounts/{id}/ai/v1/chat/completions), with an optional AI Gateway header. Kiln does not use LiteLLM's cloudflare/ provider. Read this before adding or changing any entry:

• Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models. Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to 256 output tokens (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.

• How to tell them apart: look at the model's input schema in the catalog JSON. OpenAI-format models list max_completion_tokens (and tool_choice); older models list prompt, raw and lora instead, and no max_completion_tokens:

  jq -r '.models[] | select(.task.name=="Text Generation")
    | [.name, (if (.schema.input | tostring | test("max_completion_tokens")) then "openai-format" else "OLDER - skip" end)] | @tsv' /tmp/cf_models.json

• Also skip: coding models (kimi-*-code, qwen*-coder-*), llama-guard-* (classifier), -lora bases, and models Kiln has no ModelName for unless the user asks for a new model.

• Usually a new provider on an existing model. Most Cloudflare models are already in Kiln under other providers; add a KilnModelProvider(name=ModelProviderName.cloudflare, ...) to the existing KilnModel.

• Setting flags from the catalog's properties[] (confirm every flag with the paid tests — the catalog has no JSON-mode flag):

  Cloudflare property	Kiln setting
  function_calling: "true"	leave supports_function_calling at its default; set False if the tool tests fail
  vision: "true"	supports_vision, multimodal_capable, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set supports_doc_extraction: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway
  reasoning: "true"	keep reasoning_capable=False (see Reasoning Capable Default); LiteLLM already reads reasoning_content
  reasoning_effort.supported_efforts present	required: set available_thinking_levels to exactly these values, and default_thinking_level to default_effort. See Cloudflare thinking levels
  reasoning_effort without supported_efforts	no thinking levels; the catalog does not say which values work
  require_workers_paid: "true"	no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR
  planned_deprecation_date present	do not add the model

• Thinking levels come from the catalog, and only from the catalog. Kiln sends the level as reasoning_effort. For each model, reasoning_effort in the catalog lists supported_efforts, a default_effort, and normalizes_to: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps medium to max, so a "Medium" option runs the most expensive effort.

  • Offer only values in supported_efforts, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, none still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.

  • Labels: none → "Off", low → "Low", medium → "Medium", high → "High", xhigh → "Extra High", max → "Max". List them from least to most effort.

  • default_thinking_level is default_effort, unless the entry is suggested for evals (see Judge Models Must Reason by Default).

  • mandatory: true means none is not in the list; do not add an Off level.

  • Reuse a constant when the values match exactly; else name a new one {MODEL}_CLOUDFLARE_THINKING_LEVELS.

  • Cloudflare edits these lists. After every change to a Cloudflare entry, and on every run of this skill, run the checker below. It must report no ❌ finding:

    uv run python .agents/skills/claude-maintain-models/scripts/check_thinking_levels.py

    It compares every Cloudflare and OpenRouter entry in this checkout with the provider's catalog, and exits 1 on a level or default outside supported_efforts. For Cloudflare it also exits 1 on a missing level list, or levels on a model whose catalog lists none. --provider cloudflare or --provider openrouter checks one provider. A note: line is information only.

  • Then run the paid test_thinking_level_reasoning_content for each new level. If -k "<model_enum>" collects no cloudflare/... case, cloudflare is missing from the provider list in libs/core/kiln_ai/adapters/model_adapters/test_thinking_level_paid.py; add it.

• Structured output: these models take the OpenAI response_format envelope, so start from json_schema and fall back only on test failures. Cloudflare's JSON mode doesn't support streaming.

• Don't send max_tokens. Live tests showed no truncation on the OpenAI-format models without it.

• Workers Paid: many of the newer models need the Workers Paid plan. Paid tests need an account on it.

• Rate limits: 300 requests/minute for most text models, but only 20/minute per model for Workers Paid-only models. Don't run the paid suite with many workers against one model; use -n 4 or lower.

• Slow models: Qwen 3.8 27B runs at about 23 tokens/s. Long non-streaming generations hit Cloudflare's server-side timeout (HTTP 408, code 3046, after about 2 minutes).

• No cost reporting. Kiln sends bare @cf/... ids through LiteLLM's openai/ route, which never match LiteLLM's price map (its few entries are keyed cloudflare/@cf/...), so every Cloudflare run records tokens with cost: null.

• No Model Substitution. When Cloudflare retires a model it may silently alias the old ID to a different, often more expensive, model (Kimi K2.5 → K2.6 in May 2026). The response still echoes the requested ID, so Kiln can't detect it at runtime. Cloudflare announces retirements by adding planned_deprecation_date to the model (notice has been 13–22 days). Remove or mark deprecated every Kiln Cloudflare entry that gains a planned_deprecation_date, before that date. The kiln-check-deprecation skill reports these dates.

Qwen3 / Thinking Models

• Thinking variants: reasoning_capable=True, parser=ModelParserID.r1_thinking
• No-thinking variants: formatter=ModelFormatterID.qwen3_style_no_think
• SiliconFlow may need siliconflow_enable_thinking=True/False

Thinking Levels Reference

Two providers publish the thinking levels per model, and their list is the only source for their entries: Cloudflare (see Cloudflare thinking levels) and OpenRouter (reasoning.supported_efforts in its models API, see OpenRouter (general)). For every other provider the levels must be sourced by hand. Use this lookup chain in priority order.

Lookup Chain

1. Vendor model page (most authoritative)

   • OpenAI: Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL: https://developers.openai.com/api/docs/models/{model-id}
   • Anthropic: The effort docs list levels per model. Opus 4.6 supports low, medium, high, max; Sonnet 4.6 supports low, medium, high.
   • Google Gemini: The models API returns thinking: true/false (boolean only). Levels come from docs.

2. Vercel AI Gateway docs — clean structured tables per provider:

   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/openai
   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/anthropic
   • https://vercel.com/docs/ai-gateway/capabilities/reasoning/google

3. Inherit from predecessor — if the same family/tier model has a _THINKING_LEVELS dict, the new model very likely uses the same or a superset.

4. OpenRouter reasoning — most models list their efforts:

   curl -s https://openrouter.ai/api/v1/models | jq '.data[] | select(.id == "SLUG") | {reasoning, supported_parameters}'

   reasoning.supported_efforts lists the efforts (highest first) and reasoning.default_effort the default. If reasoning is absent and supported_parameters has no reasoning, the model does not support effort levels — skip thinking levels entirely. OpenRouter documents that it maps an unsupported effort to the nearest supported one for some models, so an extra level may silently run a different effort.

5. Smoke test — as a last resort, send a request with an invalid effort level and check the error message, which often enumerates the valid values.

Judge Models Must Reason by Default

If a provider entry sets both suggested_for_evals=True and available_thinking_levels, set default_thinking_level to "medium" when the model offers a medium level, else "low". Never leave it at "none" or unset.

The V2 LLM judge runner builds the judge model's run config without a thinking_level, so the adapter falls back to the provider entry's default_thinking_level. A default of "none" therefore ships a judge that never reasons, and the eval results page shows no thinking for it.

This applies only to judge-tagged entries. A non-judge entry of the same model may keep "none" — the two are set independently per provider entry.

test_judge_models_with_thinking_levels_default_to_reasoning in libs/core/kiln_ai/adapters/test_ml_model_list.py enforces this across the whole list.

Important Distinctions

• Effort-level models (GPT-5.x, Claude 4.x, Gemini 3.x) → add available_thinking_levels dicts
• R1-style thinking models (DeepSeek R1, Qwen thinking variants) → on/off thinking, NOT effort levels. Use reasoning_capable=True + parser=ModelParserID.r1_thinking. Do NOT add thinking level dicts.
• Chat/instant models (e.g. GPT-5.3 Instant) → may not support reasoning effort at all. Verify on the vendor model page.

Existing Constants

Reuse when levels match exactly. Create a new constant only if levels differ. This is not an exhaustive list.

Sources That Do NOT Work

These were investigated and confirmed to lack thinking level data:

• LiteLLM catalog — only supports_reasoning: true/false
• Google Gemini models API — only thinking: true/false
• OpenAI /v1/models endpoint — minimal object with no capability fields

Slug Lookup Reference

Use both LiteLLM and models.dev when looking up slugs — they complement each other. LiteLLM gives you the exact slugs Kiln will use (since Kiln runs on LiteLLM), while models.dev often has broader coverage of newer or niche models with pricing, context limits, and capability details.

LiteLLM Model Catalog (https://api.litellm.ai/model_catalog)

100 free requests/day, no key needed. Supports server-side filtering: model= (substring match), provider=, mode=, supports_vision=true, supports_reasoning=true, page_size=500.

# Find all variants of a model across providers:
curl -s 'https://api.litellm.ai/model_catalog?model=MODEL_NAME&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[] | {id, provider, mode, max_input_tokens, supports_vision, supports_reasoning, supports_function_calling}'

# List all models for a provider:
curl -s 'https://api.litellm.ai/model_catalog?provider=PROVIDER&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[].id'

models.dev (https://models.dev/api.json)

Mega JSON covering 50+ providers with model IDs, pricing, context limits, capabilities, and release dates. Large file — always use curl+jq, never WebFetch.

# Search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"

# List all model IDs for a specific provider:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models | keys[]'

# Get full details for a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'

Other verified sources

• OpenRouter: curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"
• Anthropic: https://docs.anthropic.com/en/api/models/list
• Cerebras: https://inference-docs.cerebras.ai/models/overview

Lagging Providers

Fireworks, Together, SiliconFlow and Cloudflare typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, always cross-check directly — both when adding a new model and when running the Phase 1B backfill check.

Fireworks AI — model pages are the most current source. WebFetch directly:

WebFetch https://fireworks.ai/models/fireworks/{model-slug}

Or browse the catalog at https://fireworks.ai/models. Kiln slug format: accounts/fireworks/models/{model-slug}.

Together AI — the /v1/models endpoint requires an API key. The key is set in the user's shell as TOGETHERAI_API_KEY (note the AI — this matches LiteLLM's together_ai env var; the plain TOGETHER_API_KEY is usually NOT set):

# List all Together model IDs matching a term:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"

# Full record for a specific slug:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | select(.id == "SLUG")'

If TOGETHERAI_API_KEY isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.

Featherless AI — appears in neither models.dev nor the LiteLLM catalog (the catalog has only two stale Qwerky entries), so /v1/models is the sole authoritative source. It needs no API key to enumerate.

The response is enormous — ~22,000 models, the vast majority community fine-tunes. Never WebFetch it and never dump it raw. Save it once, then filter with jq:

curl -s https://api.featherless.ai/v1/models -o /tmp/feath.json
jq '.data | length' /tmp/feath.json

Restrict to official vendor orgs, or you'll drown in forks like DavidAU/Gemma3-27B-it-vl-GLM-4.7-Uncensored-Heretic:

# newest official models, excluding gated ones (gated = unusable, see Provider Quirks)
jq -r '.data[]
  | select(.is_gated | not)
  | select(.id | test("^(deepseek-ai|zai-org|Qwen|moonshotai|MiniMaxAI|mistralai|openai|google|microsoft)/"))
  | "\(.id)\tctx=\(.context_length)\t$\(.pricing.input)/\(.pricing.output)\ttools=\(.features.tool_use // false)\timg=\(.features.image_input // false)"' \
  /tmp/feath.json | sort

Look up one model, including its capability flags and exact pricing:

jq '.data[] | select(.id == "zai-org/GLM-5.2")' /tmp/feath.json

The highest-precision way to find backfill candidates is to check whether a slug Kiln already ships for together_ai / siliconflow_cn exists verbatim here — those providers use the same HuggingFace repo-id convention, so a hit needs no guessing:

jq -r '.data[] | select(.is_gated | not) | .id' /tmp/feath.json | sort > /tmp/feath_ids.txt
grep -Fx "zai-org/GLM-5.2" /tmp/feath_ids.txt   # exit 0 = safe to add

Slugs are case-sensitive (google/gemma-4-31B-it, not -31b-it). Always confirm is_gated is false and cross-check features.tool_use / features.image_input against the flags you set — don't claim capabilities the endpoint doesn't advertise.

Cloudflare Workers AI — the authoritative list is Cloudflare's public, unauthenticated catalog JSON. It is about 900 KB and can return a transient 502, so save it to a file (retry on 502) and filter with jq; never WebFetch it:

curl -s https://ai-cloudflare-com.pages.dev/api/models -o /tmp/cf_models.json

# Text-generation models, newest first, with Kiln-relevant flags
jq -r '.models[] | select(.task.name=="Text Generation")
  | (reduce .properties[] as $p ({}; .[$p.property_id]=$p.value)) as $p
  | [.name, .created_at[0:10], ($p.context_window//"-"),
     (if $p.function_calling=="true" then "tools" else "" end),
     (if $p.vision=="true" then "vision" else "" end),
     (if $p.reasoning=="true" then "reasoning" else "" end),
     (if $p.require_workers_paid=="true" then "paid" else "" end),
     (if $p.lora=="true" then "lora" else "" end),
     ($p.planned_deprecation_date//"")] | @tsv' /tmp/cf_models.json | sort -t$'\t' -k2 -r

# Full record for one model (drop the large schema)
jq '.models[] | select(.name=="@cf/zai-org/glm-5.3") | del(.schema)' /tmp/cf_models.json

Model IDs are the name field (@cf/<author>/<model>). With credentials (CLOUDFLARE_API_KEY with Workers AI access, plus CLOUDFLARE_ACCOUNT_ID), the official API returns the same records; add &include_deprecated=true to see models retired in the last three months:

curl -s "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/models/search?task=Text%20Generation&per_page=100" \
  -H "Authorization: Bearer $CLOUDFLARE_API_KEY" | jq -r '.result[].name'

• Cross-check: models.dev, provider key cloudflare-workers-ai (curl -s https://models.dev/api.json | jq '.["cloudflare-workers-ai"].models | keys[]'). It syncs hourly and is good for spotting additions, but it no longer deletes retired models, so never use it to decide a model still exists.
• Never use LiteLLM's catalog for Cloudflare. Its cloudflare/ entries are a one-time import from June 2026: they miss newer models and still list retired ones.
• Fallback: the per-model JSON files in the cloudflare/cloudflare-docs GitHub repo, src/content/workers-ai-models/*.json (same shape as the catalog).
• Also scan the Workers AI changelog for launches in the last two weeks, in case the catalog lags.

See Cloudflare (Workers AI) in Provider Quirks for the inclusion rule and flags.

SiliconFlow — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:

WebFetch https://siliconflow.com/models
WebFetch https://siliconflow.com/models/{vendor}/{model}

When you find a new reliable slug source, append it here.