Install, configure, verify, maintain, troubleshoot and optimize vLLM Semantic Router v0.5 on a Docker host (CPU, AMD or NVIDIA GPU) or a Kubernetes cluster (Helm or the operator), through its CLI and Router API. Use for a first install, configuration and routing changes, checks of routed requests, upgrades and the v0.5 config migration, secrets, logs, health, backup and restore, removal, failure diagnosis, and tuning of the decision model, recipes, caches, latency and cost with sr-bench.
100%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
The Router serves an OpenAI-compatible API and picks a model for each request.
Every routing judgment (a domain, a jailbreak, the complexity of a request, a
question you write) is a question to the decision model, System One, which
the Router's built-in model runtime serves; it starts a model only when a rule
or feature asks it. The models that answer users are the user's own endpoints,
such as vLLM, Ollama or a hosted API. vllm-sr serve runs the stack in Docker:
the Router (inference on port 8899, management API on 8080) with its model
runtime, the Dashboard (8700), sr-bench (8090) and supporting services.
This file routes you to a task. Each task names its preconditions, the reference that holds its steps, what success looks like, what to do when a step fails, and when to stop and ask the user. Every step is safe to run again.
vllm-sr serve without a complete --config: without one it
starts the Dashboard's setup mode and waits indefinitely. Pass every answer
as a flag; no command here prompts.api_key_env, --token-env).vllm-sr stack, services on its ports) without asking.vllm-sr config migrate), run that command if these rules allow it, then
rerun the step. Don't work around the message.VLLM_SR_STACK_NAME (default vllm-sr, which names its
containers) and moved by VLLM_SR_PORT_OFFSET (default 0, added to every
host port). Export both for every command of a second stack.vllm-sr commands from its config directory; the stack keeps
its state in .vllm-sr/ beside config.yaml.http://127.0.0.1:8899/v1, management
ROUTER_ORIGIN=http://127.0.0.1:8080, Dashboard http://127.0.0.1:8700.
Add the offset to each port otherwise.python3 -c; header names are case-insensitive, so use
grep -i.These commands change nothing. Run them and keep the results.
uname -sm
docker info --format '{{.ServerVersion}} {{.MemTotal}}'
python3 -c 'import sys; print(sys.version.split()[0])'
python3 -m ensurepip --version
df -h "$HOME" | tail -1
free -g 2>/dev/null | head -2
ls /dev/kfd /dev/dri/render* 2>/dev/null
nvidia-smi -L 2>/dev/null
vllm-sr --version 2>/dev/null && vllm-sr status| Check | Pass | Otherwise |
|---|---|---|
| Docker | prints a version | "permission denied": ask the user to add their account to the docker group. Not installed: ask before installing it. Podman works too: add --container-runtime podman to vllm-sr commands. |
| Python | 3.10 or newer, and ensurepip prints pip … | Ubuntu and Debian: with the user's OK, sudo apt-get install -y python3-venv. Without it the installer stops at "ensurepip is not available". |
| Disk | 15 GB free; 40 GB for a GPU platform | Ask the user. The images take about 4 GB (about 22 GB for ROCm) and the default decision model about 6 GB. |
| Memory | MemTotal of 12 GB or more, and 8 GB available | The default decision model holds about 7 GB on the CPU. Ask: leave the decision-model route out of the first configuration, or run the model on a GPU or a remote System One API (Optimization). |
vllm-sr | not installed, or status says Not running | A stack is already running. It is the user's: find its directory (below), verify it with Verification using its own decision and model names, and ask before changing it. |
Then check the host ports a stack publishes:
bash -c 'for p in 8899 8080 8700 8090 9190 50051 6379 5432 3000 9090 4317 16686; do
(exec 3<>/dev/tcp/127.0.0.1/$p) 2>/dev/null && echo "port $p is in use"; done'Nothing printed: pass. Otherwise ask the user, or run this stack beside the other one with a stack name and port offset.
Each running stack has one Router container: vllm-sr-router-container for
the default stack, NAME-vllm-sr-router-container for a stack named NAME.
Its directory is the parent of the .vllm-sr directory that Router mounts:
docker ps --format '{{.Names}}' | grep -- '-router-container$'
docker inspect vllm-sr-router-container \
--format '{{range .Mounts}}{{println .Source}}{{end}}' | grep -m1 '/\.vllm-sr$'| Task | When | Reference |
|---|---|---|
| Deploy | A first install on Docker (CPU, AMD, NVIDIA) or Kubernetes (Helm, the operator); GPUs, Envoy, engine mode, a second stack, a pinned image, a new backend | Deployment |
| Configure | Change models, signals, decisions, plugins, recipes or entrypoints of a stack | Configuration |
| Verify | Prove that a request takes the intended route and gets a complete answer | Verification |
| Maintain | Upgrade, migrate a configuration to v0.5, roll back, back up and restore, rotate secrets, read logs and health, remove | Maintenance |
| Troubleshoot | A step failed, or the stack misbehaves | Troubleshooting |
| Optimize | Improve quality, latency or cost: the decision model and its hardware, recipes, caches, judges, sr-bench comparisons | Optimization |
vllm-sr config schema reports "config_version": "v0.5";
serve exits 0 after ✓ vLLM Semantic Router is running (standalone gateway);
vllm-sr route probe … --expect-decision code-route exits 0; a preview of a
computer-science question returns "routing_decision": "cs-route", and
GET $ROUTER_ORIGIN/api/v1/inventory/model-runtime lists primary with
"ready": true.ROUTER_ORIGIN, or a config
directory for a new one; one stated intent per change.vllm-sr config get, edit a copy, then config validate, config plan
and config apply.config apply exits 0 and prints "applied": true;
vllm-sr config versions shows the new version; a preview of each affected
route returns the intended routing_decision.run `vllm-sr config migrate` ; Restart required needs a
serve. See the Configuration table of Troubleshooting.GET /v1/models (vllm-sr/auto by default).routing_decision and selected_model are the
intended ones; route probe exits 0 with "passed": true and
response.status 200; a traced preview has no signal_errors.vllm-sr --version;
a backup of config.yaml and .vllm-sr/ before any upgrade or migration.vllm-sr config migrate,
config versions and config rollback, backups, vllm-sr storage rotate,
logs and health, removal. An upgrade is: install the new CLI, migrate,
read every note and restore what was dropped, validate, then serve,
which keeps the old stack running until the new Router accepts the
configuration. A configuration that reads the caller's identity ran
behind Envoy, and takes --gateway extproc in all three commands.✓ Configuration migrated and its output
validates with Checked by the Router in …; after an upgrade the stack
runs, the new version prints and the probe passes again..vllm-sr/, rotating credentials, removing the CLI or images.vllm-sr logs router and report
what you saw."status": "completed" with every
target "complete": true and "failed": 0; a change is kept only when it
beats the baseline on the same cases and the stated objective.Report to the user:
vllm-sr --version, the channel, platform and gateway you chose;vllm-sr stop stops the stack while keeping the
config, .vllm-sr/, downloaded models, volumes and images;http://127.0.0.1:8899/v1 with model vllm-sr/auto, and
the decision model and its state;http://127.0.0.1:8700; from another machine, through
ssh -L 8700:127.0.0.1:8700 HOST. Its first visitor creates the
administrator account, so the user should open it now (or set
DASHBOARD_ADMIN_EMAIL and DASHBOARD_ADMIN_PASSWORD before serve);For the full contract, use vllm-sr --help, command help, vllm-sr config schema, and the running Router's GET /api/v1. The
Quickstart,
System One and the
v0.5 migration guide
describe the same stack for people.
vllm-project/semantic-router
8a1b356
Also appears in
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.