Deploy MiniCPM5-2B / MiniCPM5-1B with LiteRT-LM (Android / iOS / desktop / IoT)

LiteRT-LM, Google's on-device runtime built on LiteRT (formerly TensorFlow Lite). One .litertlm bundle, CPU or GPU, through a CLI, an OpenAI-compatible server, Python, Kotlin and Swift. The bundles are hosted in litert-community. Human-readable reference: docs/deployment/litert.md.

Required input

Steps

1. Install (once)

uv tool install litert-lm

2A. Run a pre-converted bundle (desktop CLI)

litert-lm run --from-huggingface-repo="${LITERT_REPO}" "${FILE}" \
    --backend "${BACKEND}" --thinking "${THINKING:-false}" --prompt "${PROMPT}"

The first run downloads the file into ~/.litert-lm/cache/huggingface/<repo>/; later runs skip the download. With thinking on, the reasoning prints between [thought] and [/thought] and the answer follows; --thinking-budget 2048 caps the reasoning (4096 for math); THINKING=true makes the model reason first. Sampling as OpenBMB recommends: --top-k 40 --top-p 0.95 --temperature 1.0 (the CLI's default top-k is 1 = greedy, so temperature alone changes nothing).

2B. Android

• No code: the AI Edge Gallery app, Model manager → + → Import from HF → paste the file's Hugging Face link.
• Your own app: implementation("com.google.ai.edge.litertlm:litertlm-android:0.17.0") from Google Maven, the libOpenCL.so <uses-native-library> entries in the manifest for the GPU, then:

val engine = Engine(EngineConfig(modelPath = path, backend = Backend.GPU(), cacheDir = context.cacheDir.absolutePath))
engine.initialize()
engine.createConversation(ConversationConfig(maxOutputToken = 1024)).use { conversation ->
    val reply = conversation.sendMessage("1+1=?")
    println(reply); println(reply.channels["thought"])
}

ConversationConfig(thinkingConfig = ThinkingConfig(enableThinking = false)) turns thinking off. Full snippet, manifest and the Galaxy S26 check: the cookbook.

2C. OpenAI-compatible server (desktop)

litert-lm import --from-huggingface-repo="${LITERT_REPO}" "${FILE}" minicpm5-2b
litert-lm serve --host 127.0.0.1 --port 9379
curl http://127.0.0.1:9379/v1/chat/completions -H "Content-Type: application/json" \
    -d '{"model": "minicpm5-2b", "messages": [{"role": "user", "content": "1+1=?"}], "reasoning_effort": "none"}'

reasoning_effort none = direct answer, any other value = thinking on; the cap field is max_completion_tokens; top_k is accepted alongside temperature / top_p / seed; "stream": true streams (server guide).

3. Validate

The reply contains 2 for 1+1=?: with --thinking false it is the whole reply; with thinking on it follows [/thought]. Through the server (2C), the same check is HTTP 200 with choices[0].message.content containing 2, and GET /v1/models lists minicpm5-2b.

Common pitfalls

• Thinking on the int4 file runs long chains: pass --thinking false for direct answers, or use the int8 file when the reasoning has to complete.
• int8 on iOS needs the increased-memory-limit entitlement (2.33 GB weight section; the default entitlements do not map it). int4's 1.28 GB main section is under the limit (checked on iPhone 17 Pro, GPU and CPU).
• First run per backend is slow: an XNNPACK weight cache is written beside the file on the CPU, kernels are compiled on the GPU (--cache no to skip the disk cache).
• Android GPU without the manifest entries: initialize() succeeds, the first message fails with Can not find OpenCL library on this device.
• Temperature without top-k does nothing: pass --top-k above 1 together with --top-p / --temperature (the default is greedy).
• Scripts: the CLI reads stdin; redirect it (< /dev/null) when running from a job with an open pipe.

Reference

docs/deployment/litert.md