Production checklist

Work through this list before a mistralrs serve deployment receives traffic from users or another service. Each item links to the page that owns the details.

Network and auth

• Bind to loopback unless the host network is private: mistralrs serve --host 127.0.0.1 --port 8080 -m <model>.
• Terminate TLS and validate credentials in a reverse proxy (nginx, Caddy, Traefik). mistral.rs has no built-in authentication - Authorization: Bearer ... headers from OpenAI clients are accepted but never validated by the server.
• Know the defaults you inherit: 50 MB request body limit, CORS allows any origin. Neither is CLI-configurable; embed mistralrs-server-core’s router builder for custom values (see embed in axum).

Reproducible startup

• Move the launch command into a TOML config and start with mistralrs from-config -f config.toml. See the TOML config reference.
• Pin versions: a container version tag (not *-latest) per the Docker guide, and be aware model ids resolve to the main revision at download time.
• Persist the model cache across restarts (volume at HF_HOME).

Resource sizing

• Run mistralrs doctor on the target host to confirm the expected accelerator and compiled features.
• Run mistralrs tune -m <model> for a starting quantization and memory plan (it estimates from configs; it does not load or benchmark the model).
• Set --max-seqs deliberately (default 32).
• Check the automatically sized KV pool under expected load. Set an explicit memory budget only when needed, and measure throughput and latency with the chosen serving limits.

Health, readiness, and metrics

• Liveness: GET /health returns 200 when the server is listening. It does not verify model load.
• Readiness: GET /v1/models includes a per-model status field (loaded, unloaded, reloading). Probe for the specific model id the caller needs, not just process liveness.
• Scrape GET /metrics: Prometheus text format with per-request counters, latency histograms, in-flight request gauges, request-body histograms, and unlabeled mistralrs_* engine metrics. Details live in observability and the HTTP API reference.
• Give startup probes a generous window; first-run model loading can take minutes.

Logging

• Default output is curated INFO startup logs, dependency warnings, and HTTP access logs for non-housekeeping requests. Use observability for request ids, Prometheus metrics, and access-log controls.
• Use -v for debug, -vv for trace, or RUST_LOG for an explicit filter.
• Only use -l/--log <path> where request and response bodies can be stored safely; it logs payloads, not just metadata.

State across restarts

• Sessions are in-memory with a 30-minute idle TTL and 128-entry capacity; they do not survive restarts. Export with GET /v1/sessions/{id} before shutdown and re-import with PUT /v1/sessions/{id} if persistence is required. See sessions.

Dynamic LoRA

• Declare adapters that must survive restart as preloads. Runtime changes do not survive a restart and must otherwise be replayed.
• Pin remote adapter revisions, preferably to commit SHAs. The CLI accepts --lora-modules '{"name":"production","path":"org/adapter","revision":"<commit>"}'.
• Keep HTTP mutation behind authentication and set MISTRALRS_LORA_ADAPTER_ROOT to an operator-owned directory. Read-only discovery does not need mutation enabled.
• Size --lora-max-adapters and --lora-max-bytes with enough headroom to replace an adapter without interrupting in-flight requests.
• Use expected_generation on replacement and unload operations, then verify the published generation through GET /v1/lora_adapters.
• Keep live mutation out of tensor-parallel deployments; preload adapters instead.

See LoRA and X-LoRA adapters for rollout, rollback, errors, and the vLLM migration table.

Multi-model

• For serving several models from one process, use mistralrs from-config with [[models]] entries and decide the default model id. See multiple models.