Auto-detection
mistralrs run -m <model> works without flags. The binary infers the
architecture and chat template from the Hugging Face repository, and
selects the accelerator from the detected hardware.
Quickstart
Fast, flexible LLM inference.
A single binary for local inference, OpenAI-compatible serving, Anthropic-compatible Messages serving, and agentic workloads.
Start with a task
Section titled “Start with a task” Quickstart Install the CLI, chat with a model, and serve an API in ten minutes.
Serve an OpenAI-compatible API Expose a model through OpenAI-compatible endpoints and the Anthropic Messages API.
Run models Load Hugging Face models and local checkpoints with automatic architecture detection.
Run GGUF models Load a local GGUF file or select an artifact from a Hugging Face repository.
Quantize a model Fit larger models on the same hardware with prebuilt UQFF (Universal Quantized File Format) files or in-situ quantization.
Build agents Tool calling, file inputs, web search, code execution, shell, OpenAI-compatible Skills, Model Context Protocol (MCP), and sessions.
Use OpenAI-compatible Skills Upload Skills, reference them from Responses requests, and run them with shell execution.
Attach files to requests Use OpenAI-compatible uploads, input_file parts, Chat file parts, and shell/code workdir mounts.
Use Python Call models from Python without running a separate server.
Use Rust Embed mistral.rs in a Rust application.
Browse examples Runnable HTTP, Python, and Rust examples organized by topic.
Features
Section titled “Features”Quantization
For a GGUF repository, --quant selects a matching artifact. For other
Hugging Face repositories, it loads a prebuilt
UQFF when available and otherwise
applies in-situ quantization.
Quantization guide
Tuning and diagnostics
mistralrs tune recommends quantization and device mapping from the model
config and detected hardware, without loading the model. mistralrs doctor
checks hardware, build features, and Hugging Face connectivity.
CLI reference
Agents
Local agent runtime with a server-side tool loop, file inputs, web search, code execution, shell, OpenAI-compatible Skills, MCP, generated media, and sessions. Agents & tools
Multi-GPU
Scale a model across GPUs and nodes: NCCL tensor parallelism, layer/P2P device mapping, multi-node NCCL, and the ring backend. Distributed inference
Web UI
Browser chat with reasoning blocks, tool, code, and shell call visualization, and inline search results. Web UI
Reference
Section titled “Reference” CLI reference Every command and flag, generated from the binary.
HTTP API Endpoints, schemas, and streaming events.
Supported models Model families and capability tables.
GGUF compatibility Supported model families, file formats, and feature combinations.
Troubleshooting Symptom-indexed fixes for build and runtime issues.