Build from source

Most users do not need this

The quickstart install script is the recommended way to install. It downloads a prebuilt binary for your platform; MISTRALRS_INSTALL_TAG=<tag> installs a specific release, and MISTRALRS_INSTALL_FROM_SOURCE=1 makes it build the latest master from source for you. Build from a checkout manually only to pin an arbitrary commit, apply a local patch, or use a feature combination the published binaries do not include.

Platform prerequisites

A source build needs Rust 1.94+ (rustup) plus, per platform:

• Linux CUDA: an NVIDIA driver matching the target CUDA version (nvidia-smi), the CUDA toolkit on PATH (nvcc --version), and libssl-dev + pkg-config (sudo apt install libssl-dev pkg-config, or sudo dnf install openssl-devel pkgconfig). For a non-standard toolkit location set CUDA_ROOT (e.g. export CUDA_ROOT=/opt/cuda-12.4); for runtime libraries in non-standard directories (common on HPC module systems) add them to LD_LIBRARY_PATH.
• macOS: the Xcode Command Line Tools (xcode-select --install); a full Xcode install is not required. Apple Silicon uses --features metal; Intel Macs use --features accelerate (or mkl if Intel MKL is installed).
• Windows: Visual Studio 2022 Build Tools (the rustup-init.exe installer can add these) and, for CUDA, the CUDA toolkit. If cargo build fails on long paths, enable long-path support: Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem' -Name LongPathsEnabled -Value 1 -Type DWord (the registry setting, not git config core.longpaths). Native Windows lacks the ring-backend and some experimental features; for full parity use WSL2 (wsl --install -d Ubuntu, verify nvidia-smi inside it, then follow the Linux steps).

Clone and build

git clone https://github.com/EricLBuehler/mistral.rs.git
cd mistral.rs

The CLI binary is in the mistralrs-cli crate:

# Release build in-place
cargo build --release --locked --features "cuda nccl flash-attn cudnn" -p mistralrs-cli

# Or install globally from the checkout
cargo install --path mistralrs-cli --locked --features "cuda nccl flash-attn cudnn"

The flags above target CUDA. On macOS use --features metal; on CPU omit --features entirely. See the cargo features reference for the full list.

CUDA 13.4 is supported. When multiple toolkits are installed, select the same toolkit for CUDA headers, libraries, nvcc, and cuTile before building:

export CUDA_HOME=/usr/local/cuda-13.4
export CUDA_ROOT="$CUDA_HOME"
export CUDA_PATH="$CUDA_HOME"
export CUDA_TOOLKIT_PATH="$CUDA_HOME"
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"

Use a driver that supports the selected toolkit’s PTX, and keep these exports in the runtime environment when using cuTile. For cuTile, the installer continues to select compatible prebuilt CUDA 13.2 or 13.3 binaries on CUDA 13.4-capable drivers.

Metal builds precompile macOS, iOS, and tvOS metallibs by default, including the GDN and Mamba/SSM kernels. For local macOS-only development, set MISTRALRS_METAL_PLATFORMS=macos; to skip Metal precompilation entirely, set MISTRALRS_METAL_PRECOMPILE=0.

Metal model loading prepares recurrent, supported quantized linear, and paged-attention pipelines for the model’s shapes and dtypes before serving requests. Preparation compiles pipelines without running a synthetic prompt or populating the KV cache. Request-specific variants can still require additional setup on first use. Disabling build-time precompilation moves shader compilation into library loading, including this preparation step.

The in-place build leaves the binary at target/release/mistralrs. The install variant copies it to ~/.cargo/bin/mistralrs, which is on PATH after a rustup install.

Feature flag combinations

Common per-platform flag strings:

Platform	--features
CPU	(none)
macOS / Metal	metal
CUDA	cuda flash-attn cudnn
CUDA multi-GPU	cuda flash-attn cudnn nccl

The full flag list, per-hardware recommendations, and per-flag effects live in the cargo features reference. Add nccl on Linux when NCCL is installed and you want CUDA multi-GPU tensor parallelism.

Developing against a local checkout

Use cargo build --release -p mistralrs-cli for incremental development.

Some tests are gated behind feature flags. Core test suite:

cargo test -p mistralrs-core -p mistralrs-quant -p mistralrs-vision

In the quantization crate, some tests run only with a specific backend feature enabled.

Python wheels

Building the Python SDK from source requires maturin:

pip install maturin[patchelf]
cd mistralrs-pyo3
maturin develop --release --features "cuda nccl flash-attn cudnn"

This installs the package into the current Python environment. maturin build produces a redistributable wheel under target/wheels/.

Version pinning in a consumer crate

To depend on the workspace directly (e.g., to use an unreleased change), add a git dependency:

[dependencies]
mistralrs = { git = "https://github.com/EricLBuehler/mistral.rs", branch = "master" }

For production, pin to a release tag or a specific commit SHA for reproducible builds.