Environment variables

User-facing environment variables read by mistralrs or its build scripts. Standard Cargo build variables such as OUT_DIR and TARGET are omitted.

Hugging Face

Variable	Purpose
HF_HOME	Root of the Hugging Face cache. Default ~/.cache/huggingface.
HF_HUB_CACHE	Hugging Face hub cache location.
HF_TOKEN	Auth token. Overrides any token saved by mistralrs login at $HF_HOME/token.
HF_HUB_TOKEN	Auth token fallback when HF_TOKEN is not set.
HF_HUB_OFFLINE	Set to 1/true/yes/on to disable all Hugging Face Hub network calls. Files and listings are then served only from $HF_HUB_CACHE/$HF_HOME/hub, and a missing file errors out. Also skips the mistralrs doctor connectivity check.

If --token-source env:NAME is used, mistral.rs reads the environment variable named by NAME as the token source.

For the offline workflow (pre-downloading models, local paths), see run models.

Logging

Variable	Purpose
RUST_LOG	Override the tracing log filter. Examples: mistralrs_core=debug,tower_http=info, trace. CLI users can usually use -v or -vv instead.
MISTRALRS_DEBUG	MISTRALRS_DEBUG=1 enables extra debug-level engine tracing.

Quantization and loading

Variable	Purpose
MISTRALRS_NO_MMAP	MISTRALRS_NO_MMAP=1 loads safetensors without mmap.
MISTRALRS_ISQ_SINGLETHREAD	If set, runs ISQ (in-situ quantization) single-threaded.

CPU runtime

Variable	Purpose
RAYON_NUM_THREADS	Sets the default CPU worker count used by Candle and Rayon-backed CPU kernels unless a more specific variable is set.
CANDLE_NUM_THREADS	Sets Candle’s CPU worker count. This overrides the fallback from RAYON_NUM_THREADS for Candle’s own thread pools.
CANDLE_CPU_MASK	Linux only. Pins CPU worker threads to a cpulist such as 15-19 or 5-9,15-19. If no explicit thread-count variable is set, the mask size also becomes the default worker count.
CANDLE_CPU_AFFINITY	Linux only. Set to 1 to try Candle’s automatic high-capacity CPU affinity mask on heterogeneous CPUs. Default is off.
CANDLE_BARRIER_POOL_SPIN_LIMIT	Advanced CPU tuning. Overrides the spin count used by Candle’s persistent barrier pool before worker threads park.

See CPU threads and affinity for examples.

Sandbox

Variable	Purpose
MISTRALRS_SANDBOX	auto, on, or off. Overrides the sandbox only when the resolved mode is auto; on and off in CLI/TOML win. See sandbox reference.

Server and UI

Variable	Purpose
MCP_CONFIG_PATH	MCP (Model Context Protocol) client configuration path used when --mcp-config is not passed.
KEEP_ALIVE_INTERVAL	SSE (Server-Sent Events) keep-alive interval in milliseconds. Falls back to the default if missing or invalid.
MISTRALRS_ALLOW_RUNTIME_LORA_UPDATING	Set to 1, true, yes, or on to enable runtime LoRA load and unload endpoints. Disabled by default; the read-only route remains registered, but the target model must have a dynamic LoRA runtime. See LoRA adapters.
MISTRALRS_LORA_ADAPTER_ROOT	Canonical directory root allowed for runtime LoRA adapter paths. Use this whenever runtime LoRA updating is enabled in production.
XDG_CACHE_HOME	Base cache directory for web UI state. The UI uses $XDG_CACHE_HOME/mistralrs.
HOME	Fallback for web UI cache path when XDG_CACHE_HOME is not set.

CUDA acceleration

Variable	Purpose
MISTRALRS_CUDA_GRAPHS	CUDA graph acceleration is enabled by default when supported. Set to 0, false, no, or off to disable. See CUDA graphs.
MISTRALRS_DFLASH_ADAPTIVE	Enabled by default for automatic CUDA DFlash serving: use the full reserved depth for up to 8 live contexts and at most 3 drafts above that. Set to 0 or false to tune depth from measured throughput instead. Applies only when --mtp-n-predict is unset.
MISTRALRS_DFLASH_ISQ	ISQ type for DFlash drafter weights (q4k, q6k, … or none for bf16); defaults to the target’s in-situ quantization type.
MISTRALRS_FLASHINFER_DECODE	Disables FlashInfer decode acceleration when set to 0, false, no, or off. Use only for compatibility troubleshooting.
MISTRALRS_GDN_DECODE_KERNEL	Overrides the CUDA GDN decode kernel for benchmarking or troubleshooting. Accepted values are auto (default), baseline, cooperative, pipelined, vmajor4, and vmajor32; an incompatible forced kernel returns an error. The value-major kernels need compute capability 8.0 or newer.
MISTRALRS_GDN_PREFILL_KERNEL	Overrides the CUDA GDN prefill kernel. Accepted values are auto (default), vmajor1, vmajor2, vmajor4, vmajor8, legacy-chunked, cutile (cuTile builds, head dim 128), and flashinfer-sm90 (Hopper builds only).
MISTRALRS_FP8_SM90_PROVIDER	Selects the SM90 provider for compatible 128x128 blockwise FP8 weights. DeepGEMM decode is selected automatically on Linux with CUDA 12.8 or newer; set cutlass to disable it, or deepgemm/auto to select it explicitly. Kernels are prepared before graph capture, with clean CUTLASS fallback when JIT or cache preparation is unavailable.
MISTRALRS_DEEPGEMM_CACHE_DIR	Overrides the owner-private, content-versioned DeepGEMM cubin and JIT-header cache. Defaults below XDG_CACHE_HOME, then ~/.cache.
MISTRALRS_DEEPGEMM_NVCC	Path to the nvcc executable used only when a DeepGEMM kernel is absent from the cache. A populated cache does not require runtime nvcc.
MISTRALRS_NO_MLA	Disables MLA acceleration for DeepSeek V2/V3 when set to 1. Use only for compatibility troubleshooting.
MISTRALRS_GGUF_AFFINE_BACKEND	Set to on to speed up GGUF matmuls at batch sizes of 8 or more. Off by default because it keeps a second copy of the quantized weights, taking that memory from the KV cache. Worth enabling for production serving with high concurrency.
MISTRALRS_MOE_BACKEND	Overrides the MoE experts backend. fused (also native, legacy, wmma) selects the fused CUDA kernels, cutile the cuTile grouped GEMM, cutlass the CUTLASS grouped GEMM, and fast the gather-based path used on Metal, CPU, and ISQ. Unset, the fastest eligible backend is chosen; on cuTile builds, checkpoints whose experts carry 128x128 block-FP8 scales use the cuTile blockwise FP8 grouped GEMM unless fast is forced.
MISTRALRS_CUTILE_TUNE	Controls the cuTile kernel autotuner that runs during warmup on cuTile builds. auto (default) reuses a persisted tuning record whose provenance (kernel source, GPU architecture, candidate set) matches and measures candidate configs for each registered kernel shape (MoE experts, FP8 GEMMs, routed LoRA) otherwise, force re-measures, and off uses the built-in policies.
MISTRALRS_CUTILE_TUNE_CACHE	Directory holding the autotuner’s records, one JSON file per kernel, shape, and GPU model. Defaults to cutile_tune under XDG_CACHE_HOME/mistralrs, then ~/.cache/mistralrs.
CUDA_TOOLKIT_PATH	CUDA toolkit root used by cuTile for headers, tileiras, and Tile IR bytecode compatibility. Recommended when multiple CUDA toolkits are installed.
CUTILE_TILEIRAS_PATH	Path to a specific tileiras executable. Takes precedence over CUDA_TOOLKIT_PATH and automatic discovery.
CUTILE_SETUP_DIAGNOSTICS	Set to 1 to print cuTile toolkit and tileiras discovery diagnostics.
CUTILE_JIT_LOG	Set to 1 to print cuTile JIT and persistent-cache diagnostics.
CUTILE_JIT_TIMING	If set, prints per-stage cuTile compilation timings and bounds-check placement counts.
CUTILE_BYTECODE_VERSION	Advanced override for the emitted Tile IR bytecode version, such as 13.2. Normally inferred from the selected toolkit or tileiras.

Multi-GPU and multi-node

Variable	Purpose
MISTRALRS_NO_NCCL	Disables NCCL for single-machine multi-GPU inference when set to 1. Also set this for ring deployments when the binary includes NCCL.
MISTRALRS_MN_GLOBAL_WORLD_SIZE	Total NCCL tensor-parallel world size across nodes. Presence of this variable enables multi-node NCCL mode.
MISTRALRS_MN_LOCAL_WORLD_SIZE	Local NCCL tensor-parallel size contributed by each node.
MISTRALRS_MN_HEAD_NUM_WORKERS	Set on the head node: number of worker nodes.
MISTRALRS_MN_HEAD_PORT	Set on the head node: listening port for worker connections.
MISTRALRS_MN_WORKER_SERVER_ADDR	Set on worker nodes: address of the head node.
MISTRALRS_MN_WORKER_ID	Set on worker nodes: worker index (0-based).
RING_CONFIG	Path to the ring backend JSON config. Setting it selects the ring backend when built with the ring feature. If the binary also has nccl, set MISTRALRS_NO_NCCL=1 as well.

See the distributed inference guide for use.

Build-time

These are read by build scripts, not at runtime.

Variable	Purpose
MISTRALRS_METAL_PRECOMPILE	MISTRALRS_METAL_PRECOMPILE=0 skips build-time precompilation for quantization, paged attention, GDN, and SSM Metal libraries. Shader compilation then happens when each library is loaded, including during model preparation. Also accepts false, no, and off.
MISTRALRS_METAL_PLATFORMS	Limits which Metal platform metallibs are precompiled. Accepts comma-separated macos, ios, tvos, or all; defaults to all platforms. For local macOS development, use MISTRALRS_METAL_PLATFORMS=macos.
CUDA_NVCC_FLAGS	Extra compiler options passed to CUDA builds.
MISTRALRS_INSTALL_TAG	Pins the installers to a specific release tag (e.g. v0.9.5): the prebuilt is downloaded from that release, and a source build checks out that git tag. Default is the latest stable release (prebuilt) or latest master (source).
MISTRALRS_INSTALL_FROM_SOURCE	MISTRALRS_INSTALL_FROM_SOURCE=1 makes the shell and PowerShell installers skip the prebuilt download and build from the latest master (bleeding edge) instead of the latest stable release.
MISTRALRS_INSTALL_NCCL	MISTRALRS_INSTALL_NCCL=1 forces the shell and PowerShell installers to add the nccl feature for CUDA builds even if NCCL is not detected.
MISTRALRS_INSTALL_NO_NCCL	MISTRALRS_INSTALL_NO_NCCL=1 makes the shell and PowerShell installers skip the nccl feature.
MISTRALRS_INSTALL_ALLOW_CUDA_MISMATCH	MISTRALRS_INSTALL_ALLOW_CUDA_MISMATCH=1 lets a source build continue when local nvcc is newer than the CUDA version reported by the NVIDIA driver.
MISTRALRS_INSTALL_YES	MISTRALRS_INSTALL_YES=1 auto-confirms every installer prompt (non-interactive installs for CI/containers; used by mistralrs update).
MISTRALRS_INSTALL_IGNORE_FFMPEG	MISTRALRS_INSTALL_IGNORE_FFMPEG=1 skips the installer’s FFmpeg step, leaving any existing FFmpeg untouched.
MISTRALRS_GIT_REVISION	Git revision embedded in the binary by the build script.

Internal

Not intended for direct use.

Variable	Purpose
__MISTRALRS_DAEMON_INTERNAL	Set by the engine on spawned worker processes.