Throughput tuning

Start with the defaults, then change one setting at a time for your workload:

mistralrs serve -m Qwen/Qwen3-4B

For server tuning, replay the same prompt and output lengths through the HTTP API at your expected concurrency. Compare throughput and latency before changing another setting. Use mistralrs bench -m <model> for single-request measurements.

Concurrency: max running sequences

Cap the number of sequences running at once. Waiting requests queue until a slot frees. Raise it for high-concurrency serving; lower it to bound per-request latency or memory.

CLI

mistralrs serve -m Qwen/Qwen3-4B --max-seqs 64

Python

from mistralrs import Runner, Which

runner = Runner(which=Which.Plain(model_id="Qwen/Qwen3-4B"), max_seqs=64)

Rust

let model = mistralrs::ModelBuilder::new("Qwen/Qwen3-4B")
    .with_max_num_seqs(64)
    .build()
    .await?;

CLI run defaults to one concurrent sequence. CLI serve, Python, and Rust default to 32.

More running sequences need more KV cache. If the pool runs out of blocks mid-generation, the scheduler preempts a running sequence, frees its blocks, and requeues it, which costs recomputation. Lower concurrency or adjust the memory budget if this happens frequently.

Mixed prefill and decode

Three limits control how new prompts share the GPU with active decodes:

• --max-num-batched-tokens bounds the total token work in a scheduler batch (default 4096).
• --max-prefill-chunk-tokens bounds each chunkable CUDA or Metal text-prompt scheduling quantum (default 512). The budget is divided across admitted prompt sequences, limiting how long active decodes wait behind one non-preemptible prefill step. Raw-logit, adapter, and multimodal prompt paths that cannot be split remain atomic; the limit does not change CPU scheduling.
• --max-decode-steps-before-prefill bounds how long a waiting prompt can be deferred while a decode batch is active (default 8). Raise it to favor inter-token latency for active requests; lower it to reduce time to first token under sustained arrivals.

Measure both output throughput and latency percentiles when changing these values. A larger prefill chunk can improve prefill throughput while increasing decode latency, especially for long prompts.

When paged attention matters

Paged attention shares a fixed cache pool across requests and is enabled by default on supported CUDA and Metal models:

• With paged attention: continuous batching and block-level prefix sharing.
• Without it (CPU or --paged-attn off): the default scheduler is first-come-first-served and batches running sequences by equal length; sequences at other lengths wait their turn and accumulate priority.

See paged attention and KV caching for setup and cache controls.

Prefix caching

Prefix caching is most useful for multi-turn chat and shared system prompts. When measuring it, include repeated prefixes in your test traffic and compare time to first token after the first request.

--prefix-cache-n controls how many prefixes are retained (default 16; 0 disables). Python uses Runner(prefix_cache_n=...), and Rust uses .with_prefix_cache_n(Some(...)). Retaining more prefixes can help workloads that cycle through many conversations.

With paged attention, matching prefixes share cache blocks. Without paging, completed sequences’ caches are retained whole. --no-kv-cache disables both the KV cache and prefix caching; --paged-attn off keeps the standard KV cache.

Qwen3.5 hybrid models on Metal can report zero prefix-cache hits for repeated identical prompts. Reuse also requires a saved recurrent-state checkpoint, and Metal does not currently create the intermediate checkpoints needed for this case.

Memory planning for automatic device mapping

With paged attention, automatic placement reserves room for weights, temporary computation, and the KV cache together. Raising concurrency or batch limits can leave less space for the cache. See memory budgets for sizing and layer mapping for placement.

CPU threads and affinity

CPU inference uses Candle’s thread pools. On Linux systems that expose heterogeneous CPU capacity through sysfs, mistral.rs defaults the CPU worker count to the high-capacity cores. On homogeneous systems it falls back to the physical core count.

Set RAYON_NUM_THREADS or CANDLE_NUM_THREADS to override the worker count:

CANDLE_NUM_THREADS=10 mistralrs bench --cpu -m meta-llama/Llama-3.2-3B-Instruct

For CPU affinity on Linux, use CANDLE_CPU_MASK with cpulist syntax. When no thread-count env var is set, the mask size also becomes the default worker count:

CANDLE_CPU_MASK=15-19 mistralrs bench --cpu -m meta-llama/Llama-3.2-3B-Instruct
CANDLE_CPU_MASK=5-9,15-19 mistralrs serve --cpu -m meta-llama/Llama-3.2-3B-Instruct

Hard affinity is not enabled by default because it can help one CPU/model pair and hurt another. To try the automatic high-capacity CPU mask without spelling out a list, set CANDLE_CPU_AFFINITY=1. The full list is in the environment variables reference.

CUDA defaults worth knowing

On CUDA, two decode fast paths are on by default and need no tuning: FlashInfer paged kernels and CUDA decode graphs. They exist as env-var switches (MISTRALRS_FLASHINFER_DECODE, MISTRALRS_CUDA_GRAPHS) only for debugging and benchmarking comparisons; details on the paged attention page.

Quantization is the other big throughput lever, traded against quality: see quantize a model.