New
v0.7.0
Highlights
- New CLI:
mistralrs-cli - Prefix Caching: We have implemented Prefix Caching for PagedAttention (#1750). This significantly accelerates multi-turn conversations and RAG workflows by reusing KV cache for shared prompt prefixes.
- Major model expanstion: Support for the Embedding Gemma, Qwen 3 Embedding, Gemma 3n, GLM-4, Granite Hybrid MoE, GLM-4 MoE, GLM-4 MoE Lite
- Dynamic model loading: Dynamic Model Loading: The server now supports loading and unloading models at runtime (#1828)
- Performance: Added support for CUDA 13.0/13.1 (#1767) and introduced highly optimized fused kernels (GEMV, GLU) and blockwise FP8 kernels for significant speedups on NVIDIA GPUs.
candle0.9.2: We have migrated to the official crates.io release of candle 0.9.2, stabilizing our backend dependencies!
New Models & Architectures
- Embedding models: Qwen 3 Embedding, Embedding Gemma
- Text models: GLM-4, GLM-4.7 Flash, Granite Hybrid, GPT-OSS