Unclaimed project

Are you a maintainer of mistral.rs? Claim this project to take control of your public changelog and roadmap.

Changelog

mistral.rs

Fast, flexible LLM inference

EricLBuehler/mistral.rs·

6.8k551RustMIT

llmrustuqff

Last updated 5 months ago

Back to changelog

NewJune 10, 2025

v0.6.0

Dockerfiles (CUDA, CPU): https://github.com/EricLBuehler/mistral.rs/pkgs/container/mistral.rs
PyPi packages (no features, cuda, mkl, metal, accelerate)

More Rust Projects

rustdesk

An open-source remote desktop application designed for self-hosting, as an alternative to TeamViewer.

118.5k

Rust

rust

Empowering everyone to build reliable and efficient software.

114.8k

Rust

tauri

Build smaller, faster, and more secure desktop and mobile applications with a web frontend.

109.2k

Rust

deno

A modern runtime for JavaScript and TypeScript.

108.2k

Rust

View all Rust projects →

Fix handling of Metal fused attn head dims by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1234
Support paged attn for vision model rust api by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1235
[Breaking] Support setting HF cache path by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1237
Support tool calling for DeepSeek models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1239
Server image processing refactor by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1244
Optimized CUDA RoPE kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1247
Typo fix (add_speial_tokens to add_special_tokens) by @edwko in https://github.com/EricLBuehler/mistral.rs/pull/1246
Fixes for UQFF + distributed layers by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1250
Automatic agentic search integration (web_search_options) by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1243
Format kernels by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1251
Add quantize guards for UQFF deserialize by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1252
Refactor cuBLASlt-related code by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1253
Update deps, bump pyo3 version by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1259
Faster cuda FP8 performance by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1257
Rust 1.86 clippy by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1260
Refactor engine arch by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1262
Revamped LoRA support - removing the Ordering system by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1263
Fast Metal-specific quantization method: AFQ by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1264
Support prequantized models from MLX by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1265
Automatic ISQ to select fastest & most accurate method by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1266
Improved usage metrics by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1267
Bump tokio from 1.44.1 to 1.44.2 by @dependabot in https://github.com/EricLBuehler/mistral.rs/pull/1270
Gather MM ops in mistralrs-quant by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1272
Improve performance of deepseek models by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1274
Implement Llama 4 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1268
Fixes for Llama 4 UQFF loading by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1275
Support sharding for UQFF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1276
Fix bug for group-topk (group_limited_greedy) in deepseek models by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1278
Support the DeepCoder model by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1279
Improved PagedAttn scheduling accuracy by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1282
Fixes for scheduling image seqs with pagedattn by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1283
update to llguidance 0.7.16 by @mmoskal in https://github.com/EricLBuehler/mistral.rs/pull/1284
Update dependencies by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1286
Much faster image inputs processing by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1289
Add more SDPA head dims for much faster SIGLIP by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1290
Show throughput in interactive mode by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1291
Unify bitwise operations by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1288
Multimodal prefix caching support! by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1298
Interactive mode improvements by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1299
Add the Qwen 3 and Qwen 3 MoE models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1285
Revamped and streaming web search support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1301
Handle vision messages or different tool call prefixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1302
Simplify prefix cacher by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1305
Use rustyline to handle non-ascii in interactive mode by @beeender in https://github.com/EricLBuehler/mistral.rs/pull/1306
Add more tools for automatic search by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1307
Fix CPU hogging in interactive mode by @beeender in https://github.com/EricLBuehler/mistral.rs/pull/1309
Add Metal precompilation support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1311
Reduce thrashing of Metal autorelease by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1313
make AdapterPaths and LoraAdapterPaths public by @Slowki in https://github.com/EricLBuehler/mistral.rs/pull/1314
Refactor KV cache manager by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1315
Add Audio and Speech model categories by @Slowki in https://github.com/EricLBuehler/mistral.rs/pull/1317
Remove has_conv2d from vision model API by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1318
Unified/automatic flash attention enabler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1319
Fix cublaslt 4d mask by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1320
Qwen VL models fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1322
Fixes for all vision models by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1323
Improved+faster LRU prefix cacher and sampler! by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1321
Inplace ISQ support and default to mmap by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1277
Fix typos by @omahs in https://github.com/EricLBuehler/mistral.rs/pull/1329
Fix Idefics 3 arch chat templating by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1330
Remove two spaces from PR comment by @szepeviktor in https://github.com/EricLBuehler/mistral.rs/pull/1331
Add automatic vision loader type by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1332
Add the Dia 1.6b TTS model! by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1304
update llguidance to 0.7.20 by @Slowki in https://github.com/EricLBuehler/mistral.rs/pull/1334
Add model category <> messages check by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1335
Improve normalization integration test by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1340
Fix streaming example print statement by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1339
Fix normalization formula in comment by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1338
Fix image_to_pixels for non-RGB images by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1337
Fix typo in expect messages by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1342
Don't use mmap on cuda by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1336
Support AWQ format models by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1350
Fix uqff dummy layer ISQ application by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1351
Disable immediate isq if write_uqff by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1352
Fixes for cuda UQFF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1354
Refactor Option references for model paths by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1347
Add a script for server benchmarking by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1355
Optimized Metal qmv_fast path by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1356
Fast sampler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1327
Fix metal parallel sampling by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1357
Add immediate isq predicates for qwen3 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1358
Regressions fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1359
Revamped and smaller readme by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1360
Add a web chat app by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1362
Add chat history support to web chat app by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1363
Refactor web chat, fix multichat image restore by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1364
Fix repeated immediate isq init by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1365
Fix missing vision weights in Mistral3 UQFF by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1366
Rolling shard creation for uqff files by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1367
Fix unstability during isq of afq by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1368
Support web chat file uploading by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1370
Add speech generation support to the web chat! by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1373
Prefix caching for PagedAttention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1369
Metal PagedAttention accuracy improvements by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1374
Handle images in paged attn scheduler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1375
Include schemas needed for chatcompletions endpoint by @matthewhaynesonline in https://github.com/EricLBuehler/mistral.rs/pull/1353
Fix case where prefix cacher returns no toks by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1377
Faster UQFF serialization by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1379
Experimental AFQ on CPU support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1380
Add CPU flash attention by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1382
Refactor attention backends by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1384
Set MacOS thread affinity for cpu attn by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1385
Faster Qwen 3 MoE support on Metal by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1387
Fix PagedAttention block leaks by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1388
Fix cuda build again by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1389
Bump version to 0.6.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1390
Fewer .contiguous calls for qwen3 moe by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1391
Allow speech models to accept batched inputs by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1393
Ring distributed backend for Metal by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1238
Add auto loader for vision/text detection by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1402
Proposal: Create Mistral.rs Server Core Lib by @matthewhaynesonline in https://github.com/EricLBuehler/mistral.rs/pull/1346
Support linear rope for llama3 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1408
Fix vllama4 uqff loading by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1409
Handle receiver disconnects by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1410
Fix Qwen3 MoE device mapping irregularities by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1411
Fix interactive mode URL parsing by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1412
Refactor auto device map by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1413
Enable runtime sampling tweaks in interactive mode by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1414
Gumbel sampling for fast sampler by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1416
Improved CPU flash attention accuracy & performance by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1417
Provide chat_templates to container users by @sempervictus in https://github.com/EricLBuehler/mistral.rs/pull/1419
Faster cpu flash attn by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1418
Web search improvements (bm25, web chat) by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1420
Propely handle consecutive searches by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1421
Update docs by @matthewhaynesonline in https://github.com/EricLBuehler/mistral.rs/pull/1422
Better tool call detection logic by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1424
Add web search hook callbacks by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1426
Fix CUDA context switching, bind thread on CudaStorage drop by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1428
Conditionally build seqlens tensors by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1429
Add AGENTS.md by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1430
Support QWen3 GGUF model by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1432
Improved paged attn prefix caching by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1434
Temporary fix for qwen3 gguf tokenizer by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1433
Add tool callback support by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1427
centralize crate dependencies by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1438
Fix bug in tokenizer created with gguf metadata by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1440
Update deps by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1441
Doc fixes by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1442
Downgrade rustyline 16.0.0 -> 15.0.0 by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1444
Support max_completion_tokens alias by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1451
Add the conformer backbone (phi4mm audio) by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1448
Fix offline cache issue for gguf models by @guoqingbao in https://github.com/EricLBuehler/mistral.rs/pull/1452
Add MCP server endpoints by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1453
MCP documentation pass by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1455
Integrate an MCP client by @EricLBuehler in https://github.com/EricLBuehler/mistral.rs/pull/1456

mistral.rs

v0.6.0

More Rust Projects

rustdesk

rust

tauri

deno

v0.6.0

🔥 Highlights from v0.6.0

What's Changed

New Contributors

More Rust Projects

rustdesk

rust

tauri

deno