KTransformers v0.5.1

🚀 Core Highlights

Optimized CPU-GPU Expert Scheduling: Introducing a flexible GPU expert mask system that enables intelligent placement of MoE experts across CPU and GPU. The new scheduling system supports multiple placement strategies (frequency-based, uniform, front-loading, random) and dynamic expert updates during inference, significantly improving throughput by up to 30% at lower GPU expert ratios.
Native Precision MoE Support with CI: Expanded native precision support for FP8 and BF16 MoE models. Run Qwen3-BF16, GLM-4.7, GLM-4.7-FP8 and more models directly in their native precision without conversion overhead, now with comprehensive CI coverage.
Unified Fine-tuning & Inference Pipeline: New end-to-end tutorial for cost-effective large model fine-tuning and inference using AutoDL cloud infrastructure. Complete the full LoRA fine-tuning and inference loop for models from 14B to 235B with minimal GPU resources.

📌 Models, Hardware & Tooling

Model support updates
- Add native precision support for MiniMax-M2, MiniMax-M2.1, MiMo, DeepSeek-V3.2, GLM-4.7-FP8.
- Extend FP8 and BF16 MoE enablement path with CI validation.
Kernel & hardware improvements

🚀 Core Highlights

Optimized CPU-GPU Expert Scheduling: Introducing a flexible GPU expert mask system that enables intelligent placement of MoE experts across CPU and GPU. The new scheduling system supports multiple placement strategies (frequency-based, uniform, front-loading, random) and dynamic expert updates during inference, significantly improving throughput by up to 30% at lower GPU expert ratios.
Native Precision MoE Support with CI: Expanded native precision support for FP8 and BF16 MoE models. Run Qwen3-BF16, GLM-4.7, GLM-4.7-FP8 and more models directly in their native precision without conversion overhead, now with comprehensive CI coverage.
Unified Fine-tuning & Inference Pipeline: New end-to-end tutorial for cost-effective large model fine-tuning and inference using AutoDL cloud infrastructure. Complete the full LoRA fine-tuning and inference loop for models from 14B to 235B with minimal GPU resources.

📌 Models, Hardware & Tooling

Model support updates
- Add native precision support for MiniMax-M2, MiniMax-M2.1, MiMo, DeepSeek-V3.2, GLM-4.7-FP8.
- Extend FP8 and BF16 MoE enablement path with CI validation.
Kernel & hardware improvements

ktransformers