v3.8.0
Welcome to LocalAI 3.8.0 !
LocalAI 3.8.0 focuses on smoothing out the user experience and exposing more power to the user without requiring restarts or complex configuration files. This release introduces a new onboarding flow and a universal model loader that handles everything from HF URLs to local files.
We’ve also improved the chat interface, addressed long-standing requests regarding OpenAI API compatibility (specifically SSE streaming standards) and exposed more granular controls for some backends (llama.cpp) and backend management.
📌 TL;DR
| Feature | Summary |
|--------|--------|
| Universal Model Import | Import directly from Hugging Face, Ollama, OCI, or local paths. Auto-detects backends and handles chat templates. |
| UI & Index Overhaul | New onboarding wizard, auto-model selection on boot, and a cleaner tabular view for model management. |
| MCP Live Streaming | New: Agent actions and tool calls are now streamed live via the Model Context Protocol—see reasoning in real-time. |
| Hot-Reloadable Settings | Modify watchdogs, API keys, P2P settings, and defaults without restarting the container. |
| Chat enhancements | Chat history and parallel conversations are now persisted in local storage. |
| Strict SSE Compliance | Fixed streaming format to exactly match OpenAI specs (resolves issues with LangChain/JS clients). |
| Advanced Config | Fine-tune context_shift, cache_ram, and parallel workers via YAML options. |
| Logprobs & Logitbias | Added token-level probability support for improved agent/eval workflows. |
Feature Breakdown
🚀 Universal Model Import (URL-based)
We have refactored how models are imported. You no longer need to manually write configuration files for common use cases. The new importer accepts URLs from Hugging Face, Ollama, and OCI registries, or local file paths also from the Web interface.
https://github.com/user-attachments/assets/230576c2-2abe-4b20-97c0-935d4ed6e7e7
- Auto-Detection: The system attempts to identify the correct backend (e.g.,
llama.cppvsdiffusers) and applies native chat templates (e.g.,llama-3,mistral) automatically by reading the model metadata. - Customization during Import: You can override defaults immediately, for example, forcing a specific quantization on a GGUF file or selecting
vLLMovertransformers. - Multimodal Support: Vision components (
mmproj) are detected and configured automatically. - File Safety: We added a safeguard to prevent the deletion of model files (blobs) if they are shared by multiple model configurations.
🎨 Complete UI Overhaul
The web interface has been redesigned for better usability and clearer management.
https://github.com/user-attachments/assets/260a566d-8ffb-4659-b8d2-966c18b3688d
- Onboarding Wizard: A guided flow helps first-time users import or install a model in under 30 seconds.
- Auto-Focus & Selection: The input field captures focus automatically, and a default model is loaded on startup so you don't start in a "no model selected" state.
- Tabular Management: Models and backends are now organized in a cleaner list view, making it easier to see what is installed.
https://github.com/user-attachments/assets/a522d85b-9c31-4fe8-a9fa-b97c17b0bb4d
🤖 Agentic Ecosystem & MCP Live Streaming
LocalAI 3.8.0 significantly upgrades support for agentic workflows using the Model Context Protocol (MCP).
- Live Action Streaming: We have added a new endpoint to stream agent results as they happen. Instead of waiting for the final output, you can now watch the agent "think": seeing tool calls, reasoning steps, and intermediate actions streamed live in the UI.
https://github.com/user-attachments/assets/857d62d1-183c-4e2f-ad15-63450702e3d4
Configuring MCP via the interface is now simplified:
https://github.com/user-attachments/assets/161387c0-5126-416d-ae3b-e2a2e4858b89
🔁 Runtime System Settings
A new Settings > System panel exposes configuration options that previously required environment variables or a restart.
https://github.com/user-attachments/assets/a1e3e205-1ec6-4d02-a86a-370bc24a74e5
- Immediate Effect: Toggling Watchdogs, P2P, and Gallery availability applies instantly.
- API Key Management: You can now generate, rotate, and expire API keys via the UI.
- Network: CORS and CSRF settings are now accessible here (note: these specific network settings still require a restart to take effect).
Note: In order to benefit from persisting runtime settings, in older LocalAI deployments it's necessary to mount the
/configurationdirectory from the container image.
⚙️ Advanced llama.cpp Configuration
For power users running large context windows or high-throughput setups, we've exposed additional underlying llama.cpp options in the YAML config. You can now tune context shifting, RAM limits for the KV cache, and parallel worker slots.
options:
- context_shift:false
- cache_ram:-1
- use_jinja:true
- parallel:2
- grpc_servers:localhost:50051,localhost:50052
📊 Logprobs & Logitbias Support
This release adds full support for logitbias and logprobs. This is critical for advanced agentic logic, Self-RAG, and evaluating model confidence / hallucination rates. It supports the OpenAI specification.
🛠️ Fixes & Improvements
OpenAI Compatibility:
- SSE Streaming: Fixed a critical issue where streaming responses were slightly non-compliant (e.g., sending empty content chunks or missing
finish_reason). This resolves integration issues withopenai-node,LangChain, andLlamaIndex. - Top_N Behavior: In the reranker,
top_ncan now be omitted or set to0to return all results, rather than defaulting to an arbitrary limit.
General Fixes:
- Model Preview: When downloading, you can now see the actual filename and size before committing to the download.
- Tool Handling: Fixed crashes when tool content is missing or malformed.
- TTS: Fixed dropdown selection states for TTS models.
- Browser Storage: Chat history is now persisted in your browser's local storage. You can switch between parallel chats, rename them, and export them to JSON.
- True Cancellation: Clicking "Stop" during a stream now correctly propagates a cancellation context to the backend (works for
llama.cpp,vLLM,transformers, anddiffusers). This immediately stops generation and frees up resources.
🚀 The Complete Local Stack for Privacy-First AI
LocalAI |
The free, Open Source OpenAI alternative. Drop-in replacement REST API compatible with OpenAI specifications for local AI inferencing. No GPU required. |
LocalAGI |
Local AI agent management platform. Drop-in replacement for OpenAI's Responses API, supercharged with advanced agentic capabilities and a no-code UI. |
❤️ Thank You
Over 35,000 stars and growing. LocalAI is a true FOSS movement — built by contributors, powered by community.
If you believe in privacy-first AI:
- ✅ Star the repo
- 💬 Contribute code, docs, or feedback
- 📣 Share with others
Your support keeps this stack alive.
✅ Full Changelog
📋 Click to expand full changelog
What's Changed
Bug fixes :bug:
- fix(reranker): respect
top_nin the request by @mkhludnev in https://github.com/mudler/LocalAI/pull/7025 - fix(chatterbox): pin numpy by @mudler in https://github.com/mudler/LocalAI/pull/7198
- fix(reranker): support omitting top_n by @mkhludnev in https://github.com/mudler/LocalAI/pull/7199
- fix(api): SSE streaming format to comply with specification by @Copilot in https://github.com/mudler/LocalAI/pull/7182
- fix(edit): propagate correctly opts when reloading by @mudler in https://github.com/mudler/LocalAI/pull/7233
- fix(reranker): llama-cpp sort score desc, crop top_n by @mkhludnev in https://github.com/mudler/LocalAI/pull/7211
- fix: handle tool errors by @mudler in https://github.com/mudler/LocalAI/pull/7271
- fix(reranker): tests and top_n check fix #7212 by @mkhludnev in https://github.com/mudler/LocalAI/pull/7284
- fix the tts model dropdown to show the currently selected model by @ErixM in https://github.com/mudler/LocalAI/pull/7306
- fix: do not delete files if used by other configured models by @mudler in https://github.com/mudler/LocalAI/pull/7235
- fix(llama.cpp): handle corner cases with tool content by @mudler in https://github.com/mudler/LocalAI/pull/7324
Exciting New Features 🎉
- feat(llama.cpp): allow to set cache-ram and ctx_shift by @mudler in https://github.com/mudler/LocalAI/pull/7009
- chore: show success toast when system prompt is updated by @shohidulbari in https://github.com/mudler/LocalAI/pull/7131
- feat(llama.cpp): consolidate options and respect tokenizer template when enabled by @mudler in https://github.com/mudler/LocalAI/pull/7120
- feat: respect context and add request cancellation by @mudler in https://github.com/mudler/LocalAI/pull/7187
- feat(ui): add wizard when p2p is disabled by @mudler in https://github.com/mudler/LocalAI/pull/7218
- feat(ui): chat stats, small visual enhancements by @mudler in https://github.com/mudler/LocalAI/pull/7223
- chore: display file names in model preview by @shohidulbari in https://github.com/mudler/LocalAI/pull/7251
- feat: import models via URI by @mudler in https://github.com/mudler/LocalAI/pull/7245
- chore(importers): small logic enhancements by @mudler in https://github.com/mudler/LocalAI/pull/7262
- feat(ui): allow to cancel ops by @mudler in https://github.com/mudler/LocalAI/pull/7264
- feat: migrate to echo and enable cancellation of non-streaming requests by @mudler in https://github.com/mudler/LocalAI/pull/7270
- feat(mcp): add LocalAI endpoint to stream live results of the agent by @mudler in https://github.com/mudler/LocalAI/pull/7274
New Contributors
- @shohidulbari made their first contribution in https://github.com/mudler/LocalAI/pull/7131
- @mkhludnev made their first contribution in https://github.com/mudler/LocalAI/pull/7025
- @ErixM made their first contribution in https://github.com/mudler/LocalAI/pull/7306
- @filipeaaoliveira made their first contribution in https://github.com/mudler/LocalAI/pull/7330
Full Changelog: https://github.com/mudler/LocalAI/compare/v3.7.0...v3.8.0
