Dynamo 0.7.0 focuses on production‑grade serving from configuration to deployment, infrastructure modernization, and expanded multimodal and performance capabilities. In this release, the path from AIConfigurator configs through planner layouts, Grove scheduling, and Kubernetes operators is hardened for real‑world operations, while the control plane is simplified with more cloud‑native building blocks. Dynamo seamlessly supports all major LLM frameworks:
SGlang
TensorRT-LLM
vLLM
Modular KV Block Manager
We are introducing KV Block Manager as a standalone pip-installable wheel, which decouples KV cache management from the serving stack, allowing more flexible integration options across inference engines and frameworks. KVBM currently supports TensorRT-LLM and vLLM with planned SGLang compatibility, and now, as a standalone implementation, can support inference frameworks like Triton. This modularity will enable operators to build the production topology that best fits their needs.
Production‑Grade Serving
Dynamo 0.7.0 strengthens the path from config to live traffic by connecting AIConfigurator deployment specs to planner layouts and Grove’s Kubernetes‑native scheduling. AIConfigurator lets you describe models and topologies once, and Grove turns those into correctly scheduled, autoscaled pods for multi‑node and disaggregated inference in your cluster, backed by finer‑grained fault tolerance, runtime health checks, and operator/CRD lifecycle automation (AIConfigurator, Grove).
Infrastructure Modernization
This release modernizes Dynamo’s infrastructure by beginning the removal of legacy NATS and etcd dependencies in favor of HTTP/TCP request‑plane transports and native Kubernetes service discovery. A filesystem‑backed key‑value store supports simple non‑distributed deployments, while refined system port configuration and a unified service discovery interface make the stack more cloud‑native, predictable to operate, and easier to test across environments.
Multimodal, Performance, and Framework Enhancements
Dynamo 0.7.0 sets the stage for multimodal support with base64 and HTTP image URL handling, unified media decoding and fetching paths, and refreshed vLLM multimodal examples for clearer setup. Performance and framework features advance with TensorRT‑LLM CUDA graphs and SGLang warmup optimizations.
First-Time Contributors
@YAMY1234 contributed a PR that adds automated prefill warmup for SGLang that reduces initial TTFT (#4058)!
@lynnmatrix contributed a PR that fixes a syntax error in dynamo.code-workspace (#4055)
Major Features & Improvements
Infrastructure Modernization
NATS Removal
Alternative Request Plane Transports: Implemented HTTP and TCP transport alternatives for the request plane via DYN_REQUEST_PLANE environment variable (#4307), enabling flexible deployment without NATS dependency and improving system simplicity.
Port Allocation Refactor: Removed port reservation system in favor of manual specification for more predictable deployments (#4142).
System Port Configuration: Deprecated DYN_SYSTEM_ENABLED in favor of DYN_SYSTEM_PORT for clearer configuration semantics (#4082).
ETCD Removal
Kubernetes-Based Service Discovery: Added native Kubernetes service discovery via EndpointSlices and metadata endpoint (#4136, #4150), enabling the Inference Gateway's Endpoint Picker to discover Dynamo workers without etcd for streamlined cloud-native deployments.
Operator Discovery Backend Configuration: Added --discovery-backend flag to the Kubernetes operator enabling selection between etcd (default) and Kubernetes native service discovery, supporting migration away from etcd dependency (#4268).
Filesystem KeyValueStore: Implemented filesystem-backed KeyValueStore for non-distributed deployments (#4138).
Multimodal
Multimodal Support: Added base64 and HTTP image URL support to vLLM workers for multimodal inference (#3967, #4114), image decoder in the frontend for preprocessing multimodal inputs (#3971), vLLM multimodal example for improved clarity and maintainability (#3634), media decoder and fetcher options in the MDC for flexible multimodal handling (#4094), media URL passthrough in OpenAI preprocessor for streamlined multimodal request handling (#3733), and multimodal example port allocation that match vLLM components for consistency (#4163).
OpenAI API
OpenAI API Compatibility: Added support for skip_special_tokens parameter in /v1/completions and /v1/chat/completions endpoints for flexible token handling (#4175) and rejection of unsupported parameters with 400 Bad Request response for /v1/chat/completions endpoint (#4021) and /v1/completions (#4140).
OpenAI API Batch Completions: Enhanced HTTP completion endpoint to accept arrays of prompts and generate multiple completions per prompt for improved throughput (#3953).
Version Upgrades
CUDA 13 Support: Upgraded TensorRT-LLM containers to CUDA 13.0-based images, enabling compatibility with next-generation GPU architectures and the latest CUDA toolkit features (#4405).
TensorRT-LLM 1.2.0rc2: Updated TensorRT-LLM to version 1.2.0rc2 with enhanced engine initialization, improved sampling parameter handling, and better ARM64 support (#4405).
SGLang 0.5.3.post4: Updated SGLang to version 0.5.3.post4 with performance improvements, bug fixes, and enhanced compatibility for distributed serving (#4227).
Performance and Framework Support
TensorRT-LLM CUDA Graphs: Implemented ForwardPassCallback API from TensorRT-LLM to register end of forward pass callbacks, enabling CUDA graphs for improved inference performance (#3297).
SGLang Warmup Optimization: Added dummy warmup request for SGLang prefill workers to reduce cold start latency (#4058).
Fault Tolerance & Observability
ETCD Resilience: Implemented ETCD high availability client failover with lease keep-alive resilience for improved distributed coordination reliability (#3868) and lease watch resilience to handle ETCD server failures gracefully (#3950).
Runtime Health Checks: Added --runtime-check flag to sanity_check.py for validating runtime environment configuration (#4102).
CPU Metrics Dashboard: Added CPU metrics to Grafana Dynamo Dashboard for comprehensive system monitoring (#3908).
Frontend Metrics: Added an output token counter to frontend metrics to improve usage tracking and observability (#4231).
vLLM Prefill Metrics: Added prefill worker metrics support for vLLM to improve observability of prefill operations (#3949).
Graceful Import Error Handling: Added try-except blocks with AttributeError handling for optional dependency imports to prevent crashes when modules are installed but missing expected attributes (#4392).
Kubernetes Deployment
Operator Deployment and Isolation: Enabled cluster-wide and namespace-restricted operators to coexist in the same cluster, updated dynamoNamespace to use Kubernetes namespace plus DGD name, and removed cluster-wide logic from namespace-restricted operators to improve security and multi-tenancy (#3966, #4126, #3934).
Operator and CRD Lifecycle Automation: Introduced the dynamoModel Custom Resource Definition for model lifecycle management and added CI checks to ensure operator code generation keeps CRDs and generated code in sync (#4166, #4139).
Profiler Storage Migration to ConfigMaps: Migrated profiler result storage from PVCs to ConfigMaps, removed deprecated PVC manipulation scripts, and eliminated PVC logic from the profiler planner to simplify deployment and reduce storage overhead (#3981, #4206, #4210).
Profiler Analysis Enhancements: Extended the profiler with cost plots and parallelization filters to support economic analysis and evaluation of different parallelization strategies for deployments (#4003, #4144).
Profiler Job Security Hardening: Added FSGroup to ProfilingJob pod security context to ensure proper volume file permissions for non-root users, improving security posture and preventing permission-related failures (#4334).
Long Prompt Testing: Added Kubernetes tests for prompts exceeding sequence length limits (#3930).
GAIE Recipe Simplification: Streamlined GAIE recipe configuration for easier deployment and maintenance (#3829).
KV Block Manager + Router
KVBM Router Integration: Added KV event consolidator for KVBM (vLLM) and router integration to enable efficient KV cache management across distributed workers (#3725).
Scheduling
Router
Lock-Free Router: Removed distributed locks dependency from Router for improved performance and reduced complexity (#4113).
vLLM Mocker Support: Added vLLM Mocker support for testing disaggregated architectures without full backend deployment (#3833) and implemented efficient serving of multiple mocker backends to improve testing and development workflows (#3997).
Planner
SGLang MoE Planner: Added support for TEP/DEP (Tensor/Data Expert Parallelism) for DeepSeek-R1 in SGLang MoE Planner (#4203).
MoE Expert Detection: Enhanced planner to automatically grab num_experts info from model metadata when available (#4060).
Other
Runtime API Privacy: Made Runtime and DistributedRuntime fields private to enforce proper encapsulation (#4193).
Triton Model Configuration: Added support for Triton model config specification in TensorModelConfig for fine-grained backend control (#3874).
NIXL Metadata Encoding: Added base64 encoding for nixl_connect metadata to support binary data transmission (#3843).
Python GIL Optimization: Released GIL for publisher to improve Python threading performance (#3747).
Documentation
DeepSeek-R1 Recipes: Restructured DeepSeek-R1 recipes and added GB200 configuration for next-generation hardware support (#3891).
Container Documentation: Updated container paths and interactive mode usage in READMEs for improved developer experience (#3927).
KVBM Troubleshooting: Updated KVBM runbooks with comprehensive troubleshooting section (#4170).
Backend File Organization: Moved backend deployment, launch, and SLURM files from components to examples for better project structure (#3849).
Observability Consolidation: Consolidated observability files including OTEL docker-compose and documentation for improved organization (#4173).
Non-Root Container Documentation: Updated README with comprehensive documentation on non-root user architecture, container security contexts, and UID/GID mapping for runtime, dev, and local-dev build targets (#4300).
vLLM Multimodal EPD Guide: Updated vLLM multimodal documentation with detailed EPD (Encode-Prefill-Decode) deployment workflows, component architecture, and end-to-end examples for disaggregated multimodal inference (#4352).
Recipe Documentation Cleanup: Cleaned up incomplete recipes with proper status indicators, clarified Kubernetes-only deployment focus, and documented missing components to improve user experience and set clear expectations (#4424).
Qwen3-235B-A22B-FP8 Recipes: Added production-ready TensorRT-LLM recipes for Qwen3-235B-A22B-FP8 model with both aggregated and disaggregated deployment configurations, including optimized MoE parallelism settings and FP8 quantization for efficient large-scale inference (#4179, #4401).
Bug Fixes
KVBM Memory Leak: Fixed GPU memory leak in KVBM that could cause out-of-memory errors in long-running deployments (#4171).
KVBM Dependency: Removed hard dependency on KVBM to allow flexible deployment configurations (#4127).
Chat Completion Whitespace: Fixed issue where whitespace was incorrectly trimmed from end of each chat completion delta in streaming aggregator (#4152).
Tool Call Deserialization: Fixed deserialization of tool call arguments to properly handle structured outputs (#4176).
Stream Finish Reasons: Fixed streaming responses to eliminate multiple finish reasons being sent (#4154).
Multi-turn Generation Prompt: Fixed multi-turn bug in should_add_generation_prompt logic that affected conversation handling (#4168).
KVBM Accuracy Test: Fixed flaky KVBM disaggregated accuracy test to improve test reliability (#4135).
ETCD Client Consistency: Updated KeyValueStore to use Dynamo's ETCD client instead of library client directly for consistent error handling (#4199).
Service Discovery Refactor: Refactored components to use service discovery interface for improved modularity and testability (#4092).
vLLM Data Parallel Port Conflicts: Fixed port allocation in vLLM data parallel workers where ZMQ endpoints were not properly offset per dp_rank, causing "Address already in use" errors in multinode deployments (#4321).
TensorRT-LLM Multinode SSH Keys: Fixed permission issues preventing non-root containers from writing SSH keys to mounted volumes, blocking MPI-based multinode communication (#4364).
TensorRT-LLM LeaderWorkerSet Support: Fixed TensorRT-LLM multinode deployments with LeaderWorkerSet by correcting pod templates for Kubernetes-native orchestration (#4477).
Metrics Publisher CPU Usage: Eliminated CPU-intensive tight loop in NATS metrics publisher by implementing event-driven publishing with proper timer resets (#4028).
SLA-Driven Autodeployment: Fixed profiler sidecar configuration handling, result collection, and deployment orchestration for reliable automated profiling and scaling (#3932).
Llama-3-70B Model Download: Fixed model download failure by correcting HuggingFace model paths and PVC configuration in llama3-70b-agg recipe (#4290).
vLLM Disaggregated Launch: Fixed launch script incorrectly enabling KV routing by default, which caused conflicts in pure prefill-decode disaggregation (#4063).
Non-Streaming Response Usage: Added missing usage field to non-streaming OpenAI responses for API compliance and token tracking (#3922).
Streaming Usage Events: Fixed streaming responses to set annotated_usage.event to None in final usage chunks for OpenAI API format compliance (#4446).
Known Issues
Grafana Dashboard - CPU Metrics Missing in Minikube Deployments
CPU metrics on the Grafana dashboard may be missing in Minikube deployments due to default kubelet configuration. Minikube's default PodAndContainerStatsFromCRI=false setting prevents cAdvisor from exporting container-level metrics required by the dashboard. Workaround: Start Minikube with minikube start --feature-gates=PodAndContainerStatsFromCRI=true to enable container-level metrics. This is not a Dynamo issue but a Minikube environment configuration limitation.
DeepSeek-R1 models with tensor parallelism (TP > 1) may fail during initialization with collective synchronization mismatch errors due to an upstream SGLang issue (SGLang #13227).
SGLang DSR1 Recipe - Model Loading from PVC Failure
When deploying the SGLang DSR1 recipe or using it as a base config in the SLA Profiler, the prefill and decode workers may fail to initialize and enter a CrashLoopBackOff state. This occurs because the model-download script places the model into a non-standard Hugging Face directory that cannot be loaded by ModelExpress. Workarounds:
Recommended: Download the HF model into a standard HF directory and set the HF_HOME environment variable to the PVC mounted path .
Provide a valid HF_TOKEN so that the model can be successfully downloaded directly.
Update --model-path to point at the directory containing the downloaded HF cache. Note: Workaround #3 is not compatible with the SLA Profiler as it currently does not support local paths.
When using the NIXL backend for disaggregated serving with SGLang (multi-node), initialization may fail with gdr_pin_buffer failed and NIXL_ERR_BACKEND errors on certain cluster configurations (e.g., environments with specific GDRCopy limitations). This typically manifests as a nixlBackendError during the startup of prefill/decode workers. Workaround: Set the environment variable UCX_TLS=^gdr_copy on the worker nodes to disable the GDRCopy transport for UCX, which resolves the initialization failure.
TensorRT-LLM pip Package Incompatibility with KVBM Wheel
As of November 26, 2025: Publicly available TensorRT-LLM pip package (v1.0.0) is incompatible with the KVBM wheel as it lacks the required connector interface present starting in TensorRT-LLM v1.1.0rc5. Recommendation is to use the TensorRT-LLM v1.1.0rc5 or later container for KVBM wheel installations.
TensorRT-LLM EPD Disaggregated Deployment Issue for Multimodal Support in Qwen2.5
Dynamo EPD disaggregated deployments (using epd_disagg.sh) with TensorRT-LLM versions 1.2.0rc2 and 1.2.0rc3 will encounter engine loading failures for Qwen2.5 models (including Qwen2-VL-7B-Instruct) with the error AttributeError: 'Qwen2VLInputProcessorBase' object has no attribute 'image_max_dim'. This is a known upstream TensorRT-LLM issue (TRT-LLM #8916, fixed in TensorRT-LLM 1.2.0rc4 and later).
TensorRT-LLM NIXL Backend Limitation with Disaggregated Deployments
Dynamo 0.7.0 ships with TensorRT-LLM 1.2.0rc2, which has incompatibility with NIXL as TensorRT-LLM’s KV cache transfer backend for disaggregated deployments. UCX is the only backend that will work. The primary root cause is version mismatch between NIXL 0.5.0 (used by TensorRT-LLM 1.2.0rc2) and the newer NIXL 0.7.1 version required by Dynamo 0.7.0. Here are specific cases and/or workarounds for this incompatibility:
Enable UCX as Backend for TensorRT-LLM KV Cache Transfer: To ensure stable operation with UCX, users should set environment variables cache_transceiver_config.backend: DEFAULT (in TensorRT-LLM 1.2.0rc2, default backend is NIXL), TRTLLM_USE_UCX_KVCACHE=1 (enable UCX for KV cache transfer) and OMPI_MCA_coll_ucc_enable=0 (avoid Python pickle response truncation issues). This is a known upstream issue that is fixed in TensorRT-LLM 1.2.0rc3 and later.
Manual Rebuild for Native NIXL Support: Users who specifically require NIXL functionality for disaggregated deployments should rebuild containers using TensorRT-LLM 1.2.0rc3, which contains the necessary NIXL version (0.7.1+) and is confirmed to be compatible with Dynamo 0.7.0. Rebuild command: ./container/build.sh --framework trtllm --tensorrtllm-git-url https://github.com/NVIDIA/TensorRT-LLM.git --tensorrtllm-commit v1.2.0-rc3. This eliminates the need for UCX fallback and provides full NIXL backend functionality. The correct setting for the environmental variable for rebuilt containers with TensorRT-LLM 1.2.0rc3 is TRTLLM_USE_UCX_KVCACHE=0. Important: TensorRT-LLM versions newer than rc3 (including the main branch) introduce breaking API changes and are NOT compatible with Dynamo 0.7.0.
Version Compatibility Requirements: Only TensorRT-LLM 1.2.0rc2 (default in Dynamo 0.7.0) and 1.2.0rc3 (manual rebuild) are supported with this release. Do not upgrade to TensorRT-LLM versions newer than rc3, as they introduce breaking API changes incompatible with Dynamo 0.7.0. Users requiring both disaggregated deployments and NIXL backend should use the rc3 rebuild option.
What’s Next
As we look ahead to Dynamo v0.8.0, we are focused on stability and expanding our KV Block Manager capabilities. Planned highlights include:
Dynamo v0.8.0:
Enhanced stability of all Dynamo components with targeted bug fixes and comprehensive performance improvements.
Improvements for KV Block Manager including object store support for distributed cache persistence and KVBM + Router support for TensorRT-LLM.
In Q1 2026
Looking into the new year, we will expand Dynamo's capabilities to support emerging AI workloads:
Agentic Workflows: Native support for multi-step reasoning and tool-calling patterns
Multimodal Serving: Enhanced support for vision-language models and multimodal pipelines
Diffusion Models: Optimized serving for image generation and diffusion workloads
Generative Recommenders: Support for large-scale recommendation systems with generative AI