🚀 SGLang Model Gateway v0.2.4 Released!

We're excited to announce SGLang Model Gateway v0.2.4 – a massive release focused on performance, security, and production-ready observability!

✨ Headline Features

⚡ Major Performance Optimizations

We've invested heavily in performance across the entire stack:

• Optimized radix tree for cache-aware load balancing – Smarter routing decisions with lower overhead
• Tokenizer optimization – Dramatically reduced CPU and memory footprint during tokenization
• Core module optimization – HTTP and gRPC routers now run leaner and faster
• Efficient OTEL implementation – Production-grade observability with minimal performance impact

🔌 Industry-First WASM Middleware Support

Programmable middleware using WebAssembly! Extend your gateway with safe, isolated plugins. Build custom routing logic, transform requests/responses, or integrate proprietary systems – all without touching core code. Your gateway, your rules.

📊 Production-Grade Observability

Full OpenTelemetry integration with distributed tracing for both HTTP and gRPC. Track requests across your entire inference stack with native trace context propagation. Finally, real visibility into your LLM infrastructure.

⚡ Built for speed. Hardened for security. Ready for production.

Gateway Changes (98 commits)

• [model-gateway] release gateway 0.2.4 (#14763) by @slin1237 in https://github.com/sgl-project/sglang/pull/14763
• [Perf] Optimize radix tree for cache-aware load balancin (#14758) by @slin1237 in https://github.com/sgl-project/sglang/pull/14758
• [SMG] perf: optimize tokenizer for reduced CPU and memory overhead (#14752) by @slin1237 in https://github.com/sgl-project/sglang/pull/14752
• [model-gateway] optimize core modules (#14751) by @slin1237 in https://github.com/sgl-project/sglang/pull/14751
• Tiny extract select_worker_min_load (#14648) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14648
• [ci][smg] fix docker release ci and add it to pr test (#14683) by @slin1237 in https://github.com/sgl-project/sglang/pull/14683
• Tiny support sgl-router http response status code metrics (#14689) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14689
• [SMG]feat: implement TokenGuardBody for managing token return (#14653) by @jimmy-evo in https://github.com/sgl-project/sglang/pull/14653
• [model-gateway] add OTEL integration to grpc router (#14671) by @slin1237 in https://github.com/sgl-project/sglang/pull/14671
• Fix cache-aware router should pick min load instead of min tenant size (#14650) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14650
• [model-gateway] Optimize memory usage in HTTP router (#14667) by @slin1237 in https://github.com/sgl-project/sglang/pull/14667
• [model-gateway] fix WASM arbitrary file read security vol (#14664) by @slin1237 in https://github.com/sgl-project/sglang/pull/14664
• [model-gateway] reduce cpu overhead in grpc router (#14663) by @slin1237 in https://github.com/sgl-project/sglang/pull/14663
• [model-gateway] reducing cpu overhead in various of places (#14658) by @slin1237 in https://github.com/sgl-project/sglang/pull/14658
• Fix dp-aware incompatible with service-discovery (#14629) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14629
• Super tiny fix unused code in router (#14618) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14618
• [model-gateway] fix WASM unbounded request/response body read vuln (#14612) by @slin1237 in https://github.com/sgl-project/sglang/pull/14612
• Super tiny remove unused select_worker_pair (#14609) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14609
• [model-gateway] refactor otel to be more efficient (#14604) by @slin1237 in https://github.com/sgl-project/sglang/pull/14604
• Tiny fix missing policy decision recording (#14605) by @fzyzcjy in https://github.com/sgl-project/sglang/pull/14605
• [model-gateway] fix WASM memory limit per module (#14600) by @slin1237 in https://github.com/sgl-project/sglang/pull/14600
• [model-gateway] reorganize metrics, logging, and otel to its own module (#14590) by @slin1237 in https://github.com/sgl-project/sglang/pull/14590

New Contributors

• @tonyluj made their first contribution in https://github.com/sgl-project/sglang/commit/6bad6a365
• @tom-jerr made their first contribution in https://github.com/sgl-project/sglang/commit/a95a38078
• @RiversJin made their first contribution in https://github.com/sgl-project/sglang/commit/2a5773440
• @jimmy-evo made their first contribution in https://github.com/sgl-project/sglang/commit/6f657070e
• @dcampora made their first contribution in https://github.com/sgl-project/sglang/commit/842807843
• @alisonshao made their first contribution in https://github.com/sgl-project/sglang/commit/cee93a6f2

Full Changelog: https://github.com/sgl-project/sglang/compare/gateway-v0.2.3...gateway-v0.2.4