Projects

SGLang

• Sparse MLA prefill — [Feature] SM90 Q8KV8 FP8 Sparse MLA Prefill Kernel and SGLang NSA Runtime Integration  roadmap I opened
  • [Kernel] Add SM90 Q8KV8 FP8 Sparse MLA Prefill JIT Kernel with Tests and Benchmark · #25751
  • [DSA] Integrate Q8KV8 FP8 Sparse MLA Prefill into the DSA Backend (DeepSeek-V3.2) · #30514
  • [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.2 & DeepSeek-V3.2: Q8-Path & Shared-Path Optimizations · #31888
  • [DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend · #32327  co-author
  • FIX: (NSA) Compute topk_indices_offset when NSA prefill flashmla_sparse is used with FP8 KV cache · #20606
• KV-cache quantization — [Feature] Multiple KVCache Quantization Enhancements  roadmap
  • feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA · #10078
  • feat: Add FP4 (E2M1) KV Cache Support for MHA · #12612
  • Refactor MHA & MLA KV caches to support FP4 · #13547
  • Support kv8 (FP8) with torch_native attention backend · #12596
  • Check KV4 compatibility with attention backends and add KV4 support to the attention_backend doc · #14467
  • Add KV4-capable backend flashmla and update server args · #14989
• MoE quantization
  • [Perf][Kernel] Fuse SiLU+Mul into NVFP4 Expert Quantization for CUTLASS MoE · #18612
• Speculative decoding
  • [Speculative] Derive the speculative max_running_requests default from the decode CUDA-graph ladder · #34290
• CUDA-free inference — Enabling CUDA-Free Support for vLLM Inference Using Triton Kernels  article, first author
  • Support non-attention path operators in Triton · #4792