Baseten engineers test: LLM-generated inference engines are up to 90% faster than vLLM
Baseten engineers, referencing the MetaInfer paper, used Claude Code (Fable 5) to automatically build an inference engine called VibeQwen for Qwen-3.6-35B-A3B (NVFP4, single B200 GPU). Single-stream decoding is 90% faster than vLLM 0.25.1, with time-to-first-token dropping from 28ms to 12ms, and throughput at 32 concurrency is 71% higher.
An AI-written inference engine is faster than vLLM; how can teams determine if it is usable?
Baseten’s performance article specifies a single B200, NVFP4 weights and vLLM 0.25.1, and compares throughput, time to first token and per-step decoding separately.
Reading AnalysisReturn to original text
A short introduction is retained here; the full text from the source website is not reposted. For specific wording and subsequent updates, please refer to the original article.
Open AIHOT original