Inference Engine Selection & Benchmark Load Testing
We benchmark vLLM and SGLang against your real traffic using tools such as GuideLLM, then deliver a selection conclusion, tuning parameters and hardware sizing advice.
Inference Engine Selection & Benchmark Load Testing
How to read the benchmarks
Independent benchmarks (such as Modal's LLM Engineer's Almanac) compared vLLM, SGLang and TensorRT-LLM across more than ten models and ten input/output length combinations on NVIDIA GPUs with default settings. The conclusion is blunt: the two are closely matched, and the difference is mostly which one delivers the feature you need sooner. Taking Llama 3.1 8B on a single H100 with default settings (the highest request rate at p95 while the first token still arrives within one second):
| Workload (input/output tokens) | Typical use | vLLM throughput | SGLang throughput | vLLM TTFT (p95) | SGLang TTFT (p95) |
|---|---|---|---|---|---|
| 1024 / 128 | RAG, summarisation | 17.08 req/s | 16.88 req/s | 0.52s | 0.83s |
| 128 / 1024 | Code generation | 5.34 req/s | 6.50 req/s | 0.82s | 0.87s |
| 2048 / 2048 | Long conversations | 1.23 req/s | 1.25 req/s | 0.65s | 0.74s |
Caveats: those tests used random snippets as prompts, so requests share almost no prefix and the prefix-reuse gains of chat and agent workloads are not reflected; they also run a single replica with default settings, so tuned results will differ.
Our method
- We benchmark with your own traffic — the best benchmark is your real workload;
- using tools such as GuideLLM, both engines run your actual prompts and we measure throughput and latency;
- we combine workload shape, hardware (NVIDIA / AMD / Intel / TPU) and ecosystem dependencies into a selection conclusion with tuning parameters and hardware sizing advice.