Cloud LLM benchmarks
- Models
- 245
- Providers
- 9
- Samples
- 70,341
- Median tok/s
- 32
- p90 tok/s
- 74
- Max tok/s
- 291
- Median spread
- 153%
- Median TTFT
- 1.40s
Throughput distribution · tok/s
- GPT-oss-safeguard-20b groq291 tok/s
- Qwen3.6-27B groq218 tok/s
- llama-3.1-8b groq218 tok/s
- qwen-3-32b groq173 tok/s
- Google: Nano Banana (Gemini 2.5 Flash Image) google161 tok/s
- llama-3.3-70b groq149 tok/s
- gemma-4-31b cerebras131 tok/s
- llama-4-scout groq120 tok/s
- nova-micro bedrock120 tok/s
- LFM2.5-8B-A1B together108 tok/s
- llama-4-maverick bedrock107 tok/s
- qwen-2-1.5b-instruct together102 tok/s
- qwen-3.5-35b-a3b deepinfra102 tok/s
- GPT-oss-120b fireworks98 tok/s
- llama-3.1-8b bedrock96 tok/s
- llama-4-scout bedrock96 tok/s
- nova-lite bedrock91 tok/s
- nova-pro bedrock91 tok/s
- llama-3.3-70b bedrock86 tok/s
- GPT-5.1-codex-mini openai86 tok/s
- GPT-5 Nano openai85 tok/s
- mistral-7b bedrock82 tok/s
Visible-token throughput, same basis as the table · curves normalised per model · 1 model's tail runs past the axis · 223 slower models not drawn, all 245 are in the table below
By provider
| Provider | Models | Median | p90 | Best | Spread | TTFT |
|---|---|---|---|---|---|---|
| groq | 6 | 173 | 291 | 291 | 133% | — |
| cerebras | 1 | 131 | 131 | 131 | 159% | 1.10 |
| bedrock | 22 | 54 | 96 | 120 | 110% | 0.37 |
| together | 20 | 54 | 78 | 108 | 189% | — |
| openai | 38 | 38 | 61 | 86 | 159% | 1.86 |
| fireworks | 20 | 38 | 63 | 98 | 185% | — |
| deepinfra | 124 | 26 | 48 | 102 | 162% | — |
| anthropic | 10 | 25 | 28 | 43 | 119% | 1.44 |
| 4 | 2 | 161 | 161 | 108% | 0.93 |
Throughput × spread · 245 models
Full results
245 of 245 modelsThroughput over time · shared scale
Method. A cron job calls each model's live API endpoint on a schedule and records what came back. Mean, min and max are over completed samples in the selected window, and n is how many there were. A missing value means the endpoint returned an error or the model was not yet in the catalogue.
Spread is (max − min) ÷ mean, so it measures run-to-run variation rather than absolute speed. TTFT is seconds to the first visible token; runs that emit only reasoning tokens are left out of that average. Mean counts visible output tokens where a provider reports them and falls back to generated throughput where it does not, which is why Gen is higher for models that think before answering.