Cloud LLM benchmarks

Method. A cron job calls each model's live API endpoint on a schedule and records what came back. Mean, min and max are over completed samples in the selected window, and n is how many there were. A missing value means the endpoint returned an error or the model was not yet in the catalogue.

Lanes. A lane is one way of reaching a model: a provider, a catalogue id and a transport (a direct key, or served through OpenRouter). Since 2026-08-16 most measurement runs through OpenRouter; the direct lanes it replaced are retired, not deleted. By default the page shows live lanes and one row per model at each provider, spoken for by the lane measured most recently; All shows every lane in the window, retired ones tagged off.

Spread is (max − min) ÷ mean, so it measures run-to-run variation rather than absolute speed. TTFT is seconds to the first visible token; runs that emit only reasoning tokens are left out of that average. Mean counts visible output tokens where a provider reports them and falls back to generated throughput where it does not, which is why Gen is higher for models that think before answering. Decode is the generation rate once tokens are flowing, with time to first token and reasoning excluded; only streams that deliver a few tokens per chunk can carry it, so a dash means the stream was too coarse to time.