OpenInfer ← Benchmark

Benchmark telemetry

Sellable throughput under sustained load

Identical open-loop load driven at both fleets in parallel — a dedicated one-model-per-node baseline vs. OpenInfer co-locating every model on each node — measuring the tokens delivered within the latency SLA.

3.3×

more sellable tokens than baseline — 1.1M vs 351k delivered within the latency SLA

Latest run · 2026-09-01 20:17 UTC

Run summary avg 4.0× min 1.7× max 11.1× 83 runs 2026-08-01 → 2026-09-01

Results

Sellable tok/s

Baseline
2.9k
486.9/node
OpenInfer
9.5k
1.6k/node
Delta
3.3×

Sellable tokens

Baseline
351k
58.4k/node
OpenInfer
1.1M
190k/node
Delta
3.3×

Peak fleet GPU

Baseline
32%
OpenInfer
98%
Delta
+66 pts

Active concurrency over time

in-flight
Baseline OpenInfer
0 200 400 600 800 0s 20s 40s 60s 80s 100s 120s

Throughput over time

tok/s
Baseline OpenInfer active concurrency
0 10k 20k 30k 40k 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Fleet GPU utilization over time

% util
Baseline OpenInfer active concurrency
0 25 50 75 100 0s 20s 40s 60s 80s 100s 120s 0 200 400 600 800

Over-capacity rejections over time

rejected/s
Baseline OpenInfer
0 100 200 300 400 0s 20s 40s 60s 80s 100s 120s

Results

Totals and per node

Metric Baseline total Per node OpenInfer total Per node
Total tokens 808k 135k 1.4M 232k
Aggregate tok/s 6.7k 1.1k 11.6k 1.9k
Requests (steps) 16,598 2,766 7,058 1,176
Over capacity 12,557 75.7% 100 1.4%

Latency & throughput

Per model · p95 / tok-s / over-capacity

Model Baseline p95 / tps / over-cap OpenInfer p95 / tps / over-cap
Qwen3.5-27B 5.7s / 10 / 95.4% 3.9s / 26.9 / 0%
Qwen3.5-9B 1.1s / 42.2 / 0% 2.1s / 75.5 / 0%
Llama-3.2-1B-Instruct 3.9s / 135.8 / 30.3% 3.1s / 127.5 / 0%
Phi-4-14B 5.8s / 39.7 / 70.7% 4.9s / 45.7 / 4.8%
Gemma4-12B-IT 910ms / 48.5 / 94.4% 2.6s / 55.4 / 3.4%
Qwen3-1.7B 1.8s / 130.8 / 25.3% 1.2s / 166.2 / 0%

Load balance

Requests & utilization per model

Requests admitted, Served (2xx), and GPU aggregated across the card(s) serving each model. OpenInfer models are pooled across ×N cards — that spread is why OpenInfer sheds far fewer 429/503s than baseline's single card per model as concurrency climbs. Raw request counts are dominated by fast rejections under overload, so compare Served, not Requests.

Model Requests Served Over cap Tokens Avg GPU % Peak GPU %
Baseline Qwen3.5-27B 5,066 233 95.4% 46.6k 21.7 100
Baseline Qwen3.5-9B 384 384 0% 76.8k 14.5 97
Baseline Llama-3.2-1B-Instruct 1,980 1,380 30.3% 276k 13.2 89
Baseline Phi-4-14B 1,780 521 70.7% 104k 14.4 97
Baseline Gemma4-12B-IT 5,782 323 94.4% 64.6k 17.2 97
Baseline Qwen3-1.7B 1,606 1,200 25.3% 240k 13.1 89
OpenInfer Qwen3.5-27B ×6 cards 637 637 0% 127k 81.2 99
OpenInfer Qwen3.5-9B ×6 cards 607 607 0% 121k 81.2 99
OpenInfer Llama-3.2-1B-Instruct ×6 cards 1,920 1,920 0% 384k 81.2 99
OpenInfer Phi-4-14B ×6 cards 1,700 1,618 4.8% 324k 81.2 99
OpenInfer Gemma4-12B-IT ×6 cards 530 512 3.4% 102k 81.2 99
OpenInfer Qwen3-1.7B ×6 cards 1,664 1,664 0% 333k 81.2 99