Docs / Benchmarks

Benchmarks

nuvec-lm against mlx-lm on the same Mac: decode, time to first token up to 256K tokens, peak memory, load time and concurrency for twelve models from 0.6B to 35B parameters, with the machine-readable record.

nuvec’s performance target is MLX on the same machine. Every number here comes from just bench-parity, which runs both engines on the same Mac in one session and reports losses as well as wins. The record, with every sample: 0.5.0 full tier, M4 Pro 48 GB (JSON). Each record is validated against its schema, and index.json lists all of them.

Method🔗

  • Two tiers. just bench-parity (the quick tier, about 45 minutes) gates each release: one model per architecture (dense, hybrid, mixture of experts, gpt-oss), decode, time to first token at 2,048 tokens, memory, warm load and 4 concurrent requests. just bench-parity --tier full adds the whole roster, 32K tokens and longer, cold loads and 1 and 8 requests; it takes hours and runs in pieces (--budget, --resume). Each record names its tier.
  • Regressions. The previous release runs as a third engine in the same session, and every cell compares the current tree with it, best sample against best sample. A cell more than 3% slower (or larger) is measured again with a fresh pass of every engine; if it stays below, the record lists it under summary.regressions and the command fails.
  • Same work. Both engines get the same mlx-community checkpoint and the same token ids, written once from mlx-lm’s tokenizer. Decoding is greedy, and the end-of-sequence token is ignored, so every run generates its 128 tokens.
  • Fresh processes. Every measurement runs in its own child process, and the engines alternate (nuvec, the previous release, mlx-lm, and again). Cells show medians; the record keeps every sample.
  • Ratios are oriented so that above 1.00× favours nuvec: speeds are nuvec / mlx-lm, and times and memory are mlx-lm / nuvec. ⚠️ marks a speed or time cell below the 0.98× gate (run-to-run spread is about ±2%). The goal is 1.0×.
  • Prefill is prompt tokens over the time to the first token. Decode is the tokens after the first over the time they took. The prefill geometric mean leaves out the 25-token rows, which mostly measure the first token.
  • Peak memory is MemoryStats::peak_with_mapped for nuvec (the allocator high-water mark plus mapped files, counted whole) and mx.get_peak_memory() for mlx-lm (which includes the weights it copied). Each phase runs in a fresh process. The record also keeps nuvec’s allocator peak, active and mapped bytes, and the OS peak resident size of both processes.
  • Load is the time from the start of load to a ready model and to the first token of a 16-token prompt: one priming run, then three samples. Cold load needs sudo -n /usr/sbin/purge without a password (add a sudoers rule for /usr/sbin/purge); without it, cold cells stay empty.
  • Speculative decoding decodes the 25-token prompt with nuvec’s drafters (prompt lookup on every model; MTP heads on Qwen 3.5–3.8 when scripts/extract_mtp.py has extracted their weights) against mlx-lm’s plain decode of the same prompt. mlx-lm has no MTP drafter for these models, so these cells are not gated.
  • Concurrency serves 1, 4 or 8 prompts of 512 tokens together and generates 128 tokens each: nuvec’s Batcher against mlx-lm’s BatchGenerator. “Same tokens as alone” checks nuvec’s batched output against each request generated by itself.

Results: 0.5.0 (full tier)🔗

Apple M4 Pro, 48 GB, macOS 26.6.2; nuvec 0.5.0, mlx 0.32.4, mlx-lm 0.32.2; 2026-10-31, with 16 GB of swap in use at the start. nuvec / mlx-lm, above 1.00× favours nuvec; ⚠️ marks cells below the 0.98× gate.

The geometric means are decode 1.01×, prefill 1.01×, time to first token 1.02× at 2,048 and 32,768 tokens (0.98× at 65,536, Qwen3.6-35B-A3B), warm load 1.80×, peak memory 1.02×, and concurrent serving 1.01× / 1.00× / 1.00× at 1 / 4 / 8 requests. At 261,952 tokens nuvec prefills Qwen3.6-35B-A3B in 1,224 s; mlx-lm ran out of GPU memory.

modelkinddecode (25 / 961 / 3,841 tokens)prefill (961 / 3,841)TTFT 2K / 32Kpeak memorywarm load
Qwen3-0.6Bdense1.10× / 1.10× / 1.06×1.05× / 1.03×1.01× / 1.01×1.00×2.71×
Llama-3.2-3Bdense0.99× / 1.00× / 1.00×1.01× / 1.00×1.00× / 1.01×1.09×2.39×
Llama-3.1-8Bdense1.01× / 1.00× / 1.00×1.01× / 1.00×1.00× / 1.00×1.03×1.66×
Qwen3-8Bdense1.00× / 1.01× / 1.01×1.01× / 1.00×1.00× / 1.01×1.03×1.31×
Qwen3.5-9Bhybrid1.00× / 1.00× / 1.00×1.00× / 1.00×1.00× / 0.99×1.01×1.83×
Qwen3.8-27Bhybrid1.00× / 1.01× / 1.05×1.00× / 1.01×1.01× / 1.00×0.98×1.25×
Qwen3-30B-A3Bmoe1.00× / 1.00× / 1.00×1.00× / 1.00×0.99× / 1.00×1.01×1.20×
Qwen3.6-35B-A3Bhybrid-moe0.99× / 1.00× / 1.00×0.99× / 0.99×1.01× / 0.98×1.00×1.46×
gpt-oss-20b MXFP4moe0.99× / 0.99× / 0.98×1.02× / 1.04×1.04× / 1.03×1.02×1.47×
gpt-oss-20b 8-bitmoe1.01× / 1.01× / 1.00×1.01× / 1.02×1.02× / 1.02×1.01×3.60×
Gemma 4 E4Bdense0.98× / 1.00× / 1.00×1.03× / 1.07×1.07× / 1.07×1.06×2.55×
Gemma 4 26B-A4Bmoe1.01× / 1.01× / 1.01×1.01× / 1.08×1.06× / 1.08×1.01×1.49×

Peak memory is the geometric mean over a model’s phases (load, decode, prefill at 2,048 and 32,768 tokens); warm load is the time to the first token.

Concurrency🔗

Aggregate throughput of nuvec’s Batcher against mlx-lm’s BatchGenerator, 512-token prompts, 128 tokens each:

model1 request4 requests8 requests
Qwen3-0.6B1.06×0.98× ⚠️0.99×
Llama-3.2-3B1.00×1.00×1.00×
Llama-3.1-8B1.01×1.01×1.00×
Qwen3-8B1.00×1.01×1.01×
Qwen3.5-9B1.00×1.00×1.01×
Qwen3.8-27B1.01×1.00×1.01×
Qwen3-30B-A3B1.00×0.98×1.01×
Qwen3.6-35B-A3B0.99×0.98× ⚠️0.99×
gpt-oss-20b MXFP41.00×1.00×1.00×
gpt-oss-20b 8-bit1.03×1.08×1.01×
Gemma 4 E4B1.02×1.02×1.01×
Gemma 4 26B-A4B1.02×0.99×1.00×

Products route as MLX’s, so a request’s batched tokens may differ in their low bits from its tokens alone, as mlx-lm’s do; BatcherOptions::batch_invariant serves each request its solo tokens.

Kernels🔗

just bench-gemm, bench-quant, bench-fast and bench-compile run the same kernels in nuvec and MLX: sustained asynchronous bursts after a warm-up, best of three, programs alternating. They report nuvec/MLX time (lower is better).

Run them yourself🔗

The benchmarks need uv, which installs the pinned MLX, and the models in your Hugging Face cache:

just bench-parity                                  # every section, cached roster models
just bench-parity mlx-community/Qwen3-8B-4bit --sections decode,ttft
just bench-parity --resume --budget 540            # continue a long run in steps
just bench-validate                                # check records against the schema
just bench-compare base.json candidate.json        # fails on a ratio drop above 2%
just bench-quant gqmm                              # kernel benchmarks take a name filter

Swap in use makes the numbers noisy; each record stores the swap in use at its start.

✏️ Edit this page on GitHub