Corrections

Every error we have found in our own data or method, what it affected, and what changed. If you find one we have not listed, tell us — being wrong in public and fixed within hours is worth more than pretending to be right.

Our published error bands described the fit, not the error

high2026-07-28

What was wrong: Every modeled speed on the site carried a band — 4.5% on Apple Silicon, 7.3% on consumer NVIDIA. Those figures came from how closely our curve matched the small controlled dataset it was fitted on. They were never a measurement of how well it predicts anything else, and we presented them as though they were.
Impact: Scored against 737 independent measured runs harvested from public benchmark threads, the real mean error is 15.7% on consumer NVIDIA and 22.9% on Apple Silicon — two to five times what we published. The median is closer to the old claim (7.6% and 7.4%), so the typical prediction was roughly as good as advertised, but the tail was far worse than anyone reading the band would expect.
Fix: Bands on the site are now the measured error, not the fitted error, and the full table is published on the methodology page including the 90th percentile. A band nobody can check is a claim, not a band.

Mixture-of-experts models were scored as dense

medium2026-07-28

What was wrong: The first version of our scoring harness passed every model through the dense path, so an MoE model that reads a small fraction of its weights per token was predicted as though it read all of them.
Impact: It made our own accuracy look considerably worse than it is — mean error on consumer NVIDIA read 23.9% instead of 15.7% — and would have led us to blame the roofline for our own mislabelling.
Fix: Scoring now uses the resolved architecture's expert counts, and skips any model that looks like an MoE but whose architecture we could not resolve, rather than scoring our own ignorance.

The compute buffer ignored context entirely and was several times too small

high2026-07-28

What was wrong: Our working-buffer term accepted a context length and never used it, so on the fit-matrix page — whose entire purpose is sweeping context — that component of memory never moved. It also scaled on hidden size rather than feed-forward width, which understated it badly on models with wide feed-forward layers. Against a published llama.cpp allocation report it came out roughly ten times low at a large micro-batch, and several times low at the default.
Impact: A systematic underestimate of a few hundred megabytes to a couple of gigabytes, landing exactly on the 8, 12, 16 and 24 GB fit boundaries where the answer flips. Some configurations were reported as fitting when they would not.
Fix: Rebuilt to scale with feed-forward width, micro-batch and, on the non-flash-attention path, context. The constants are still ours rather than fitted, and the methodology page now says so plainly instead of claiming they were regressed.

Apple hardware was discounted twice and under-reported by about 30%

high2026-07-28

What was wrong: Apple Silicon carries a measured wired-memory ceiling of roughly three quarters of nameplate. We stored that ceiling and then applied a separate driver-overhead haircut on top of it, so a 128 GB machine was treated as having about 89 GB usable against a real figure near 96 GB.
Impact: False 'does not fit' answers on the single most asked-about hardware in local AI, on 55 accelerator pages.
Fix: A measured ceiling is now used directly. The overhead fraction applies only where no ceiling is known.

Every NVIDIA card shared one speed calibration, and unified-memory AMD parts used the CPU curve

high2026-07-28

What was wrong: Speed constants were fitted per backend, but the mapping from hardware to backend was too coarse. Every NVIDIA part received the consumer fit, including datacenter cards whose kernels behave differently on small models. Worse, any AMD part without dedicated VRAM was routed to the CPU curve — putting Strix Halo, which has 256 GB/s of its own memory bandwidth, on an efficiency intended for system RAM.
Impact: Datacenter cards were out by 16–25%, beyond our own stated error band. Unified-memory AMD systems were understated by roughly four times.
Fix: Separate calibration classes for consumer and datacenter NVIDIA, dedicated-VRAM and unified AMD, and Intel. Each carries its own error band.

Embedding models were shown a KV cache they do not have

medium2026-07-28

What was wrong: Fourteen architectures — the BERT family, embedding models, and diffusion-language models — allocate no KV cache at all in llama.cpp. We rendered a context-versus-cache table for them anyway.
Impact: A category error rather than a precision error, on a modality we advertise covering.
Fix: These architectures now say plainly that no cache exists and why, instead of showing a curve.

Speed estimates halved the KV term at a stated context

medium2026-07-28

What was wrong: Bytes-read-per-token averaged the cache over a whole generation by halving it. That is reasonable for a generation average but wrong for every page we actually publish, all of which ask what happens at a specific context depth, where the window is full.
Impact: Generation speed was optimistic at long context, increasingly so as context grew.
Fix: The full cache is counted at the stated depth. The residual this was absorbing is efficiency, and it belongs in the efficiency constant where it can be seen.

Hybrid linear-attention layers were counted as full attention

high2026-07-28

What was wrong: Our KV formula recognised two kinds of layer, full and sliding-window. Models in the Qwen3.5/3.6 family are hybrids: most of their layers use linear attention, which keeps a fixed-size recurrent state rather than a per-token cache. We were treating those as ordinary attention layers.
Impact: KV cache was overstated roughly 4× for the most-downloaded model family in our index. Qwen3.6-27B at 128K context was reported as needing 32.00 GiB of cache when the real figure is 8.00 GiB — only 16 of its 64 layers cache anything that grows with context.
Fix: Layers are now classified into three kinds. Recurrent layers contribute a constant, not a context-scaled term.

Strix Halo memory bandwidth listed as 273 GB/s

medium2026-07-28

What was wrong: Several secondary sources report AMD's Ryzen AI Max (Strix Halo) at 273 GB/s. That figure is DGX Spark's LPDDR5X-8533 bandwidth appearing in the wrong row. Strix Halo is a 256-bit bus at 8000 MT/s, which is 256.0 GB/s, and AMD states 256.
Impact: Modeled token rates for Strix Halo were about 7% optimistic.
Fix: Corrected to 256 GB/s, cross-checked by recomputing bus width × data rate for every unified-memory part.

Intel Arc Pro tensor throughput taken at face value

medium2026-07-28

What was wrong: Wikipedia's Arc Pro column is headed 'XMX Half Precision (TFLOPS)' but its values are Intel's published peak INT8 TOPS, confirmed against Intel's own specification pages.
Impact: Tensor FP16 throughput would have been overstated 2× on exactly the cards this audience is most interested in — the B50, B60 and B70.
Fix: Values halved and recorded with an explicit accumulate mode. The mislabelling is noted in our source attribution.

MXFP4 quantization type index

low2026-07-28

What was wrong: The widely-copied ggml type table lists MXFP4 at enum index 36. The current llama.cpp source defines it as 39.
Impact: Every tensor in a gpt-oss quantization would have been labelled with the wrong type name in our histograms.
Fix: The type table is now parsed from llama.cpp source in CI rather than hardcoded, so it cannot drift silently.

Comparison multipliers were quoted from secondary analysis

medium2026-07-28

What was wrong: We had prepared to publish that a flat formula overstates DeepSeek-V3's KV cache by 71×. Recomputing from the model's real configuration gives 41×. The discrepancy comes from which incorrect formula you assume a competitor uses — a choice, not a fact.
Impact: No incorrect figure was published, but it would have been. A site selling correctness cannot afford to be wrong about how wrong someone else is.
Fix: Absolute values are ours and verified. Comparison multipliers now require querying the named source and quoting what it actually returns.