Skip to content
← Blog
2026.02.18 benchmarksmethodology 1 min read

A benchmark without conditions is a rumor

Why every number we publish carries its operating conditions, and why standalone performance claims are worse than no claim at all.

VoskenAI · Feb 18, 2026

“16 Gbps.” It looks great on a slide. It means almost nothing on its own.

Throughput at how many bytes per cycle? Which device? Measured, or a design target? One engine or sixteen? Each of those answers can move the number by an order of magnitude. A figure without its conditions is not data; it is a rumor that happens to be numeric.

The conditions are the claim

We publish benchmarks as a row, never as a headline:

metricvalueconditions
Throughput · sparse9.5 Gbpsmeasured · cocotb + Verilator RTL sim · 250 MHz model
Throughput · dense7.4 Gbpsmeasured · adversarial single-char pattern
Clock250 MHzAWS F2 · Virtex UltraScale+ VU47P

The right-hand column is the part that lets an engineer reproduce, compare, or reject the number. Remove it and you have removed the only thing that made the number trustworthy. Note the dense row: we publish the adversarial case next to the favourable one, because the workload you actually run is somewhere in between - and a vendor who only shows you one of those rows has already answered the question of how they will treat you later.

Why we would rather say less

It is tempting to quote the best configuration as if it were the typical one. We do not, for a simple reason: the people we want as customers will notice, and the ones who do not notice are not the ones who keep us in business.

Specificity is slower to write and less impressive at a glance. It is also the only kind of claim that survives a technical review, and a technical review is exactly where these decisions get made.

Want the evidence behind the words?

See Verification Evidence