One of the Fabric disk benchmarks produced a throughput number which looked almost too good. It was.
The test wrote zero-filled data to ZFS. ZFS quite reasonably compressed it, so the benchmark moved very few real bytes and reported the result as storage throughput. The graph was accurate about the test it had run; the test just was not asking the question I thought it was.
Fixing this meant checking the data, the bytes which actually reached each stage and whether the measurement path itself changed the system. I ran into similar problems with container memory limits and a synthetic receiver which was slower than writing to disk because it allocated buffers differently.
I am now rather suspicious of a benchmark which produces a delightful number before it can explain where the bytes went.