r/programming • u/mttd • 11d ago
Pitfalls of Benchmarking on Modern Systems
https://stefan-marr.de/2026/08/pitfalls-of-benchmarking-on-modern-systems/10
u/StunningHeart7004 11d ago
if u increase the number of iterations then don't most of the problems disappear?
16
u/avinthakur080 11d ago edited 11d ago
The noise surely reduces (better, gets attenuated) but different issues appear:
- Benchmarks will take much more time
- You never know at what iteration to stop.
On a stable machine, you may stop early and on a noisier machine, you need to run more.A simple solution is to measure the noise and stop when it reaches below a threshold or the bench times out.
Even better solution is to not only measure the noise but also develop ways to clean the sample by removing outliers.8
u/matthieum 11d ago
Which is why in Rust, I really like
criterion, by default:
- It'll do a warm-up run of ~3s to estimate an average time the benchmark takes.
- Then compute the number of iterations for a roughly 5s total run.
- And finally delivers not just a mean, but various statistics, with outliers eliminated, and even compares to the last time the benchmark was run.
Most of the times, it's good enough to get a solid order of magnitude on the performance on the code being benchmarked.
The strategy the Rust compiler benchmark test suite uses is slightly different. Instead they measure instruction counts as a proxy of performance, as that tends to be a lot more stable, and "close enough" to wall times in general. The difficulty is understanding when it diverges from wall times...
5
u/yawkat 10d ago
This is normal functionality in JVM benchmark frameworks like JMH as well. But to resolve the issues in the article, you don't just need many iterations in the same process, you need many different processes. If running on the cloud, you even need many different VMs. If benchmark results are affected by ASLR for example, a loop in one process will not help.
2
u/matthieum 10d ago
ASLR is a tough one, yes.
This reminds me of a video by Lemire (?) demonstrating a new benchmark/profile method to avoid the effect of different environment variables changing the stack frames alignment and therefore the performance. I think it was named Co<something>, but I can't put my hands on it :'(
2
u/buerkle 10d ago
This one? It's a great talk.
1
u/matthieum 9d ago
Yes! Damn, that's why looking for Lemire didn't bring it up, I had the author wrong :'(
3
u/avinthakur080 10d ago
Criterion is really good, particularly the reporting, which was very enlightening.
However, it feels to me like its development is stuck.
Recently, I enjoyed nanonebench in C++ and would want criterion to borrow some features of it. Criterion can also improve on the conciseness of its reporting system.
41
u/crisp_lynx_370 11d ago
i spent an embarrassing amount of time trying to figure out why my benchmarks were giving me wildly inconsistent results before i realized i hadnt pinned the process to a core and turbo boost was just doing whatever it wanted. would have saved me a few days if id read something like this earlier.