Go developers who have written benchmarks using the testing package might have encountered some of its various pitfalls. Go 1.24 introduces a new way to write benchmarks that’s just as easy to use, but at the same time far more robust: testing.B.Loop.
Traditionally, Go benchmarks are written using a loop from 0 to b.N:
func Benchmark(b *testing.B) {
for range b.N {
... code to measure ...
}
}
Using b.Loop instead is a trivial change:
func Benchmark(b *testing.B) {
for b.Loop() {
... code to measure ...
}
}
testing.B.Loop has many benefits:
- It prevents unwanted compiler optimizations within the benchmark loop.
- It automatically excludes setup and cleanup code from benchmark timing.
- Code can’t accidentally depend on the total number of iterations or the current iteration.
These were all easy mistakes to make with b.N-style benchmarks that would silently result in bogus benchmark results. As an added bonus, b.Loop-style benchmarks even complete in less time!
Let’s explore the advantages of testing.B.Loop and how to effectively utilize it.
Old benchmark loop problems
Before Go 1.24, while the basic structure of a benchmark was simple, more sophisticated benchmarks required more care:
func Benchmark(b *testing.B) {
... setup ...
b.ResetTimer() // if setup may be expensive
for range b.N {
... code to measure ...
... use sinks or accumulation to prevent dead-code elimination ...
}
b.StopTimer() // if cleanup or reporting may be expensive
... cleanup ...
... report ...
}
If setup or cleanup are non-trivial, the developer needs to surround the benchmark loop with ResetTimer and/or StopTimer calls. These are easy to forget, and even if the developer remembers they may be necessary, it can be difficult to judge whether setup or cleanup are “expensive enough” to require them.
Without these, the testing package can only time the entire benchmark function. If a benchmark function omits them, the setup and cleanup code will be included in the overall time measurement, silently skewing the final benchmark result.
There is another, more subtle pitfall that requires deeper understanding: (Example source)
func isCond(b byte) bool {
if b%3 == 1 && b%7 == 2 && b%17 == 11 && b%31 == 9 {
return true
}
return false
}
func BenchmarkIsCondWrong(b *testing.B) {
for range b.N {
isCond(201)
}
}
In this example, the user might observe isCond executing in sub-nanosecond time. CPUs are fast, but not that fast! This seemingly anomalous result stems from the fact that isCond is inlined, and since its result is never used, the compiler eliminates it as dead code. As a result, this benchmark doesn’t measure isCond at all; it measures how long it takes to do nothing.
In this case, the sub-nanosecond result is a clear red flag, but in more complex benchmarks, partial dead-code elimination can lead to results that look reasonable but still aren’t measuring what was intended.
How testing.B.Loop helps
Unlike a b.N-style benchmark, testing.B.Loop is able to track when it is first called in a benchmark when the final iteration ends. The b.ResetTimer at the loop’s start and b.StopTimer at its end are integrated into testing.B.Loop, eliminating the need to manually manage the benchmark timer for setup and cleanup code.
Furthermore, the Go compiler now detects loops where the condition is just a call to testing.B.Loop and prevents dead code elimination within the loop. In Go 1.24, this is implemented by disallowing inlining into the body of such a loop, but we plan to improve this in the future.
Another nice feature of testing.B.Loop is its one-shot ramp-up approach. With a b.N-style benchmark, the testing package must call the benchmark function several times with different values of b.N, ramping up until the measured time reached a threshold.
In contrast, b.Loop can simply run the benchmark loop until it reaches the time threshold, and only needs to call the benchmark function once. Internally, b.Loop still uses a ramp-up process to amortize measurement overhead, but this is hidden from the caller and can be more efficient.
Certain constraints of the b.N-style loop still apply to the b.Loop-style loop. It remains the user’s responsibility to manage the timer within the benchmark loop, when necessary: (Example source)
func BenchmarkSortInts(b *testing.B) {
ints := make([]int, N)
for b.Loop() {
b.StopTimer()
fillRandomInts(ints)
b.StartTimer()
slices.Sort(ints)
}
}
In this example, to benchmark the in-place sorting performance of slices.Sort,a randomly initialized array is required for each iteration. The user must still manually manage the timer in such cases.
Furthermore, there still needs to be exactly one such loop in the benchmark function body (a b.N-style loop cannot coexist with a b.Loop-style loop), and every iteration of the loop should do the same thing.
When to use
The testing.B.Loop method is now the preferred way to write benchmarks:
func Benchmark(b *testing.B) {
... setup ...
for b.Loop() {
// optional timer control for in-loop setup/cleanup
... code to measure ...
}
... cleanup ...
}
testing.B.Loop offers faster, more accurate, and more intuitive benchmarking.
Acknowledgements
A huge thank you to everyone in the community who provided feedback on the proposal issue and reported bugs as this feature was released! I’m also grateful to Eli Bendersky for his helpful blog summaries. And finally a big thank you to Austin Clements, Cherry Mui and Michael Pratt for their review, thoughtful work on the design options and documentation improvements. Thank you all for your contributions!
Credits: Junyang Shao
Photo by Will Paterson on Unsplash
This article is available on
