silk.bench.harness.
The harness of a Silk program whose benchmarks a benchmark task drives: it answers the driven protocol and times each run.
When to use#
Use a Harness in a program that holds several pieces of work to time, when the samples
of two implementations should be taken in turns from processes that stay alive. Declare the
program in a suite with silk.bench.suite.Implementation.driven. A program that does one
piece of work from start to exit needs no harness: a command suite times it as a process.
Details#
A program gives the harness the names of its benchmarks and then loops on
Harness.next. Each time it returns true, the program runs the body of
Harness.name exactly Harness.iterations times. The harness times from the return of
next to the next call, and answers with a result line in ns/op.
Started by a benchmark task, the harness follows the commands of the task, which decides
how many iterations a sample takes and takes the samples of the implementations in turns.
The dialogue is the one silk.bench.transport describes, over the Transport service.
Started by hand, the same program is a benchmark of its own: for each name the harness
finds an iteration count that takes about 50 ms, or what Harness.sample sets, then
takes 10 samples, and prints them as result lines, which a suite reads with Implementation.resultLines and tools such as
benchstat read too.
Harness.reset starts the timing again after setup that should not count, and
Harness.stop and Harness.start leave a stretch out. Harness.check reports what
the run computed, so the task can tell when two implementations did different work,
Harness.keep stops the compiler from removing or precomputing the work, and
Harness.value adds a value in another unit to the result line.
Gotchas#
A body whose result is never used may be removed by the compiler, and a body that computes
the same value every time may be run once. Give the result of each iteration, and its
input, to Harness.keep.
Examples#
import silk.bench.harness { Harness }
import silk.bench.transport { StreamTransport, Transport, TransportError }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.system.host_input { HostInput }
import silk.system.monotonic_clock { MonotonicClock }
import silk.system.monotonic_clock_native { OsMonotonicClock }
effect fn benchmarks() -> ()
! TransportError | OutOfMemoryError
? &mut Transport | &mut Allocator | &mut MonotonicClock | &mut HostInput {
let mut harness = run Harness.open(&["Sum/n=1000", "Squares/n=1000"])
while run harness.next() {
let mut left = harness.iterations()
let square = harness.name() == "Squares/n=1000"
let mut total: u64 = 0
while left > 0 {
// One iteration: the work whose time is reported. The compiler cannot know the
// length, so it runs the loop every time.
let length: u64 = 1000
Harness.keep<u64>(&length)
total = 0
let mut at: u64 = 0
while at < length {
if square {
total = total + at * at
} else {
total = total + at
}
at = at + 1
}
Harness.keep<u64>(&total)
left = left - 1
}
// The same for any number of iterations, so two implementations can be compared.
run harness.checkNumber(total)
}
}
effect fn failed(error: TransportError | OutOfMemoryError) -> i32 {
drop error
return 1
}
effect fn finished() -> i32
! TransportError | OutOfMemoryError
? &mut Transport | &mut Allocator | &mut MonotonicClock | &mut HostInput {
run benchmarks()
return 0
}
pub effect fn main() -> i32
? &mut HostInput {
let mut allocator = Allocator.systemAllocatorProvider()
let mut transport = StreamTransport.standard()
let mut clock = OsMonotonicClock.make()
return run Effect.catchAll(finished(), failed)
|> Effect.provideMut<Transport>(&mut transport)
|> Effect.provideMut<MonotonicClock>(&mut clock)
|> Effect.provideMut<Allocator>(&mut allocator)
}Import as Harness with import silk.bench.harness { Harness }.
Public declarations: 1.
Harness#
pub struct HarnessThe benchmarks of a program, and the run that is in progress.
Implementation Harness#
impl HarnessAssociated function make#
pub effect fn make(names: &[string], driven: bool) -> Harness
! OutOfMemoryError
? &mut AllocatorReturns a harness for the benchmarks names. With driven, it follows the commands that
arrive over the Transport; without, it runs every benchmark itself and sends the result
lines.
Errors#
OutOfMemoryError if the Allocator cannot supply the names.
Associated function open#
pub effect fn open(names: &[string]) -> Harness
! OutOfMemoryError
? &mut Allocator | &mut HostInputReturns a harness for the benchmarks names that is driven when a benchmark task started
the program, which the task says with the environment variable BENCH_DRIVE.
Errors#
OutOfMemoryError if the Allocator cannot supply the names.
Method next#
pub effect fn next(self: &mut Self) -> bool
! TransportError | OutOfMemoryError
? &mut Transport | &mut Allocator | &mut MonotonicClockFinishes the run in progress, if there is one, and returns whether there is another run
to do. After true, run the body of Harness.name Harness.iterations times and
call next again.
Errors#
TransportErrorif a line cannot be carried.OutOfMemoryErrorif theAllocatorcannot supply a line.
Method sample#
pub fn sample(self: &mut Self, milliseconds: u64) -> ()Sets how long one sample should take when nobody drives the harness, in milliseconds. A driven harness takes the iteration counts of its task and ignores this.
Method name#
pub fn name(self: &Self) -> stringReturns the name of the benchmark to run, as it was given to the harness.
Method position#
pub fn position(self: &Self) -> usizeReturns the position of the benchmark to run in the names the harness was given.
Method iterations#
pub fn iterations(self: &Self) -> u64Returns how many times to run the body.
Method measured#
pub fn measured(self: &mut Self, nanoseconds: u64) -> ()Gives the harness the time of the run in nanoseconds, for a body that timed itself, and stops the timing of the harness.
Method failed#
pub effect fn failed(self: &mut Self, text: string) -> ()
! OutOfMemoryError
? &mut AllocatorSays that the run failed, with text as the reason on one line. The harness answers with
an error: line and no result line.
Errors#
OutOfMemoryError if the Allocator cannot supply the copy.
Method reset#
pub effect fn reset(self: &mut Self) -> ()
? &mut MonotonicClockStarts the timing of the run again from zero, after setup that should not count.
Method stop#
pub effect fn stop(self: &mut Self) -> ()
? &mut MonotonicClockStops the timing, until Harness.start. Stopping twice changes nothing.
Method start#
pub effect fn start(self: &mut Self) -> ()
? &mut MonotonicClockStarts the timing again after Harness.stop. Starting twice changes nothing.
Method check#
pub effect fn check(self: &mut Self, text: string) -> ()
! OutOfMemoryError
? &mut AllocatorReports text as what the run computed, such as a checksum. A benchmark task does not
compare implementations whose checks differ.
Errors#
OutOfMemoryError if the Allocator cannot supply the copy.
Method checkNumber#
pub effect fn checkNumber(self: &mut Self, number: u64) -> ()
! OutOfMemoryError
? &mut AllocatorReports number as what the run computed. See Harness.check.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Method value#
pub effect fn value(self: &mut Self, unit: string, amount: f64) -> ()
! OutOfMemoryError
? &mut AllocatorAdds amount in unit to the result line of the run, such as 1024 B/op. A unit holds
no space.
Errors#
OutOfMemoryError if the Allocator cannot supply the copy.
Associated function keep#
pub fn keep<T>(value: &T) -> ()Makes the compiler treat value as read and written, so it can neither remove the work
that computed it nor assume what it holds. Give it the result of each iteration, and an
input the compiler could otherwise compute with.
Gotchas#
On a target without inline assembly, which is every target but Linux on x86_64 and aarch64 and macOS on aarch64, the value is only made to live in memory, which stops less.
Associated function grown#
pub fn grown(iterations: u64, elapsed: u64, target: u64) -> u64Returns the iteration count to try after iterations took elapsed nanoseconds, when a
sample should take target: enough to reach the target with a fifth to spare, at least
twice and at most a hundred times as many.
Category: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.