silk.bench.harness.

The harness of a Silk program whose benchmarks a benchmark task drives: it answers the driven protocol and times each run.

When to use#

Use a Harness in a program that holds several pieces of work to time, when the samples of two implementations should be taken in turns from processes that stay alive. Declare the program in a suite with silk.bench.suite.Implementation.driven. A program that does one piece of work from start to exit needs no harness: a command suite times it as a process.

Details#

A program gives the harness the names of its benchmarks and then loops on Harness.next. Each time it returns true, the program runs the body of Harness.name exactly Harness.iterations times. The harness times from the return of next to the next call, and answers with a result line in ns/op.

Started by a benchmark task, the harness follows the commands of the task, which decides how many iterations a sample takes and takes the samples of the implementations in turns. The dialogue is the one silk.bench.transport describes, over the Transport service.

Started by hand, the same program is a benchmark of its own: for each name the harness finds an iteration count that takes about 50 ms, or what Harness.sample sets, then takes 10 samples, and prints them as result lines, which a suite reads with Implementation.resultLines and tools such as benchstat read too.

Harness.reset starts the timing again after setup that should not count, and Harness.stop and Harness.start leave a stretch out. Harness.check reports what the run computed, so the task can tell when two implementations did different work, Harness.keep stops the compiler from removing or precomputing the work, and Harness.value adds a value in another unit to the result line.

Gotchas#

A body whose result is never used may be removed by the compiler, and a body that computes the same value every time may be run once. Give the result of each iteration, and its input, to Harness.keep.

Examples#

silk
import silk.bench.harness { Harness }
import silk.bench.transport { StreamTransport, Transport, TransportError }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.system.host_input { HostInput }
import silk.system.monotonic_clock { MonotonicClock }
import silk.system.monotonic_clock_native { OsMonotonicClock }

effect fn benchmarks() -> ()
  ! TransportError | OutOfMemoryError
  ? &mut Transport | &mut Allocator | &mut MonotonicClock | &mut HostInput {
  let mut harness = run Harness.open(&["Sum/n=1000", "Squares/n=1000"])
  while run harness.next() {
    let mut left = harness.iterations()
    let square = harness.name() == "Squares/n=1000"
    let mut total: u64 = 0
    while left > 0 {
      // One iteration: the work whose time is reported. The compiler cannot know the
      // length, so it runs the loop every time.
      let length: u64 = 1000
      Harness.keep<u64>(&length)
      total = 0
      let mut at: u64 = 0
      while at < length {
        if square {
          total = total + at * at
        } else {
          total = total + at
        }
        at = at + 1
      }
      Harness.keep<u64>(&total)
      left = left - 1
    }

    // The same for any number of iterations, so two implementations can be compared.
    run harness.checkNumber(total)
  }
}

effect fn failed(error: TransportError | OutOfMemoryError) -> i32 {
  drop error
  return 1
}

effect fn finished() -> i32
  ! TransportError | OutOfMemoryError
  ? &mut Transport | &mut Allocator | &mut MonotonicClock | &mut HostInput {
  run benchmarks()
  return 0
}

pub effect fn main() -> i32
  ? &mut HostInput {
  let mut allocator = Allocator.systemAllocatorProvider()
  let mut transport = StreamTransport.standard()
  let mut clock = OsMonotonicClock.make()
  return run Effect.catchAll(finished(), failed)
    |> Effect.provideMut<Transport>(&mut transport)
    |> Effect.provideMut<MonotonicClock>(&mut clock)
    |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as Harness with import silk.bench.harness { Harness }.

Public declarations: 1.

Harness#

silk
pub struct Harness

The benchmarks of a program, and the run that is in progress.

Implementation Harness#

silk
impl Harness

Associated function make#

silk
pub effect fn make(names: &[string], driven: bool) -> Harness
    ! OutOfMemoryError
    ? &mut Allocator

Returns a harness for the benchmarks names. With driven, it follows the commands that arrive over the Transport; without, it runs every benchmark itself and sends the result lines.

Errors#

OutOfMemoryError if the Allocator cannot supply the names.

Associated function open#

silk
pub effect fn open(names: &[string]) -> Harness
    ! OutOfMemoryError
    ? &mut Allocator | &mut HostInput

Returns a harness for the benchmarks names that is driven when a benchmark task started the program, which the task says with the environment variable BENCH_DRIVE.

Errors#

OutOfMemoryError if the Allocator cannot supply the names.

Method next#

silk
pub effect fn next(self: &mut Self) -> bool
    ! TransportError | OutOfMemoryError
    ? &mut Transport | &mut Allocator | &mut MonotonicClock

Finishes the run in progress, if there is one, and returns whether there is another run to do. After true, run the body of Harness.name Harness.iterations times and call next again.

Errors#

  • TransportError if a line cannot be carried.

  • OutOfMemoryError if the Allocator cannot supply a line.

Method sample#

silk
pub fn sample(self: &mut Self, milliseconds: u64) -> ()

Sets how long one sample should take when nobody drives the harness, in milliseconds. A driven harness takes the iteration counts of its task and ignores this.

Method name#

silk
pub fn name(self: &Self) -> string

Returns the name of the benchmark to run, as it was given to the harness.

Method position#

silk
pub fn position(self: &Self) -> usize

Returns the position of the benchmark to run in the names the harness was given.

Method iterations#

silk
pub fn iterations(self: &Self) -> u64

Returns how many times to run the body.

Method measured#

silk
pub fn measured(self: &mut Self, nanoseconds: u64) -> ()

Gives the harness the time of the run in nanoseconds, for a body that timed itself, and stops the timing of the harness.

Method failed#

silk
pub effect fn failed(self: &mut Self, text: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Says that the run failed, with text as the reason on one line. The harness answers with an error: line and no result line.

Errors#

OutOfMemoryError if the Allocator cannot supply the copy.

Method reset#

silk
pub effect fn reset(self: &mut Self) -> ()
    ? &mut MonotonicClock

Starts the timing of the run again from zero, after setup that should not count.

Method stop#

silk
pub effect fn stop(self: &mut Self) -> ()
    ? &mut MonotonicClock

Stops the timing, until Harness.start. Stopping twice changes nothing.

Method start#

silk
pub effect fn start(self: &mut Self) -> ()
    ? &mut MonotonicClock

Starts the timing again after Harness.stop. Starting twice changes nothing.

Method check#

silk
pub effect fn check(self: &mut Self, text: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Reports text as what the run computed, such as a checksum. A benchmark task does not compare implementations whose checks differ.

Errors#

OutOfMemoryError if the Allocator cannot supply the copy.

Method checkNumber#

silk
pub effect fn checkNumber(self: &mut Self, number: u64) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Reports number as what the run computed. See Harness.check.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Method value#

silk
pub effect fn value(self: &mut Self, unit: string, amount: f64) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds amount in unit to the result line of the run, such as 1024 B/op. A unit holds no space.

Errors#

OutOfMemoryError if the Allocator cannot supply the copy.

Associated function keep#

silk
pub fn keep<T>(value: &T) -> ()

Makes the compiler treat value as read and written, so it can neither remove the work that computed it nor assume what it holds. Give it the result of each iteration, and an input the compiler could otherwise compute with.

Gotchas#

On a target without inline assembly, which is every target but Linux on x86_64 and aarch64 and macOS on aarch64, the value is only made to live in memory, which stops less.

Associated function grown#

silk
pub fn grown(iterations: u64, elapsed: u64, target: u64) -> u64

Returns the iteration count to try after iterations took elapsed nanoseconds, when a sample should take target: enough to reach the target with a fifth to spare, at least twice and at most a hundred times as many.

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page