silk.bench.suite.

Describes what a benchmark run compares: suites, their implementations, the programs to time, and where the reports go.

When to use#

Build a Suite in a project's tasks.silk and give it to silk.bench.bench.Bench.perform. A suite compares implementations of one piece of work: a Silk program and the same program in another language, two versions of one program, or one program with different options.

Details#

A command suite, from Suite.command, times whole processes: any program in any language is an implementation, with nothing to adopt. An Implementation is one program with its arguments. A Call is a program a hook runs without timing it: Suite.setup and Suite.cleanup run once around the suite, and Implementation.prepare and Implementation.conclude run around every timed run.

Every path is a silk.build.step.Deferred<Path> from the build system, resolved when the task runs: a package file from Deferred<Path>.source, a host path from Deferred<Path>.external, or the output of a build step, such as a compilation's artifact() or a Run step that builds a program in another language. The steps a suite's paths come from run before anything is timed, with the build system's cache, so a second run rebuilds nothing that did not change.

A program that measures itself says so with Implementation.resultLines: the suite then reads the result lines the program prints, one sample a line, and reports what the lines say in their own unit, such as ns/op or req/s. A load generator, a server benchmark, or a loop that times one function inside a larger program reports this way. Implementation.googleBenchmark and Implementation.criterion say the same of a program that prints the table of Google Benchmark or the lines of a Rust benchmark.

A program that holds several benchmarks and can run each on request says so with Implementation.driven: the suite keeps it alive, decides how many iterations a sample takes, and takes the samples of the implementations in turns.

Suite.parameter runs the suite once for each value: {name} in an argument is replaced by the value, and each combination of values is its own benchmark, named suite/name=value. Suite.cpus does the same for the number of processors a program may run on, as the parameter cpus.

Suite.view declares how the reports show the results, as a silk.bench.view.View: a line against a parameter, every sample, a comparison against another implementation.

Suite.output declares where a report goes: an Output is a Format and a place, standard output or a file. A suite without outputs prints the terminal table.

Examples#

silk
import silk.bench.suite { Format, Implementation, Output, Suite }
import silk.build.graph { BuildError, Compilation, Options }
import silk.build.step { Deferred }
import silk.mem.allocator { Allocator }
import silk.system.filesystem { FileSystem, Path }
import silk.system.host_input { HostInput }

// Compares this package's program with a Rust build of the same work, on two inputs.
effect fn jsonParse(options: &Options) -> Suite
  ! BuildError
  ? &mut Allocator | &mut FileSystem | &mut HostInput {
  let compilation = run Compilation.fromPackage(options)
  let mut silk = Implementation.program(run compilation.artifact())
  run silk.path(run Deferred<Path>.source("data/{file}.json"))

  let mut rust = Implementation.program(run Deferred<Path>.source("rust/target/release/parse"))
  run rust.path(run Deferred<Path>.source("data/{file}.json"))

  let mut suite = run Suite.command("json-parse")
  run suite.implementation("silk", move silk)
  run suite.implementation("rust", move rust)
  run suite.reference("rust")
  run suite.parameter("file", &["twitter", "canada"])
  run suite.output(Output.stdout(Format.Terminal))
  run suite.output(Output.file(run Deferred<Path>.source("build/bench/json.html"), Format.Html))
  return move suite
}

Import as Suite with import silk.bench.suite { Suite }.

Public declarations: 10.

Part#

silk
pub union Part

One argument of a Call: text, or a path the build system resolves.

Text#

silk
Text

Text passed as it is, after {name} placeholders are replaced.

Field value#

silk
pub value: String

Location#

silk
Location

A path, passed as the absolute path it resolves to.

Field path#

silk
pub path: Deferred<Path>

Call#

silk
pub struct Call

A program with its arguments, working directory, and environment.

Details#

A call inherits the environment of the task and adds the variables set with Call.variable. Without Call.directory, it runs in the package directory.

Field executable#

silk
pub executable: Deferred<Path>

The program.

Field parts#

silk
pub parts: List<Part>

The arguments, in order.

Field workingDirectory#

silk
pub workingDirectory: Option<Deferred<Path>>

The working directory, if the call sets one.

Field variables#

silk
pub variables: List<Pair>

The environment variables the call adds.

Implementation Call#

silk
impl Call

Associated function program#

silk
pub fn program(program: Deferred<Path>) -> Call

Returns a call of the program at program, with no arguments.

Associated function command#

silk
pub effect fn command(name: string) -> Call
    ! StepFailure | FileError | HostInputError | OutOfMemoryError
    ? &mut HostInput | &mut Allocator | &mut FileSystem

Returns a call of the program named name, found now: on PATH for a name without a /, or as a path relative to the package directory.

Errors#

StepError if no such program exists.

Method argument#

silk
pub effect fn argument(self: &mut Self, value: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds the argument value. {name} in it is replaced by the value of the suite parameter name.

Method arguments#

silk
pub effect fn arguments(self: &mut Self, values: &[string]) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds each of values as an argument, as Call.argument does.

Method path#

silk
pub effect fn path(self: &mut Self, value: Deferred<Path>) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds an argument that is the absolute path value resolves to. The step that produces the path, if any, runs before the suite is timed.

Method directory#

silk
pub fn directory(self: &mut Self, value: Deferred<Path>) -> ()

Sets the working directory.

Method variable#

silk
pub effect fn variable(self: &mut Self, name: string, value: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Sets the environment variable name to value for the program.

Dialect#

silk
pub enum Dialect

The format in which a program prints its own results.

ResultLines#

silk
ResultLines

Result lines, as silk.bench.lines describes them.

GoogleBenchmark#

silk
GoogleBenchmark

The console table of Google Benchmark.

Criterion#

silk
Criterion

The lines of Criterion, or of Rust's built-in benchmark harness.

Implementation#

silk
pub struct Implementation

One program a suite times, with the hooks around each of its runs.

Field call#

silk
pub call: Call

The program that is timed.

Field before#

silk
pub before: List<Call>

The calls that run before every timed run, without being timed.

Field after#

silk
pub after: List<Call>

The calls that run after every timed run, without being timed.

Field version#

silk
pub version: Option<String>

The revision recorded for the implementation.

Field reports#

silk
pub reports: bool

Whether the program prints result lines, which then are what the suite measures.

Field dialect#

silk
pub dialect: Dialect

The format of what the program prints, when it prints its results.

Field drives#

silk
pub drives: bool

Whether a benchmark task drives the program: it is started once, and asked for one sample at a time.

Implementation Implementation#

silk
impl Implementation

Associated function of#

silk
pub fn of(call: Call) -> Implementation

Returns an implementation that times call.

Associated function program#

silk
pub fn program(program: Deferred<Path>) -> Implementation

Returns an implementation that times the program at program, such as a compilation's artifact() or the output of a Run step.

Associated function command#

silk
pub effect fn command(name: string) -> Implementation
    ! StepFailure | FileError | HostInputError | OutOfMemoryError
    ? &mut HostInput | &mut Allocator | &mut FileSystem

Returns an implementation that times the program named name, found as Call.command finds it.

Errors#

StepError if no such program exists.

Method argument#

silk
pub effect fn argument(self: &mut Self, value: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds the argument value, as Call.argument does.

Method arguments#

silk
pub effect fn arguments(self: &mut Self, values: &[string]) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds each of values as an argument, as Call.argument does.

Method path#

silk
pub effect fn path(self: &mut Self, value: Deferred<Path>) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds an argument that is a resolved path, as Call.path does.

Method directory#

silk
pub fn directory(self: &mut Self, value: Deferred<Path>) -> ()

Sets the working directory of the timed program.

Method variable#

silk
pub effect fn variable(self: &mut Self, name: string, value: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Sets the environment variable name to value for the timed program.

Method prepare#

silk
pub effect fn prepare(self: &mut Self, call: Call) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a call that runs before every timed run of this implementation, such as one that removes a cache. It is not timed. A call that fails stops the run.

Method conclude#

silk
pub effect fn conclude(self: &mut Self, call: Call) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a call that runs after every timed run of this implementation. It is not timed. A call that fails stops the run.

Method revision#

silk
pub effect fn revision(self: &mut Self, text: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Records text as the revision of this implementation, such as a commit hash or a version number, so a report says what was measured.

Method resultLines#

silk
pub fn resultLines(self: &mut Self) -> ()

Says that the program prints result lines, as silk.bench.lines describes them, and that the suite measures what those lines report, not the time of the process.

Details#

Each line is one sample of the benchmark its name gives, so one run of the program may report many benchmarks and many samples of each. The program writes the lines to standard output, or to the file the environment variable BENCH_OUT names, which the suite sets for every timed run. Lines that are not result lines are ignored.

The suite reads four configuration keys. check: says what the program computed, such as a checksum or a count: implementations whose checks differ for one benchmark are not compared, and the task fails. revision: is recorded as the implementation's revision when the suite gives none. note: is added to the notes of the run. view: suggests a view, as silk.bench.view.View.parse reads it, which the reports show when the suite declares none.

Method googleBenchmark#

silk
pub fn googleBenchmark(self: &mut Self) -> ()

Says that the program is a Google Benchmark program, and that the suite measures what its console table reports, as Implementation.resultLines does for result lines.

Details#

Each row of the table is one sample: its Time column is reported as ns/op and its CPU column as cpu-ns/op. The rows Google Benchmark adds for repetitions are left out. silk.bench.lines.Lines.fromGoogleBenchmark does the reading. The program must print the table to standard output, which is what it does without --benchmark_format.

Method criterion#

silk
pub fn criterion(self: &mut Self) -> ()

Says that the program is a Rust benchmark, run by Criterion or by the built-in harness, and that the suite measures what it reports, as Implementation.resultLines does for result lines.

Details#

Each name time: [low estimate high] line of Criterion is one sample, the estimate in the middle, and so is each test name ... bench: 1,240 ns/iter (+/- 12) line. Both are reported as ns/op. silk.bench.lines.Lines.fromCriterion does the reading. The program must print to standard output, as cargo bench does.

Method driven#

silk
pub fn driven(self: &mut Self) -> ()

Says that the program is a harness the task drives, as silk.bench.transport describes: the task starts it with BENCH_DRIVE=1, asks it for its benchmarks, and asks for one sample at a time over its standard input and output.

Details#

The task finds how many iterations of each benchmark take the suite's sample time, and then, in each round, takes one sample of each benchmark from every implementation, in a random order. The programs stay alive between samples, and each is started again a few times over a run, so that one unlucky start does not color every sample. The report is in the unit of the harness, ns/op.

A Silk program answers with silk.bench.harness.Harness. A program in another language reads the commands from its standard input and prints result lines, which takes a loop of a few dozen lines.

Format#

silk
pub enum Format

How a report is written.

Terminal#

silk
Terminal

The table for a terminal, of silk.bench.render.Render.terminal. It has color when it goes to a terminal that takes it.

Markdown#

silk
Markdown

Markdown, of silk.bench.render.Render.markdown.

Html#

silk
Html

One static HTML page with charts, of silk.bench.html.Html.page.

Report#

silk
Report

The report with its statistics as JSON, a silk.bench.record.Report, for a renderer of your own.

Result#

silk
Result

The raw samples as JSON, a silk.bench.record.RunResult.

Lines#

silk
Lines

The samples as result lines, of silk.bench.lines.Lines.fromResult, the text format of Go's testing package that tools such as benchstat read.

History#

silk
History

One line for each row of the report, of silk.bench.history.History.lines, to add to a history file with Output.appending.

Bencher#

silk
Bencher

Bencher Metric Format, of silk.bench.history.History.bencher, the JSON the Bencher service reads.

Destination#

silk
pub union Destination

Where a report goes.

Stdout#

silk
Stdout

Standard output.

File#

silk
File

A file. Its directories are made when they are missing.

Field path#

silk
pub path: Deferred<Path>

Field append#

silk
pub append: bool

Output#

silk
pub struct Output

One report of a suite: a format and a destination.

Field format#

silk
pub format: Format

How the report is written.

Field destination#

silk
pub destination: Destination

Where it goes.

Implementation Output#

silk
impl Output

Associated function stdout#

silk
pub fn stdout(format: Format) -> Output

Returns an output that prints the report in format on standard output.

Associated function file#

silk
pub fn file(path: Deferred<Path>, format: Format) -> Output

Returns an output that writes the report in format to the file at path, replacing it.

Method appending#

silk
pub fn appending(self: Self) -> Output

Returns self writing to the end of its file instead of replacing it, for a log that grows with every run. An output to standard output is returned as it is.

Parameter#

silk
pub struct Parameter

A parameter of a suite: a name and the values the suite runs with.

Field name#

silk
pub name: String

The name, which {name} in an argument refers to.

Field values#

silk
pub values: List<String>

The values, in the order they run.

Named#

silk
pub struct Named

An implementation with the name a suite gave it.

Field name#

silk
pub name: String

The name.

Field implementation#

silk
pub implementation: Implementation

The implementation.

Suite#

silk
pub struct Suite

A set of implementations of the same work, compared to each other.

Details#

The fields are public so a tool can read a suite. Build one with Suite.command and the methods below.

Field name#

silk
pub name: String

The name, which starts the name of each of its benchmarks and of its result file.

Field implementations#

silk
pub implementations: List<Named>

The implementations, in the order they were added.

Field baseline#

silk
pub baseline: Option<String>

The name of the implementation the others are compared to. Without one, it is the first.

Field parameters#

silk
pub parameters: List<Parameter>

The parameters, in the order they were added.

Field processors#

silk
pub processors: List<u32>

The processor counts to run with. Empty means no limit.

Field rounds#

silk
pub rounds: Option<u64>

The number of measured rounds, if the suite sets it.

Field warmups#

silk
pub warmups: Option<u64>

The number of warm-up rounds, if the suite sets it.

Field sampling#

silk
pub sampling: Option<u64>

How long one sample of a driven implementation should take, in milliseconds, if the suite sets it.

Field notes#

silk
pub notes: List<String>

The notes recorded with every run.

Field recorded#

silk
pub recorded: List<String>

The environment variables whose values are recorded with every run.

Field before#

silk
pub before: List<Call>

The calls that run once before anything is timed.

Field after#

silk
pub after: List<Call>

The calls that run once after everything is timed.

Field outputs#

silk
pub outputs: List<Output>

The reports. Empty means the terminal table on standard output.

Field views#

silk
pub views: List<View>

The views. Empty means the report picks by the shape of the results.

Implementation Suite#

silk
impl Suite

Associated function command#

silk
pub effect fn command(name: string) -> Suite
    ! OutOfMemoryError
    ? &mut Allocator

Returns an empty suite named name that times whole processes. name becomes part of file names, so use letters, digits, -, _ and ..

Method implementation#

silk
pub effect fn implementation(self: &mut Self, name: string, implementation: Implementation) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds implementation under name. Names must differ inside a suite.

Method reference#

silk
pub effect fn reference(self: &mut Self, name: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Names the implementation the others are compared to. Without it, the first one added is the reference.

Method parameter#

silk
pub effect fn parameter(self: &mut Self, name: string, values: &[string]) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds the parameter name with values. The suite runs once for each value, and once for each combination when it has several parameters.

Method cpus#

silk
pub effect fn cpus(self: &mut Self, counts: &[u32]) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Runs the suite once for each of counts, with every program limited to that many processors, as the parameter cpus. The limit is an affinity mask over the first processors the task may use.

Gotchas#

Only Linux can limit processors. On another system, a suite with this setting fails before anything is timed.

Method runs#

silk
pub fn runs(self: &mut Self, count: u64) -> ()

Sets the number of measured rounds. Every implementation runs once in each round, in a random order. The default is 10. --runs on the command line wins.

Method warmup#

silk
pub fn warmup(self: &mut Self, count: u64) -> ()

Sets the number of warm-up rounds, which fill caches before the measured rounds and are in no statistic. The default is 1. --warmup on the command line wins.

Method sample#

silk
pub fn sample(self: &mut Self, milliseconds: u64) -> ()

Sets how long one sample of a driven implementation should take, in milliseconds: the task asks for as many iterations as fill that time. The default is 50.

Method note#

silk
pub effect fn note(self: &mut Self, text: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Records text with every run of the suite, such as what a reader should know about the setup.

Method record#

silk
pub effect fn record(self: &mut Self, name: string) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Records the value of the environment variable name with every run.

Method setup#

silk
pub effect fn setup(self: &mut Self, call: Call) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a call that runs once before anything is timed. A call that fails stops the run.

Method cleanup#

silk
pub effect fn cleanup(self: &mut Self, call: Call) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a call that runs once after everything is timed, also when a run failed.

Method output#

silk
pub effect fn output(self: &mut Self, output: Output) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a report. A suite with at least one output prints nothing else: add Output.stdout(Format.Terminal) to keep the table.

Method view#

silk
pub effect fn view(self: &mut Self, view: View) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Adds a view: one more way the reports show the results. A suite with at least one view gets no other: without any, a report picks views by the shape of the results.

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page