silk.bench.suite.
Describes what a benchmark run compares: suites, their implementations, the programs to time, and where the reports go.
When to use#
Build a Suite in a project's tasks.silk and give it to silk.bench.bench.Bench.perform. A
suite compares implementations of one piece of work: a Silk program and the same program in
another language, two versions of one program, or one program with different options.
Details#
A command suite, from Suite.command, times whole processes: any program in any language
is an implementation, with nothing to adopt. An Implementation is one program with its
arguments. A Call is a program a hook runs without timing it: Suite.setup and
Suite.cleanup run once around the suite, and Implementation.prepare and
Implementation.conclude run around every timed run.
Every path is a silk.build.step.Deferred<Path> from the build system, resolved when the
task runs: a package file from Deferred<Path>.source, a host path from
Deferred<Path>.external, or the output of a build step, such as a compilation's
artifact() or a Run step that builds a program in another language. The steps a suite's
paths come from run before anything is timed, with the build system's cache, so a second run
rebuilds nothing that did not change.
A program that measures itself says so with Implementation.resultLines: the suite then
reads the result lines the program prints, one sample a line, and reports what the lines
say in their own unit, such as ns/op or req/s. A load generator, a server benchmark, or
a loop that times one function inside a larger program reports this way.
Implementation.googleBenchmark and Implementation.criterion say the same of a program
that prints the table of Google Benchmark or the lines of a Rust benchmark.
A program that holds several benchmarks and can run each on request says so with
Implementation.driven: the suite keeps it alive, decides how many iterations a sample
takes, and takes the samples of the implementations in turns.
Suite.parameter runs the suite once for each value: {name} in an argument is replaced
by the value, and each combination of values is its own benchmark, named
suite/name=value. Suite.cpus does the same for the number of processors a program may
run on, as the parameter cpus.
Suite.view declares how the reports show the results, as a silk.bench.view.View: a
line against a parameter, every sample, a comparison against another implementation.
Suite.output declares where a report goes: an Output is a Format and a place,
standard output or a file. A suite without outputs prints the terminal table.
Examples#
import silk.bench.suite { Format, Implementation, Output, Suite }
import silk.build.graph { BuildError, Compilation, Options }
import silk.build.step { Deferred }
import silk.mem.allocator { Allocator }
import silk.system.filesystem { FileSystem, Path }
import silk.system.host_input { HostInput }
// Compares this package's program with a Rust build of the same work, on two inputs.
effect fn jsonParse(options: &Options) -> Suite
! BuildError
? &mut Allocator | &mut FileSystem | &mut HostInput {
let compilation = run Compilation.fromPackage(options)
let mut silk = Implementation.program(run compilation.artifact())
run silk.path(run Deferred<Path>.source("data/{file}.json"))
let mut rust = Implementation.program(run Deferred<Path>.source("rust/target/release/parse"))
run rust.path(run Deferred<Path>.source("data/{file}.json"))
let mut suite = run Suite.command("json-parse")
run suite.implementation("silk", move silk)
run suite.implementation("rust", move rust)
run suite.reference("rust")
run suite.parameter("file", &["twitter", "canada"])
run suite.output(Output.stdout(Format.Terminal))
run suite.output(Output.file(run Deferred<Path>.source("build/bench/json.html"), Format.Html))
return move suite
}Import as Suite with import silk.bench.suite { Suite }.
Public declarations: 10.
Part#
pub union PartOne argument of a Call: text, or a path the build system resolves.
Text#
TextText passed as it is, after {name} placeholders are replaced.
Field value#
pub value: StringLocation#
LocationA path, passed as the absolute path it resolves to.
Field path#
pub path: Deferred<Path>Call#
pub struct CallA program with its arguments, working directory, and environment.
Details#
A call inherits the environment of the task and adds the variables set with
Call.variable. Without Call.directory, it runs in the package directory.
Field executable#
pub executable: Deferred<Path>The program.
Field parts#
pub parts: List<Part>The arguments, in order.
Field workingDirectory#
pub workingDirectory: Option<Deferred<Path>>The working directory, if the call sets one.
Field variables#
pub variables: List<Pair>The environment variables the call adds.
Implementation Call#
impl CallAssociated function program#
pub fn program(program: Deferred<Path>) -> CallReturns a call of the program at program, with no arguments.
Associated function command#
pub effect fn command(name: string) -> Call
! StepFailure | FileError | HostInputError | OutOfMemoryError
? &mut HostInput | &mut Allocator | &mut FileSystemReturns a call of the program named name, found now: on PATH for a name without a
/, or as a path relative to the package directory.
Errors#
StepError if no such program exists.
Method argument#
pub effect fn argument(self: &mut Self, value: string) -> ()
! OutOfMemoryError
? &mut AllocatorAdds the argument value. {name} in it is replaced by the value of the suite parameter
name.
Method arguments#
pub effect fn arguments(self: &mut Self, values: &[string]) -> ()
! OutOfMemoryError
? &mut AllocatorAdds each of values as an argument, as Call.argument does.
Method path#
pub effect fn path(self: &mut Self, value: Deferred<Path>) -> ()
! OutOfMemoryError
? &mut AllocatorAdds an argument that is the absolute path value resolves to. The step that produces
the path, if any, runs before the suite is timed.
Method directory#
pub fn directory(self: &mut Self, value: Deferred<Path>) -> ()Sets the working directory.
Method variable#
pub effect fn variable(self: &mut Self, name: string, value: string) -> ()
! OutOfMemoryError
? &mut AllocatorSets the environment variable name to value for the program.
Dialect#
pub enum DialectThe format in which a program prints its own results.
ResultLines#
ResultLinesResult lines, as silk.bench.lines describes them.
GoogleBenchmark#
GoogleBenchmarkThe console table of Google Benchmark.
Criterion#
CriterionThe lines of Criterion, or of Rust's built-in benchmark harness.
Implementation#
pub struct ImplementationOne program a suite times, with the hooks around each of its runs.
Field call#
pub call: CallThe program that is timed.
Field before#
pub before: List<Call>The calls that run before every timed run, without being timed.
Field after#
pub after: List<Call>The calls that run after every timed run, without being timed.
Field version#
pub version: Option<String>The revision recorded for the implementation.
Field reports#
pub reports: boolWhether the program prints result lines, which then are what the suite measures.
Field dialect#
pub dialect: DialectThe format of what the program prints, when it prints its results.
Field drives#
pub drives: boolWhether a benchmark task drives the program: it is started once, and asked for one sample at a time.
Implementation Implementation#
impl ImplementationAssociated function of#
pub fn of(call: Call) -> ImplementationReturns an implementation that times call.
Associated function program#
pub fn program(program: Deferred<Path>) -> ImplementationReturns an implementation that times the program at program, such as a compilation's
artifact() or the output of a Run step.
Associated function command#
pub effect fn command(name: string) -> Implementation
! StepFailure | FileError | HostInputError | OutOfMemoryError
? &mut HostInput | &mut Allocator | &mut FileSystemReturns an implementation that times the program named name, found as Call.command
finds it.
Errors#
StepError if no such program exists.
Method argument#
pub effect fn argument(self: &mut Self, value: string) -> ()
! OutOfMemoryError
? &mut AllocatorAdds the argument value, as Call.argument does.
Method arguments#
pub effect fn arguments(self: &mut Self, values: &[string]) -> ()
! OutOfMemoryError
? &mut AllocatorAdds each of values as an argument, as Call.argument does.
Method path#
pub effect fn path(self: &mut Self, value: Deferred<Path>) -> ()
! OutOfMemoryError
? &mut AllocatorAdds an argument that is a resolved path, as Call.path does.
Method directory#
pub fn directory(self: &mut Self, value: Deferred<Path>) -> ()Sets the working directory of the timed program.
Method variable#
pub effect fn variable(self: &mut Self, name: string, value: string) -> ()
! OutOfMemoryError
? &mut AllocatorSets the environment variable name to value for the timed program.
Method prepare#
pub effect fn prepare(self: &mut Self, call: Call) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a call that runs before every timed run of this implementation, such as one that removes a cache. It is not timed. A call that fails stops the run.
Method conclude#
pub effect fn conclude(self: &mut Self, call: Call) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a call that runs after every timed run of this implementation. It is not timed. A call that fails stops the run.
Method revision#
pub effect fn revision(self: &mut Self, text: string) -> ()
! OutOfMemoryError
? &mut AllocatorRecords text as the revision of this implementation, such as a commit hash or a version
number, so a report says what was measured.
Method resultLines#
pub fn resultLines(self: &mut Self) -> ()Says that the program prints result lines, as silk.bench.lines describes them, and that
the suite measures what those lines report, not the time of the process.
Details#
Each line is one sample of the benchmark its name gives, so one run of the program may
report many benchmarks and many samples of each. The program writes the lines to standard
output, or to the file the environment variable BENCH_OUT names, which the suite sets
for every timed run. Lines that are not result lines are ignored.
The suite reads four configuration keys. check: says what the program computed, such
as a checksum or a count: implementations whose checks differ for one benchmark are not
compared, and the task fails. revision: is recorded as the implementation's revision
when the suite gives none. note: is added to the notes of the run.
view: suggests a view, as silk.bench.view.View.parse reads it, which the reports show
when the suite declares none.
Method googleBenchmark#
pub fn googleBenchmark(self: &mut Self) -> ()Says that the program is a Google Benchmark program, and that the suite measures what its
console table reports, as Implementation.resultLines does for result lines.
Details#
Each row of the table is one sample: its Time column is reported as ns/op and its
CPU column as cpu-ns/op. The rows Google Benchmark adds for repetitions are left out.
silk.bench.lines.Lines.fromGoogleBenchmark does the reading. The program must print the
table to standard output, which is what it does without --benchmark_format.
Method criterion#
pub fn criterion(self: &mut Self) -> ()Says that the program is a Rust benchmark, run by Criterion or by the built-in harness,
and that the suite measures what it reports, as Implementation.resultLines does for
result lines.
Details#
Each name time: [low estimate high] line of Criterion is one sample, the estimate in
the middle, and so is each test name ... bench: 1,240 ns/iter (+/- 12) line. Both are
reported as ns/op. silk.bench.lines.Lines.fromCriterion does the reading. The program
must print to standard output, as cargo bench does.
Method driven#
pub fn driven(self: &mut Self) -> ()Says that the program is a harness the task drives, as silk.bench.transport describes:
the task starts it with BENCH_DRIVE=1, asks it for its benchmarks, and asks for one
sample at a time over its standard input and output.
Details#
The task finds how many iterations of each benchmark take the suite's sample time, and
then, in each round, takes one sample of each benchmark from every implementation, in a
random order. The programs stay alive between samples, and each is started again a few
times over a run, so that one unlucky start does not color every sample. The report is in
the unit of the harness, ns/op.
A Silk program answers with silk.bench.harness.Harness. A program in another language
reads the commands from its standard input and prints result lines, which takes a loop of
a few dozen lines.
Format#
pub enum FormatHow a report is written.
Terminal#
TerminalThe table for a terminal, of silk.bench.render.Render.terminal. It has color when it
goes to a terminal that takes it.
Markdown#
MarkdownMarkdown, of silk.bench.render.Render.markdown.
Html#
HtmlOne static HTML page with charts, of silk.bench.html.Html.page.
Report#
ReportThe report with its statistics as JSON, a silk.bench.record.Report, for a renderer of
your own.
Result#
ResultThe raw samples as JSON, a silk.bench.record.RunResult.
Lines#
LinesThe samples as result lines, of silk.bench.lines.Lines.fromResult, the text format of
Go's testing package that tools such as benchstat read.
History#
HistoryOne line for each row of the report, of silk.bench.history.History.lines, to add to a
history file with Output.appending.
Bencher#
BencherBencher Metric Format, of silk.bench.history.History.bencher, the JSON the Bencher
service reads.
Destination#
pub union DestinationWhere a report goes.
Stdout#
StdoutStandard output.
File#
FileA file. Its directories are made when they are missing.
Field path#
pub path: Deferred<Path>Field append#
pub append: boolOutput#
pub struct OutputOne report of a suite: a format and a destination.
Field format#
pub format: FormatHow the report is written.
Field destination#
pub destination: DestinationWhere it goes.
Implementation Output#
impl OutputAssociated function stdout#
pub fn stdout(format: Format) -> OutputReturns an output that prints the report in format on standard output.
Associated function file#
pub fn file(path: Deferred<Path>, format: Format) -> OutputReturns an output that writes the report in format to the file at path, replacing it.
Method appending#
pub fn appending(self: Self) -> OutputReturns self writing to the end of its file instead of replacing it, for a log that
grows with every run. An output to standard output is returned as it is.
Parameter#
pub struct ParameterA parameter of a suite: a name and the values the suite runs with.
Field name#
pub name: StringThe name, which {name} in an argument refers to.
Field values#
pub values: List<String>The values, in the order they run.
Named#
pub struct NamedAn implementation with the name a suite gave it.
Field name#
pub name: StringThe name.
Field implementation#
pub implementation: ImplementationThe implementation.
Suite#
pub struct SuiteA set of implementations of the same work, compared to each other.
Details#
The fields are public so a tool can read a suite. Build one with Suite.command and the
methods below.
Field name#
pub name: StringThe name, which starts the name of each of its benchmarks and of its result file.
Field implementations#
pub implementations: List<Named>The implementations, in the order they were added.
Field baseline#
pub baseline: Option<String>The name of the implementation the others are compared to. Without one, it is the first.
Field parameters#
pub parameters: List<Parameter>The parameters, in the order they were added.
Field processors#
pub processors: List<u32>The processor counts to run with. Empty means no limit.
Field rounds#
pub rounds: Option<u64>The number of measured rounds, if the suite sets it.
Field warmups#
pub warmups: Option<u64>The number of warm-up rounds, if the suite sets it.
Field sampling#
pub sampling: Option<u64>How long one sample of a driven implementation should take, in milliseconds, if the suite sets it.
Field notes#
pub notes: List<String>The notes recorded with every run.
Field recorded#
pub recorded: List<String>The environment variables whose values are recorded with every run.
Field before#
pub before: List<Call>The calls that run once before anything is timed.
Field after#
pub after: List<Call>The calls that run once after everything is timed.
Field outputs#
pub outputs: List<Output>The reports. Empty means the terminal table on standard output.
Field views#
pub views: List<View>The views. Empty means the report picks by the shape of the results.
Implementation Suite#
impl SuiteAssociated function command#
pub effect fn command(name: string) -> Suite
! OutOfMemoryError
? &mut AllocatorReturns an empty suite named name that times whole processes. name becomes part of
file names, so use letters, digits, -, _ and ..
Method implementation#
pub effect fn implementation(self: &mut Self, name: string, implementation: Implementation) -> ()
! OutOfMemoryError
? &mut AllocatorAdds implementation under name. Names must differ inside a suite.
Method reference#
pub effect fn reference(self: &mut Self, name: string) -> ()
! OutOfMemoryError
? &mut AllocatorNames the implementation the others are compared to. Without it, the first one added is the reference.
Method parameter#
pub effect fn parameter(self: &mut Self, name: string, values: &[string]) -> ()
! OutOfMemoryError
? &mut AllocatorAdds the parameter name with values. The suite runs once for each value, and once for
each combination when it has several parameters.
Method cpus#
pub effect fn cpus(self: &mut Self, counts: &[u32]) -> ()
! OutOfMemoryError
? &mut AllocatorRuns the suite once for each of counts, with every program limited to that many
processors, as the parameter cpus. The limit is an affinity mask over the first
processors the task may use.
Gotchas#
Only Linux can limit processors. On another system, a suite with this setting fails before anything is timed.
Method runs#
pub fn runs(self: &mut Self, count: u64) -> ()Sets the number of measured rounds. Every implementation runs once in each round, in a
random order. The default is 10. --runs on the command line wins.
Method warmup#
pub fn warmup(self: &mut Self, count: u64) -> ()Sets the number of warm-up rounds, which fill caches before the measured rounds and are
in no statistic. The default is 1. --warmup on the command line wins.
Method sample#
pub fn sample(self: &mut Self, milliseconds: u64) -> ()Sets how long one sample of a driven implementation should take, in milliseconds: the task asks for as many iterations as fill that time. The default is 50.
Method note#
pub effect fn note(self: &mut Self, text: string) -> ()
! OutOfMemoryError
? &mut AllocatorRecords text with every run of the suite, such as what a reader should know about the
setup.
Method record#
pub effect fn record(self: &mut Self, name: string) -> ()
! OutOfMemoryError
? &mut AllocatorRecords the value of the environment variable name with every run.
Method setup#
pub effect fn setup(self: &mut Self, call: Call) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a call that runs once before anything is timed. A call that fails stops the run.
Method cleanup#
pub effect fn cleanup(self: &mut Self, call: Call) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a call that runs once after everything is timed, also when a run failed.
Method output#
pub effect fn output(self: &mut Self, output: Output) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a report. A suite with at least one output prints nothing else: add
Output.stdout(Format.Terminal) to keep the table.
Method view#
pub effect fn view(self: &mut Self, view: View) -> ()
! OutOfMemoryError
? &mut AllocatorAdds a view: one more way the reports show the results. A suite with at least one view gets no other: without any, a report picks views by the shape of the results.
Category: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.