silk.bench.bench.
Runs benchmark suites from a project's tasks and writes their reports.
When to use#
Use Bench.perform in a task of src/tasks.silk to compare programs: a Silk program with
the same program in another language, two versions of one program, or one program with
different options. There is no separate benchmark command: a project declares a task, by
custom named Bench, and runs it with silk task bench.
Details#
A task builds its suites with silk.bench.suite and gives them to Bench.perform with
the BenchOptions it read from the command line. For each suite, in order:
The build steps the suite's paths come from run, with the build system's cache.
The suite's setup calls run.
For each benchmark, the warm-up rounds run and then the measured rounds. Every implementation runs once in each round, as a new process, in a random order, so a slow minute of the computer does not fall on one implementation. Each run records its time from start to exit, its processor time, and its peak memory.
The suite's cleanup calls run, also after a failure.
The result file with every raw sample is written to
build/bench/<suite>-<time>.json.The statistics are computed and each declared output is written. A suite without outputs prints the terminal table.
An implementation declared with Implementation.resultLines is measured by the result lines
its program prints, as silk.bench.lines describes them: each line is a sample, each name the
lines carry is a benchmark of its own, and its table is in the unit of the lines. The task
reads the lines from the program's standard output and from the file it names in the
environment variable BENCH_OUT. After every suite has been reported, the task fails if two
implementations gave different check: values for one benchmark.
A program built on another benchmark library is read the same way without changing it:
Implementation.googleBenchmark reads the console table of Google Benchmark, and
Implementation.criterion reads what Criterion or Rust's built-in harness prints. Their
rows become result lines first, so a Google Benchmark row, a Criterion line and a Silk
bench fn that carry the same name are compared in one table.
An implementation declared with Implementation.driven is a harness the task drives, as
silk.bench.transport describes: the task starts it once, with BENCH_DRIVE set to 1,
asks it for the names of its benchmarks, finds for each the iteration count that makes one
sample last the suite's Suite.sample time (50 ms unless set), and then asks every
implementation for one sample in each round, in a random order. A harness is restarted
a few times in a benchmark's rounds, so the layout of one process does not decide a result.
A Silk harness is a program that uses silk.bench.harness, or one whose main runs the
bench fn declarations of its modules with silk.bench.bencher.Benchmarks.main: compile it
with Compilation.fromPackage(&options) and root set to its entry file, and give the
artifact to Implementation.program.
Every run records the revision it measured: --revision, or else the commit that HEAD
names in the Git repository the package is in. --record adds the run to the suite's
history, bench/history/<suite>.jsonl, a text file to commit. --history runs nothing and
prints each suite's history as the tables of silk.bench.history, also as a page with charts
with --html and as Markdown with --markdown. --log "text" runs nothing and adds the
text to the journal, bench/journal.md, under the revision, or under the day without one;
the history shows it beside the first run of that revision.
A report shows the views the suite declares with Suite.view, as silk.bench.view
describes them. A suite without views shows the ones its programs suggest in view: lines,
and without those the report picks by the shape of the results. A history view reads the
suite's history file and adds the run being reported.
A run of a program that exits with a status other than 0 is kept in the result file, counted in the report, and left out of the statistics. After every suite has been reported, the task fails if any measured run failed.
The statistics are those of silk.bench.statistics: the median with a 95% interval, and for
each implementation the ratio to the suite's reference with an interval and a verdict.
--noise sets how large a difference must be to count; the default is 1%.
--save <name> also keeps the result of each suite under a name, in
build/bench/saved/<suite>/<name>.json. --baseline <name> adds the implementations of the
run kept under that name to the report, named <name> for a suite with one implementation
and <name>:<implementation> otherwise, and compares the new samples with them. So
--save main before a change and --baseline main after it shows what the change did.
--instrument callgrind also runs each implementation's program once under Valgrind's
Callgrind, after the timed rounds, and adds the instructions it executed to the
implementation's row. Instruction counts barely vary between runs, so they show a small
change in one implementation that timing cannot. They compare two languages poorly, because
instructions differ in cost. The count covers the program from start to exit and not the
processes it starts. A driven program is not counted, and a program that prints result lines
gets the count only when it reports one benchmark. valgrind must be on PATH.
--counters reads the counters of the processor and the kernel over every timed run, on
Linux on x86_64 and aarch64, and adds the median of each to the implementation's row:
instructions, cycles, cache-misses, branch-misses and page-faults, all in user
mode. They cover the program and the processes it starts and cost the run
nothing measurable. A counter the system does not offer is left out without an error: a
virtual machine or a container often offers only the last, and the hardware counters
need /proc/sys/kernel/perf_event_paranoid at 2 or lower.
Gotchas#
Processes can be measured on Linux with GNU libc and on macOS.
Statistics cannot repair a noisy computer. Close other programs, and distrust a run whose report shows a load well above 0.
A function cannot be named
run, so the entry point isBench.perform.
Examples#
import silk.bench.bench { Bench, BenchError, BenchOptions }
import silk.bench.suite { Implementation, Suite }
import silk.build.graph { Compilation, Options }
import silk.build.step { Deferred }
import silk.cli.command { Command, CommandLine }
import silk.cli.task { Task, TaskSet }
import silk.mem.allocator { Allocator }
import silk.system.child_process { ChildProcess }
import silk.system.filesystem { FileSystem, Path }
import silk.system.host_input { HostInput }
union Tasks {
Bench { pub options: Options, pub bench: BenchOptions },
}
impl CommandLine for Tasks {
static fn command() -> Command<Tasks> {
return Command<Tasks>.make("tasks", "This project's tasks")
|> Command.variant("Bench", "Compare this program with the C one")
|> Command.flatten("options")
|> Command.flatten("bench")
}
}
impl TaskSet for Tasks {
effect fn perform(self: Tasks) -> ()
! BenchError
? &mut HostInput | &mut Allocator | &mut ChildProcess | &mut FileSystem {
match move self {
Tasks.Bench { options, bench } => {
let compilation = run Compilation.fromPackage(&options)
let mut silk = Implementation.program(run compilation.artifact())
run silk.path(run Deferred<Path>.source("data/words.txt"))
let mut c = Implementation.program(run Deferred<Path>.source("c/count"))
run c.path(run Deferred<Path>.source("data/words.txt"))
let mut suite = run Suite.command("count-words")
run suite.implementation("silk", move silk)
run suite.implementation("c", move c)
run suite.reference("c")
run Bench.perform(&bench, &[move suite])
}
}
}
}
pub effect fn main() -> i32 ? &mut HostInput {
return run Task.main<Tasks>()
}silk task bench --release --runs 20 --html then builds the program optimized, runs 20
rounds, prints the table, and writes build/bench/count-words.html.
Import as Bench with import silk.bench.bench { Bench }.
Public declarations: 3.
BenchError#
pub type BenchError = TaskError
| StepFailure
| SpawnError
| Cancelled
| FileError
| HostInputError
| OutOfMemoryError
| WriterError
| SchemaErrorEvery way a benchmark task can fail. silk.cli.task.Task.main reports each of them.
BenchOptions#
pub struct BenchOptionsThe command-line options every benchmark task shares. A task embeds them with
Command.flatten.
Field filter#
pub filter: List<String>Run only the suites whose name contains this text. Repeat it to select more.
Field runs#
pub runs: Option<u64>The number of measured rounds, over what a suite sets.
Field warmup#
pub warmup: Option<u64>The number of warm-up rounds, over what a suite sets.
Field color#
pub color: StringWhen the terminal table has color: auto, always, or never.
Field html#
pub html: boolAlso write each suite's page with charts to build/bench/<suite>.html.
Field markdown#
pub markdown: boolAlso write each suite's Markdown report to build/bench/<suite>.md.
Field lines#
pub lines: boolAlso write each suite's samples as result lines to build/bench/<suite>.txt.
Field note#
pub note: List<String>Notes recorded with the run. Repeat it to add more.
Field label#
pub label: Option<String>A label for the run, such as before or after.
Field revision#
pub revision: Option<String>The revision of the project the run measures. Without it, the commit that HEAD names in
the Git repository the package is in.
Field record#
pub record: boolAlso add the run to the suite's history, bench/history/<suite>.jsonl.
Field history#
pub history: boolPrint each suite's history and run nothing.
Field log#
pub log: Option<String>Add this text to the journal, bench/journal.md, under the revision, and run nothing.
Field noise#
pub noise: StringHow large a difference must be to count, in percent.
Field save#
pub save: Option<String>Also keep each suite's result under this name, as build/bench/saved/<suite>/<name>.json.
Field baseline#
pub baseline: Option<String>Compare each suite with the result kept under this name.
Field instrument#
pub instrument: Option<String>Also run each implementation's program once under this instrument and report what it
counted. callgrind counts instructions with Valgrind.
Field counters#
pub counters: boolAlso read the counters of the processor and the kernel over each timed run, where the system offers them: instructions, cycles, cache misses, branch misses and page faults.
Implementation BenchOptions#
impl BenchOptionsAssociated function make#
pub effect fn make() -> Self
! OutOfMemoryError
? &mut AllocatorReturns the options of a command line that gives none of them.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Implementation CommandLine for BenchOptions#
impl CommandLine for BenchOptionsOperation command#
static fn command() -> Command<Self>Bench#
pub struct BenchThe owner type of the benchmark entry point. It has no fields and is never constructed.
Implementation Bench#
impl BenchAssociated function perform#
pub effect fn perform(options: &BenchOptions, suites: &[Suite]) -> ()
! BenchError
? &mut HostInput | &mut Allocator | &mut FileSystemRuns each of suites that options selects and writes its reports, as the module
describes.
Errors#
TaskErrorwith status 2 if a suite is not valid, such as one without implementations or with a reference it does not have, or if--noiseis not a number.TaskErrorif a setup, prepare, conclude or cleanup call fails, if a program cannot be started, or, after every suite has been reported, if a measured run failed.StepFailureif a build step a path comes from fails.FileErrorif a report cannot be written.
Category: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.