silk.bench.bench.

Runs benchmark suites from a project's tasks and writes their reports.

When to use#

Use Bench.perform in a task of src/tasks.silk to compare programs: a Silk program with the same program in another language, two versions of one program, or one program with different options. There is no separate benchmark command: a project declares a task, by custom named Bench, and runs it with silk task bench.

Details#

A task builds its suites with silk.bench.suite and gives them to Bench.perform with the BenchOptions it read from the command line. For each suite, in order:

  1. The build steps the suite's paths come from run, with the build system's cache.

  2. The suite's setup calls run.

  3. For each benchmark, the warm-up rounds run and then the measured rounds. Every implementation runs once in each round, as a new process, in a random order, so a slow minute of the computer does not fall on one implementation. Each run records its time from start to exit, its processor time, and its peak memory.

  4. The suite's cleanup calls run, also after a failure.

  5. The result file with every raw sample is written to build/bench/<suite>-<time>.json.

  6. The statistics are computed and each declared output is written. A suite without outputs prints the terminal table.

An implementation declared with Implementation.resultLines is measured by the result lines its program prints, as silk.bench.lines describes them: each line is a sample, each name the lines carry is a benchmark of its own, and its table is in the unit of the lines. The task reads the lines from the program's standard output and from the file it names in the environment variable BENCH_OUT. After every suite has been reported, the task fails if two implementations gave different check: values for one benchmark.

A program built on another benchmark library is read the same way without changing it: Implementation.googleBenchmark reads the console table of Google Benchmark, and Implementation.criterion reads what Criterion or Rust's built-in harness prints. Their rows become result lines first, so a Google Benchmark row, a Criterion line and a Silk bench fn that carry the same name are compared in one table.

An implementation declared with Implementation.driven is a harness the task drives, as silk.bench.transport describes: the task starts it once, with BENCH_DRIVE set to 1, asks it for the names of its benchmarks, finds for each the iteration count that makes one sample last the suite's Suite.sample time (50 ms unless set), and then asks every implementation for one sample in each round, in a random order. A harness is restarted a few times in a benchmark's rounds, so the layout of one process does not decide a result. A Silk harness is a program that uses silk.bench.harness, or one whose main runs the bench fn declarations of its modules with silk.bench.bencher.Benchmarks.main: compile it with Compilation.fromPackage(&options) and root set to its entry file, and give the artifact to Implementation.program.

Every run records the revision it measured: --revision, or else the commit that HEAD names in the Git repository the package is in. --record adds the run to the suite's history, bench/history/<suite>.jsonl, a text file to commit. --history runs nothing and prints each suite's history as the tables of silk.bench.history, also as a page with charts with --html and as Markdown with --markdown. --log "text" runs nothing and adds the text to the journal, bench/journal.md, under the revision, or under the day without one; the history shows it beside the first run of that revision.

A report shows the views the suite declares with Suite.view, as silk.bench.view describes them. A suite without views shows the ones its programs suggest in view: lines, and without those the report picks by the shape of the results. A history view reads the suite's history file and adds the run being reported.

A run of a program that exits with a status other than 0 is kept in the result file, counted in the report, and left out of the statistics. After every suite has been reported, the task fails if any measured run failed.

The statistics are those of silk.bench.statistics: the median with a 95% interval, and for each implementation the ratio to the suite's reference with an interval and a verdict. --noise sets how large a difference must be to count; the default is 1%.

--save <name> also keeps the result of each suite under a name, in build/bench/saved/<suite>/<name>.json. --baseline <name> adds the implementations of the run kept under that name to the report, named <name> for a suite with one implementation and <name>:<implementation> otherwise, and compares the new samples with them. So --save main before a change and --baseline main after it shows what the change did.

--instrument callgrind also runs each implementation's program once under Valgrind's Callgrind, after the timed rounds, and adds the instructions it executed to the implementation's row. Instruction counts barely vary between runs, so they show a small change in one implementation that timing cannot. They compare two languages poorly, because instructions differ in cost. The count covers the program from start to exit and not the processes it starts. A driven program is not counted, and a program that prints result lines gets the count only when it reports one benchmark. valgrind must be on PATH.

--counters reads the counters of the processor and the kernel over every timed run, on Linux on x86_64 and aarch64, and adds the median of each to the implementation's row: instructions, cycles, cache-misses, branch-misses and page-faults, all in user mode. They cover the program and the processes it starts and cost the run nothing measurable. A counter the system does not offer is left out without an error: a virtual machine or a container often offers only the last, and the hardware counters need /proc/sys/kernel/perf_event_paranoid at 2 or lower.

Gotchas#

  • Processes can be measured on Linux with GNU libc and on macOS.

  • Statistics cannot repair a noisy computer. Close other programs, and distrust a run whose report shows a load well above 0.

  • A function cannot be named run, so the entry point is Bench.perform.

Examples#

silk
import silk.bench.bench { Bench, BenchError, BenchOptions }
import silk.bench.suite { Implementation, Suite }
import silk.build.graph { Compilation, Options }
import silk.build.step { Deferred }
import silk.cli.command { Command, CommandLine }
import silk.cli.task { Task, TaskSet }
import silk.mem.allocator { Allocator }
import silk.system.child_process { ChildProcess }
import silk.system.filesystem { FileSystem, Path }
import silk.system.host_input { HostInput }

union Tasks {
  Bench { pub options: Options, pub bench: BenchOptions },
}

impl CommandLine for Tasks {
  static fn command() -> Command<Tasks> {
    return Command<Tasks>.make("tasks", "This project's tasks")
      |> Command.variant("Bench", "Compare this program with the C one")
      |> Command.flatten("options")
      |> Command.flatten("bench")
  }
}

impl TaskSet for Tasks {
  effect fn perform(self: Tasks) -> ()
    ! BenchError
    ? &mut HostInput | &mut Allocator | &mut ChildProcess | &mut FileSystem {
    match move self {
      Tasks.Bench { options, bench } => {
        let compilation = run Compilation.fromPackage(&options)
        let mut silk = Implementation.program(run compilation.artifact())
        run silk.path(run Deferred<Path>.source("data/words.txt"))
        let mut c = Implementation.program(run Deferred<Path>.source("c/count"))
        run c.path(run Deferred<Path>.source("data/words.txt"))

        let mut suite = run Suite.command("count-words")
        run suite.implementation("silk", move silk)
        run suite.implementation("c", move c)
        run suite.reference("c")
        run Bench.perform(&bench, &[move suite])
      }
    }
  }
}

pub effect fn main() -> i32 ? &mut HostInput {
  return run Task.main<Tasks>()
}

silk task bench --release --runs 20 --html then builds the program optimized, runs 20 rounds, prints the table, and writes build/bench/count-words.html.

Import as Bench with import silk.bench.bench { Bench }.

Public declarations: 3.

BenchError#

silk
pub type BenchError = TaskError
  | StepFailure
  | SpawnError
  | Cancelled
  | FileError
  | HostInputError
  | OutOfMemoryError
  | WriterError
  | SchemaError

Every way a benchmark task can fail. silk.cli.task.Task.main reports each of them.

BenchOptions#

silk
pub struct BenchOptions

The command-line options every benchmark task shares. A task embeds them with Command.flatten.

Field filter#

silk
pub filter: List<String>

Run only the suites whose name contains this text. Repeat it to select more.

Field runs#

silk
pub runs: Option<u64>

The number of measured rounds, over what a suite sets.

Field warmup#

silk
pub warmup: Option<u64>

The number of warm-up rounds, over what a suite sets.

Field color#

silk
pub color: String

When the terminal table has color: auto, always, or never.

Field html#

silk
pub html: bool

Also write each suite's page with charts to build/bench/<suite>.html.

Field markdown#

silk
pub markdown: bool

Also write each suite's Markdown report to build/bench/<suite>.md.

Field lines#

silk
pub lines: bool

Also write each suite's samples as result lines to build/bench/<suite>.txt.

Field note#

silk
pub note: List<String>

Notes recorded with the run. Repeat it to add more.

Field label#

silk
pub label: Option<String>

A label for the run, such as before or after.

Field revision#

silk
pub revision: Option<String>

The revision of the project the run measures. Without it, the commit that HEAD names in the Git repository the package is in.

Field record#

silk
pub record: bool

Also add the run to the suite's history, bench/history/<suite>.jsonl.

Field history#

silk
pub history: bool

Print each suite's history and run nothing.

Field log#

silk
pub log: Option<String>

Add this text to the journal, bench/journal.md, under the revision, and run nothing.

Field noise#

silk
pub noise: String

How large a difference must be to count, in percent.

Field save#

silk
pub save: Option<String>

Also keep each suite's result under this name, as build/bench/saved/<suite>/<name>.json.

Field baseline#

silk
pub baseline: Option<String>

Compare each suite with the result kept under this name.

Field instrument#

silk
pub instrument: Option<String>

Also run each implementation's program once under this instrument and report what it counted. callgrind counts instructions with Valgrind.

Field counters#

silk
pub counters: bool

Also read the counters of the processor and the kernel over each timed run, where the system offers them: instructions, cycles, cache misses, branch misses and page faults.

Implementation BenchOptions#

silk
impl BenchOptions

Associated function make#

silk
pub effect fn make() -> Self
    ! OutOfMemoryError
    ? &mut Allocator

Returns the options of a command line that gives none of them.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Implementation CommandLine for BenchOptions#

silk
impl CommandLine for BenchOptions

Operation command#

silk
static fn command() -> Command<Self>

Bench#

silk
pub struct Bench

The owner type of the benchmark entry point. It has no fields and is never constructed.

Implementation Bench#

silk
impl Bench

Associated function perform#

silk
pub effect fn perform(options: &BenchOptions, suites: &[Suite]) -> ()
    ! BenchError
    ? &mut HostInput | &mut Allocator | &mut FileSystem

Runs each of suites that options selects and writes its reports, as the module describes.

Errors#

  • TaskError with status 2 if a suite is not valid, such as one without implementations or with a reference it does not have, or if --noise is not a number.

  • TaskError if a setup, prepare, conclude or cleanup call fails, if a program cannot be started, or, after every suite has been reported, if a measured run failed.

  • StepFailure if a build step a path comes from fails.

  • FileError if a report cannot be written.

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page