silk.bench.record.

The data of a benchmark run: the result file with every raw sample, and the report with the statistics a renderer draws.

When to use#

Read a RunResult to work with the raw samples of a run, such as to compare two saved runs. Read a Report to draw a run in a renderer of your own: every number a table or a chart needs is already in it, so a renderer does no statistics.

Details#

Both types are silk.schema.schema.Schema types, written as JSON with silk.schema.json.SchemaJson. silk.bench.bench writes the result as build/bench/<suite>-<time>.json, and the report wherever a suite declares an output with silk.bench.suite.Format.Report.

Times are nanoseconds and memory is bytes. A raw sample holds whole numbers; the statistics of a report are f64. Dates are RFC 3339 text in UTC, such as 2026-11-19T14:03:22Z.

A program that prints result lines is measured by what it reports, not by its process: each line is a Reading, and a report gives every name the lines carry a table of its own, in the unit of the lines.

A result keeps the views of its suite as text, one line each, as silk.bench.view.View.text writes them. A report holds each view resolved into a Chart: panels of series of points, with the statistic and the whiskers of every point already computed.

A run never drops a sample. A warm-up run is kept with warmup set, and a run whose program failed is kept with its exit status. A report counts the failed runs of a row in failures and leaves both kinds out of its statistics.

Examples#

silk
import silk.bench.record { Report }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.schema.issue { SchemaError }
import silk.schema.json { SchemaJson }

// The fastest median of the first benchmark of a report, in nanoseconds.
effect fn fastest(text: &[u8]) -> f64 ! SchemaError | OutOfMemoryError ? &mut Allocator {
  let report = run SchemaJson.decode<Report>(text)
  let rows = report.benchmarks.asSlice()[0].rows.asSlice()
  let mut best = rows[0].median
  let mut at: usize = 1
  while at < rows.length {
    if rows[at].median < best {
      best = rows[at].median
    }
    at = at + 1
  }
  return best
}

// Text that is not a report is an error, never a guess.
effect fn check() -> i32 ! SchemaError | OutOfMemoryError ? &mut Allocator {
  let best = run fastest(b"{}")
  if best > 0.0 {
    return 1
  }
  return 2
}

effect fn failed(error: SchemaError | OutOfMemoryError) -> i32 {
  drop error
  return 0
}

pub fn main() -> i32 {
  let mut allocator = Allocator.systemAllocatorProvider()
  return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as Report with import silk.bench.record { Report }.

Public declarations: 19.

PROTOCOL#

silk
pub const PROTOCOL: u64

The version of the result and report files this module writes.

Pair#

silk
pub struct Pair

A name with a value: a parameter of a benchmark, or a recorded environment variable.

Field name#

silk
pub name: String

The name.

Field value#

silk
pub value: String

The value.

Implementation Schema for Pair#

silk
impl Schema for Pair

Environment#

silk
pub struct Environment

The computer a run was measured on.

Field system#

silk
pub system: String

The kernel name, such as Linux or Darwin.

Field release#

silk
pub release: String

The kernel release, such as 6.8.0-45-generic.

Field machine#

silk
pub machine: String

The hardware name, such as x86_64 or arm64.

Field processor#

silk
pub processor: String

The model text of the first processor, or unknown.

Field processors#

silk
pub processors: u64

The number of logical processors online.

Field usable#

silk
pub usable: u64

The number of processors the run could use, which a container or an affinity mask lowers.

Field memory#

silk
pub memory: u64

The physical memory, in bytes.

Field load#

silk
pub load: f64

The 1 minute load average when the run started. A value well above 0 on an idle computer means something else was running.

Field variables#

silk
pub variables: List<Pair>

The environment variables the suite asked to record.

Implementation Schema for Environment#

silk
impl Schema for Environment

Subject#

silk
pub struct Subject

One implementation of a suite, as it was run.

Field name#

silk
pub name: String

The name the suite gave it.

Field command#

silk
pub command: String

The program and its arguments, as one line for a reader. Placeholders are not filled in.

Field revision#

silk
pub revision: Option<String>

The revision the suite recorded for it, such as a commit hash or a version.

Implementation Schema for Subject#

silk
impl Schema for Subject

Sample#

silk
pub struct Sample

One timed run of one program.

Field implementation#

silk
pub implementation: String

The name of the implementation.

Field round#

silk
pub round: u64

The round the run belongs to, from 0. Every implementation runs once in each round.

Field position#

silk
pub position: u64

The position of the run inside its round, from 0. The order is random in each round.

Field warmup#

silk
pub warmup: bool

Whether the run was a warm-up, which statistics leave out.

Field wall#

silk
pub wall: u64

The time from start to exit, in nanoseconds.

Field user#

silk
pub user: u64

The processor time in user mode, in nanoseconds, summed over the threads of the program and the children it waited for.

Field system#

silk
pub system: u64

The processor time in kernel mode, in nanoseconds.

Field peak#

silk
pub peak: u64

The peak resident memory, in bytes.

Field exit#

silk
pub exit: i64

The exit status, or -1 when a signal ended the program.

Field signal#

silk
pub signal: i64

The signal that ended the program, or 0.

Field counts#

silk
pub counts: List<Metric>

What the system's counters measured over the run, such as instructions or page-faults. Empty unless the run asked for counters with --counters.

Implementation Schema for Sample#

silk
impl Schema for Sample

Metric#

silk
pub struct Metric

One value of a reading with its unit.

Field unit#

silk
pub unit: String

The unit as the program wrote it, such as ns/op, B/op or req/s.

Field value#

silk
pub value: f64

The value, for one iteration.

Implementation Schema for Metric#

silk
impl Schema for Metric

Reading#

silk
pub struct Reading

One sample a program reported as a result line, as silk.bench.lines reads it.

Field name#

silk
pub name: String

The name the program gave the sample, without the leading Benchmark.

Field implementation#

silk
pub implementation: String

The name of the implementation whose run printed the line.

Field round#

silk
pub round: u64

The round of the run that printed the line, from 0.

Field warmup#

silk
pub warmup: bool

Whether that run was a warm-up, which statistics leave out.

Field iterations#

silk
pub iterations: u64

The number of iterations the sample covers.

Field values#

silk
pub values: List<Metric>

The values of the sample, in the order the program wrote them.

Field check#

silk
pub check: Option<String>

What the program said it computed, from its last check: line before this one. Two implementations with different checks for one name did not do the same work.

Implementation Schema for Reading#

silk
impl Schema for Reading

Count#

silk
pub struct Count

What an instrument counted over one whole run of an implementation's program, such as the instructions --instrument callgrind reports.

Field implementation#

silk
pub implementation: String

The name of the implementation.

Field unit#

silk
pub unit: String

What was counted, such as instructions.

Field value#

silk
pub value: f64

The count for one run of the program, from start to exit.

Implementation Schema for Count#

silk
impl Schema for Count

Measured#

silk
pub struct Measured

The samples of one combination of a suite's parameters.

Field name#

silk
pub name: String

The name: the suite's name, then /name=value for each parameter.

Field parameters#

silk
pub parameters: List<Pair>

The parameters of this combination, in the order the suite declared them.

Field samples#

silk
pub samples: List<Sample>

Every run, in the order it happened.

Field readings#

silk
pub readings: List<Reading>

Every result line the runs printed, in the order they were read. Empty when no implementation reports result lines.

Field counts#

silk
pub counts: List<Count>

What an instrument counted for each implementation, one run each. Empty when the run named no instrument.

Implementation Schema for Measured#

silk
impl Schema for Measured

RunResult#

silk
pub struct RunResult

A whole run of one suite with its raw samples: the content of a result file.

Field protocol#

silk
pub protocol: u64

The version of this format, PROTOCOL.

Field suite#

silk
pub suite: String

The name of the suite.

Field id#

silk
pub id: String

The run's identity: its start time and a random part, safe in a file name.

Field label#

silk
pub label: Option<String>

The label the run was given, such as before or after.

Field revision#

silk
pub revision: Option<String>

The revision of the project the run measured, such as a commit hash.

Field started#

silk
pub started: String

When the run started, as RFC 3339 text in UTC.

Field ended#

silk
pub ended: String

When the run ended, as RFC 3339 text in UTC.

Field notes#

silk
pub notes: List<String>

The notes of the suite and of the command line.

Field environment#

silk
pub environment: Environment

The computer the run was measured on.

Field implementations#

silk
pub implementations: List<Subject>

The implementations, in the order the suite declared them.

Field reference#

silk
pub reference: String

The name of the implementation the others are compared to.

Field runs#

silk
pub runs: u64

The number of measured rounds asked for.

Field warmups#

silk
pub warmups: u64

The number of warm-up rounds asked for.

Field floor#

silk
pub floor: u64

The smallest peak memory the run could see, in bytes, or 0 when there is none. On Linux a new process starts with the peak of the process that started it, so a program that uses less memory than the benchmark task reads as the task's own peak.

Field benchmarks#

silk
pub benchmarks: List<Measured>

The benchmarks, one for each combination of parameters.

Field views#

silk
pub views: List<String>

The views to show the results with, one line each. Empty means the report picks by the shape of the results.

Implementation Schema for RunResult#

silk
impl Schema for RunResult

Comparison#

silk
pub struct Comparison

How one implementation compares to the reference on one benchmark.

Field value#

silk
pub value: f64

The median divided by the reference's median. For a time, 0.5 is twice as fast.

Field low#

silk
pub low: f64

The low end of the 95% interval of the ratio.

Field high#

silk
pub high: f64

The high end of the 95% interval of the ratio.

Field verdict#

silk
pub verdict: String

faster, slower, or same when the interval does not clear the noise threshold.

Implementation Schema for Comparison#

silk
impl Schema for Comparison

Row#

silk
pub struct Row

The statistics of one implementation on one benchmark.

Field implementation#

silk
pub implementation: String

The name of the implementation.

Field reference#

silk
pub reference: bool

Whether this is the reference the other rows are compared to.

Field runs#

silk
pub runs: u64

The number of values the statistics cover: the measured runs that succeeded, or the result lines those runs printed for this benchmark.

Field failures#

silk
pub failures: u64

The number of measured runs whose program failed. They are in no statistic.

Field median#

silk
pub median: f64

The median, in the table's unit: for process times, the time from start to exit in nanoseconds.

Field minimum#

silk
pub minimum: f64

The smallest value, in the table's unit.

Field maximum#

silk
pub maximum: f64

The largest value, in the table's unit.

Field mean#

silk
pub mean: f64

The mean, in the table's unit.

Field low#

silk
pub low: f64

The low end of the 95% interval of the median, in the table's unit.

Field high#

silk
pub high: f64

The high end of the 95% interval of the median, in the table's unit.

Field outliers#

silk
pub outliers: u64

How many runs lie far outside the others. They are counted, never removed.

Field user#

silk
pub user: f64

The median processor time in user mode, in nanoseconds. 0 in a table of result lines.

Field system#

silk
pub system: f64

The median processor time in kernel mode, in nanoseconds. 0 in a table of result lines.

Field peak#

silk
pub peak: f64

The median peak resident memory, in bytes.

Field ratio#

silk
pub ratio: Option<Comparison>

The comparison to the reference. Absent for the reference itself, for a row with no successful run, and in a table whose checks differ.

Field times#

silk
pub times: List<f64>

Every value the statistics cover, in the table's unit, in the order they were measured, for a renderer that draws the distribution.

Field others#

silk
pub others: List<Metric>

The median of each other unit the row's readings carry, in the order the units first appear. Empty for a table of process times.

Field check#

silk
pub check: Option<String>

What the implementation said it computed, when its readings carry a check.

Implementation Schema for Row#

silk
impl Schema for Row

Table#

silk
pub struct Table

The rows of one benchmark.

Field name#

silk
pub name: String

The name of the benchmark.

Field parameters#

silk
pub parameters: List<Pair>

The parameters of the benchmark.

Field unit#

silk
pub unit: String

The unit of the statistics of the rows: ns for the time of a whole process, or the unit of the result lines, such as ns/op or req/s.

Field higher#

silk
pub higher: bool

Whether a larger value is better, as for req/s. A ratio above 1 is then faster.

Field mismatch#

silk
pub mismatch: bool

Whether two rows carry different checks. The rows then have no ratio: the implementations did not do the same work.

Field rows#

silk
pub rows: List<Row>

One row for each implementation, in the order the suite declared them.

Implementation Schema for Table#

silk
impl Schema for Table

Point#

silk
pub struct Point

One value of a Series.

Field x#

silk
pub x: String

The entry of the axis the point belongs to, such as 1024, p99 or a benchmark name.

Field at#

silk
pub at: f64

Where the point is along the axis: the number x holds when the axis is numeric, and else the position of x among the entries of the panel, from 0.

Field value#

silk
pub value: f64

The value, in the unit of the chart, or as a ratio when the chart is normalized.

Field low#

silk
pub low: f64

The low end of the whisker. Equal to value when there is none.

Field high#

silk
pub high: f64

The high end of the whisker. Equal to value when there is none.

Field faded#

silk
pub faded: bool

Whether the point is a warm-up run of a distribution, which a renderer dims.

Implementation Schema for Point#

silk
impl Schema for Point

Series#

silk
pub struct Series

One line, or one group of bars, of a Panel.

Field name#

silk
pub name: String

The name, such as the implementation.

Field points#

silk
pub points: List<Point>

The points, in the order of the axis.

Implementation Schema for Series#

silk
impl Schema for Series

Panel#

silk
pub struct Panel

One separate chart of a Chart.

Field title#

silk
pub title: String

What the panel shows, such as a benchmark name. Empty when the chart has one panel that covers everything.

Field series#

silk
pub series: List<Series>

The series, in the order they first appear in the results.

Implementation Schema for Panel#

silk
impl Schema for Panel

Chart#

silk
pub struct Chart

A view of a suite with every number computed: what a renderer draws.

Field kind#

silk
pub kind: String

The kind of the view, such as line or x-flame.

Field title#

silk
pub title: String

The heading.

Field y#

silk
pub y: String

The unit of the values.

Field x#

silk
pub x: String

What runs along the axis.

Field series#

silk
pub series: String

What each series is.

Field facet#

silk
pub facet: String

What each panel is. Empty when the panels are whatever x and series leave.

Field stat#

silk
pub stat: String

The statistic of the samples each point shows.

Field error#

silk
pub error: String

What the whiskers show: interval, range or none.

Field scale#

silk
pub scale: String

The scale: linear, log or log-log.

Field zero#

silk
pub zero: bool

Whether the value axis starts at zero.

Field normalize#

silk
pub normalize: String

The implementation every value is divided by. Empty when the values are in the unit y.

Field sort#

silk
pub sort: String

The order of the entries of x: value, -value, or empty for the order they ran in.

Field higher#

silk
pub higher: bool

Whether a larger value is better, as for req/s.

Field numeric#

silk
pub numeric: bool

Whether every entry of x is a number, so the axis is numeric.

Field fields#

silk
pub fields: List<Pair>

The fields the view set, also the ones the report does not know, for a renderer of your own.

Field panels#

silk
pub panels: List<Panel>

The panels. A history gets its panels from silk.bench.history.History.drawn, which knows the recorded runs.

Implementation Schema for Chart#

silk
impl Schema for Chart

Overall#

silk
pub struct Overall

How one implementation compares to the reference over a whole suite.

Field implementation#

silk
pub implementation: String

The name of the implementation.

Field ratio#

silk
pub ratio: f64

The geometric mean of its ratios to the reference, as time: the ratio of a table where a larger value is better is inverted first, so below 1 always means faster.

Field benchmarks#

silk
pub benchmarks: u64

The number of benchmarks the mean covers.

Implementation Schema for Overall#

silk
impl Schema for Overall

Report#

silk
pub struct Report

A run of one suite with its statistics: everything a renderer draws.

Field protocol#

silk
pub protocol: u64

The version of this format, PROTOCOL.

Field suite#

silk
pub suite: String

The name of the suite.

Field id#

silk
pub id: String

The run's identity, the same as in its result file.

Field label#

silk
pub label: Option<String>

The label the run was given.

Field revision#

silk
pub revision: Option<String>

The revision of the project the run measured, such as a commit hash.

Field started#

silk
pub started: String

When the run started, as RFC 3339 text in UTC.

Field ended#

silk
pub ended: String

When the run ended, as RFC 3339 text in UTC.

Field notes#

silk
pub notes: List<String>

The notes of the suite and of the command line.

Field environment#

silk
pub environment: Environment

The computer the run was measured on.

Field implementations#

silk
pub implementations: List<Subject>

The implementations, in the order the suite declared them.

Field reference#

silk
pub reference: String

The name of the implementation the others are compared to.

Field noise#

silk
pub noise: f64

The noise threshold of the verdicts, as a fraction: 0.01 is 1%.

Field runs#

silk
pub runs: u64

The number of measured rounds asked for.

Field floor#

silk
pub floor: u64

The smallest peak memory the run could see, in bytes, or 0 when there is none. A row whose peak is not above it used at most this much.

Field benchmarks#

silk
pub benchmarks: List<Table>

The benchmarks, one for each combination of parameters.

Field overall#

silk
pub overall: List<Overall>

The geometric mean of each implementation's ratios, without the reference. Empty when the suite has one benchmark, where the mean would repeat the ratio.

Field views#

silk
pub views: List<Chart>

The views of the suite, resolved. Without declared views: a ratio when several implementations ran several benchmarks, and a line for each parameter whose values are all numbers.

Implementation Schema for Report#

silk
impl Schema for Report

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page