silk.bench.record.
The data of a benchmark run: the result file with every raw sample, and the report with the statistics a renderer draws.
When to use#
Read a RunResult to work with the raw samples of a run, such as to compare two saved
runs. Read a Report to draw a run in a renderer of your own: every number a table or a
chart needs is already in it, so a renderer does no statistics.
Details#
Both types are silk.schema.schema.Schema types, written as JSON with
silk.schema.json.SchemaJson. silk.bench.bench writes the result as
build/bench/<suite>-<time>.json, and the report wherever a suite declares an output with
silk.bench.suite.Format.Report.
Times are nanoseconds and memory is bytes. A raw sample holds whole numbers; the statistics
of a report are f64. Dates are RFC 3339 text in UTC, such as 2026-11-19T14:03:22Z.
A program that prints result lines is measured by what it reports, not by its process: each
line is a Reading, and a report gives every name the lines carry a table of its own, in
the unit of the lines.
A result keeps the views of its suite as text, one line each, as silk.bench.view.View.text
writes them. A report holds each view resolved into a Chart: panels of series of points,
with the statistic and the whiskers of every point already computed.
A run never drops a sample. A warm-up run is kept with warmup set, and a run whose program
failed is kept with its exit status. A report counts the failed runs of a row in failures
and leaves both kinds out of its statistics.
Examples#
import silk.bench.record { Report }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.schema.issue { SchemaError }
import silk.schema.json { SchemaJson }
// The fastest median of the first benchmark of a report, in nanoseconds.
effect fn fastest(text: &[u8]) -> f64 ! SchemaError | OutOfMemoryError ? &mut Allocator {
let report = run SchemaJson.decode<Report>(text)
let rows = report.benchmarks.asSlice()[0].rows.asSlice()
let mut best = rows[0].median
let mut at: usize = 1
while at < rows.length {
if rows[at].median < best {
best = rows[at].median
}
at = at + 1
}
return best
}
// Text that is not a report is an error, never a guess.
effect fn check() -> i32 ! SchemaError | OutOfMemoryError ? &mut Allocator {
let best = run fastest(b"{}")
if best > 0.0 {
return 1
}
return 2
}
effect fn failed(error: SchemaError | OutOfMemoryError) -> i32 {
drop error
return 0
}
pub fn main() -> i32 {
let mut allocator = Allocator.systemAllocatorProvider()
return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}Import as Report with import silk.bench.record { Report }.
Public declarations: 19.
PROTOCOL#
pub const PROTOCOL: u64The version of the result and report files this module writes.
Pair#
pub struct PairA name with a value: a parameter of a benchmark, or a recorded environment variable.
Field name#
pub name: StringThe name.
Field value#
pub value: StringThe value.
Implementation Schema for Pair#
impl Schema for PairEnvironment#
pub struct EnvironmentThe computer a run was measured on.
Field system#
pub system: StringThe kernel name, such as Linux or Darwin.
Field release#
pub release: StringThe kernel release, such as 6.8.0-45-generic.
Field machine#
pub machine: StringThe hardware name, such as x86_64 or arm64.
Field processor#
pub processor: StringThe model text of the first processor, or unknown.
Field processors#
pub processors: u64The number of logical processors online.
Field usable#
pub usable: u64The number of processors the run could use, which a container or an affinity mask lowers.
Field memory#
pub memory: u64The physical memory, in bytes.
Field load#
pub load: f64The 1 minute load average when the run started. A value well above 0 on an idle computer means something else was running.
Field variables#
pub variables: List<Pair>The environment variables the suite asked to record.
Implementation Schema for Environment#
impl Schema for EnvironmentSubject#
pub struct SubjectOne implementation of a suite, as it was run.
Field name#
pub name: StringThe name the suite gave it.
Field command#
pub command: StringThe program and its arguments, as one line for a reader. Placeholders are not filled in.
Field revision#
pub revision: Option<String>The revision the suite recorded for it, such as a commit hash or a version.
Implementation Schema for Subject#
impl Schema for SubjectSample#
pub struct SampleOne timed run of one program.
Field implementation#
pub implementation: StringThe name of the implementation.
Field round#
pub round: u64The round the run belongs to, from 0. Every implementation runs once in each round.
Field position#
pub position: u64The position of the run inside its round, from 0. The order is random in each round.
Field warmup#
pub warmup: boolWhether the run was a warm-up, which statistics leave out.
Field wall#
pub wall: u64The time from start to exit, in nanoseconds.
Field user#
pub user: u64The processor time in user mode, in nanoseconds, summed over the threads of the program and the children it waited for.
Field system#
pub system: u64The processor time in kernel mode, in nanoseconds.
Field peak#
pub peak: u64The peak resident memory, in bytes.
Field exit#
pub exit: i64The exit status, or -1 when a signal ended the program.
Field signal#
pub signal: i64The signal that ended the program, or 0.
Field counts#
pub counts: List<Metric>What the system's counters measured over the run, such as instructions or
page-faults. Empty unless the run asked for counters with --counters.
Implementation Schema for Sample#
impl Schema for SampleMetric#
pub struct MetricOne value of a reading with its unit.
Field unit#
pub unit: StringThe unit as the program wrote it, such as ns/op, B/op or req/s.
Field value#
pub value: f64The value, for one iteration.
Implementation Schema for Metric#
impl Schema for MetricReading#
pub struct ReadingOne sample a program reported as a result line, as silk.bench.lines reads it.
Field name#
pub name: StringThe name the program gave the sample, without the leading Benchmark.
Field implementation#
pub implementation: StringThe name of the implementation whose run printed the line.
Field round#
pub round: u64The round of the run that printed the line, from 0.
Field warmup#
pub warmup: boolWhether that run was a warm-up, which statistics leave out.
Field iterations#
pub iterations: u64The number of iterations the sample covers.
Field values#
pub values: List<Metric>The values of the sample, in the order the program wrote them.
Field check#
pub check: Option<String>What the program said it computed, from its last check: line before this one. Two
implementations with different checks for one name did not do the same work.
Implementation Schema for Reading#
impl Schema for ReadingCount#
pub struct CountWhat an instrument counted over one whole run of an implementation's program, such as the
instructions --instrument callgrind reports.
Field implementation#
pub implementation: StringThe name of the implementation.
Field unit#
pub unit: StringWhat was counted, such as instructions.
Field value#
pub value: f64The count for one run of the program, from start to exit.
Implementation Schema for Count#
impl Schema for CountMeasured#
pub struct MeasuredThe samples of one combination of a suite's parameters.
Field name#
pub name: StringThe name: the suite's name, then /name=value for each parameter.
Field parameters#
pub parameters: List<Pair>The parameters of this combination, in the order the suite declared them.
Field samples#
pub samples: List<Sample>Every run, in the order it happened.
Field readings#
pub readings: List<Reading>Every result line the runs printed, in the order they were read. Empty when no implementation reports result lines.
Field counts#
pub counts: List<Count>What an instrument counted for each implementation, one run each. Empty when the run named no instrument.
Implementation Schema for Measured#
impl Schema for MeasuredRunResult#
pub struct RunResultA whole run of one suite with its raw samples: the content of a result file.
Field protocol#
pub protocol: u64The version of this format, PROTOCOL.
Field suite#
pub suite: StringThe name of the suite.
Field id#
pub id: StringThe run's identity: its start time and a random part, safe in a file name.
Field label#
pub label: Option<String>The label the run was given, such as before or after.
Field revision#
pub revision: Option<String>The revision of the project the run measured, such as a commit hash.
Field started#
pub started: StringWhen the run started, as RFC 3339 text in UTC.
Field ended#
pub ended: StringWhen the run ended, as RFC 3339 text in UTC.
Field notes#
pub notes: List<String>The notes of the suite and of the command line.
Field environment#
pub environment: EnvironmentThe computer the run was measured on.
Field implementations#
pub implementations: List<Subject>The implementations, in the order the suite declared them.
Field reference#
pub reference: StringThe name of the implementation the others are compared to.
Field runs#
pub runs: u64The number of measured rounds asked for.
Field warmups#
pub warmups: u64The number of warm-up rounds asked for.
Field floor#
pub floor: u64The smallest peak memory the run could see, in bytes, or 0 when there is none. On Linux a new process starts with the peak of the process that started it, so a program that uses less memory than the benchmark task reads as the task's own peak.
Field benchmarks#
pub benchmarks: List<Measured>The benchmarks, one for each combination of parameters.
Field views#
pub views: List<String>The views to show the results with, one line each. Empty means the report picks by the shape of the results.
Implementation Schema for RunResult#
impl Schema for RunResultComparison#
pub struct ComparisonHow one implementation compares to the reference on one benchmark.
Field value#
pub value: f64The median divided by the reference's median. For a time, 0.5 is twice as fast.
Field low#
pub low: f64The low end of the 95% interval of the ratio.
Field high#
pub high: f64The high end of the 95% interval of the ratio.
Field verdict#
pub verdict: Stringfaster, slower, or same when the interval does not clear the noise threshold.
Implementation Schema for Comparison#
impl Schema for ComparisonRow#
pub struct RowThe statistics of one implementation on one benchmark.
Field implementation#
pub implementation: StringThe name of the implementation.
Field reference#
pub reference: boolWhether this is the reference the other rows are compared to.
Field runs#
pub runs: u64The number of values the statistics cover: the measured runs that succeeded, or the result lines those runs printed for this benchmark.
Field failures#
pub failures: u64The number of measured runs whose program failed. They are in no statistic.
Field median#
pub median: f64The median, in the table's unit: for process times, the time from start to exit in nanoseconds.
Field minimum#
pub minimum: f64The smallest value, in the table's unit.
Field maximum#
pub maximum: f64The largest value, in the table's unit.
Field mean#
pub mean: f64The mean, in the table's unit.
Field low#
pub low: f64The low end of the 95% interval of the median, in the table's unit.
Field high#
pub high: f64The high end of the 95% interval of the median, in the table's unit.
Field outliers#
pub outliers: u64How many runs lie far outside the others. They are counted, never removed.
Field user#
pub user: f64The median processor time in user mode, in nanoseconds. 0 in a table of result lines.
Field system#
pub system: f64The median processor time in kernel mode, in nanoseconds. 0 in a table of result lines.
Field peak#
pub peak: f64The median peak resident memory, in bytes.
Field ratio#
pub ratio: Option<Comparison>The comparison to the reference. Absent for the reference itself, for a row with no successful run, and in a table whose checks differ.
Field times#
pub times: List<f64>Every value the statistics cover, in the table's unit, in the order they were measured, for a renderer that draws the distribution.
Field others#
pub others: List<Metric>The median of each other unit the row's readings carry, in the order the units first appear. Empty for a table of process times.
Field check#
pub check: Option<String>What the implementation said it computed, when its readings carry a check.
Implementation Schema for Row#
impl Schema for RowTable#
pub struct TableThe rows of one benchmark.
Field name#
pub name: StringThe name of the benchmark.
Field parameters#
pub parameters: List<Pair>The parameters of the benchmark.
Field unit#
pub unit: StringThe unit of the statistics of the rows: ns for the time of a whole process, or the unit
of the result lines, such as ns/op or req/s.
Field higher#
pub higher: boolWhether a larger value is better, as for req/s. A ratio above 1 is then faster.
Field mismatch#
pub mismatch: boolWhether two rows carry different checks. The rows then have no ratio: the implementations did not do the same work.
Field rows#
pub rows: List<Row>One row for each implementation, in the order the suite declared them.
Implementation Schema for Table#
impl Schema for TablePoint#
pub struct PointOne value of a Series.
Field x#
pub x: StringThe entry of the axis the point belongs to, such as 1024, p99 or a benchmark name.
Field at#
pub at: f64Where the point is along the axis: the number x holds when the axis is numeric, and
else the position of x among the entries of the panel, from 0.
Field value#
pub value: f64The value, in the unit of the chart, or as a ratio when the chart is normalized.
Field low#
pub low: f64The low end of the whisker. Equal to value when there is none.
Field high#
pub high: f64The high end of the whisker. Equal to value when there is none.
Field faded#
pub faded: boolWhether the point is a warm-up run of a distribution, which a renderer dims.
Implementation Schema for Point#
impl Schema for PointSeries#
pub struct SeriesOne line, or one group of bars, of a Panel.
Field name#
pub name: StringThe name, such as the implementation.
Field points#
pub points: List<Point>The points, in the order of the axis.
Implementation Schema for Series#
impl Schema for SeriesPanel#
pub struct PanelOne separate chart of a Chart.
Field title#
pub title: StringWhat the panel shows, such as a benchmark name. Empty when the chart has one panel that covers everything.
Field series#
pub series: List<Series>The series, in the order they first appear in the results.
Implementation Schema for Panel#
impl Schema for PanelChart#
pub struct ChartA view of a suite with every number computed: what a renderer draws.
Field kind#
pub kind: StringThe kind of the view, such as line or x-flame.
Field title#
pub title: StringThe heading.
Field y#
pub y: StringThe unit of the values.
Field x#
pub x: StringWhat runs along the axis.
Field series#
pub series: StringWhat each series is.
Field facet#
pub facet: StringWhat each panel is. Empty when the panels are whatever x and series leave.
Field stat#
pub stat: StringThe statistic of the samples each point shows.
Field error#
pub error: StringWhat the whiskers show: interval, range or none.
Field scale#
pub scale: StringThe scale: linear, log or log-log.
Field zero#
pub zero: boolWhether the value axis starts at zero.
Field normalize#
pub normalize: StringThe implementation every value is divided by. Empty when the values are in the unit y.
Field sort#
pub sort: StringThe order of the entries of x: value, -value, or empty for the order they ran in.
Field higher#
pub higher: boolWhether a larger value is better, as for req/s.
Field numeric#
pub numeric: boolWhether every entry of x is a number, so the axis is numeric.
Field fields#
pub fields: List<Pair>The fields the view set, also the ones the report does not know, for a renderer of your own.
Field panels#
pub panels: List<Panel>The panels. A history gets its panels from silk.bench.history.History.drawn, which
knows the recorded runs.
Implementation Schema for Chart#
impl Schema for ChartOverall#
pub struct OverallHow one implementation compares to the reference over a whole suite.
Field implementation#
pub implementation: StringThe name of the implementation.
Field ratio#
pub ratio: f64The geometric mean of its ratios to the reference, as time: the ratio of a table where a larger value is better is inverted first, so below 1 always means faster.
Field benchmarks#
pub benchmarks: u64The number of benchmarks the mean covers.
Implementation Schema for Overall#
impl Schema for OverallReport#
pub struct ReportA run of one suite with its statistics: everything a renderer draws.
Field protocol#
pub protocol: u64The version of this format, PROTOCOL.
Field suite#
pub suite: StringThe name of the suite.
Field id#
pub id: StringThe run's identity, the same as in its result file.
Field label#
pub label: Option<String>The label the run was given.
Field revision#
pub revision: Option<String>The revision of the project the run measured, such as a commit hash.
Field started#
pub started: StringWhen the run started, as RFC 3339 text in UTC.
Field ended#
pub ended: StringWhen the run ended, as RFC 3339 text in UTC.
Field notes#
pub notes: List<String>The notes of the suite and of the command line.
Field environment#
pub environment: EnvironmentThe computer the run was measured on.
Field implementations#
pub implementations: List<Subject>The implementations, in the order the suite declared them.
Field reference#
pub reference: StringThe name of the implementation the others are compared to.
Field noise#
pub noise: f64The noise threshold of the verdicts, as a fraction: 0.01 is 1%.
Field runs#
pub runs: u64The number of measured rounds asked for.
Field floor#
pub floor: u64The smallest peak memory the run could see, in bytes, or 0 when there is none. A row whose
peak is not above it used at most this much.
Field benchmarks#
pub benchmarks: List<Table>The benchmarks, one for each combination of parameters.
Field overall#
pub overall: List<Overall>The geometric mean of each implementation's ratios, without the reference. Empty when the suite has one benchmark, where the mean would repeat the ratio.
Field views#
pub views: List<Chart>The views of the suite, resolved. Without declared views: a ratio when several
implementations ran several benchmarks, and a line for each parameter whose values are
all numbers.
Implementation Schema for Report#
impl Schema for ReportCategory: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.