silk.bench.history.
The history of a benchmark suite: one line of text for each result of each run, the journal a person keeps beside it, and the table that shows progress over time.
When to use#
Use History.lines to add a run to a history file, and History.parse with
History.terminal or History.markdown to show how a suite moved from run to run.
silk.bench.bench does both for a task: --record appends to
bench/history/<suite>.jsonl, and --history prints the table.
Details#
A history file is JSON Lines: each line is one Entry, the statistics of one
implementation on one benchmark in one run. Lines are only ever added, so the file diffs and
merges in a repository like any text. An entry holds no raw samples: those stay in the
result file of the run.
An entry carries the ratio to the reference measured in the same run. That ratio is the
steadiest number to follow over time, because both sides shared the computer and its noise.
Medians of two runs are compared only when both ran on the same computer, by
Entry.computer: a change is called faster or slower when the 95% intervals of the two
medians do not overlap, and is a plain percentage otherwise.
The journal is a Markdown file a person writes, bench/journal.md: each ## heading is a
revision or a date, and the text under it says what changed there, such as "two-pass parse"
or "moved to a 4-core container". The table shows that text beside the first run of that
revision or date.
Examples#
import silk.bench.history { Journal }
import silk.core.effect { Effect }
import silk.core.option { Option }
import silk.mem.allocator { Allocator, OutOfMemoryError }
effect fn check() -> i32 ! OutOfMemoryError ? &mut Allocator {
let text = run Journal.added("# Journal\n", "1503ff0", "two-pass parse")
let notes = run Journal.parse(text.view())
return match Journal.find(notes.asSlice(), "1503ff0") {
Option<string>.Some { value } => if value == "two-pass parse" { 0 } else { 1 }
Option<string>.None => 2
}
}
effect fn failed(error: OutOfMemoryError) -> i32 {
drop error
return 3
}
pub fn main() -> i32 {
let mut allocator = Allocator.systemAllocatorProvider()
return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}Import as History with import silk.bench.history { History }.
Public declarations: 4.
Entry#
pub struct EntryOne line of a history file: the statistics of one implementation on one benchmark in one run.
Field suite#
pub suite: StringThe name of the suite.
Field benchmark#
pub benchmark: StringThe name of the benchmark.
Field implementation#
pub implementation: StringThe name of the implementation.
Field unit#
pub unit: StringThe unit of the statistics: ns for the time of a whole process, or the unit of the
result lines.
Field higher#
pub higher: boolWhether a larger value is better.
Field date#
pub date: StringWhen the run started, as RFC 3339 text in UTC.
Field id#
pub id: StringThe run's identity, the same as in its result file.
Field revision#
pub revision: Option<String>The revision of the project the run measured, such as a commit hash.
Field version#
pub version: Option<String>The revision recorded for the implementation itself, such as the version of a library in another language.
Field label#
pub label: Option<String>The label the run was given.
Field computer#
pub computer: StringThe computer of the run in one line: system, hardware, processor, and the number of processors the run could use. Medians are compared only between equal computers.
Field runs#
pub runs: u64The number of values the statistics cover.
Field median#
pub median: f64The median, in the unit.
Field low#
pub low: f64The low end of the 95% interval of the median.
Field high#
pub high: f64The high end of the 95% interval of the median.
Field minimum#
pub minimum: f64The smallest value.
Field reference#
pub reference: StringThe name of the implementation the run compared the others to.
Field ratio#
pub ratio: Option<f64>The median divided by the reference's median in the same run. Absent for the reference itself and for a result that was not compared.
Implementation Schema for Entry#
impl Schema for EntryNote#
pub struct NoteOne section of a journal: what a person wrote about a revision or a date.
Field key#
pub key: StringThe heading: a revision or a date.
Field text#
pub text: StringThe text under the heading, as one line.
Journal#
pub struct JournalThe owner type of the journal functions. It has no fields and is never constructed.
Implementation Journal#
impl JournalAssociated function parse#
pub effect fn parse(text: string) -> List<Note>
! OutOfMemoryError
? &mut AllocatorReturns the sections of the journal text: one Note for each ## heading, with the
lines under it joined by spaces. Text before the first such heading is left out.
Errors#
OutOfMemoryError if the Allocator cannot supply the notes.
Associated function added#
pub effect fn added(text: string, key: string, entry: string) -> String
! OutOfMemoryError
? &mut AllocatorReturns the journal text with entry added under the heading key. When the last
heading of the journal is key, the entry joins its section; otherwise a new section
starts at the end.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Associated function find#
pub fn find<'notes>(notes: &'notes [Note], key: string) -> Option<string<'notes>>Returns the text of the first section of notes whose heading is key, or whose heading
and key start alike when one is a shortened form of the other, as with a commit hash of
7 and of 40 characters. A shortened form has at least 7 characters.
History#
pub struct HistoryThe owner type of the history functions. It has no fields and is never constructed.
Implementation History#
impl HistoryAssociated function entries#
pub effect fn entries(report: &Report) -> List<Entry>
! OutOfMemoryError
? &mut AllocatorReturns one entry for each row of report that has a successful run, in the order of the
report.
Errors#
OutOfMemoryError if the Allocator cannot supply the entries.
Associated function lines#
pub effect fn lines(report: &Report) -> String
! SchemaError | WriterError | OutOfMemoryError
? &mut AllocatorReturns the entries of report as the lines to add to a history file: one JSON object a
line, each line ended.
Errors#
OutOfMemoryErrorif theAllocatorcannot supply the text.SchemaErrororWriterErrorif an entry cannot be encoded, which a value of a report that is not a finite number causes.
Associated function bencher#
pub effect fn bencher(report: &Report) -> String
! OutOfMemoryError
? &mut AllocatorReturns report in Bencher Metric Format, the JSON that the Bencher service reads: one
object whose keys are <benchmark>/<implementation>, each with one measure that holds
the median as value and its 95% interval as lower_value and upper_value.
Details#
The measure is latency for a time, which Bencher takes in nanoseconds, and otherwise the
unit as the report has it, such as req/s. Rows with no successful run are left out.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Associated function drawn#
pub effect fn drawn(report: &mut Report, recorded: &[Entry]) -> ()
! OutOfMemoryError
? &mut AllocatorGives every history chart of report its panels: one for each benchmark, with a line for
each implementation over the runs of recorded and then the run of report itself.
Details#
A run is named by its revision, or by its day when it has none. A chart shows the entries in its own unit, so a chart of a unit the history does not hold stays empty. A normalized chart shows the ratio each entry recorded, when its reference is the chart's.
Errors#
OutOfMemoryError if the Allocator cannot supply the panels.
Associated function parse#
pub effect fn parse(text: string) -> List<Entry>
! SchemaError | OutOfMemoryError
? &mut AllocatorReturns the entries of the history file text, in the order of its lines. Empty lines are
skipped.
Errors#
SchemaErrorif a line is not an entry.OutOfMemoryErrorif theAllocatorcannot supply the entries.
Associated function terminal#
pub effect fn terminal(entries: &[Entry], notes: &[Note], color: bool) -> String
! OutOfMemoryError
? &mut AllocatorReturns the history entries as tables for a terminal: one for each implementation on
each benchmark, a row a run, oldest first. notes are the sections of the journal. With
color, the text holds ANSI escape codes.
Details#
The columns are the date, the revision, the median, the change from the run before on the same computer, the ratio to the reference of the same run, and the journal's text for the revision or the day, shown at the first run that has it.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Associated function page#
pub effect fn page(suite: string, entries: &[Entry], notes: &[Note]) -> String
! OutOfMemoryError
? &mut AllocatorReturns the history entries of the suite suite as one static HTML page, in the style
of silk.bench.html: for each benchmark a chart over the runs, oldest first, and the
tables of History.terminal.
Details#
A chart draws one line for each implementation: its ratio to the reference of the same run, with a dashed line at 1, or its median from zero when the benchmark has no ratio. A dotted vertical line with a number marks the first run of a revision or a day the journal has text for, and the text is listed under the chart.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Associated function markdown#
pub effect fn markdown(entries: &[Entry], notes: &[Note]) -> String
! OutOfMemoryError
? &mut AllocatorReturns the history entries as Markdown: a heading and a table for each implementation
on each benchmark, with the columns of History.terminal.
Errors#
OutOfMemoryError if the Allocator cannot supply the text.
Category: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.