silk.bench.history.

The history of a benchmark suite: one line of text for each result of each run, the journal a person keeps beside it, and the table that shows progress over time.

When to use#

Use History.lines to add a run to a history file, and History.parse with History.terminal or History.markdown to show how a suite moved from run to run. silk.bench.bench does both for a task: --record appends to bench/history/<suite>.jsonl, and --history prints the table.

Details#

A history file is JSON Lines: each line is one Entry, the statistics of one implementation on one benchmark in one run. Lines are only ever added, so the file diffs and merges in a repository like any text. An entry holds no raw samples: those stay in the result file of the run.

An entry carries the ratio to the reference measured in the same run. That ratio is the steadiest number to follow over time, because both sides shared the computer and its noise. Medians of two runs are compared only when both ran on the same computer, by Entry.computer: a change is called faster or slower when the 95% intervals of the two medians do not overlap, and is a plain percentage otherwise.

The journal is a Markdown file a person writes, bench/journal.md: each ## heading is a revision or a date, and the text under it says what changed there, such as "two-pass parse" or "moved to a 4-core container". The table shows that text beside the first run of that revision or date.

Examples#

silk
import silk.bench.history { Journal }
import silk.core.effect { Effect }
import silk.core.option { Option }
import silk.mem.allocator { Allocator, OutOfMemoryError }

effect fn check() -> i32 ! OutOfMemoryError ? &mut Allocator {
  let text = run Journal.added("# Journal\n", "1503ff0", "two-pass parse")
  let notes = run Journal.parse(text.view())
  return match Journal.find(notes.asSlice(), "1503ff0") {
    Option<string>.Some { value } => if value == "two-pass parse" { 0 } else { 1 }
    Option<string>.None => 2
  }
}

effect fn failed(error: OutOfMemoryError) -> i32 {
  drop error
  return 3
}

pub fn main() -> i32 {
  let mut allocator = Allocator.systemAllocatorProvider()
  return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as History with import silk.bench.history { History }.

Public declarations: 4.

Entry#

silk
pub struct Entry

One line of a history file: the statistics of one implementation on one benchmark in one run.

Field suite#

silk
pub suite: String

The name of the suite.

Field benchmark#

silk
pub benchmark: String

The name of the benchmark.

Field implementation#

silk
pub implementation: String

The name of the implementation.

Field unit#

silk
pub unit: String

The unit of the statistics: ns for the time of a whole process, or the unit of the result lines.

Field higher#

silk
pub higher: bool

Whether a larger value is better.

Field date#

silk
pub date: String

When the run started, as RFC 3339 text in UTC.

Field id#

silk
pub id: String

The run's identity, the same as in its result file.

Field revision#

silk
pub revision: Option<String>

The revision of the project the run measured, such as a commit hash.

Field version#

silk
pub version: Option<String>

The revision recorded for the implementation itself, such as the version of a library in another language.

Field label#

silk
pub label: Option<String>

The label the run was given.

Field computer#

silk
pub computer: String

The computer of the run in one line: system, hardware, processor, and the number of processors the run could use. Medians are compared only between equal computers.

Field runs#

silk
pub runs: u64

The number of values the statistics cover.

Field median#

silk
pub median: f64

The median, in the unit.

Field low#

silk
pub low: f64

The low end of the 95% interval of the median.

Field high#

silk
pub high: f64

The high end of the 95% interval of the median.

Field minimum#

silk
pub minimum: f64

The smallest value.

Field reference#

silk
pub reference: String

The name of the implementation the run compared the others to.

Field ratio#

silk
pub ratio: Option<f64>

The median divided by the reference's median in the same run. Absent for the reference itself and for a result that was not compared.

Implementation Schema for Entry#

silk
impl Schema for Entry

Note#

silk
pub struct Note

One section of a journal: what a person wrote about a revision or a date.

Field key#

silk
pub key: String

The heading: a revision or a date.

Field text#

silk
pub text: String

The text under the heading, as one line.

Journal#

silk
pub struct Journal

The owner type of the journal functions. It has no fields and is never constructed.

Implementation Journal#

silk
impl Journal

Associated function parse#

silk
pub effect fn parse(text: string) -> List<Note>
    ! OutOfMemoryError
    ? &mut Allocator

Returns the sections of the journal text: one Note for each ## heading, with the lines under it joined by spaces. Text before the first such heading is left out.

Errors#

OutOfMemoryError if the Allocator cannot supply the notes.

Associated function added#

silk
pub effect fn added(text: string, key: string, entry: string) -> String
    ! OutOfMemoryError
    ? &mut Allocator

Returns the journal text with entry added under the heading key. When the last heading of the journal is key, the entry joins its section; otherwise a new section starts at the end.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Associated function find#

silk
pub fn find<'notes>(notes: &'notes [Note], key: string) -> Option<string<'notes>>

Returns the text of the first section of notes whose heading is key, or whose heading and key start alike when one is a shortened form of the other, as with a commit hash of 7 and of 40 characters. A shortened form has at least 7 characters.

History#

silk
pub struct History

The owner type of the history functions. It has no fields and is never constructed.

Implementation History#

silk
impl History

Associated function entries#

silk
pub effect fn entries(report: &Report) -> List<Entry>
    ! OutOfMemoryError
    ? &mut Allocator

Returns one entry for each row of report that has a successful run, in the order of the report.

Errors#

OutOfMemoryError if the Allocator cannot supply the entries.

Associated function lines#

silk
pub effect fn lines(report: &Report) -> String
    ! SchemaError | WriterError | OutOfMemoryError
    ? &mut Allocator

Returns the entries of report as the lines to add to a history file: one JSON object a line, each line ended.

Errors#

  • OutOfMemoryError if the Allocator cannot supply the text.

  • SchemaError or WriterError if an entry cannot be encoded, which a value of a report that is not a finite number causes.

Associated function bencher#

silk
pub effect fn bencher(report: &Report) -> String
    ! OutOfMemoryError
    ? &mut Allocator

Returns report in Bencher Metric Format, the JSON that the Bencher service reads: one object whose keys are <benchmark>/<implementation>, each with one measure that holds the median as value and its 95% interval as lower_value and upper_value.

Details#

The measure is latency for a time, which Bencher takes in nanoseconds, and otherwise the unit as the report has it, such as req/s. Rows with no successful run are left out.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Associated function drawn#

silk
pub effect fn drawn(report: &mut Report, recorded: &[Entry]) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Gives every history chart of report its panels: one for each benchmark, with a line for each implementation over the runs of recorded and then the run of report itself.

Details#

A run is named by its revision, or by its day when it has none. A chart shows the entries in its own unit, so a chart of a unit the history does not hold stays empty. A normalized chart shows the ratio each entry recorded, when its reference is the chart's.

Errors#

OutOfMemoryError if the Allocator cannot supply the panels.

Associated function parse#

silk
pub effect fn parse(text: string) -> List<Entry>
    ! SchemaError | OutOfMemoryError
    ? &mut Allocator

Returns the entries of the history file text, in the order of its lines. Empty lines are skipped.

Errors#

  • SchemaError if a line is not an entry.

  • OutOfMemoryError if the Allocator cannot supply the entries.

Associated function terminal#

silk
pub effect fn terminal(entries: &[Entry], notes: &[Note], color: bool) -> String
    ! OutOfMemoryError
    ? &mut Allocator

Returns the history entries as tables for a terminal: one for each implementation on each benchmark, a row a run, oldest first. notes are the sections of the journal. With color, the text holds ANSI escape codes.

Details#

The columns are the date, the revision, the median, the change from the run before on the same computer, the ratio to the reference of the same run, and the journal's text for the revision or the day, shown at the first run that has it.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Associated function page#

silk
pub effect fn page(suite: string, entries: &[Entry], notes: &[Note]) -> String
    ! OutOfMemoryError
    ? &mut Allocator

Returns the history entries of the suite suite as one static HTML page, in the style of silk.bench.html: for each benchmark a chart over the runs, oldest first, and the tables of History.terminal.

Details#

A chart draws one line for each implementation: its ratio to the reference of the same run, with a dashed line at 1, or its median from zero when the benchmark has no ratio. A dotted vertical line with a number marks the first run of a revision or a day the journal has text for, and the text is listed under the chart.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Associated function markdown#

silk
pub effect fn markdown(entries: &[Entry], notes: &[Note]) -> String
    ! OutOfMemoryError
    ? &mut Allocator

Returns the history entries as Markdown: a heading and a table for each implementation on each benchmark, with the columns of History.terminal.

Errors#

OutOfMemoryError if the Allocator cannot supply the text.

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page