silk.bench.statistics.

The statistics a benchmark report is made of: medians, bootstrap intervals, ratios between two sets of samples, and the verdict a ratio supports.

When to use#

Use these functions to summarize samples you measured yourself, or to write a renderer that needs a number silk.bench.report does not compute. A suite run by silk.bench.bench already has them applied.

Details#

A summary is built on the median, because a few slow runs on a busy computer move a mean far more than they move a median. An interval is a 95% percentile bootstrap: the samples are drawn again with replacement many times, and the interval holds the middle 95% of the medians that come out. The draws use a fixed seed, so the same samples always give the same interval.

A ratio compares a candidate to a reference: 0.5 means the candidate took half the time. Its interval comes from drawing both sides again. A Verdict calls a difference only when the whole interval is beyond the noise threshold, so a ratio of 0.99 with an interval of 0.97 to 1.01 is Same.

Examples#

silk
import silk.bench.statistics { Statistics, Verdict }
import silk.collections.list { List }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.num.usize

// Ten runs near 100 against ten runs near 200: the candidate is about twice as fast.
effect fn check() -> i32 ! OutOfMemoryError ? &mut Allocator {
  let mut fast = List<f64>.make()
  let mut slow = List<f64>.make()
  let mut at: usize = 0
  while at < 10 {
    run fast.append(100.0 + usize.toF64(at))
    run slow.append(200.0 + usize.toF64(at))
    at = at + 1
  }
  let ratio = run Statistics.ratio(fast.asSlice(), slow.asSlice(), 0.01)
  if ratio.verdict == Verdict.Faster && ratio.high < 0.6 {
    return 0
  }
  return 1
}

effect fn failed(error: OutOfMemoryError) -> i32 {
  drop error
  return 2
}

pub fn main() -> i32 {
  let mut allocator = Allocator.systemAllocatorProvider()
  return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as Statistics with import silk.bench.statistics { Statistics }.

Public declarations: 4.

Verdict#

silk
pub enum Verdict

What a ratio to a reference supports.

Faster#

silk
Faster

The whole interval is below the noise threshold: the candidate is faster.

Slower#

silk
Slower

The whole interval is above the noise threshold: the candidate is slower.

Same#

silk
Same

The interval reaches into the noise threshold: no difference was found.

Summary#

silk
pub struct Summary

The summary of one set of samples.

Field count#

silk
pub count: usize

The number of samples.

Field median#

silk
pub median: f64

The median.

Field minimum#

silk
pub minimum: f64

The smallest sample.

Field maximum#

silk
pub maximum: f64

The largest sample.

Field mean#

silk
pub mean: f64

The arithmetic mean.

Field low#

silk
pub low: f64

The low end of the 95% bootstrap interval of the median.

Field high#

silk
pub high: f64

The high end of the 95% bootstrap interval of the median.

Field outliers#

silk
pub outliers: usize

How many samples lie more than 1.5 interquartile ranges outside the quartiles. They are counted, never removed.

Implementation Copy for Summary#

silk
impl Copy for Summary

Ratio#

silk
pub struct Ratio

A candidate's median divided by a reference's median, with its interval and verdict.

Field value#

silk
pub value: f64

The ratio of the medians. Below 1 means the candidate's values are smaller.

Field low#

silk
pub low: f64

The low end of the 95% bootstrap interval.

Field high#

silk
pub high: f64

The high end of the 95% bootstrap interval.

Field verdict#

silk
pub verdict: Verdict

What the interval supports at the noise threshold the ratio was made with.

Implementation Copy for Ratio#

silk
impl Copy for Ratio

Statistics#

silk
pub struct Statistics

The owner type of the statistics functions. It has no fields and is never constructed.

Implementation Statistics#

silk
impl Statistics

Associated function medianOfSorted#

silk
pub fn medianOfSorted(sorted: &[f64]) -> f64

Returns the median of sorted, which must be in ascending order, or 0 for no samples.

Associated function quantileOfSorted#

silk
pub fn quantileOfSorted(sorted: &[f64], fraction: f64) -> f64

Returns the value at fraction (0 to 1) of sorted, which must be in ascending order, between neighbors in a straight line. Returns 0 for no samples.

Associated function sorted#

silk
pub effect fn sorted(samples: &[f64]) -> List<f64>
    ! OutOfMemoryError
    ? &mut Allocator

Returns a copy of samples in ascending order, for Statistics.quantileOfSorted.

Errors#

OutOfMemoryError if the Allocator cannot supply the copy.

Associated function median#

silk
pub effect fn median(samples: &[f64]) -> f64
    ! OutOfMemoryError
    ? &mut Allocator

Returns the median of samples, in any order, or 0 for no samples.

Errors#

OutOfMemoryError if the Allocator cannot supply the sorted copy.

Associated function summary#

silk
pub effect fn summary(samples: &[f64]) -> Summary
    ! OutOfMemoryError
    ? &mut Allocator

Returns the summary of samples: count, median, extremes, mean, the 95% bootstrap interval of the median, and the number of outliers. Every field is 0 for no samples.

Errors#

OutOfMemoryError if the Allocator cannot supply the working copies.

Associated function ratio#

silk
pub effect fn ratio(candidate: &[f64], reference: &[f64], noise: f64) -> Ratio
    ! OutOfMemoryError
    ? &mut Allocator

Returns the median of candidate divided by the median of reference, with its 95% bootstrap interval and the verdict at the noise threshold noise, a fraction such as 0.01 for 1%.

Details#

The verdict is Faster when the whole interval is below 1 - noise, Slower when it is above 1 + noise, and Same otherwise. With no samples on either side, or a reference median of 0, the ratio is 1 and the verdict is Same.

Errors#

OutOfMemoryError if the Allocator cannot supply the working copies.

Associated function geometricMean#

silk
pub fn geometricMean(ratios: &[f64]) -> f64

Returns the geometric mean of ratios, the right mean for ratios: a suite that is twice as fast on one benchmark and half as fast on another comes out at 1. Returns 1 for no ratios. Ratios that are not positive are left out.

Category: Benchmarks.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page