silk.bench.statistics.
The statistics a benchmark report is made of: medians, bootstrap intervals, ratios between two sets of samples, and the verdict a ratio supports.
When to use#
Use these functions to summarize samples you measured yourself, or to write a renderer that
needs a number silk.bench.report does not compute. A suite run by silk.bench.bench already
has them applied.
Details#
A summary is built on the median, because a few slow runs on a busy computer move a mean far more than they move a median. An interval is a 95% percentile bootstrap: the samples are drawn again with replacement many times, and the interval holds the middle 95% of the medians that come out. The draws use a fixed seed, so the same samples always give the same interval.
A ratio compares a candidate to a reference: 0.5 means the candidate took half the time. Its
interval comes from drawing both sides again. A Verdict calls a difference only when the
whole interval is beyond the noise threshold, so a ratio of 0.99 with an interval of 0.97 to
1.01 is Same.
Examples#
import silk.bench.statistics { Statistics, Verdict }
import silk.collections.list { List }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.num.usize
// Ten runs near 100 against ten runs near 200: the candidate is about twice as fast.
effect fn check() -> i32 ! OutOfMemoryError ? &mut Allocator {
let mut fast = List<f64>.make()
let mut slow = List<f64>.make()
let mut at: usize = 0
while at < 10 {
run fast.append(100.0 + usize.toF64(at))
run slow.append(200.0 + usize.toF64(at))
at = at + 1
}
let ratio = run Statistics.ratio(fast.asSlice(), slow.asSlice(), 0.01)
if ratio.verdict == Verdict.Faster && ratio.high < 0.6 {
return 0
}
return 1
}
effect fn failed(error: OutOfMemoryError) -> i32 {
drop error
return 2
}
pub fn main() -> i32 {
let mut allocator = Allocator.systemAllocatorProvider()
return run Effect.catchAll(check(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}Import as Statistics with import silk.bench.statistics { Statistics }.
Public declarations: 4.
Verdict#
pub enum VerdictWhat a ratio to a reference supports.
Faster#
FasterThe whole interval is below the noise threshold: the candidate is faster.
Slower#
SlowerThe whole interval is above the noise threshold: the candidate is slower.
Same#
SameThe interval reaches into the noise threshold: no difference was found.
Summary#
pub struct SummaryThe summary of one set of samples.
Field count#
pub count: usizeThe number of samples.
Field median#
pub median: f64The median.
Field minimum#
pub minimum: f64The smallest sample.
Field maximum#
pub maximum: f64The largest sample.
Field mean#
pub mean: f64The arithmetic mean.
Field low#
pub low: f64The low end of the 95% bootstrap interval of the median.
Field high#
pub high: f64The high end of the 95% bootstrap interval of the median.
Field outliers#
pub outliers: usizeHow many samples lie more than 1.5 interquartile ranges outside the quartiles. They are counted, never removed.
Implementation Copy for Summary#
impl Copy for SummaryRatio#
pub struct RatioA candidate's median divided by a reference's median, with its interval and verdict.
Field value#
pub value: f64The ratio of the medians. Below 1 means the candidate's values are smaller.
Field low#
pub low: f64The low end of the 95% bootstrap interval.
Field high#
pub high: f64The high end of the 95% bootstrap interval.
Field verdict#
pub verdict: VerdictWhat the interval supports at the noise threshold the ratio was made with.
Implementation Copy for Ratio#
impl Copy for RatioStatistics#
pub struct StatisticsThe owner type of the statistics functions. It has no fields and is never constructed.
Implementation Statistics#
impl StatisticsAssociated function medianOfSorted#
pub fn medianOfSorted(sorted: &[f64]) -> f64Returns the median of sorted, which must be in ascending order, or 0 for no samples.
Associated function quantileOfSorted#
pub fn quantileOfSorted(sorted: &[f64], fraction: f64) -> f64Returns the value at fraction (0 to 1) of sorted, which must be in ascending order,
between neighbors in a straight line. Returns 0 for no samples.
Associated function sorted#
pub effect fn sorted(samples: &[f64]) -> List<f64>
! OutOfMemoryError
? &mut AllocatorReturns a copy of samples in ascending order, for Statistics.quantileOfSorted.
Errors#
OutOfMemoryError if the Allocator cannot supply the copy.
Associated function median#
pub effect fn median(samples: &[f64]) -> f64
! OutOfMemoryError
? &mut AllocatorReturns the median of samples, in any order, or 0 for no samples.
Errors#
OutOfMemoryError if the Allocator cannot supply the sorted copy.
Associated function summary#
pub effect fn summary(samples: &[f64]) -> Summary
! OutOfMemoryError
? &mut AllocatorReturns the summary of samples: count, median, extremes, mean, the 95% bootstrap interval
of the median, and the number of outliers. Every field is 0 for no samples.
Errors#
OutOfMemoryError if the Allocator cannot supply the working copies.
Associated function ratio#
pub effect fn ratio(candidate: &[f64], reference: &[f64], noise: f64) -> Ratio
! OutOfMemoryError
? &mut AllocatorReturns the median of candidate divided by the median of reference, with its 95%
bootstrap interval and the verdict at the noise threshold noise, a fraction such as 0.01
for 1%.
Details#
The verdict is Faster when the whole interval is below 1 - noise, Slower when it is
above 1 + noise, and Same otherwise. With no samples on either side, or a reference
median of 0, the ratio is 1 and the verdict is Same.
Errors#
OutOfMemoryError if the Allocator cannot supply the working copies.
Associated function geometricMean#
pub fn geometricMean(ratios: &[f64]) -> f64Returns the geometric mean of ratios, the right mean for ratios: a suite that is twice as
fast on one benchmark and half as fast on another comes out at 1. Returns 1 for no ratios.
Ratios that are not positive are left out.
Category: Benchmarks.
Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.