silk.compress.brotli_encode.

When to use#

Use a BrotliEncoder to write .br files or HTTP bodies with Content-Encoding: br, in steps and with fixed memory. Use BrotliEncoder.encode to compress one complete buffer. Decode with silk.compress.brotli.Brotli.

Details#

Up to quality 9 the encoder finds matches with the bucket hash tables of the reference brotli (more and longer buckets as the quality rises), checks the recent distances first, and looks one or more bytes ahead for a better match from quality 2. Qualities 10 and 11 collect all matches from a binary tree and choose the commands with the shortest path search of the reference, priced by estimated literal costs, and at quality 11 a second time with the costs of the first result. They also copy words of the static dictionary, with the transforms that keep the start of the word, and pick the postfix bits and direct codes of the distances. Each meta-block gets its own prefix codes for literals, commands, and distances. From quality 5 the literals use one prefix code for each cluster of the 64 contexts of their two previous bytes, in UTF-8 or signed context mode, and the commands and distances of a meta-block are split into blocks of up to 64 types with their own prefix codes, with 4 contexts by copy length for the distances. At qualities 10 and 11 the literals are also split, into up to 32 types, from more histograms. From quality 4 the counts are smoothed toward runs before the prefix codes are built. A meta-block that does not shrink is stored uncompressed. BrotliEncoder.make allocates all storage one time. Later calls do not allocate and do not keep a borrow of your buffers. The output is deterministic but is not the same bytes that the reference brotli writes.

Examples#

Compress a buffer#

import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.core.effect { Effect }
import silk.compress.brotli_encode { BrotliEncodeError, BrotliEncoder, BrotliOptions }

pub effect fn main() -> i32
  ! OutOfMemoryError | BrotliEncodeError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let input = b"abcabcabcabcabcabcabcabcabcabcabcabc"
  let mut output: [u8; 64] = [0; 64]
  let written = run BrotliEncoder.encode(input, &mut output, BrotliOptions.make(5))
    |> Effect.provideMut<Allocator>(&mut allocator)
  if written == 0 || written >= input.length {
    return 1
  }
  return 0
}

Import as BrotliEncoder with import silk.compress.brotli_encode { BrotliEncoder }.

Public declarations: 7.

BrotliFlush#

pub enum BrotliFlush

What a call to BrotliEncoder.step must finish before it returns.

The values match the operations PROCESS, FLUSH, and FINISH of the reference BrotliEncoderCompressStream.

Continue#

Continue

Compress as the encoder sees fit. Output can lag behind the input by up to one meta-block.

Flush#

Flush

Write all input and pad the stream to a whole byte, so that a decoder can decode all of it.

End#

End

Write all input and end the stream. After it finishes, the encoder accepts no more input.

BrotliEncodeStatus#

pub enum BrotliEncodeStatus

What the encoder needs after a successful BrotliEncoder.step.

NeedInput#

NeedInput

All input is consumed and the flush was Continue. Call again with more input, or with another flush.

NeedOutput#

NeedOutput

The output buffer is full. Call again with a new output buffer, the input that was not consumed, and the same flush.

Flushed#

Flushed

All input is consumed and written up to a whole byte, after BrotliFlush.Flush.

Finished#

Finished

The stream is complete.

BrotliEncodeProgress#

pub struct BrotliEncodeProgress

The result of one successful BrotliEncoder.step.

Field consumed#

pub consumed: usize

The number of input bytes that this call consumed, from the start of the input.

Field written#

pub written: usize

The number of output bytes that this call wrote, from the start of the output.

Field status#

pub status: BrotliEncodeStatus

What the encoder needs next.

BrotliEncodeErrorKind#

pub enum BrotliEncodeErrorKind

The cause of a BrotliEncodeError.

InvalidOptions#

InvalidOptions

The quality or window size is outside its range.

InvalidUse#

InvalidUse

The call is not allowed now: input after the stream finished, or any step after a failure.

OutputFull#

OutputFull

BrotliEncoder.encode filled the output buffer before the stream was complete.

Implementation Debug for BrotliEncodeErrorKind#

impl Debug for BrotliEncodeErrorKind

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the name of the variant, such as InvalidOptions.

BrotliEncodeError#

pub struct BrotliEncodeError

An encoding failure, with the progress of the call before it failed.

Gotchas#

After a failed BrotliEncoder.step, discard the stream. Each later step fails with InvalidUse until you call BrotliEncoder.reset.

Field kind#

pub kind: BrotliEncodeErrorKind

The cause of the failure.

Field consumed#

pub consumed: usize

The number of input bytes that the call consumed before it failed.

Field written#

pub written: usize

The number of output bytes that the call wrote before it failed.

Implementation Debug for BrotliEncodeError#

impl Debug for BrotliEncodeError

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the fields, as BrotliEncodeError { kind: ..., consumed: ..., written: ... }.

BrotliOptions#

pub struct BrotliOptions

The settings of a BrotliEncoder.

Field quality#

pub quality: u32

The effort, from 0 to 11. 0 is the fastest and 11 the smallest. The reference brotli defaults to 11.

Field windowBits#

pub windowBits: u32

The base-2 logarithm of the window, from 10 to 24. Matches reach back up to the window size less 16 bytes. The stream header records it, and the decoder needs a window of that size.

Implementation BrotliOptions#

impl BrotliOptions

Associated function make#

pub fn make(quality: u32) -> BrotliOptions

Returns options for quality with the 4 MiB window (22 bits) that the reference brotli uses by default.

BrotliEncoder#

pub struct BrotliEncoder

A streaming Brotli encoder that owns its window, match tables, and output buffer.

When to use#

Use it to compress data that arrives in parts, as the reference BrotliEncoderCompressStream does. Make one with BrotliEncoder.make, call BrotliEncoder.step with each part of the input, and end with BrotliFlush.End. Call BrotliEncoder.reset to write another stream with the same storage.

Details#

BrotliEncoder.make allocates all storage one time: twice the window or twice the meta-block (64 KiB to 256 KiB, and at most the window from 64 KiB), whichever is larger, and about two meta-blocks of command and output buffers. Up to quality 9 it adds the hash buckets (512 KiB at quality 0 and 32 MiB at 9, less when the window is small), and from quality 5 about 5 MiB for literal contexts and block splitting. Qualities 10 and 11 add one more window, a binary tree of 8 bytes for each byte of the window, about 115 bytes for each byte of the meta-block for matches, the shortest path search, and block splitting (30 MiB with 256 KiB meta-blocks), and about 7 MiB for histogram clustering and a dictionary index. Later calls do not allocate and do not keep a borrow of your buffers, so you can move the encoder between threads between calls.

Implementation BrotliEncoder#

impl BrotliEncoder

Associated function make#

pub effect fn make(options: BrotliOptions) -> BrotliEncoder
    ! OutOfMemoryError | BrotliEncodeError
    ? &mut Allocator

Returns an encoder for one stream with options, with storage from the ambient Allocator.

Errors#

• BrotliEncodeError with InvalidOptions if the quality is above 11 or the window bits are not 10 to 24, checked before the allocation.

• OutOfMemoryError if the Allocator fails.

Method reset#

pub fn reset(self: &mut Self) -> ()

Starts a new stream with the same options and storage. It does not allocate. It drops the input and output that the encoder holds, and a failure.

Method step#

pub fn step(self: &mut Self, input: &[u8], output: &mut [u8], flush: BrotliFlush) -> Result<
    BrotliEncodeProgress,
    BrotliEncodeError,
  >

Compresses from input into output and returns how much it consumed and wrote.

Details#

On the next call, give only the input after the consumed prefix. You can give a new output buffer of any length. Call with BrotliFlush.Continue while more input follows. A flush finishes only when the step returns Flushed, or Finished for BrotliFlush.End. Until then, call again with the same flush. A flush that has to pad the stream to a whole byte writes an empty metadata meta-block, as the reference brotli does.

Errors#

InvalidUse with input after the stream finished, or after an earlier failure.

Associated function encodedBound#

pub fn encodedBound(length: usize) -> Option<usize>

Returns the most bytes that BrotliEncoder.encode can write for length input bytes with any options, or None if that count does not fit in usize.

Details#

An output buffer of this size never fails with OutputFull. The bound is the input, 5 bytes of header and padding for each 64 KiB meta-block or part of one, and 3 bytes for the stream header, its end, and the last padding.

Associated function encode#

pub effect fn encode(input: &[u8], output: &mut [u8], options: BrotliOptions) -> usize
    ! OutOfMemoryError | BrotliEncodeError
    ? &mut Allocator

Compresses all of input into the start of output as one stream with options, and returns the encoded length.

Details#

It makes a BrotliEncoder with storage from the ambient Allocator, runs one step with BrotliFlush.End, and releases the storage before it returns. The window shrinks to the smallest one, at least 10 bits, whose reach holds the input, so small inputs need little memory and the decoder needs a small window.

Errors#

• OutputFull if output fills before the stream is complete. It never occurs when output holds at least BrotliEncoder.encodedBound(input.length) bytes. Discard the partial stream.

• InvalidOptions if an option is outside its range.

• OutOfMemoryError if the Allocator fails.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.