silk.compress.zstd_encode.

When to use#

Use a ZstdEncoder to write .zst files or HTTP bodies with Content-Encoding: zstd, in steps and with fixed memory. Use ZstdEncoder.encode to compress one complete buffer. Decode with silk.compress.zstd.Zstd.

Details#

Each level uses the parameters and match strategy of the reference zstd for large inputs: one hash table for levels 1 and 2, two hash tables for 3 and 4, lazy matching over hash chains for 5 to 12, a binary tree with lazy matching for 13 to 15, and a binary tree with an optimal parser that prices literals and sequences for 16 to 22. The window grows from 512 KiB at level 1 to 128 MiB at level 22. Literals use Huffman codes, and sequences use FSE tables made for each block, or the predefined tables when those are smaller. A block that does not shrink is stored raw, and a block of one repeated byte is stored as RLE. ZstdEncoder.make allocates all storage one time. Later calls do not allocate and do not keep a borrow of your buffers. The output is deterministic but is not the same bytes that the reference zstd writes.

Examples#

Compress a buffer#

import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.core.effect { Effect }
import silk.compress.zstd_encode { ZstdEncodeError, ZstdEncoder, ZstdOptions }

pub effect fn main() -> i32
  ! OutOfMemoryError | ZstdEncodeError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let input = b"abcabcabcabcabcabcabcabcabcabcabcabc"
  let mut output: [u8; 64] = [0; 64]
  let written = run ZstdEncoder.encode(input, &mut output, ZstdOptions.make(3))
    |> Effect.provideMut<Allocator>(&mut allocator)
  // A frame starts with the magic number 28 b5 2f fd.
  if output[0] != 40 || output[3] != 253 || written >= input.length {
    return 1
  }
  return 0
}

Import as ZstdEncoder with import silk.compress.zstd_encode { ZstdEncoder }.

Public declarations: 7.

ZstdFlush#

pub enum ZstdFlush

What a call to ZstdEncoder.step must finish before it returns.

The values match the end directives of the reference ZSTD_compressStream2.

Continue#

Continue

Compress as the encoder sees fit. Output can lag behind the input by up to one block.

Flush#

Flush

Write all input in complete blocks, so that a decoder can decode all of it.

End#

End

Write all input, end the frame, and write the checksum if it is on. After it finishes, the encoder accepts no more input.

ZstdEncodeStatus#

pub enum ZstdEncodeStatus

What the encoder needs after a successful ZstdEncoder.step.

NeedInput#

NeedInput

All input is consumed and the flush was Continue. Call again with more input, or with another flush.

NeedOutput#

NeedOutput

The output buffer is full. Call again with a new output buffer, the input that was not consumed, and the same flush.

Flushed#

Flushed

All input is consumed and written in complete blocks, after ZstdFlush.Flush.

Finished#

Finished

The frame is complete.

ZstdEncodeProgress#

pub struct ZstdEncodeProgress

The result of one successful ZstdEncoder.step.

Field consumed#

pub consumed: usize

The number of input bytes that this call consumed, from the start of the input.

Field written#

pub written: usize

The number of output bytes that this call wrote, from the start of the output.

Field status#

pub status: ZstdEncodeStatus

What the encoder needs next.

ZstdEncodeErrorKind#

pub enum ZstdEncodeErrorKind

The cause of a ZstdEncodeError.

InvalidOptions#

InvalidOptions

The level or window size is outside its range.

InvalidUse#

InvalidUse

The call is not allowed now: a pledged size after the frame started, input after the frame finished, a frame whose size differs from the pledged size, or any step after a failure.

OutputFull#

OutputFull

ZstdEncoder.encode filled the output buffer before the frame was complete.

Implementation Debug for ZstdEncodeErrorKind#

impl Debug for ZstdEncodeErrorKind

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the name of the variant, such as InvalidOptions.

ZstdEncodeError#

pub struct ZstdEncodeError

An encoding failure, with the progress of the call before it failed.

Gotchas#

After a failed ZstdEncoder.step, discard the frame. Each later step fails with InvalidUse until you call ZstdEncoder.reset.

Field kind#

pub kind: ZstdEncodeErrorKind

The cause of the failure.

Field consumed#

pub consumed: usize

The number of input bytes that the call consumed before it failed.

Field written#

pub written: usize

The number of output bytes that the call wrote before it failed.

Implementation Debug for ZstdEncodeError#

impl Debug for ZstdEncodeError

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the fields, as ZstdEncodeError { kind: ..., consumed: ..., written: ... }.

ZstdOptions#

pub struct ZstdOptions

The settings of a ZstdEncoder.

Field level#

pub level: u32

The effort, from 1 to 22. 1 is the fastest, 22 the smallest, and 3 is the default of the reference zstd.

Field windowLog#

pub windowLog: u32

The base-2 logarithm of the match window, from 10 to 27. A larger window finds older matches and uses more memory. The decoder needs a window of the same size.

Field checksum#

pub checksum: bool

Whether the frame ends with the low 32 bits of the XXH64 of the content.

Field contentSize#

pub contentSize: bool

Whether the frame header records the content size when the encoder knows it: from ZstdEncoder.setPledgedSize, or when the first step ends the frame.

Implementation ZstdOptions#

impl ZstdOptions

Associated function make#

pub fn make(level: u32) -> ZstdOptions

Returns options for level with the window of the reference zstd for that level (512 KiB for level 1, 1 MiB for 2, 2 MiB for 3 to 8, 4 MiB for 9 to 16, 8 MiB for 17 to 19, and 32, 64, and 128 MiB for 20, 21, and 22), no checksum, and the content size recorded when known.

ZstdEncoder#

pub struct ZstdEncoder

A streaming Zstandard encoder that owns its window, match tables, and output buffer.

When to use#

Use it to compress data that arrives in parts, as the reference ZSTD_compressStream2 does. Make one with ZstdEncoder.make, call ZstdEncoder.step with each part of the input, and end with ZstdFlush.End. Call ZstdEncoder.reset to write another frame with the same storage.

Details#

ZstdEncoder.make allocates all storage one time: twice the window, the hash and chain tables (64 KiB together at level 1, 640 MiB at level 22, and smaller when the window is), about 640 KiB of block buffers, and 160 KiB more for the optimal parser of levels 16 to 22. Later calls do not allocate and do not keep a borrow of your buffers, so you can move the encoder between threads between calls.

Implementation ZstdEncoder#

impl ZstdEncoder

Associated function make#

pub effect fn make(options: ZstdOptions) -> ZstdEncoder
    ! OutOfMemoryError | ZstdEncodeError
    ? &mut Allocator

Returns an encoder for one frame with options, with storage from the ambient Allocator.

Errors#

• ZstdEncodeError with InvalidOptions if the level is not 1 to 22 or the window log is not 10 to 27, checked before the allocation.

• OutOfMemoryError if the Allocator fails.

Method reset#

pub fn reset(self: &mut Self) -> ()

Starts a new frame with the same options and storage. It does not allocate. It drops the input and output that the encoder holds, the pledged size, and a failure.

Method setPledgedSize#

pub fn setPledgedSize(self: &mut Self, size: u64) -> Result<(), ZstdEncodeError>

Declares the exact size of the frame's content before the first step, as the reference ZSTD_CCtx_setPledgedSrcSize does. The frame header then records the size.

Errors#

InvalidUse after the first step or after a failure. The step that ends the frame fails with InvalidUse if the content has another size.

Method step#

pub fn step(self: &mut Self, input: &[u8], output: &mut [u8], flush: ZstdFlush) -> Result<
    ZstdEncodeProgress,
    ZstdEncodeError,
  >

Compresses from input into output and returns how much it consumed and wrote.

Details#

On the next call, give only the input after the consumed prefix. You can give a new output buffer of any length. Call with ZstdFlush.Continue while more input follows. A flush finishes only when the step returns Flushed, or Finished for ZstdFlush.End. Until then, call again with the same flush.

Errors#

InvalidUse with input after the frame finished, a frame whose size differs from the pledged size, or after an earlier failure.

Associated function encodedBound#

pub fn encodedBound(length: usize) -> Option<usize>

Returns the most bytes that ZstdEncoder.encode can write for length input bytes with any options, or None if that count does not fit in usize.

Details#

An output buffer of this size never fails with OutputFull. The bound is the frame header (at most 18 bytes), 3 bytes per 128 KiB block or part of one, the input, and a 4-byte checksum.

Associated function encode#

pub effect fn encode(input: &[u8], output: &mut [u8], options: ZstdOptions) -> usize
    ! OutOfMemoryError | ZstdEncodeError
    ? &mut Allocator

Compresses all of input into the start of output as one frame with options, and returns the encoded length.

Details#

It makes a ZstdEncoder with storage from the ambient Allocator, runs one step with ZstdFlush.End, and releases the storage before it returns. The window shrinks to the smallest power of two, at least 1 KiB, that holds the input, so small inputs need little memory and the decoder needs a small window. The frame header records the content size if options.contentSize is set.

Errors#

• OutputFull if output fills before the frame is complete. It never occurs when output holds at least ZstdEncoder.encodedBound(input.length) bytes. Discard the partial frame.

• InvalidOptions if an option is outside its range.

• OutOfMemoryError if the Allocator fails.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.