Browse the docs

silk.compress.zstd.

Decompress Zstandard data (RFC 8878) in steps, with fixed memory and limits that you set.

When to use#

Use Zstd to read .zst files or HTTP bodies with Content-Encoding: zstd. Make a decoder with Zstd.make, and call Zstd.step until it returns Finished. This module does not compress.

Details#

The decoder supports all block types and all entropy modes, and more than one frame, including skippable frames. Zstd.make allocates all storage one time: the history window, three 128 KiB staging buffers, and the entropy tables. Zstd.step does not allocate and keeps no borrow of your buffers. The ZstdLimits apply to the full input, across all frames.

The end of a frame is not the end of the input. The decoder returns Finished only when the input is marked final, all of it is consumed, and each frame is complete.

Errors#

  • OutOfMemoryError from Zstd.make if the Allocator fails.

  • ZstdError if the configuration is not valid, the data is not valid, a limit is exceeded, or the calls do not follow the rules of Zstd.step. reason gives the cause and offset the input position.

Gotchas#

Output is not verified until Finished, because a checksum or a truncated frame can fail later. A failure is permanent: drop the decoder and make a new one for another stream. Dictionaries are not supported.

Examples#

Decode a frame with one raw block#

silk
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.core.effect { Effect }
import silk.core.result { Result }
import silk.compress.zstd { Zstd, ZstdError, ZstdLimits, ZstdProgress, ZstdStatus }

pub effect fn main() -> i32
  ! ZstdError | OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let mut decoder = run Zstd.make(ZstdLimits.make()) |> Effect.provideMut<Allocator>(&mut allocator)

  // Magic number, a header for a 2-byte frame, a last raw block of 2 bytes, then "hi".
  let input = b"\x28\xb5\x2f\xfd\x20\x02\x11\x00\x00hi"
  let mut output: [u8; 16] = [0; 16]
  return match Zstd.step(&mut decoder, &input, &mut output, true) {
    Result<ZstdProgress, ZstdError>.Success { value } => {
      if value.status != ZstdStatus.Finished || value.written != 2 || output[0] != 104
        || output[1] != 105 {
        return 1
      }
      return 0
    }
    Result<ZstdProgress, ZstdError>.Failure { error: _ } => 1
  }
}

Import as Zstd with import silk.compress.zstd { Zstd }.

Public declarations: 6.

ZstdLimits#

silk
pub struct ZstdLimits

The resource limits for one complete input, across all frames. ZstdLimits.make gives defaults.

Field inputBytes#

silk
pub inputBytes: u64

The maximum number of input bytes to consume, with headers, checksums, and skippable frames.

Field outputBytes#

silk
pub outputBytes: u64

The maximum number of decoded bytes, across all frames.

Field frames#

silk
pub frames: u64

The maximum number of frames, regular and skippable.

Field skippableBytes#

silk
pub skippableBytes: u64

The maximum number of payload bytes in all skippable frames.

Field windowBytes#

silk
pub windowBytes: usize

The size of the history buffer that Zstd.make allocates. A frame that asks for a larger window fails with WindowLimit. It must not be 0.

Field workspaceBytes#

silk
pub workspaceBytes: usize

The storage budget other than the window: staging buffers, decoder state, and entropy tables. If it is too small, Zstd.make fails with WorkspaceLimit.

Implementation ZstdLimits#

silk
impl ZstdLimits

Associated function make#

silk
pub fn make() -> Self

Returns default limits: 64 MiB of input, 256 MiB of output, 1024 frames, 8 MiB of skippable data, an 8 MiB window, and a 1 MiB workspace.

ZstdStatus#

silk
pub enum ZstdStatus

What the decoder needs after a successful step.

NeedInput#

silk
NeedInput

All given input is consumed. Call again with more input, or with finalInput set if there is no more.

NeedOutput#

silk
NeedOutput

The output buffer is full. Call again with an output buffer that is not empty, and the input that was not consumed.

Finished#

silk
Finished

The final input is consumed and each frame is complete and verified.

ZstdProgress#

silk
pub struct ZstdProgress

The result of one successful step.

Field consumed#

silk
pub consumed: usize

The number of input bytes that this call consumed. Give only the rest of the input on the next call.

Field written#

silk
pub written: usize

The number of output bytes that this call wrote. Only this prefix of the output changed.

Field status#

silk
pub status: ZstdStatus

What the decoder needs next.

ZstdReason#

silk
pub enum ZstdReason

The cause of a ZstdError.

None#

silk
None

No failure. The decoder uses it internally. A returned error never has it.

InvalidConfiguration#

silk
InvalidConfiguration

The window size is 0, or the total storage size does not fit in usize.

InvalidState#

silk
InvalidState

A call changes the final input after it was set, adds input after Finished, or uses a decoder that lost its storage.

InvalidMagic#

silk
InvalidMagic

The input does not start with the magic number of a Zstandard frame or a skippable frame.

ReservedFrame#

silk
ReservedFrame

The reserved bit of a frame header is set.

ReservedBlock#

silk
ReservedBlock

A block uses the reserved block type.

BlockSize#

silk
BlockSize

A block is larger than the frame allows, or its size is not valid.

MalformedBlock#

silk
MalformedBlock

The entropy data or the sequences of a compressed block are not valid. blockReason gives more detail.

UnsupportedDictionary#

silk
UnsupportedDictionary

A frame asks for a dictionary. Dictionaries are not supported.

Truncated#

silk
Truncated

The final input ends before the last frame is complete.

Checksum#

silk
Checksum

The frame checksum is not the XXH64 (seed 0) of the decoded bytes.

ContentSize#

silk
ContentSize

The decoded size of a frame is not the content size in its header.

InputLimit#

silk
InputLimit

The consumed input would go over ZstdLimits.inputBytes.

OutputLimit#

silk
OutputLimit

The decoded output would go over ZstdLimits.outputBytes.

FrameLimit#

silk
FrameLimit

The number of frames would go over ZstdLimits.frames.

SkippableLimit#

silk
SkippableLimit

The skippable payload would go over ZstdLimits.skippableBytes.

WindowLimit#

silk
WindowLimit

A frame asks for a window larger than ZstdLimits.windowBytes.

WorkspaceLimit#

silk
WorkspaceLimit

ZstdLimits.workspaceBytes is too small for the decoder storage.

Implementation Debug for ZstdReason#

silk
impl Debug for ZstdReason

Operation debug#

silk
effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the name of the variant, such as None.

ZstdError#

silk
pub struct ZstdError

A decoding failure. It is permanent for the decoder.

Each later call returns the same reason and offset with no progress. The output that earlier calls wrote is not valid.

Field reason#

silk
pub reason: ZstdReason

The cause of the failure.

Field blockReason#

silk
pub blockReason: BlockReason

For MalformedBlock, the part of the block that is not valid. For other reasons, it is BlockReason.None.

Field offset#

silk
pub offset: u64

The input position where the decoder found the problem, counted from the start of the full input.

Field consumed#

silk
pub consumed: usize

The number of input bytes that this call consumed before it failed.

Field written#

silk
pub written: usize

The number of output bytes that this call wrote before it failed.

Implementation Debug for ZstdError#

silk
impl Debug for ZstdError

Operation debug#

silk
effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the fields, as ZstdError { reason: ..., blockReason: ..., ... }.

Zstd#

silk
pub struct Zstd

A Zstandard decoder for one complete input. It owns all its storage.

Dropping it releases the storage. Each regular frame starts with a new history, new entropy tables, and a new checksum, but the limits count across all frames.

When to use#

Use it to read .zst files or HTTP bodies with Content-Encoding: zstd. Make a decoder with Zstd.make, and call Zstd.step until it returns Finished. It does not compress.

Details#

It supports all block types and all entropy modes, and more than one frame, including skippable frames. Zstd.make allocates all storage one time. Zstd.step does not allocate and keeps no borrow of your buffers. The end of a frame is not the end of the input: the decoder returns Finished only when the input is marked final, all of it is consumed, and each frame is complete.

Gotchas#

Output is not verified until Finished, because a checksum or a truncated frame can fail later. A failure is permanent: drop the decoder and make a new one for another stream. Dictionaries are not supported.

Implementation Zstd#

silk
impl Zstd

Associated function make#

silk
pub effect fn make(limits: ZstdLimits) -> Self ! ZstdError | OutOfMemoryError ? &mut Allocator

Returns a decoder that uses limits, with storage from the ambient Allocator.

It allocates the window, three 128 KiB staging buffers, and the entropy tables. Zstd.step does not allocate.

Errors#

  • ZstdError with WorkspaceLimit if workspaceBytes is too small, or with InvalidConfiguration if windowBytes is 0 or the total size does not fit in usize. These checks come before the allocation.

  • OutOfMemoryError if the Allocator fails.

Method step#

silk
pub fn step(self: &mut Self, input: &[u8], output: &mut [u8], finalInput: bool) -> Result<
    ZstdProgress,
    ZstdError,
  >

Decodes from input into output and returns how much it consumed and wrote.

On the next call, give only the input after the consumed prefix, and use only the written prefix of the output. Set finalInput when input holds all remaining data. After that, each call must set finalInput and give exactly the input that is not consumed yet. NeedOutput with an empty output makes no progress.

Errors#

  • ZstdError if the data is not valid, a ZstdLimits value would be exceeded, the final input ends early (Truncated), or the call changes the final input (InvalidState).

Gotchas#

Output is not verified until Finished. A failure is permanent for this decoder.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2.

View the source of this page