silk.http.content.

When to use#

Make a ResponseContext from a response head, and a CodingPlan from it. Then decode the body with a Decoder when you have the wire bytes, or with withReader over a BufferedDuplex. Use acceptEncoding to make the Accept-Encoding field of a request. silk.http.exchange and silk.http.fetch use this module for you.

Details#

With Mode.Raw, the body is returned as it was sent. With Mode.Decode, the codings in Content-Encoding are decoded in reverse order. A response can have at most four codings. CodingPlan.make checks all limits and memory before it reads the body. The decoder also removes the body framing, so give it the bytes after the head.

Errors#

• ContentError tells the reason and the byte counts so far. A coding error tells where the coding is in Content-Encoding.

• OutOfMemoryError if the Allocator fails.

Gotchas#

Decoded bytes are not final until the state is End. A later checksum, trailer, or framing error can still fail. A step can return no bytes, so call it again until the state is End. This module does not check that the body belongs to the head. silk.http.exchange does.

Examples#

Decode a gzip body#

import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.core.effect { Effect }
import silk.http.message { Method }
import silk.http.body { TrailerPolicy }
import silk.http.content {
  CodingPlan,
  ContentError,
  ContentProgressState,
  DecodeFailure,
  DecodeProgress,
  Decoder,
  Mode,
  ResponseContext
}
import silk.http.exchange { Limits as ClientLimits }
import silk.http.fetch { FetchOptions }
import silk.http.head { ParseError, ParsedResponse, ResponseHead, ResponseParser, parseResponse }
import silk.http.body { BodyError }
import silk.core.result { Result }
import silk.collections.slice { Slice }
import silk.num.usize

effect fn program() -> i32
  ! OutOfMemoryError
  ? &mut Allocator {
  let limits = ClientLimits.defaults()
  let wire = b"HTTP/1.1 200 OK\r\nContent-Encoding: gzip\r\nContent-Length: 28\r\n\r\n"
  let parsed = match run parseResponse(&wire, limits.head) {
    Result<ParsedResponse, ParseError>.Success { value } => move value
    Result<ParsedResponse, ParseError>.Failure { error: _ } => {
      return 1
    }
  }
  let head = match ResponseParser.head(&parsed.parser) {
    Result<ResponseHead, ParseError>.Success { value } => value
    Result<ResponseHead, ParseError>.Failure { error: _ } => {
      return 1
    }
  }
  let context = match ResponseContext.make(head, Method.get(), TrailerPolicy.defaultPolicy()) {
    Result<ResponseContext, BodyError>.Success { value } => move value
    Result<ResponseContext, BodyError>.Failure { error: _ } => {
      return 1
    }
  }
  let options = FetchOptions.defaults()
  let plan = match CodingPlan.make(
    move context,
    Mode.Decode,
    limits.responseBody,
    move options.contentLimits
  ) {
    Result<CodingPlan, ContentError>.Success { value } => move value
    Result<CodingPlan, ContentError>.Failure { error: _ } => {
      return 1
    }
  }
  let made = run Effect.result(Decoder.make(move plan))
  let mut decoder = match move made {
    Result.Success { value } => move value
    Result.Failure { error } => {
      drop error
      return 1
    }
  }

  // gzip of "hello hello hello"
  let body: [u8; 28] = [
    31,
    139,
    8,
    0,
    0,
    0,
    0,
    0,
    2,
    255,
    203,
    72,
    205,
    201,
    201,
    87,
    200,
    64,
    144,
    0,
    128,
    136,
    249,
    229,
    17,
    0,
    0,
    0,
  ]

  // A step can return no bytes. Call it again until the state is End.
  let mut chunk: [u8; 64] = [0; 64]
  let mut offset = usize.ZERO
  let mut total = usize.ZERO
  let mut first: u8 = 0
  let mut done = false
  let mut steps = 0
  while !done && steps < 1000 {
    let rest = Slice.view<u8>(&body, offset, body.length - offset)
    let step = match Decoder.step(&mut decoder, rest, &mut chunk, true) {
      Result<DecodeProgress, DecodeFailure>.Success { value } => move value
      Result<DecodeProgress, DecodeFailure>.Failure { error: _ } => {
        return 1
      }
    }
    offset = offset + step.consumed
    if step.progress.written > usize.ZERO && total == usize.ZERO {
      first = chunk[0]
    }
    total = total + step.progress.written
    done = step.progress.state == ContentProgressState.End
    steps = steps + 1
  }
  if !done || total != 17 || first != 104 {
    return 1
  }
  return 0
}

effect fn failed<E>(error: E) -> i32 {
  drop error
  return 1
}

pub fn main() -> i32 {
  let mut allocator = Allocator.systemAllocatorProvider()
  return run Effect.catchAll(program(), failed) |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as ContentReader with import silk.http.content { ContentReader }.

Public declarations: 25.

Mode#

pub enum Mode

Whether to decode the content.

Raw#

Raw

Return the body as it was sent. Content-Encoding is not read.

Decode#

Decode

Decode the codings in Content-Encoding.

Coding#

pub enum Coding

A content coding that this module knows.

Identity#

Identity

identity: no change. It still counts in maxDepth.

Gzip#

Gzip

gzip. More gzip members after the first are decoded too.

XGzip#

XGzip

x-gzip, the old name of gzip.

Deflate#

Deflate

deflate, as a zlib stream. Raw DEFLATE without the zlib wrapper fails.

Zstd#

Zstd

zstd, within the window and memory limits.

CodingPosition#

pub struct CodingPosition

Where a coding is in the Content-Encoding fields.

Field wireIndex#

pub wireIndex: usize

The position of the coding over all Content-Encoding fields, from 0.

Field fieldIndex#

pub fieldIndex: usize

The position of the field in the response head, from 0.

Field elementIndex#

pub elementIndex: usize

The position of the coding in its field, from 0.

Field offset#

pub offset: usize

The byte offset of the coding, or the bad byte, in the field value.

Implementation Copy for CodingPosition#

impl Copy for CodingPosition

Implementation Debug for CodingPosition#

impl Debug for CodingPosition

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the fields, as CodingPosition { wireIndex: ..., fieldIndex: ..., ... }.

Limits#

pub struct Limits

The limits for content decoding.

Field maxEncoded#

pub maxEncoded: u64

The maximum body bytes before decoding, after the framing is removed.

Field maxIntermediate#

pub maxIntermediate: u64

The maximum total bytes from all decoding stages except the last.

Field maxDecoded#

pub maxDecoded: u64

The maximum decoded bytes.

Field maxOwned#

pub maxOwned: usize

The maximum memory for the decoders, without the memory of the body decoder.

Field maxDepth#

pub maxDepth: usize

The maximum number of codings, including identity. It is from 1 to 4.

Field intermediateCapacity#

pub intermediateCapacity: usize

The size of the buffer between two stages. It is from 1 to 65536.

Field inflate#

pub inflate: InflateLimits

The limits for each gzip or deflate decoder.

Field zstd#

pub zstd: ZstdLimits

The limits for each Zstandard decoder.

ContentLimitKind#

pub enum ContentLimitKind

The limit that failed. See ContentReason.InvalidLimit.

CodingDepth#

CodingDepth

maxDepth.

IntermediateCapacity#

IntermediateCapacity

intermediateCapacity.

OwnedBytes#

OwnedBytes

maxOwned.

EncodedBytes#

EncodedBytes

maxEncoded.

IntermediateBytes#

IntermediateBytes

maxIntermediate.

DecodedBytes#

DecodedBytes

maxDecoded.

Implementation Debug for ContentLimitKind#

impl Debug for ContentLimitKind

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the name of the variant, such as CodingDepth.

ContentReason#

pub union ContentReason<'head>

The reason that content decoding failed.

InvalidLimit#

InvalidLimit

A limit is not valid, or a byte count exceeded it.

Field limit#

pub limit: ContentLimitKind

The limit.

Field allowed#

pub allowed: u64

The value of the limit.

Field attempted#

pub attempted: u64

The value that was found.

CodingSyntax#

CodingSyntax

An item in Content-Encoding is empty or not valid.

Field position#

pub position: CodingPosition

Where the item is.

UnsupportedCoding#

UnsupportedCoding

A coding has no decoder, for example br.

Field position#

pub position: CodingPosition

Where the coding is.

Field token#

pub token: &'head [u8]

The name of the coding, from the response head.

PartialResponse#

PartialResponse

A 206 partial response has a coding other than identity, and Mode.Decode cannot decode a part.

Field position#

pub position: CodingPosition

The first coding that is not identity.

Tunnel#

Tunnel

The body is a tunnel, not content.

Body#

Body

The body framing failed.

Field error#

pub error: BodyError

The error from silk.http.body.

Buffer#

Buffer

Reading from the transport failed.

Field error#

pub error: BufferError

The error, with its transport cause.

Inflate#

Inflate

Gzip or deflate decoding failed.

Field stage#

pub stage: usize

The decoding stage, from 0. Stage 0 decodes the last coding.

Field wireIndex#

pub wireIndex: usize

The position of the coding in Content-Encoding, from 0.

Field error#

pub error: DecodeError

The error from silk.compress.inflate.

Zstandard#

Zstandard

Zstandard decoding failed.

Field stage#

pub stage: usize

The decoding stage, from 0. Stage 0 decodes the last coding.

Field wireIndex#

pub wireIndex: usize

The position of the coding in Content-Encoding, from 0.

Field error#

pub error: ZstdError

The error from silk.compress.zstd.

TrailingEncodedData#

TrailingEncodedData

There were more bytes after the end of a compressed stream.

Field stage#

pub stage: usize

The decoding stage, from 0. Stage 0 decodes the last coding.

Field wireIndex#

pub wireIndex: usize

The position of the coding in Content-Encoding, from 0.

ArithmeticOverflow#

ArithmeticOverflow

A byte count overflowed.

InvalidState#

InvalidState

The reader had already failed or stopped.

Implementation Debug for ContentReason<'head>#

impl<'head> Debug for ContentReason<'head>

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the name of the variant and its fields, such as InvalidLimit { ... }.

ContentError#

pub struct ContentError<'head>

The error from content decoding, with the reason and the byte counts so far. A coding error tells where the coding is in Content-Encoding.

Field reason#

pub reason: ContentReason<'head>

The reason.

Field written#

pub written: usize

The decoded bytes written by the call before the error.

Field totalEncoded#

pub totalEncoded: u64

The total body bytes read, after the framing is removed.

Field totalIntermediate#

pub totalIntermediate: u64

The total bytes from all stages except the last.

Field totalDecoded#

pub totalDecoded: u64

The total decoded bytes written.

Implementation Debug for ContentError<'head>#

impl<'head> Debug for ContentError<'head>

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the fields, as ContentError { reason: ..., written: ..., ... }.

ContentProgressState#

pub enum ContentProgressState

Whether the content is complete.

Data#

Data

There is more content. The bytes so far are not final.

End#

End

The body and all decoding completed, and all checks passed.

ContentProgress#

pub struct ContentProgress

The progress of a read.

Field written#

pub written: usize

The decoded bytes written by this call.

Field totalEncoded#

pub totalEncoded: u64

The total body bytes read, after the framing is removed.

Field totalIntermediate#

pub totalIntermediate: u64

The total bytes from all stages except the last.

Field totalDecoded#

pub totalDecoded: u64

The total decoded bytes written.

Field state#

pub state: ContentProgressState

Whether the content is complete.

ReuseDisposition#

pub enum ReuseDisposition

Whether the connection can be used again after this response.

EligibleAfterCompletion#

EligibleAfterCompletion

Yes, but only after the content completes.

Reusable#

Reusable

Yes. The body had a clear end, there was no problem, and the content completed.

NonReusable#

NonReusable

No. The body ends when the connection closes, or the framing had a problem.

AppliedPlan#

pub struct AppliedPlan

A summary of a CodingPlan.

Field mode#

pub mode: Mode

Whether the content is decoded.

Field codingCount#

pub codingCount: usize

The number of codings in Content-Encoding, including identity.

Field decoderCount#

pub decoderCount: usize

The number of decoders that run, without identity.

Field reuse#

pub reuse: ReuseDisposition

Whether the connection can be used again.

Implementation Copy for AppliedPlan#

impl Copy for AppliedPlan

RepresentationView#

pub struct RepresentationView<'owner, 'head>

Information about the decoded content. The headers of the response are not changed.

Implementation RepresentationView<'owner, 'head>#

impl<'owner, 'head> RepresentationView<'owner, 'head>

Method originalHead#

pub fn originalHead(self: &Self) -> ResponseHead<'head>

Returns the response head as received, with its Content-Encoding and Content-Length.

Method appliedPlan#

pub fn appliedPlan(self: &Self) -> AppliedPlan

Returns the summary of the plan.

Method anomaly#

pub fn anomaly(self: &Self) -> Anomaly

Returns a framing problem of the response, if any.

Method decodingApplied#

pub fn decodingApplied(self: &Self) -> bool

Returns true if at least one decoder runs.

Method decodedLength#

pub fn decodedLength(self: &Self) -> Option<u64>

Returns the decoded length after completion, 0 for a response without a body, or None before completion.

ContentCompletion#

pub struct ContentCompletion<'owner>

Proof that the body and all decoding completed.

Implementation ContentCompletion<'owner>#

impl<'owner> ContentCompletion<'owner>

Method bodyKind#

pub fn bodyKind(self: &Self) -> BodyCompletionKind

Returns how the body ended: at a clear end, or when the connection closed.

Method decodedBytes#

pub fn decodedBytes(self: &Self) -> u64

Returns the number of decoded bytes.

Method anomaly#

pub fn anomaly(self: &Self) -> Anomaly

Returns a framing problem of the response, if any.

Method reuse#

pub fn reuse(self: &Self) -> ReuseDisposition

Returns whether the connection can be used again.

ResponseContext#

pub struct ResponseContext<'head, 'method, 'names>

A response head with its request method and trailer policy, ready for a CodingPlan.

Implementation ResponseContext<'head, 'method, 'names>#

impl<'head, 'method, 'names> ResponseContext<'head, 'method, 'names>

Associated function make#

pub fn make(
    head: ResponseHead<'head>,
    requestMethod: Method<'method>,
    trailerPolicy: TrailerPolicy<'names>,
  ) -> Result<ResponseContext<'head, 'method, 'names>, BodyError>

Selects the body framing of head, and returns a context. Fails if the framing is not valid.

CodingPlan#

pub struct CodingPlan<'head, 'method, 'names>

A checked plan to decode one response body. Use it one time.

Details#

With Mode.Raw, the body is returned as it was sent. With Mode.Decode, the codings in Content-Encoding (gzip, deflate, and Zstandard) are decoded in reverse order. A response can have at most four codings. CodingPlan.make checks all limits and memory before the body is read.

Implementation CodingPlan<'head, 'method, 'names>#

impl<'head, 'method, 'names> CodingPlan<'head, 'method, 'names>

Associated function make#

pub fn make(
    context: ResponseContext<'head, 'method, 'names>,
    mode: Mode,
    bodyLimits: BodyLimits,
    limits: Limits,
  ) -> Result<CodingPlan<'head, 'method, 'names>, ContentError<'head>>

Reads Content-Encoding in Mode.Decode, and checks all limits and memory.

It does not allocate or read the body.

Method originalHead#

pub fn originalHead(self: &Self) -> ResponseHead<'head>

Returns the response head as received.

Method selectionAnomaly#

pub fn selectionAnomaly(self: &Self) -> Anomaly

Returns a framing problem of the response, if any.

Method selectedFraming#

pub fn selectedFraming(self: &Self) -> Framing

Returns the body framing.

Method applied#

pub fn applied(self: &Self) -> AppliedPlan

Returns the summary of the plan.

Method ownedBytes#

pub fn ownedBytes(self: &Self) -> usize

Returns the memory that the plan needs, as counted against maxOwned.

EnabledCodings#

pub struct EnabledCodings

The codings to name in an automatic Accept-Encoding field. x-gzip is never named.

Field gzip#

pub gzip: bool

Name gzip.

Field deflate#

pub deflate: bool

Name deflate.

Field zstd#

pub zstd: bool

Name zstd.

Implementation Copy for EnabledCodings#

impl Copy for EnabledCodings

EmptyEncodingPolicy#

pub enum EmptyEncodingPolicy

What to send when no coding is enabled.

Identity#

Identity

Send Accept-Encoding: identity.

Omit#

Omit

Send no Accept-Encoding field.

AcceptEncoding#

pub union AcceptEncoding<'value>

How to make the Accept-Encoding field.

Automatic#

Automatic

Name the enabled codings, in the order gzip, deflate, zstd. Only in Mode.Decode.

Field enabled#

pub enabled: EnabledCodings

The codings to name.

Field empty#

pub empty: EmptyEncodingPolicy

What to send when no coding is enabled.

Override#

Override

Send this value. The response is still checked as in the other cases.

Field value#

pub value: &'value [u8]

The field value.

Omit#

Omit

Send no Accept-Encoding field.

acceptEncoding#

pub fn acceptEncoding<'value>(
  mode: Mode,
  policy: AcceptEncoding<'value>,
  limits: HeaderLimits,
) -> Result<Option<Header<'value>>, ValueError>

Returns the Accept-Encoding field for mode and policy, or None when no field is sent.

validateLimits#

pub fn validateLimits<'head>(limits: &Limits) -> Result<(), ContentError<'head>>

Checks limits without allocation or I/O.

Details#

maxDepth must be from 1 to 4, and intermediateCapacity from 1 to 65536. The other limits can be 0, and are checked during planning or decoding. The limits of a decoder are checked when a plan selects it.

Decoder#

pub struct Decoder

Decodes a body from wire bytes that you give, without I/O.

Details#

The decoder owns its state. Bytes that step does not use, such as the next HTTP message, stay with you. The decoder also removes the body framing, so give it the bytes after the head.

Gotchas#

Decoded bytes are not final until the state is End. A later checksum, trailer, or framing error can still fail. A step can return no bytes, so call it again until the state is End. It does not check that the body belongs to the head. silk.http.exchange does.

DecodeProgress#

pub struct DecodeProgress

The result of a Decoder.step.

Field consumed#

pub consumed: usize

The input bytes that the step used.

Field progress#

pub progress: ContentProgress

The decoded bytes and the totals.

Field needsInput#

pub needsInput: bool

true when the decoder needs more input before it can continue.

DecodeFailure#

pub struct DecodeFailure

The error from a Decoder.step.

Field consumed#

pub consumed: usize

The input bytes that the step used before the error.

Field error#

pub error: ContentError<'static>

The error.

Implementation Decoder#

impl Decoder

Associated function make#

pub effect fn make<'head, 'method, 'names>(plan: CodingPlan<'head, 'method, 'names>) -> Decoder
  ! ContentError<'head> | OutOfMemoryError
  ? &mut Allocator

Returns a decoder for plan, and allocates its memory.

Method step#

pub fn step(self: &mut Self, input: &[u8], output: &mut [u8], endOfInput: bool) -> Result<
    DecodeProgress,
    DecodeFailure,
  >

Decodes from input into output, and returns how much it used and wrote.

Details#

Set endOfInput to true when no bytes come after input. The decoded bytes are final only after the state is End. A step can write no bytes, so call it again. An empty output does nothing.

Method progress#

pub fn progress(self: &Self) -> ContentProgress

Returns the totals so far.

Method completion#

pub fn completion<'owner>(self: &'owner Self) -> Result<
    Option<ContentCompletion<'owner>>,
    ContentError<'static>,
  >

Returns proof of completion, or None if the content is not complete.

Method trailers#

pub fn trailers<'owner>(self: &'owner Self) -> Result<
    Option<Trailers<'owner>>,
    ContentError<'static>,
  >

Returns the trailers, after the content completes.

Method abandon#

pub fn abandon(self: &mut Self) -> ()

Stops the decoder. Later calls fail.

ContentReader#

pub struct ContentReader<'reader, 'transport, 'head, P>

Reads the content of a body in the callback of withReader.

Implementation ContentReader<'reader, 'transport, 'head, P>#

impl<'reader, 'transport, 'head, P> ContentReader<'reader, 'transport, 'head, P>

Method representation#

pub fn representation<'owner>(
    self: &'owner Self,
  ) -> Result<RepresentationView<'owner, 'head>, ContentError<'head>>

Returns information about the response head and the plan.

Method completion#

pub fn completion<'owner>(
    self: &'owner Self,
  ) -> Result<Option<ContentCompletion<'owner>>, ContentError<'head>>

Returns proof of completion, after the body and all decoders finish.

Method trailers#

pub fn trailers<'owner>(
    self: &'owner Self,
  ) -> Result<Option<Trailers<'owner>>, ContentError<'head>>

Returns the trailers, after the content completes.

Method readSome#

pub effect fn readSome(
    self: &mut Self,
    output: &mut [u8],
    deadline: Option<Instant>,
  ) -> ContentProgress
  ! ContentError<'head>
  ? &mut MonotonicClock
  where &'transport mut P provides &ByteDuplex from &mut ByteDuplex | &mut MonotonicClock

Reads decoded bytes into output. The bytes are final only after the state is End. deadline is on the MonotonicClock.

withReader#

pub effect fn withReader<
  'reader,
  'transport: 'reader,
  'head: 'reader,
  'method: 'reader,
  'names: 'reader,
  'callback,
  A,
  E,
  ?CallbackRequirements,
  P,
>(
  plan: CodingPlan<'head, 'method, 'names>,
  body: &'reader mut BufferedDuplex<'transport, P>,
  callback: for<
    'call,
    'readerView: 'call,
    'transportView: 'readerView,
    'headView: 'readerView,
  > once fn<'callback>(
    &'call mut ContentReader<'readerView, 'transportView, 'headView, P>,
  ) -> once Effect<'call; A ! E ? CallbackRequirements>,
) -> A
! E | ContentError<'head> | OutOfMemoryError
? CallbackRequirements | &mut Allocator
where &'transport mut P provides &ByteDuplex from &mut ByteDuplex | &mut MonotonicClock

Gives a ContentReader for the body in body to callback, with the decoders of plan.

Gotchas#

Decoded bytes are not final until the body ends. A later checksum, trailer, or framing error can still fail.

Category: HTTP.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2.