silk.msgpack.scanner.

Read MessagePack data as a sequence of tokens, without building a tree and without allocating.

When to use#

Use MsgpackScanner to read MessagePack bytes that are in one slice, token by token, when you do not need a tree. To get a tree, use silk.msgpack.value.Msgpack.parse. To read your own types, use silk.schema.msgpack.SchemaMsgpack.decode.

Details#

Each call of MsgpackScanner.next reads one value header. A scalar is complete in its token. An MsgpackToken.Array token is followed by the tokens of count values, and an MsgpackToken.Map token by the tokens of count keys, each followed by its value. The scanner does not track that nesting; the caller counts.

Integers of every width become MsgpackToken.Integer, or MsgpackToken.Unsigned above the largest i64. A 32-bit float is widened to f64. A string or binary token gives a MsgpackSpan of its payload in the input; the UTF-8 of a string is checked. All offsets are byte offsets from the start of the input.

Errors#

  • MsgpackError with a MsgpackReason and the offset of the problem. After an error, the scanner cannot continue. Make a new one.

Gotchas#

The scanner does not limit nesting and does not check for duplicate map keys. An extension value, such as a timestamp (type -1), is returned as its type and payload, not interpreted.

Examples#

Sum the integers of an array#

silk
import silk.core.result { Result }
import silk.msgpack.scanner { MsgpackError, MsgpackScanner, MsgpackToken }

fn sum(input: &[u8]) -> i64 {
  let mut scanner = MsgpackScanner.make(input)
  let mut total: i64 = 0
  while true {
    match move MsgpackScanner.next(&mut scanner) {
      Result<MsgpackToken, MsgpackError>.Failure {..} => {
        return -1
      }
      Result<MsgpackToken, MsgpackError>.Success { value } => match move value {
        MsgpackToken.Integer { value: number } => {
          total = total + number
        }
        MsgpackToken.EndOfInput => {
          return total
        }
        _ => {}
      }
    }
  }
  return total
}

pub fn main() -> i32 {
  // An array of 3 elements: 1, 200 (as a `uint 8`) and 5.
  if sum(b"\x93\x01\xcc\xc8\x05") != 206 {
    return 1
  }
  // The array says 3 elements but the input ends after one.
  if sum(b"\x93\x01") != -1 {
    return 2
  }
  return 0
}

Import as MsgpackScanner with import silk.msgpack.scanner { MsgpackScanner }.

Public declarations: 6.

MAX_DEPTH#

silk
pub const MAX_DEPTH: usize

The largest number of arrays and maps that silk.msgpack.value.Msgpack.parse and silk.schema.msgpack.SchemaMsgpack accept nested in one another: 64.

MsgpackSpan#

silk
pub struct MsgpackSpan

A range of bytes in the input: the payload of a string, binary or extension value.

Field offset#

silk
pub offset: usize

The offset of the first byte.

Field length#

silk
pub length: usize

The number of bytes.

Implementation Copy for MsgpackSpan#

silk
impl Copy for MsgpackSpan

MsgpackReason#

silk
pub enum MsgpackReason

The cause of a MsgpackError.

UnexpectedEnd#

silk
UnexpectedEnd

The input ends inside a value, or an array or map says it has more elements than the input has bytes left.

ReservedByte#

silk
ReservedByte

A value starts with the byte 0xc1, which MessagePack never uses.

InvalidUtf8#

silk
InvalidUtf8

A string is not valid UTF-8.

TrailingBytes#

silk
TrailingBytes

More bytes follow the one value that the input must hold.

DepthExceeded#

silk
DepthExceeded

More than MAX_DEPTH (64) arrays and maps are nested.

TooLong#

silk
TooLong

A string, binary value, extension payload, array or map is longer than MessagePack can write: more than 4294967295 bytes or elements.

Implementation Debug for MsgpackReason#

silk
impl Debug for MsgpackReason

Operation debug#

silk
effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the name of the variant, such as UnexpectedEnd.

MsgpackError#

silk
pub struct MsgpackError

A MessagePack error: the cause and the byte position where it occurred.

Field reason#

silk
pub reason: MsgpackReason

The cause of the error.

Field offset#

silk
pub offset: usize

The byte position of the problem in the input. An error from writing, which has no input, uses 0.

Implementation Copy for MsgpackError#

silk
impl Copy for MsgpackError

Implementation Debug for MsgpackError#

silk
impl Debug for MsgpackError

Operation debug#

silk
effect fn debug(self: &Self, formatter: &mut Formatter) -> ()
    ! WriterError
    ? &mut Writer

Writes the fields, as MsgpackError { reason: ..., offset: ... }.

MsgpackToken#

silk
pub union MsgpackToken

One MessagePack token: a complete scalar, or the header of an array or a map.

Nil#

silk
Nil

nil.

Boolean#

silk
Boolean

true or false.

Field value#

silk
pub value: bool

The value.

Integer#

silk
Integer

An integer that fits in i64, whatever width it was written in.

Field value#

silk
pub value: i64

The value.

Unsigned#

silk
Unsigned

An integer above the largest i64, written as a uint 64.

Field value#

silk
pub value: u64

The value.

Float#

silk
Float

A float 64, or a float 32 widened to f64.

Field value#

silk
pub value: f64

The value.

String#

silk
String

A string. Its UTF-8 is checked.

Field span#

silk
pub span: MsgpackSpan

The range of the text in the input, without the header.

Binary#

silk
Binary

Binary data.

Field span#

silk
pub span: MsgpackSpan

The range of the data in the input, without the header.

Array#

silk
Array

The start of an array. The tokens of count values follow.

Field count#

silk
pub count: usize

The number of elements.

Map#

silk
Map

The start of a map. The tokens of count keys follow, each followed by its value.

Field count#

silk
pub count: usize

The number of key and value pairs.

Extension#

silk
Extension

An extension value, such as a timestamp (type -1). It is not interpreted.

Field kind#

silk
pub kind: i8

The extension type. MessagePack reserves the negative types.

Field span#

silk
pub span: MsgpackSpan

The range of the payload in the input, without the header and the type.

EndOfInput#

silk
EndOfInput

The input has no more bytes. Later reads also return it.

Implementation Copy for MsgpackToken#

silk
impl Copy for MsgpackToken

MsgpackScanner#

silk
pub struct MsgpackScanner<'input>

A reader of the MessagePack tokens of one slice.

When to use#

Use it to read MessagePack without a tree: call MsgpackScanner.next until it returns MsgpackToken.EndOfInput.

Details#

The scanner borrows the input and holds only its position, so MsgpackScanner.fork is cheap. It reads values one after another and does not require the input to be one value; the caller decides what follows a complete value.

Gotchas#

After an error, the scanner cannot continue. Make a new one.

Implementation MsgpackScanner<'input>#

silk
impl<'input> MsgpackScanner<'input>

Associated function make#

silk
pub fn make(input: &'input [u8]) -> MsgpackScanner<'input>

Returns a scanner at the start of input.

Method next#

silk
pub fn next(self: &mut MsgpackScanner<'input>) -> Result<MsgpackToken, MsgpackError>

Returns the next token, or EndOfInput when no bytes are left.

Details#

A string, binary or extension token is returned only when its whole payload is in the input. An array or map token is returned only when the input has at least one byte left for each element it counts, so count is never larger than the input.

Errors#

  • MsgpackError with UnexpectedEnd if the input ends inside the value, ReservedByte for the byte 0xc1, or InvalidUtf8 for a string that is not UTF-8. The scanner cannot continue.

Method peek#

silk
pub fn peek(self: &mut MsgpackScanner<'input>) -> Result<MsgpackToken, MsgpackError>

Returns the next token without consuming it. The next call of next returns the same token.

Errors#

  • MsgpackError with UnexpectedEnd if the input ends inside the value, ReservedByte for the byte 0xc1, or InvalidUtf8 for a string that is not UTF-8. The scanner cannot continue.

Method fork#

silk
pub fn fork(self: &MsgpackScanner<'input>) -> MsgpackScanner<'input>

Returns a scanner at the same position as self, which reads the same tokens next without moving self.

When to use#

Use it to look ahead more than one token, such as to find a key of a map before reading the entries before it.

Method offset#

silk
pub fn offset(self: &MsgpackScanner<'input>) -> usize

Returns the byte offset of the last token that next or peek returned: the offset of its first header byte. Use it for errors.

Method end#

silk
pub fn end(self: &MsgpackScanner<'input>) -> usize

Returns the byte offset after the last token that next or peek returned: after the payload of a scalar, or after the header of an array or a map.

Method length#

silk
pub fn length(self: &MsgpackScanner<'input>) -> usize

Returns the length of the input in bytes.

Method slice#

silk
pub fn slice(self: &MsgpackScanner<'input>, span: MsgpackSpan) -> &'input [u8]

Returns the input bytes of span, such as the UTF-8 of a string token or the data of a binary token.

Gotchas#

Traps if span is not inside the input. A span from a token of this scanner always is.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page