silk.schema.msgpack.

Read and write Schema types as MessagePack.

When to use#

Call SchemaMsgpack.decode to read a value of a silk.schema.schema.Schema type from MessagePack bytes, SchemaMsgpack.decodeAll to report every problem in the bytes at once, and SchemaMsgpack.encode to write a value to the ambient Writer. Use MessagePack instead of silk.schema.json.SchemaJson when the reader is a program and size or binary data matters. Use silk.msgpack.value.Msgpack for MessagePack whose shape is not known in advance.

Details#

A struct is a map with one text key per field, in declaration order; a union follows the same tag rules as in JSON. An Option that is None is nil, and a None field is left out. A silk.collections.bytes.Bytes field is MessagePack binary data, not Base64 text. Integers are written in the shortest format and read from any width, and a float 32 is read as f64.

Decoding streams over the bytes with silk.msgpack.scanner.MsgpackScanner and builds the value directly, with no tree in between. Each map is scanned once into an index of its entries, so entries may come in any order, and entries the type does not name are skipped, or reported when its spec is Spec.strict. Every problem carries the span of the bytes it is about.

Gotchas#

A map with a key that is not text cannot be read as a struct. An extension value, such as a timestamp (type -1), has no schema type and is reported as IssueReason.InvalidType where a field reads it; it is skipped where no field does. Nesting is limited to 64 levels.

silk.schema.report.SchemaReport shows problems as lines and columns of text, which MessagePack does not have. Show a problem by its path and by the byte offsets of its issue's span.

Examples#

Write a struct and read it back#

silk
import silk.collections.bytes { Bytes }
import silk.collections.slice { Slice }
import silk.core.effect { Effect }
import silk.io.writer { Writer, WriterError }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.schema.issue { SchemaError }
import silk.schema.msgpack { SchemaMsgpack }
import silk.schema.schema { Schema }

struct Capture {
  bytes: [u8; 64],
  length: usize,
}

impl Writer for Capture {
  effect fn writeAll(self: &mut Self, value: &[u8]) -> () ! WriterError ? &mut Writer {
    if self.length + value.length > 64 { fail Writer.failure() }
    Slice.copyInto<u8>(&mut self.bytes, self.length, value)
    self.length = self.length + value.length
    return ()
  }

  effect fn flush(self: &mut Self) -> () ! WriterError ? &mut Writer { return () }
}

struct Frame {
  pub id: u32,
  pub payload: Bytes,
}

impl Schema for Frame {}

effect fn check() -> i32 ! SchemaError | WriterError | OutOfMemoryError ? &mut Allocator {
  let frame = Frame { id: 7, payload: run Bytes.copy(b"\xff\x00") }
  let mut capture = Capture { bytes: [0; 64], length: 0 }
  run SchemaMsgpack.encode<Frame>(&frame) |> Effect.provideMut<Writer>(&mut capture)
  let written = Slice.view<u8>(&capture.bytes, 0, capture.length)
  // {"id": 7, "payload": bin(ff 00)}
  if written != b"\x82\xa2id\x07\xa7payload\xc4\x02\xff\x00" {
    return 1
  }
  let again = run SchemaMsgpack.decode<Frame>(written)
  if again.id != 7 || again.payload.asSlice() != b"\xff\x00" {
    return 2
  }
  return 0
}

pub effect fn main() -> i32 ! SchemaError | WriterError | OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  return run check() |> Effect.provideMut<Allocator>(&mut allocator)
}

Import as SchemaMsgpack with import silk.schema.msgpack { SchemaMsgpack }.

Public declarations: 3.

SchemaMsgpack#

silk
pub struct SchemaMsgpack

The owner of the MessagePack entry points. It has no fields and is never constructed.

Implementation SchemaMsgpack#

silk
impl SchemaMsgpack

Associated function decode#

silk
pub effect fn decode<T: Schema>(bytes: &[u8]) -> T
    ! SchemaError | OutOfMemoryError
    ? &mut Allocator

Reads one value of type T from the MessagePack bytes, stopping at the first problem.

Details#

The whole input must be exactly one MessagePack value. A struct is read from a map with text keys, in any order; integers are read from any width, and Bytes from binary data.

Errors#

  • SchemaError if the bytes are not valid MessagePack, hold more than one value, or do not hold a T. Its path names the part that failed, and its issue has the span of the bytes it is about, as byte offsets into bytes.

  • OutOfMemoryError if the Allocator cannot supply the value.

Associated function decodeAll#

silk
pub effect fn decodeAll<T: Schema>(bytes: &[u8]) -> T
    ! SchemaErrors | OutOfMemoryError
    ? &mut Allocator

Reads one value of type T from the MessagePack bytes, reporting every problem.

Details#

A field, element or variant that fails is recorded and skipped, and reading goes on with the next one, so one call lists every problem, in input order: a missing key, a wrong type, a number out of range and a failed rule alike. Bytes that are not one valid MessagePack value are the one problem reported. At most MAX_PROBLEMS (64) are listed.

Errors#

  • SchemaErrors with every problem found.

  • OutOfMemoryError if the Allocator cannot supply the value.

Associated function encode#

silk
pub effect fn encode<T: Schema>(value: &T) -> ()
    ! SchemaError | WriterError | OutOfMemoryError
    ? &mut Writer | &mut Allocator

Writes value as MessagePack to the ambient Writer, without flushing it.

Details#

A struct is written as a map with one text key per field, in declaration order, without its None fields. Every value takes the shortest format that holds it, and a float is always a float 64. Nothing is buffered: each value goes to the Writer as it is reached.

Errors#

  • SchemaError if MessagePack cannot hold the value: text, bytes or a list longer than 4294967295, or a value nested more than 64 levels deep. Its path is empty.

  • WriterError if the Writer fails. Bytes that it accepted before the failure stay written.

  • OutOfMemoryError if a codec needs memory that the Allocator cannot supply.

MsgpackSchemaReader#

silk
pub struct MsgpackSchemaReader<'input>

A SchemaReader over MessagePack bytes.

When to use#

SchemaMsgpack.decode and SchemaMsgpack.decodeAll use it. Use it directly with silk.schema.schema.Schemas.read to read a value that more bytes follow, which SchemaMsgpack.decode refuses.

Implementation MsgpackSchemaReader<'input>#

silk
impl<'input> MsgpackSchemaReader<'input>

Associated function make#

silk
pub fn make(input: &'input [u8]) -> MsgpackSchemaReader<'input>

Returns a reader at the start of input that stops at the first problem.

Associated function collectingAll#

silk
pub fn collectingAll(input: &'input [u8]) -> MsgpackSchemaReader<'input>

Returns a reader at the start of input that records every problem instead of stopping at the first. Read them with MsgpackSchemaReader.intoErrors.

Gotchas#

A collecting read goes on after a failed part by reading past it, so input must be valid MessagePack; SchemaMsgpack.decodeAll checks that first.

Method intoErrors#

silk
pub fn intoErrors(self: Self) -> List<SchemaError>

Returns the problems the reader recorded, in the order they are in the input.

Implementation MsgpackSchemaReader<'input>#

silk
impl<'input> MsgpackSchemaReader<'input>

Implementation SchemaReader for MsgpackSchemaReader<'input>#

silk
impl<'input> SchemaReader for MsgpackSchemaReader<'input>

Operation path#

silk
fn path(self: &mut Self) -> &mut SchemaPath

Returns the path of the part being read.

Operation report#

silk
effect fn report(self: &mut Self, issue: Issue) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Fails with the issue, or records it and returns when the reader collects every problem.

Operation collecting#

silk
fn collecting(self: &Self) -> bool

Returns whether the reader records every problem.

Operation lastSpan#

silk
fn lastSpan(self: &Self) -> Option<Span>

Returns the span of the last complete value read.

Operation lastKind#

silk
fn lastKind(self: &Self) -> ValueKind

Returns the kind of the last complete value read.

Operation mark#

silk
effect fn mark(self: &mut Self) -> usize
    ! OutOfMemoryError
    ? &mut Allocator

Remembers the open maps and arrays and the scanner's position.

Operation recover#

silk
effect fn recover(self: &mut Self, mark: usize, skip: bool) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Returns to the mark, closing the maps and arrays opened since, and reads past the value there when skip is true.

Operation release#

silk
fn release(self: &mut Self, mark: usize) -> ()

Forgets the mark and every later one.

Operation boolean#

silk
effect fn boolean(self: &mut Self) -> bool
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads true or false.

Operation signed#

silk
effect fn signed(self: &mut Self) -> i64
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads an integer of any width as an i64. A float is not an integer, even a whole one.

Operation unsigned#

silk
effect fn unsigned(self: &mut Self) -> u64
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads an integer of any width as a u64. A float is not an integer, even a whole one.

Operation float#

silk
effect fn float(self: &mut Self) -> f64
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads a float, or an integer converted to the nearest f64.

Operation text#

silk
effect fn text(self: &mut Self) -> String
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads a MessagePack string. Binary data is not text.

Operation bytes#

silk
effect fn bytes(self: &mut Self) -> Bytes
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads MessagePack binary data. A string is not binary data, and is not read as Base64.

Operation null#

silk
effect fn null(self: &mut Self) -> bool
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads past nil and returns true, or returns false for another value.

Operation beginList#

silk
effect fn beginList(self: &mut Self) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads an array header.

Operation nextElement#

silk
effect fn nextElement(self: &mut Self) -> bool
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Returns true at the next element, or closes the array and returns false after the last.

Operation beginObject#

silk
effect fn beginObject(self: &mut Self) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reads the map's entries into an index once, so member finds each without scanning.

Errors#

  • Issue with IssueReason.InvalidType for a value that is not a map, or for a map with a key that is not text.

Operation member#

silk
effect fn member(self: &mut Self, key: string) -> bool
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Finds the entry key in the map's index and leaves the reader at its value; the first one when the map repeats the key. A missing key leaves the map's header as the last span.

Operation nextMember#

silk
effect fn nextMember(self: &mut Self) -> Option<String>
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Returns the key of the next entry no member or nextMember call found, and leaves the reader at its value. Returns None after every entry has been visited.

Operation rejectUnread#

silk
effect fn rejectUnread(self: &mut Self, expected: string<'static>) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Reports every entry no member call found, at its key.

Operation endObject#

silk
effect fn endObject(self: &mut Self) -> ()
    ! Issue | OutOfMemoryError
    ? &mut Allocator

Jumps past the whole map being read.

MsgpackSchemaWriter#

silk
pub struct MsgpackSchemaWriter

A SchemaWriter that writes MessagePack to the ambient Writer.

When to use#

SchemaMsgpack.encode uses it. Use it directly to write a value inside MessagePack that the program writes itself with silk.msgpack.output.MsgpackOutput.

Details#

It holds no bytes: every call writes straight to the Writer. An array or map header carries the count the schema gives beginList and beginObject.

Implementation MsgpackSchemaWriter#

silk
impl MsgpackSchemaWriter

Associated function make#

silk
pub fn make() -> MsgpackSchemaWriter

Returns a writer for one top-level value.

Implementation SchemaWriter for MsgpackSchemaWriter#

silk
impl SchemaWriter for MsgpackSchemaWriter

Operation boolean#

silk
effect fn boolean(self: &mut Self, value: bool) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes true or false in one byte.

Operation signed#

silk
effect fn signed(self: &mut Self, value: i64) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the integer in the shortest format that holds it.

Operation unsigned#

silk
effect fn unsigned(self: &mut Self, value: u64) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the integer in the shortest format that holds it.

Operation float#

silk
effect fn float(self: &mut Self, value: f64) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the number as a float 64, including NaN and the infinities.

Operation text#

silk
effect fn text(self: &mut Self, value: string) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes a MessagePack string.

Errors#

  • Issue with IssueReason.InvalidValue if value is longer than 4294967295 bytes.

  • WriterError if the Writer fails.

Operation bytes#

silk
effect fn bytes(self: &mut Self, value: &[u8]) -> ()
    ! Issue | WriterError | OutOfMemoryError
    ? &mut Writer | &mut Allocator

Writes MessagePack binary data, not Base64 text.

Errors#

  • Issue with IssueReason.InvalidValue if value is longer than 4294967295 bytes.

  • WriterError if the Writer fails.

Operation null#

silk
effect fn null(self: &mut Self) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes nil.

Operation beginList#

silk
effect fn beginList(self: &mut Self, count: usize) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the header of an array of count elements.

Errors#

  • Issue with IssueReason.InvalidValue if count is above 4294967295, or if 64 arrays and maps are already open.

  • WriterError if the Writer fails.

Operation element#

silk
effect fn element(self: &mut Self) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes nothing: MessagePack has no separator between elements.

Operation endList#

silk
effect fn endList(self: &mut Self) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes nothing: the array's header said how many elements it has.

Operation beginObject#

silk
effect fn beginObject(self: &mut Self, count: usize) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the header of a map of count entries.

Errors#

  • Issue with IssueReason.InvalidValue if count is above 4294967295, or if 64 arrays and maps are already open.

  • WriterError if the Writer fails.

Operation key#

silk
effect fn key(self: &mut Self, name: string) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes the key as a MessagePack string. Its value comes next.

Operation endObject#

silk
effect fn endObject(self: &mut Self) -> ()
    ! Issue | WriterError
    ? &mut Writer

Writes nothing: the map's header said how many entries it has.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.

View the source of this page