silk.json.scanner.

When to use#

Use JsonScanner to read a complete document that is in one slice, token by token. It is the input of Deserialize implementations. Use JsonScanState when the bytes arrive in pieces and you keep the parts of each token yourself. To read from a BufferedInput, use silk.json.reader.JsonReader. To get a tree, use silk.json.value.Json.parse.

Details#

The scanner checks the full JSON grammar (RFC 8259) as it reads, including UTF-8 in strings and the nesting of arrays and objects. A string or number token gives a JsonSpan in the source. Strings keep their source spelling. The token says if the string has escapes, so you can decode it with silk.json.text.JsonText.decode only when needed. All offsets are byte offsets from the start of the document.

Errors#

• JsonError with a JsonReason and the offset of the problem. After an error, the state cannot continue. Make a new one.

Gotchas#

Nesting is limited to MAX_DEPTH (64) levels. The scanner does not check for duplicate keys. silk.json.value.Json.parse does.

Examples#

Read the tokens of an array#

import silk.json.scanner { JsonError, JsonScanner, JsonToken }
import silk.core.result { Result }

fn countNumbers(input: &[u8]) -> i32 {
  let mut scanner = JsonScanner.make(input)
  let mut count = 0
  while true {
    match move JsonScanner.next(&mut scanner) {
      Result<JsonToken, JsonError>.Failure { error } => {
        return -1
      }
      Result<JsonToken, JsonError>.Success { value } => match move value {
        JsonToken.Number { span } => {
          count = count + 1
        }
        JsonToken.EndOfInput => {
          return count
        }
        _ => {}
      }
    }
  }
  return count
}

pub fn main() -> i32 {
  if countNumbers(b"[1, 2.5, -3e2]") != 3 || countNumbers(b"[1,]") != -1 {
    return 1
  }
  return 0
}

Import as JsonScanner with import silk.json.scanner { JsonScanner }.

Public declarations: 9.

MAX_DEPTH#

pub const MAX_DEPTH: usize

The largest number of arrays and objects that can be open at one time: 64.

JsonSpan#

pub struct JsonSpan

A range of bytes in the document. Offsets are from the start of the document.

Field offset#

pub offset: usize

The offset of the first byte from the start of the document.

Field length#

pub length: usize

The number of bytes.

Implementation Copy for JsonSpan#

impl Copy for JsonSpan

JsonReason#

pub enum JsonReason

Why JSON reading or writing failed.

UnexpectedByte#

UnexpectedByte

A byte that is not allowed at this position.

UnterminatedString#

UnterminatedString

A string has no closing quote.

InvalidEscape#

InvalidEscape

A backslash escape is not valid, or a \u surrogate pair is incomplete.

InvalidUtf8#

InvalidUtf8

A string contains bytes that are not valid UTF-8.

InvalidNumber#

InvalidNumber

A number does not follow the JSON number grammar, such as 01 or 1..

DuplicateKey#

DuplicateKey

An object has the same key two times.

InvalidValue#

InvalidValue

A value in a tree has a child of the wrong kind.

TypeMismatch#

TypeMismatch

The JSON is valid, but the value does not have the type that the reader asked for.

NumberOutOfRange#

NumberOutOfRange

A number does not fit in the requested numeric type.

MissingField#

MissingField

An object does not have a member that the reader needs.

UnknownField#

UnknownField

An object has a member that the requested type does not have.

UnexpectedEnd#

UnexpectedEnd

The input ended before the value was complete.

DepthExceeded#

DepthExceeded

The nesting is deeper than the limit.

TrailingContent#

TrailingContent

More content follows the value.

OffsetOverflow#

OffsetOverflow

A byte offset does not fit in usize.

TokenTooLarge#

TokenTooLarge

A token does not fit in the buffer that the caller gave for it.

Implementation Debug for JsonReason#

impl Debug for JsonReason

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the name of the variant, such as UnexpectedByte.

JsonError#

pub struct JsonError

The error from a JSON operation: the reason and, when known, the position and buffer sizes.

Gotchas#

After a read error, the scanner or reader that returned it cannot continue.

Field reason#

pub reason: JsonReason

Why the operation failed.

Field offset#

pub offset: Option<usize>

The byte offset of the problem, or of the end of the input. Absent for some output errors.

Field required#

pub required: Option<usize>

For TokenTooLarge, the buffer size that the token needs, if known.

Field available#

pub available: Option<usize>

For TokenTooLarge, the buffer size that was available.

Implementation Debug for JsonError#

impl Debug for JsonError

Operation debug#

effect fn debug(self: &Self, formatter: &mut Formatter) -> () ! WriterError ? &mut Writer

Writes the fields, as JsonError { reason: ..., offset: ..., ... }.

Implementation JsonError#

impl JsonError

Associated function at#

pub fn at(reason: JsonReason, offset: usize) -> JsonError

Returns an error with reason at byte offset. Deserialize implementations use it.

JsonToken#

pub union JsonToken

One JSON token. String and number tokens give the range of their source text.

ObjectBegin#

ObjectBegin

{

ObjectEnd#

ObjectEnd

}

ArrayBegin#

ArrayBegin

[

ArrayEnd#

ArrayEnd

]

KeySeparator#

KeySeparator

: between a key and its value.

ValueSeparator#

ValueSeparator

, between elements or members.

String#

String

A string. Its UTF-8 and escapes are checked.

Field span#

pub span: JsonSpan

The source range, including the quotes.

Field escaped#

pub escaped: bool

true if the string has at least one backslash escape.

Number#

Number

A number. It is not converted. Use silk.text.format.Format to read its text.

Field span#

pub span: JsonSpan

The source range.

True#

True

true

False#

False

false

Null#

Null

null

EndOfInput#

EndOfInput

The value and any whitespace after it are complete. Later reads also return it.

Implementation Copy for JsonToken#

impl Copy for JsonToken

JsonScanEvent#

pub union JsonScanEvent

What one step of JsonScanState.step found.

Token#

Token

A complete token.

Field token#

pub token: JsonToken

The token. Its offsets are from the start of the document.

NeedMore#

NeedMore

The step used all bytes of the piece. Give it the next piece.

EndOfInput#

EndOfInput

The document is complete.

JsonScanStep#

pub struct JsonScanStep

The result of one step of JsonScanState.step.

Field consumed#

pub consumed: usize

The number of bytes of this piece that the step used.

Field fragment#

pub fragment: JsonSpan

The bytes of the current token inside this piece. It can be empty. Keep them if the token continues in the next piece.

Field event#

pub event: JsonScanEvent

A token, a request for more input, or the end of the document.

JsonScanState#

pub struct JsonScanState

The state of a JSON scan that reads its input in pieces.

When to use#

Use it when the bytes arrive in pieces and you keep the parts of each token yourself. When the whole document is in one slice, use JsonScanner.

Details#

The scanner checks the full JSON grammar (RFC 8259) as it reads, including UTF-8 in strings and the nesting of arrays and objects, up to MAX_DEPTH (64) levels.

It does not keep a reference to the input. Call JsonScanState.step with the rest of the current piece until it returns NeedMore or EndOfInput. After NeedMore, give it the next piece.

JsonScanner#

pub struct JsonScanner<'input>

A JSON scanner over a complete document in one slice. Make one with JsonScanner.make.

When to use#

Use it to read a document that is in one slice, token by token, without building a tree and without allocating. It is the input of Deserialize implementations. When the bytes arrive in pieces, use JsonScanState. To read from a BufferedInput, use silk.json.reader.JsonReader. To get a tree, use silk.json.value.Json.parse.

Details#

The scanner checks the full JSON grammar (RFC 8259) as it reads, including UTF-8 in strings and the nesting of arrays and objects, up to MAX_DEPTH (64) levels. A string or number token gives a JsonSpan in the source. Strings keep their source spelling. The token says if the string has escapes, so you can decode it with silk.json.text.JsonText.decode only when needed.

Gotchas#

The scanner does not check for duplicate keys. silk.json.value.Json.parse does. After an error, the scanner cannot continue. Make a new one.

Implementation JsonScanState#

impl JsonScanState

Associated function make#

pub fn make() -> JsonScanState

Returns the state for the start of one JSON document. It does not allocate.

Method step#

pub fn step(self: &mut JsonScanState, chunk: &[u8], final: bool) -> Result<JsonScanStep, JsonError>

Reads from chunk up to the end of one token, the end of the chunk, or the end of the document.

Details#

After a Token event, call step again with the bytes after consumed. Set final to true for the last piece and for all of its rest, also when the rest is empty.

Errors#

• JsonError if the input is not valid JSON. After an error, the state cannot continue.

Gotchas#

NeedMore uses all of the piece. A number at the end of a piece that is not final is not complete yet, because the next byte could be a digit. The byte after the number stays for the next call.

Implementation JsonScanner<'input>#

impl<'input> JsonScanner<'input>

Associated function make#

pub fn make(input: &'input [u8]) -> JsonScanner<'input>

Returns a scanner at the start of input. input must be one complete document.

Method next#

pub fn next(self: &mut JsonScanner<'input>) -> Result<JsonToken, JsonError>

Returns the next token, or EndOfInput when the document and the whitespace after it are complete.

Errors#

• JsonError if the input is not valid JSON, or UnexpectedEnd if it ends early.

Method peek#

pub fn peek(self: &mut JsonScanner<'input>) -> Result<JsonToken, JsonError>

Returns the next token without consuming it. The next call to next returns the same token.

Method offset#

pub fn offset(self: &JsonScanner<'input>) -> usize

Returns the byte offset of the last token that next or peek returned. Use it for errors.

Method length#

pub fn length(self: &JsonScanner<'input>) -> usize

Returns the length of the input in bytes.

Method slice#

pub fn slice(self: &JsonScanner<'input>, span: JsonSpan) -> &'input [u8]

Returns the source bytes of span, such as the text of a string or number token.

Category: Data formats.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2.