silk.text.utf8_scan.

When to use#

Use decode in a hot loop that keeps byte offsets, such as a segmentation or line breaking scan. Use String.nextScalar in silk.text.string for ordinary iteration over a string, and silk.text.utf8_decoder for bytes that arrive in pieces.

Details#

decode reads the scalar at a byte offset and gives the offset after it. It accepts only the well-formed sequences of RFC 3629. An ill-formed byte, or a sequence that ends early, gives U+FFFD and moves one byte, so a scan always moves forward. The function does not allocate.

Examples#

Decode the scalars of a byte string#

import silk.text.utf8_scan

pub fn main() -> i32 {
  let bytes = b"a\xc3\xa9\xff"
  let first = utf8_scan.decode(bytes, 0)
  let second = utf8_scan.decode(bytes, first.next)
  let third = utf8_scan.decode(bytes, second.next)
  if first.scalar != 0x61 || second.scalar != 0xe9 || third.scalar != 0xfffd || third.next != 4 {
    return 1
  }
  return 0
}

Import as utf8_scan with import silk.text.utf8_scan as utf8_scan.

Public declarations: 2.

Decoded#

pub struct Decoded

One decoded scalar and the byte offset after it, as decode gives it.

Field scalar#

pub scalar: u32

The scalar value, or U+FFFD for an ill-formed byte.

Field next#

pub next: usize

The byte offset after the scalar: the offset of the next scalar, or the length of the bytes.

Implementation Copy for Decoded#

impl Copy for Decoded

decode#

pub fn decode(bytes: &[u8], at: usize) -> Decoded

Decodes the UTF-8 scalar that starts at byte offset at of bytes.

Details#

A well-formed sequence gives its scalar and the offset after it. An ill-formed byte, an overlong form, a surrogate, a value past U+10FFFF, or a sequence that ends early gives U+FFFD and at + 1. At or past the end of bytes, the result is U+FFFD and the length of bytes. ASCII decodes inline; longer sequences take one call.

Category: Text.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.