silk.text.unicode.

When to use#

Use normalization where text with different encodings of the same characters must compare equal, such as user names or file names from different systems. Use Unicode.normalizeNfc for storage and comparison. Use Unicode.normalizeNfd when a consumer needs the decomposed form.

Details#

Normalization is explicit. string equality compares the exact UTF-8 bytes and does not normalize. Each normalization function returns a new String and needs an Allocator. The results are the same on each target. Unicode.dataVersion gives the Unicode version of the data.

Gotchas#

This module implements canonical normalization only. It does not implement compatibility normalization (NFKC or NFKD), locale-sensitive comparison, grapheme segmentation, or case folding.

Examples#

Make two encodings of the same text compare equal#

import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.core.effect { Effect }
import silk.text.string { String }
import silk.text.unicode { Unicode }

pub effect fn main() -> i32
  ! OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()

  // "e" followed by a combining acute accent, and the precomposed "é".
  let composed = run Unicode.normalizeNfc("e\u{301}")
    |> Effect.provideMut<Allocator>(&mut allocator)
  let decomposed = run Unicode.normalizeNfd("é") |> Effect.provideMut<Allocator>(&mut allocator)
  if String.view(&composed) != "é" || String.view(&decomposed) != "e\u{301}" {
    return 1
  }
  return 0
}

Import as Unicode with import silk.text.unicode { Unicode }.

Public declarations: 1.

Unicode#

pub struct Unicode

The owner type of the Unicode canonical normalization operations (NFC and NFD). It has no fields and is never constructed.

When to use#

Use normalization where text with different encodings of the same characters must compare equal, such as user names or file names from different systems. Use Unicode.normalizeNfc for storage and comparison. Use Unicode.normalizeNfd when a consumer needs the decomposed form.

Details#

Normalization is explicit. string equality compares the exact UTF-8 bytes and does not normalize. The results are the same on each target.

Gotchas#

Only canonical normalization is implemented. There is no compatibility normalization (NFKC or NFKD), locale-sensitive comparison, grapheme segmentation, or case folding.

Implementation Unicode#

impl Unicode

Associated function dataVersion#

pub fn dataVersion() -> string<'static>

Returns the Unicode version of the normalization data, such as "17.0.0".

Details#

A later version of the data can change normalization results for new characters.

Associated function longestDecomposition#

pub fn longestDecomposition() -> usize

Returns the maximum number of scalars in the full canonical decomposition of one scalar.

Associated function canonicalCombiningClass#

pub fn canonicalCombiningClass(
    /// Unicode scalar value represented as its unsigned code point.
    scalar: u32,
  ) -> u32

Returns the canonical combining class of a code point. Returns 0 for a starter or an unassigned code point.

Parameter scalar#

scalar: u32

Unicode scalar value represented as its unsigned code point.

Associated function normalizeNfd#

pub effect fn normalizeNfd(value: string) -> String ! OutOfMemoryError ? &mut Allocator

Returns the Normalization Form D (NFD) of value: each character fully decomposed, with marks in canonical order.

When to use#

Use normalizeNfd when a consumer needs base characters and marks as separate scalars. Use Unicode.normalizeNfc for storage and comparison.

Details#

The function does not change value. Two canonically equivalent inputs give equal output.

Errors#

• OutOfMemoryError if the Allocator cannot supply the storage for the result.

Associated function normalizeNfc#

pub effect fn normalizeNfc(value: string) -> String ! OutOfMemoryError ? &mut Allocator

Returns the Normalization Form C (NFC) of value: decomposed, put in canonical order, and then composed again.

When to use#

Use normalizeNfc before you store or compare text that can come in different encodings. Most text from keyboards and the web is already NFC.

Details#

The function does not change value. Two canonically equivalent inputs give equal output.

Errors#

• OutOfMemoryError if the Allocator cannot supply the storage for the result.

Category: Text.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2.