silk.text.segmentation.

When to use#

Use graphemes or GraphemeCursor to find user-perceived characters, for example to move a text cursor or to truncate text. Use words for word selection and sentences for sentence selection. Use silk.text.line_break to find where a line can wrap.

Details#

The functions implement the default (untailored) rules of UAX #29 revision 49. They take a string and report byte offsets into it. A boundary list is appended to a caller-owned list. It is sorted and includes 0 and the text's byte length once each, even for empty text. The scans use constant extra storage and look at each scalar a bounded number of times.

Gotchas#

The rules are not tailored for a language. Word boundaries do not split Thai, Lao, Khmer, Myanmar, Chinese or Japanese text, and sentence boundaries do not know abbreviations.

Examples#

Count the user-perceived characters of a text#

import silk.collections.list { List }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.text.segmentation

pub effect fn main() -> i32
  ! OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let mut boundaries = List<usize>.make()
  // "e" with a combining acute accent, a family emoji joined with ZWJ, and "x".
  run segmentation.graphemes("e\u{301}\u{1f468}\u{200d}\u{1f469}\u{200d}\u{1f467}x", &mut boundaries)
    |> Effect.provideMut<Allocator>(&mut allocator)
  if boundaries.length() - 1 != 3 {
    return 1
  }
  return 0
}

Import as segmentation with import silk.text.segmentation as segmentation.

Public declarations: 5.

GraphemeCursor#

pub struct GraphemeCursor

A forward cursor over the extended grapheme clusters of one text. It does not allocate.

When to use#

Use a GraphemeCursor to step through user-perceived characters one at a time, without a list of boundaries. Use graphemes to get all boundaries at once.

Details#

Make the cursor with GraphemeCursor.make and pass the same text to each call of GraphemeCursor.next. The cursor implements the default extended grapheme cluster rules of UAX #29 for Unicode 18.0.0, including the GB9c rule for Indic conjuncts and the emoji rules.

Gotchas#

The cursor keeps only a byte offset and rule state. Passing a different text to a later call gives meaningless boundaries; it does not trap.

Implementation GraphemeCursor#

impl GraphemeCursor

Associated function make#

pub fn make() -> Self

Returns a cursor at the start of a text.

Method next#

pub fn next(self: &mut Self, text: string) -> usize

Moves to the next extended grapheme cluster boundary in text and returns its byte offset.

Details#

The first call returns the end of the first cluster, never 0. After the last cluster, each call returns text's byte length. For empty text, each call returns 0.

Method nextBytes#

pub fn nextBytes(self: &mut Self, text: &[u8]) -> usize

Moves to the next extended grapheme cluster boundary in the UTF-8 bytes text and returns its byte offset.

When to use#

Use nextBytes in a scan that holds UTF-8 bytes rather than a string. Use GraphemeCursor.next otherwise.

Details#

The result is the same as GraphemeCursor.next for the same text. An ill-formed byte counts as U+FFFD and moves one byte.

nextGrapheme#

pub fn nextGrapheme(text: string, start: usize) -> usize

Returns the byte offset of the extended grapheme cluster boundary that follows start in text.

Details#

start must be a cluster boundary, such as 0 or a result of an earlier call. At or past the end of text, the result is text's byte length. The function does not allocate.

Gotchas#

A start that is not a cluster boundary is not detected. The scan starts there as if a new cluster began, so the result can split a cluster.

graphemes#

pub effect fn graphemes(text: string, output: &mut List<usize>) -> ()
  ! OutOfMemoryError
  ? &mut Allocator

Appends the extended grapheme cluster boundaries of text to output, as byte offsets.

When to use#

Use graphemes to count or split user-perceived characters, for example to move a text cursor or to truncate text without cutting an accent or an emoji from its base.

Details#

The boundaries follow the default extended grapheme cluster rules of UAX #29 for Unicode 18.0.0. They are sorted and include 0 and text's byte length once each, so empty text gives one boundary, 0. The number of clusters is the number of boundaries minus 1. The function does not clear output, so clear it first to reuse its storage.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

words#

pub effect fn words(text: string, output: &mut List<usize>) -> ()
  ! OutOfMemoryError
  ? &mut Allocator

Appends the word boundaries of text to output, as byte offsets.

When to use#

Use words for double-click selection, word counts, or moving a cursor by words.

Details#

The boundaries follow the default word boundary rules of UAX #29 for Unicode 18.0.0. They are sorted and include 0 and text's byte length once each, so empty text gives one boundary, 0. Spaces and punctuation are segments of their own: filter the segments that contain letters or digits to count words. The function does not clear output, so clear it first to reuse its storage.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

Gotchas#

The default rules do not split Thai, Lao, Khmer, Myanmar, Chinese or Japanese text into words. Each run of those characters stays one segment.

sentences#

pub effect fn sentences(text: string, output: &mut List<usize>) -> ()
  ! OutOfMemoryError
  ? &mut Allocator

Appends the sentence boundaries of text to output, as byte offsets.

Details#

The boundaries follow the default sentence boundary rules of UAX #29 for Unicode 18.0.0. They are sorted and include 0 and text's byte length once each, so empty text gives one boundary, 0. A sentence keeps its trailing spaces and closing punctuation. The function does not clear output, so clear it first to reuse its storage.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

Gotchas#

The default rules do not know abbreviations, so "Mr. Smith" has a boundary after "Mr. ".

Category: Text.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.