silk.text.word_dictionary.

When to use#

Use IcuDictionary with line_break.breaksWithDictionary in silk.text.line_break to wrap text that can contain Thai, Lao, Khmer or Myanmar. These scripts do not separate words with spaces, so the line breaker needs a dictionary to find where a line can end.

Details#

The engine is a port of the dictionary break engines of ICU release 78.1 (dictbe.cpp), and the data is ICU's Thai, Lao, Khmer and Burmese word lists, packed as minimal tries in silk.text.word_dictionary_thai, silk.text.word_dictionary_lao, silk.text.word_dictionary_khmer and silk.text.word_dictionary_myanmar. Segmentation allocates only to grow the caller's list.

Gotchas#

The dictionaries add about 2.9 MB to a program that uses this module. Only import it when the text can contain these scripts.

Examples#

Wrap Thai text#

import silk.collections.list { List }
import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.text.line_break
import silk.text.line_break { BreakOpportunity, LineOptions }
import silk.text.word_dictionary { IcuDictionary }

pub effect fn main() -> i32
  ! OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let dictionary = IcuDictionary.make()
  let mut opportunities = List<BreakOpportunity>.make()
  // "Thai language" without a space between the two words.
  run line_break.breaksWithDictionary(
    "\u{e20}\u{e32}\u{e29}\u{e32}\u{e44}\u{e17}\u{e22}",
    LineOptions.make(),
    &dictionary,
    &mut opportunities,
  )
    |> Effect.provideMut<Allocator>(&mut allocator)
  // A line can end after the first word, at byte 12, and must end at the end.
  let items = opportunities.asSlice()
  if items.length != 2 || items[0].offset != 12 {
    return 1
  }
  return 0
}

Import as IcuDictionary with import silk.text.word_dictionary { IcuDictionary }.

Public declarations: 1.

IcuDictionary#

pub struct IcuDictionary

A word segmenter for Thai, Lao, Khmer and Myanmar text, with the dictionaries and the engines of ICU 78.1.

When to use#

Pass an IcuDictionary to line_break.breaksWithDictionary in silk.text.line_break to wrap text that can contain Thai, Lao, Khmer or Myanmar, whose words are not separated by spaces.

Details#

The value has no fields: the dictionaries are static data, so IcuDictionary.make does not allocate. The engine is a port of the ICU dictionary break engines: an ordered three-word lookahead, the combination of unknown text with short words, the resynchronization sets of each language, marks kept with the word before them, and the Thai suffix rules.

Gotchas#

A program that uses IcuDictionary links about 2.9 MB of dictionary data. There is no dictionary for Chinese or Japanese.

Implementation IcuDictionary#

impl IcuDictionary

Associated function make#

pub fn make() -> Self

Returns the segmenter. It does not allocate.

Implementation DictionarySegmenter for IcuDictionary#

impl DictionarySegmenter for IcuDictionary

Operation breaks#

effect fn breaks(self: &Self, text: string, output: &mut List<usize>) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Implementation IcuDictionary#

impl IcuDictionary

Method wordBreaks#

pub effect fn wordBreaks(self: &Self, text: string, output: &mut List<usize>) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Appends the dictionary word boundaries of text to output, as sorted byte offsets that include 0 and text's byte length once each.

Details#

Only runs of Thai, Lao, Khmer and Myanmar letters are split into words. Other text, such as Latin words, spaces and punctuation, is not split, and a run of one script followed by letters of the same script continues the word. The function allocates only to grow output.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

Gotchas#

These are not the UAX #29 word boundaries of silk.text.segmentation; they are the dictionary words that line breaking uses.

Category: Text.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.