silk.text.bidi.

When to use#

Use BidiAnalysis to find the direction and embedding level of each part of a text that can contain Arabic, Hebrew or other right-to-left scripts. Use reorder to put the parts of a wrapped line in visual order.

Details#

BidiAnalysis.analyze implements the whole algorithm of UAX #9 revision 52: explicit embeddings, overrides and isolates, overflow handling, bracket pairs (N0), and the removal of formatting characters (X9). Its results are paragraphs and level runs over the byte offsets of the text. Paragraphs without right-to-left characters, Arabic digits or explicit formatting take a fast path. analyzeClasses and analyzeCodepoints resolve one paragraph of classes or code points, for conformance tests.

Gotchas#

The analysis applies rule L1 only with each paragraph as one line, in BidiRun.unwrappedLevel. After you wrap a paragraph, apply rule L1 to each line yourself before reorder.

Examples#

Find the right-to-left part of a text#

import silk.core.effect { Effect }
import silk.mem.allocator { Allocator, OutOfMemoryError }
import silk.text.bidi { BidiAnalysis, ParagraphDirection }

pub effect fn main() -> i32
  ! OutOfMemoryError {
  let mut allocator = Allocator.systemAllocatorProvider()
  let mut analysis = BidiAnalysis.make()
  // "abc " followed by the Hebrew letters alef, bet and gimel.
  run analysis.analyze("abc \u{5d0}\u{5d1}\u{5d2}", ParagraphDirection.Auto)
    |> Effect.provideMut<Allocator>(&mut allocator)
  let runs = analysis.runs()
  // The Hebrew run starts at byte 4 and has the right-to-left level 1.
  if runs.length != 2 || runs[1].start != 4 || runs[1].level != 1 {
    return 1
  }
  return 0
}

Import as bidi with import silk.text.bidi as bidi.

Public declarations: 8.

ParagraphDirection#

pub enum ParagraphDirection

The base direction of the paragraphs of a text.

Auto#

Auto

Each paragraph takes the direction of its first strong character (rules P2 and P3), and is left-to-right when it has none.

Ltr#

Ltr

Each paragraph is left-to-right, embedding level 0.

Rtl#

Rtl

Each paragraph is right-to-left, embedding level 1.

BidiParagraph#

pub struct BidiParagraph

One paragraph of a text, as BidiAnalysis.paragraphs gives it.

Details#

A paragraph ends after each paragraph separator, such as LF or U+2029 (rule P1). CR and LF each end a paragraph, so CRLF gives an empty paragraph between them.

Field start#

pub start: usize

The byte offset where the paragraph starts.

Field end#

pub end: usize

The byte offset where the paragraph ends, after its paragraph separator.

Field level#

pub level: u8

The paragraph embedding level: 0 for left-to-right, 1 for right-to-left.

Field hasStrong#

pub hasStrong: bool

True if the paragraph has a strong character outside isolates (rule P2). An ParagraphDirection.Auto paragraph without one is left-to-right.

Implementation Copy for BidiParagraph#

impl Copy for BidiParagraph

BidiRun#

pub struct BidiRun

A maximal range of a text with the same bidi results, as BidiAnalysis.runs gives it. A run never crosses a paragraph.

Field start#

pub start: usize

The byte offset where the run starts.

Field end#

pub end: usize

The byte offset where the run ends.

Field level#

pub level: u8

The resolved embedding level before the line rule L1. Shape the run with this level: even is left-to-right, odd is right-to-left.

Field unwrappedLevel#

pub unwrappedLevel: u8

The level after rule L1 when the whole paragraph is one line. Trailing spaces and separators get the paragraph level.

Field removed#

pub removed: bool

True for characters that rule X9 removes: explicit formatting characters and boundary neutrals. They keep their bytes and take the level of the preceding character.

Implementation Copy for BidiRun#

impl Copy for BidiRun

BidiWorkspace#

pub struct BidiWorkspace

The per-character results of one paragraph, as analyzeClasses and analyzeCodepoints give them.

When to use#

Use a BidiWorkspace to resolve one paragraph of bidi classes or code points and read the level of each character, as the Unicode conformance tests do. Use BidiAnalysis for text.

Details#

The workspace keeps its storage across calls. Each analysis replaces its contents. Character index is the position in the input of the analysis.

Implementation BidiWorkspace#

impl BidiWorkspace

Associated function make#

pub fn make() -> Self

Returns an empty workspace. It does not allocate.

Method length#

pub fn length(self: &Self) -> usize

Returns the number of characters of the last analysis.

Method level#

pub fn level(self: &Self, index: usize) -> u8

Returns the resolved embedding level of character index, before the line rule L1. Traps if index is not less than BidiWorkspace.length.

Method unwrappedLevel#

pub fn unwrappedLevel(self: &Self, index: usize) -> u8

Returns the level of character index after rule L1, with the whole paragraph as one line. Traps if index is not less than BidiWorkspace.length.

Method isRemoved#

pub fn isRemoved(self: &Self, index: usize) -> bool

Returns true if rule X9 removes character index: an explicit formatting character or a boundary neutral. Traps if index is not less than BidiWorkspace.length.

Method visualOrder#

pub effect fn visualOrder(self: &Self, output: &mut List<usize>) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Appends the character indices of the whole paragraph to output in visual order (rule L2), with the paragraph as one line. Characters that rule X9 removes are left out.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

BidiAnalysis#

pub struct BidiAnalysis

The results of the Unicode Bidirectional Algorithm (UAX #9) for one text: its paragraphs and its level runs.

When to use#

Use a BidiAnalysis to lay out text that can mix left-to-right and right-to-left scripts, such as English with Arabic or Hebrew. Shape each BidiRun in its direction, then order the runs of each line with reorder.

Details#

BidiAnalysis.analyze fills the analysis. It implements the whole algorithm: explicit embeddings, overrides and isolates, bracket pairs, and the removal of formatting characters (X9). The runs cover the text in logical order. An analysis keeps its storage, so reuse one analysis for many texts.

Implementation BidiAnalysis#

impl BidiAnalysis

Associated function make#

pub fn make() -> Self

Returns an empty analysis. It does not allocate.

Method paragraphs#

pub fn paragraphs(self: &Self) -> &[BidiParagraph]

Returns the paragraphs of the last analyzed text, in order. Empty text has one empty paragraph.

Method runs#

pub fn runs(self: &Self) -> &[BidiRun]

Returns the runs of the last analyzed text, in logical order. They cover the whole text without gaps.

Method runAt#

pub fn runAt(self: &Self, offset: usize) -> Option<BidiRun>

Returns the run that contains byte offset, or None at or past the end of the text.

Method paragraphAt#

pub fn paragraphAt(self: &Self, offset: usize) -> Option<BidiParagraph>

Returns the paragraph that contains byte offset, or None at or past the end of the text.

Method analyze#

pub effect fn analyze(self: &mut Self, text: string, direction: ParagraphDirection) -> ()
    ! OutOfMemoryError
    ? &mut Allocator

Replaces the results with those for text, with direction as the base direction of each paragraph.

Details#

A paragraph ends after each paragraph separator; CR and LF each count (rule P1). With ParagraphDirection.Auto, each paragraph takes its own direction (rules P2 and P3). Offsets in the results are byte offsets into text. The analysis keeps its storage, so a later call allocates only when a text needs more.

Errors#

• OutOfMemoryError if the Allocator cannot grow the storage of the analysis.

analyzeClasses#

pub effect fn analyzeClasses(
  classes: &[BidiClass],
  direction: ParagraphDirection,
  workspace: &mut BidiWorkspace,
) -> u8
  ! OutOfMemoryError
  ? &mut Allocator

Resolves one paragraph of bidi classes into workspace, and returns the paragraph embedding level.

When to use#

Use analyzeClasses to test the algorithm against class sequences, as the Unicode BidiTest.txt file gives them. Use BidiAnalysis.analyze for text.

Details#

The input is one paragraph: a paragraph separator class does not start a new one. Classes have no bracket identity, so no bracket pairs are found.

Errors#

• OutOfMemoryError if the Allocator cannot grow workspace.

analyzeCodepoints#

pub effect fn analyzeCodepoints(
  codepoints: &[u32],
  direction: ParagraphDirection,
  workspace: &mut BidiWorkspace,
) -> u8
  ! OutOfMemoryError
  ? &mut Allocator

Resolves one paragraph of code points into workspace, and returns the paragraph embedding level.

When to use#

Use analyzeCodepoints to test the algorithm against code point sequences, as the Unicode BidiCharacterTest.txt file gives them. Use BidiAnalysis.analyze for text.

Details#

The input is one paragraph: a paragraph separator does not start a new one. Bracket pairs use the Unicode paired-bracket properties.

Errors#

• OutOfMemoryError if the Allocator cannot grow workspace.

reorder#

pub effect fn reorder(levels: &[u8], output: &mut List<usize>) -> ()
  ! OutOfMemoryError
  ? &mut Allocator

Appends the indices of levels to output in visual order (rule L2).

When to use#

Use reorder for each line after wrapping: take the levels of the characters or runs of the line, apply rule L1 to them, and reorder.

Details#

levels holds the embedding level of each item of one line, in logical order, after rule L1. Leave out the characters that rule X9 removes. The function reverses each maximal sequence at or above each odd level, from the highest level down. It does not allocate scratch storage.

Errors#

• OutOfMemoryError if the Allocator cannot grow output.

Category: Text.

Profiles: aarch64-apple-darwin, aarch64-unknown-linux-gnu, aarch64-unknown-linux-gnu-no-libc, x86_64-unknown-linux-gnu, x86_64-unknown-linux-gnu-no-libc, wasm32-unknown-unknown, wasm32-wasip1, wasm32-wasip2, x86_64-pc-windows-msvc, wasm64-unknown-unknown.