comm-central/third_party/rust/icu_normalizer/src
Name Description Size
error.rs Normalizer-specific error 1210
lib.rs Normalizing text into Unicode Normalization Forms. This module is published as its own crate ([`icu_normalizer`](https://docs.rs/icu_normalizer/latest/icu_normalizer/)) and as part of the [`icu`](https://docs.rs/icu/latest/icu/) crate. See the latter for more details on the ICU4X project. # Implementation notes The normalizer operates on a lazy iterator over Unicode scalar values (Rust `char`) internally and iterating over guaranteed-valid UTF-8, potentially-invalid UTF-8, and potentially-invalid UTF-16 is a step that doesn’t leak into the normalizer internals. Ill-formed byte sequences are treated as U+FFFD. The normalizer data layout is not based on the ICU4C design at all. Instead, the normalization data layout is a clean-slate design optimized for the concept of fusing the NFD decomposition into the collator. That is, the decomposing normalizer is a by-product of the collator-motivated data layout. Notably, the decomposition data structure is optimized for a starter decomposing to itself, which is the most common case, and for a starter decomposing to a starter and a non-starter on the Basic Multilingual Plane. Notably, in this case, the collator makes use of the knowledge that the second character of such a decomposition is a non-starter. Therefore, decomposition into two starters is handled by generic fallback path that looks the decomposition from an array by offset and length instead of baking a BMP starter pair directly into a trie value. The decompositions into non-starters are hard-coded. At present in Unicode, these appear to be special cases falling into three categories: 1. Deprecated combining marks. 2. Particular Tibetan vowel sings. 3. NFKD only: half-width kana voicing marks. Hopefully Unicode never adds more decompositions into non-starters (other than a character decomposing to itself), but if it does, a code update is needed instead of a mere data update. The composing normalizer builds on the decomposing normalizer by performing the canonical composition post-processing per spec. As an optimization, though, the composing normalizer attempts to pass through already-normalized text consisting of starters that never combine backwards and that map to themselves if followed by a character whose decomposition starts with a starter that never combines backwards. As a difference with ICU4C, the composing normalizer has only the simplest possible passthrough (only one inversion list lookup per character in the best case) and the full decompose-then-canonically-compose behavior, whereas ICU4C has other paths between these extremes. The ICU4X collator doesn't make use of the FCD concept at all in order to avoid doing the work of checking whether the FCD condition holds. 121189
properties.rs Access to the Unicode properties or property-based operations that are required for NFC and NFD. Applications should generally use the full normalizers that are provided at the top level of this crate. However, the APIs in this module are provided for callers such as HarfBuzz that specifically want access to the raw canonical composition operation e.g. for use in a glyph-availability-guided custom normalizer. 21430
provider.rs 🚧 \[Unstable\] Data provider struct definitions for this ICU4X component. <div class="stab unstable"> 🚧 This code is considered unstable; it may change at any time, in breaking or non-breaking ways, including in SemVer minor releases. While the serde representation of data structs is guaranteed to be stable, their Rust representation might not be. Use with caution. </div> Read more about data providers: [`icu_provider`] 8906
uts46.rs Bundles the part of UTS 46 that makes sense to implement as a normalization. This is meant to be used as a building block of an UTS 46 implementation, such as the `idna` crate. 5724