Glossary Development

Unicode Normalization

Unicode normalization converts text that can be written in more than one way, such as an accented letter, into a single standard form so that equal-looking strings compare as equal.

Last reviewed:

AdSense Placeholder
Slot: header_reference_page

In more detail

Unicode allows é as one precomposed character (U+00E9) or as e followed by a combining accent (U+0301). There are four normalization forms: NFC composes where possible, NFD decomposes everything, and NFKC and NFKD also replace compatibility characters (the ligature fi becomes fi, the superscript ² becomes 2). NFC is the usual choice for storing and exchanging text.

Why it matters

Without normalization, two identical-looking strings can have different lengths and fail equality checks, break database lookups and produce duplicate usernames or file names, and the compatibility forms are also used in security checks for look-alike characters. The Unicode Normalizer converts between the forms and lists the code points.

Example

é precomposed has length 1; its NFD form e + U+0301 has length 2. After NFC both are the single character U+00E9 and compare equal.

AdSense Placeholder
Slot: footer_leaderboard