How Unicode and UTF-8 Work

Unicode gives every character a number called a code point, written U+0041 for A. There are 1,114,112 possible code points, arranged in 17 planes of 65,536. An encoding such as UTF-8 or UTF-16 turns code points into bytes. Look characters up with the Unicode Character Inspector and clean text with the Unicode Normalizer; the dictionary entry is Unicode.

UTF-8 in one table

UTF-8 is backward compatible with ASCII (the first 128 code points are one byte, identical to ASCII), self-synchronising (you can find a character boundary from any byte) and the default for the web. UTF-16 uses 2 bytes for most characters and a surrogate pair of two 2-byte units for code points above U+FFFF, such as emoji.

Examples, verified

One character, several code points

• é can be one code point (U+00E9) or two: e plus a combining acute accent (U+0301). The text looks the same but its length is 1 or 2, and a plain comparison fails. Normalisation converts both to one form: NFC composes, NFD decomposes; NFKC and NFKD also fold compatibility characters (fi becomes fi, ² becomes 2).
• What people see as one character (a grapheme cluster) can be many code points: the family emoji is 5 code points joined by zero-width joiners and takes 18 bytes in UTF-8.
• String length depends on the unit: bytes, UTF-16 units, code points or graphemes can all differ for the same text.

Mojibake and other traps

• Mojibake: UTF-8 bytes read as Latin-1 turn naïve into naïve. Always declare the encoding (<meta charset="utf-8">, an HTTP header, or your editor setting).
• Invisible characters (zero-width spaces, bidirectional controls) can hide in copied text and in identifiers; find them with the Invisible Character Detector.
• Lookalikes: Cyrillic а is not Latin a. Domain names handle this with Punycode, see Punycode.
• A UTF-8 byte order mark (EF BB BF) at the start of a file is allowed but usually unwanted.