utf8Luau

The utf8 library encodes, decodes, counts and iterates the Unicode characters in a UTF-8 string.

A string holds bytes, and text in a string is UTF-8 by convention. A character outside ASCII takes two to four bytes. #s and the string library count bytes rather than characters.

Its decoding functions treat overlong encodings, surrogates (U+D800 to U+DFFF) and values above U+10FFFF as invalid UTF-8.

Returns a string containing the UTF-8 encoding of every codepoint passed in, where a codepoint is the number Unicode assigns to a character: 72 for H, 233 for é and 0x1F600 for 😀. Because ASCII characters encode to a single byte and everything else takes two to four, the three characters below add up to seven bytes.

Copyconst face = utf8.char(72, 233, 0x1F600)
print(face)
print(#face)

Hé😀
7

Every codepoint has to fall between 0 and 0x10FFFF, the highest one Unicode defines, and anything outside that range raises value out of range. Fractions are truncated rather than rejected, and calling utf8.char with no arguments returns an empty string.

Surrogates (0xD800 to 0xDFFF) are the one gap in that range: they aren't characters at all but the halves of the pairs UTF-16 uses to write codepoints above 0xFFFF. Even though utf8.len, utf8.codes and utf8.codepoint reject the result as invalid UTF-8, utf8.char encodes them anyway.

Copyconst surrogate = utf8.char(0xD800)
print(#surrogate)
print(utf8.len(surrogate))

3
nil	1

A pattern for string.gmatch and the other pattern functions that matches one UTF-8 character at a time. It's the same string Lua 5.3 defined, "[\0-\x7F\xC2-\xF4][\x80-\xBF]*": one byte that can start a character (ASCII, or a lead byte from 0xC2 to 0xF4) followed by any number of continuation bytes from 0x80 to 0xBF.

Copyfor character in string.gmatch("ñ€", utf8.charpattern) do
    print(`{character} takes {#character} bytes`)
end

ñ takes 2 bytes
€ takes 3 bytes

Because the * never checks how many continuation bytes the lead byte calls for, the pattern only works on text that's already valid UTF-8. On malformed input it glues extra continuation bytes onto the character before them and skips stray ones without complaint. When a string comes from somewhere you don't trust, check it with utf8.len first: on bad input it returns nil and the position of the first bad byte instead of a count.

Copyconst broken = "A\xC3\xA9\xA9B"
for character in string.gmatch(broken, utf8.charpattern) do
    print(#character)
end
print(utf8.len(broken))

1
3
1
nil	4

Returns an iterator for a for loop that gives the byte index and the codepoint of each character in s. Invalid UTF-8 raises an error.

Copyfor index, codepoint in utf8.codes("hé!") do
    print(`{index}: {codepoint}`)
end

1: 104
2: 233
4: 33

Returns the codepoint of each character that starts between byte indices i and j. i defaults to 1 and j defaults to i. With one index or none, codepoint returns the codepoint of the character that starts there.

Negative indices count back from the end. An index outside the string or invalid UTF-8 raises an error.

Copyprint(utf8.codepoint("é"))
const first, second = utf8.codepoint("hé", 1, -1)
print(first)
print(second)

233
104
233

Returns the number of characters that start between byte indices i and j. i defaults to 1 and j defaults to -1, the last byte. Negative indices count back from the end.

When len finds invalid UTF-8, it returns nil followed by the index of the first invalid byte.

Copyprint(#"héllo")
print(utf8.len("héllo"))
const count, bad_index = utf8.len("h\xFFi")
print(count)
print(bad_index)

6
5
nil
2

Returns the byte index where the nth character counting from byte index i starts. A positive n counts forward and 1 is the character at i. A negative n counts backward from i. When n is 0, offset returns the start of the character that byte i belongs to.

i defaults to 1 when n is positive and to #s + 1 otherwise. utf8.offset(s, -1) gives the start of the last character. When there's no such character, offset returns nil. A nonzero n with an i in the middle of a character raises an error.

Copyconst word = "héllo"
print(utf8.offset(word, 3))
print(utf8.offset(word, -1))
print(utf8.offset(word, 0, 3))
print(utf8.offset(word, 9))

4
6
2
nil