@bold(About UTF-8 string routines)

Some parts of the Allegro API, such as the font routines, expect Unicode strings encoded in UTF-8. The UTF8 basic routines are provided to help you work with UTF-8 strings, however it does @italic(not) mean you need to use them.

Briefly, Unicode is a standard consisting of a large character set of over 100,000 characters, and rules, such as how to sort strings. A code point is the integer value of a character, but not all code points are characters, as some code points have other uses. Unlike legacy character sets, the set of code points is open ended and more are assigned with time.

Clearly it is impossible to represent each code point with a 8-bit byte (limited to 256 code points) or even a 16-bit integer (limited to 65536 code points). It is possible to store code points in a 32-bit integers but it is space inefficient, and not actually that useful (at least, when handling the full complexity of Unicode; Allegro only does the very basics). There exist different Unicode Transformation Formats for encoding code points into smaller code units. The most important transformation formats are UTF-8 and UTF-16.

UTF-8 is a @italic(variable-length) encoding which encodes each code point to between one and four 8-bit bytes each. UTF-8 has many nice properties, but the main advantages are that it is backwards compatible with C strings, and ASCII characters (code points in the range 0-127) are encoded in UTF-8 exactly as they would be in ASCII.

UTF-16 is another variable-length encoding, but encodes each code point to one or two 16-bit words each. It is, of course, not compatible with traditional C strings. Allegro does not generally use UTF-16 strings.

Here is a diagram of the representation of the word "ål", with a NUL terminator, in both UTF-8 and UTF-16.

@table(
  @rowhead(@cell(String)     @cell(å)              @cell(l)              @cell(NUL))
  @row(@cell(Code points)    @cell(U+00E5 @(229@)) @cell(U+006C @(108@)) @cell(U+0000 @(0@)))
  @row(@cell(UTF-8 bytes)    @cell(0xC3, 0xA5)     @cell(0x6C)           @cell(0x00))
  @row(@cell(UTF-16LE bytes) @cell(0xE5, 0x00)     @cell(0x6C, 0x00)     @cell(0x00, 0x00))
)

You can see the aforementioned properties of UTF-8. The first code point U+00E5 ("å") is outside of the ASCII range (0-127) so is encoded to multiple code units -- it requires two bytes. U+006C ("l") and U+0000 (NUL) both exist in the ASCII range so take exactly one byte each, as in a pure ASCII string. A zero byte never appears except to represent the NUL character, so many functions which expect C-style strings will work with UTF-8 strings without modification.

On the other hand, UTF-16 represents each code point by either one or two 16-bit code units (two or four bytes). The representation of each 16-bit code unit depends on the byte order; here we have demonstrated little endian.

Both UTF-8 and UTF-16 are self-synchronising. Starting from any offset within a string, it is efficient to find the beginning of the previous or next code point.

Not all sequences of bytes or 16-bit words are valid UTF-8 and UTF-16 strings respectively. UTF-8 also has an additional problem of overlong forms, where a code point value is encoded using more bytes than is strictly necessary. This is invalid and needs to be guarded against.

In the "ustr" functions, be careful whether a function takes code unit (byte) or code point indices. In general, all position parameters are in code unit offsets. This may be surprising, but if you think about it, it is required for good performance. (It also means some functions will work even if they do not contain UTF-8, since they only care about storing bytes, so you may actually store arbitrary data in the ALLEGRO_USTRs.)

For actual text processing, where you want to specify positions with code point indices, you should use @link(al_ustr_offset) to find the code unit offset position. However, most of the time you would probably just work with byte offsets.
