Encodings and detection

The conversion pipeline

Every file takes the same path:

1. Detect the encoding (skipped when you give the source encoding).
2. Decode to Unicode. Invalid bytes become � and are counted.
3. Entities: with /cu (or the setting), &#NNNN; and &#xHHHH; become text.
4. Script: convert according to the script setting and engine; see Conversion engines.
5. Charset: web-page file types only.
6. Encode into the target encoding, handling characters it lacks.
7. Write to a temporary file and rename it. Nothing is written when the bytes would not change.

Supported encodings

Encoding	Notes
Big5	Traditional Chinese, Taiwan and Hong Kong (Windows code page 950)
GBK	Simplified Chinese (code page 936); it also has Traditional characters
UTF-8	BOM optional
UTF-16 LE / BE	Always written with a BOM
HZ	7-bit GB2312 from early e-mail and Usenet (RFC 1843)
Shift_JIS, EUC-JP, ISO-2022-JP	Japanese

Big5, GBK and the Japanese encodings go through encoding_rs, the library Firefox uses, following the WHATWG Encoding Standard. HZ is implemented in RustZ.

Detection

Detection checks, in order (NUL bytes, ASCII and UTF-8 validity are checked over the whole file; UTF-16 and the scoring use up to 256 KB):

1. BOM: EF BB BF is UTF-8, FF FE is UTF-16 LE, FE FF is UTF-16 BE.

2. UTF-16 without a BOM: mostly-ASCII UTF-16 has a zero in every other byte. If more than 30% of one side is zero and less than 5% of the other, it is UTF-16.

3. Binary: anything else containing NUL bytes is treated as binary and skipped during detection.

4. 7-bit encodings: with no byte above 0x7F, an escape sequence such as ESC $ B means ISO-2022-JP, ~{ … ~} means HZ, and otherwise it is plain ASCII (handled as UTF-8).

5. UTF-8: if the whole sample is valid UTF-8, it is UTF-8.

6. Scoring double-byte encodings: the sample is decoded as Big5, GBK, Shift_JIS and EUC-JP, and RustZ counts how many of the resulting characters are common ones:

   • Big5 and GBK use ConvertZ's own frequency tables — the 144 most common characters and punctuation marks, one table per script.
   • Japanese counts the share of hiragana and katakana.
   • Invalid bytes, private-use characters and (for Shift_JIS) runs of half-width katakana count against an encoding.

   Scoring starts just before the first non-ASCII byte, so a long English header does not push the Chinese text out of the sample. The highest score wins. With fewer than three common characters there is too little to go on: RustZ picks the encoding with the fewest invalid bytes and marks it as a guess (an orange "Big5?" in the app).

Short files

As ConvertZ's own manual warned, Big5 and GBK can be confused when a file has fewer than about a hundred Chinese characters. When a short file shows up as a guess, check the preview and pick the source encoding by hand if needed.

Missing characters

When the target encoding has no code for a character (such as 发 in Big5), RustZ tries, in order:

1. A substitute, from ConvertZ's table of equivalents. Writing Big5, Simplified characters become Traditional (发 → 發) and variant forms from GBK's extension area become forms Big5 has. Writing GBK or HZ, characters become their Simplified or GBK forms. Half-width katakana become full-width.
2. An entity: web-page file types get &#NNNN;, which browsers display as the original character.
3. A question mark: other files get ?.

At the end RustZ reports how many characters needed step 2 or 3.

Charset declarations in web pages

In web-page file types, RustZ rewrites these declarations to the target encoding:

• <meta charset="big5">, <meta http-equiv="Content-Type" content="text/html; charset=big5">, and charset=big5 in code.
• The XML declaration <?xml version="1.0" encoding="big5"?>.

Only values that are recognised encoding names are replaced (a variable assignment like charset = name is left alone). The names written are big5, gbk, utf-8, utf-16, shift_jis, euc-jp, iso-2022-jp and hz-gb-2312.

Entities

中 (decimal) and 中 (hex) both become 中. Values below U+0080 are left alone, so < (<) and friends never turn back into HTML syntax.