An example of an invalid utf8 string?

Question

I am testing how some of my codes process bad data, and I need several series of bytes that are invalid UTF-8.

Can you post some and, ideally, explanations of why they are bad / where did you get them?

Answer 1

See Markus Kuhn UTF-8 Decoder Capabilities and Stress Test File

You will find examples of many UTF-8 irregularities, including single start bytes, missing continuation bytes, overlap sequences, etc.

Answer 2

In PHP:

$examples = array( 'Valid ASCII' => "a", 'Valid 2 Octet Sequence' => "\xc3\xb1", 'Invalid 2 Octet Sequence' => "\xc3\x28", 'Invalid Sequence Identifier' => "\xa0\xa1", 'Valid 3 Octet Sequence' => "\xe2\x82\xa1", 'Invalid 3 Octet Sequence (in 2nd Octet)' => "\xe2\x28\xa1", 'Invalid 3 Octet Sequence (in 3rd Octet)' => "\xe2\x82\x28", 'Valid 4 Octet Sequence' => "\xf0\x90\x8c\xbc", 'Invalid 4 Octet Sequence (in 2nd Octet)' => "\xf0\x28\x8c\xbc", 'Invalid 4 Octet Sequence (in 3rd Octet)' => "\xf0\x90\x28\xbc", 'Invalid 4 Octet Sequence (in 4th Octet)' => "\xf0\x28\x8c\x28", 'Valid 5 Octet Sequence (but not Unicode!)' => "\xf8\xa1\xa1\xa1\xa1", 'Valid 6 Octet Sequence (but not Unicode!)' => "\xfc\xa1\xa1\xa1\xa1\xa1", ); 

From http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php#54805

Answer 3

The idea of ​​patterns of incorrectly formed byte sequences can be obtained from a table of well-formed byte sequences. See " Table 3-7. Well-formed UTF-8 byte sequences " in Unicode Standard 6.2.

  Code Points First Byte Second Byte Third Byte Fourth Byte U+0000 - U+007F 00 - 7F U+0080 - U+07FF C2 - DF 80 - BF U+0800 - U+0FFF E0 A0 - BF 80 - BF U+1000 - U+CFFF E1 - EC 80 - BF 80 - BF U+D000 - U+D7FF ED 80 - 9F 80 - BF U+E000 - U+FFFF EE - EF 80 - BF 80 - BF U+10000 - U+3FFFF F0 90 - BF 80 - BF 80 - BF U+40000 - U+FFFFF F1 - F3 80 - BF 80 - BF 80 - BF U+100000 - U+10FFFF F4 80 - 8F 80 - BF 80 - BF 

Here are examples created from U + 24B62. I used them to report an error: Error # 65045 mb_convert_encoding crashes a well-formed character

 // U+24B62: "\xF0\xA4\xAD\xA2" "\xF0\xA4\xAD" ."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2" "\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD" 

In different libraries, you can see a simplification of the range of finite bytes ([0x80, 0xBF]).

 // U+0800 - U+0FFF \xE0\x80\x80 // U+D000 - U+D7FF \xED\xBF\xBF // U+10000 - U+3FFFF \xF0\x80\x80\x80 // U+100000 - U+10FFFF \xF4\xBF\xBF\xBF 

Answer 4

̆ was especially evil. I see this as combined on ubuntu.

Breve separators

Answer 5

Fuzz Testing - generate a random sequence of octets. You will most likely get some illegal sequences sooner rather than later.