root / json-framework-3.2.0 / Tests / Data / invalid / encoding / utf8 / input @ 3ebe9884
History | View | Annotate | Download (20.3 kB)
1 |
["UTF-8 decoder capability and stress test\n----------------------------------------\n\nMarkus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19\n\nThis test file can help you examine, how your UTF-8 decoder handles\nvarious types of correct, malformed, or otherwise interesting UTF-8\nsequences. This file is not meant to be a conformance test. It does\nnot prescribes any particular outcome and therefore there is no way to\n\"pass\" or \"fail\" this test file, even though the texts suggests a\npreferable decoder behaviour at some places. The aim is instead to\nhelp you think about and test the behaviour of your UTF-8 on a\nsystematic collection of unusual inputs. Experience so far suggests\nthat most first-time authors of UTF-8 decoders find at least one\nserious problem in their decoder by using this file.\n\nThe test lines below cover boundary conditions, malformed UTF-8\nsequences as well as correctly encoded UTF-8 sequences of Unicode code\npoints that should never occur in a correct UTF-8 file.\n\nAccording to ISO 10646-1:2000, sections D.7 and 2.3c, a device\nreceiving UTF-8 shall interpret a \"malformed sequence in the same way\nthat it interprets a character that is outside the adopted subset\" and\n\"characters that are not within the adopted subset shall be indicated\nto the user\" by a receiving device. A quite commonly used approach in\nUTF-8 decoders is to replace any malformed UTF-8 sequence by a\nreplacement character (U+FFFD), which looks a bit like an inverted\nquestion mark, or a similar symbol. It might be a good idea to\nvisually distinguish a malformed UTF-8 sequence from a correctly\nencoded Unicode character that is just not available in the current\nfont but otherwise fully legal, even though ISO 10646-1 doesn't\nmandate this. In any case, just ignoring malformed sequences or\nunavailable characters does not conform to ISO 10646, will make\ndebugging more difficult, and can lead to user confusion.\n\nPlease check, whether a malformed UTF-8 sequence is (1) represented at\nall, (2) represented by exactly one single replacement character (or\nequivalent signal), and (3) the following quotation mark after an\nillegal UTF-8 sequence is correctly displayed, i.e. proper\nresynchronization takes place immageately after any malformed\nsequence. This file says \"THE END\" in the last line, so if you don't\nsee that, your decoder crashed somehow before, which should always be\ncause for concern.\n\nAll lines in this file are exactly 79 characters long (plus the line\nfeed). In addition, all lines end with \"|\", except for the two test\nlines 2.1.1 and 2.2.1, which contain non-printable ASCII controls\nU+0000 and U+007F. If you display this file with a fixed-width font,\nthese \"|\" characters should all line up in column 79 (right margin).\nThis allows you to test quickly, whether your UTF-8 decoder finds the\ncorrect number of characters in every line, that is whether each\nmalformed sequences is replaced by a single replacement character.\n\nNote that as an alternative to the notion of malformed sequence used\nNote that as an alternative to the notion of malformed sequence used\npreferable) solution to represent each individual byte of a malformed\nsequence by a replacement character. If you follow this strategy in\nyour decoder, then please ignore the \"|\" column.\n\n\nHere come the tests: |\n |\n1 Some correct UTF-8 text |\n1 Some correct UTF-8 text |\nYou should see the Greek word 'kosme': \"?????\" |\n |\n2 Boundary condition test cases |\n |\n2.1 First possible sequence of a certain length |\n |\n2.1.1 1 byte (U-00000000): \"NUL\" \n2.1.2 2 bytes (U-00000080): \"?\" |\n2.1.3 3 bytes (U-00000800): \"?\" |\n2.1.4 4 bytes (U-00010000): \"?\" |\n2.1.5 5 bytes (U-00200000): \"?????\" |\n2.1.6 6 bytes (U-04000000): \"??????\" |\n |\n2.2 Last possible sequence of a certain length |\n |\n2.2.1 1 byte (U-0000007F): \"\" \n2.2.2 2 bytes (U-000007FF): \"?\" |\n2.2.3 3 bytes (U-0000FFFF): \"?\" |\n2.2.4 4 bytes (U-001FFFFF): \"????\" |\n2.2.5 5 bytes (U-03FFFFFF): \"?????\" |\n2.2.6 6 bytes (U-7FFFFFFF): \"??????\" |\n |\n2.3 Other boundary conditions |\n |\n2.3.1 U-0000D7FF = ed 9f bf = \"?\" |\n2.3.2 U-0000E000 = ee 80 80 = \"?\" |\n2.3.3 U-0000FFFD = ef bf bd = \"?\" |\n2.3.4 U-0010FFFF = f4 8f bf bf = \"?\" |\n2.3.5 U-00110000 = f4 90 80 80 = \"????\" |\n |\n3 Malformed sequences |\n |\n3.1 Unexpected continuation bytes |\n |\nEach unexpected continuation byte should be separately signalled as a |\nmalformed sequence of its own. |\n |\n3.1.1 First continuation byte 0x80: \"?\" |\n3.1.2 Last continuation byte 0xbf: \"?\" |\n |\n3.1.3 2 continuation bytes: \"??\" |\n3.1.4 3 continuation bytes: \"???\" |\n3.1.5 4 continuation bytes: \"????\" |\n3.1.6 5 continuation bytes: \"?????\" |\n3.1.7 6 continuation bytes: \"??????\" |\n3.1.8 7 continuation bytes: \"???????\" |\n |\n3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |\n |\n \"???????????????? |\n ???????????????? |\n ???????????????? |\n ????????????????\" |\n |\n3.2 Lonely start characters |\n |\n3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |\n each followed by a space character: |\n |\n \"? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? |\n ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \" |\n |\n3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |\n each followed by a space character: |\n |\n \"? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? \" |\n |\n3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |\n each followed by a space character: |\n |\n \"? ? ? ? ? ? ? ? \" |\n |\n3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |\n each followed by a space character: |\n |\n \"? ? ? ? \" |\n |\n3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |\n each followed by a space character: |\n |\n \"? ? \" |\n |\n3.3 Sequences with last continuation byte missing |\n |\nAll bytes of an incomplete sequence should be signalled as a single |\nmalformed sequence, i.e., you should see only a single replacement |\ncharacter in each of the next 10 tests. (Characters as in section 2) |\n |\n3.3.1 2-byte sequence with last byte missing (U+0000): \"?\" |\n3.3.2 3-byte sequence with last byte missing (U+0000): \"??\" |\n3.3.3 4-byte sequence with last byte missing (U+0000): \"???\" |\n3.3.4 5-byte sequence with last byte missing (U+0000): \"????\" |\n3.3.5 6-byte sequence with last byte missing (U+0000): \"?????\" |\n3.3.6 2-byte sequence with last byte missing (U-000007FF): \"?\" |\n3.3.7 3-byte sequence with last byte missing (U-0000FFFF): \"?\" |\n3.3.8 4-byte sequence with last byte missing (U-001FFFFF): \"???\" |\n3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): \"????\" |\n3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): \"?????\" |\n |\n3.4 Concatenation of incomplete sequences |\n |\nAll the 10 sequences of 3.3 concatenated, you should see 10 malformed |\nsequences being signalled: |\n |\n \"?????????????????????????????\" |\n |\n3.5 Impossible bytes |\n |\nThe following two bytes cannot appear in a correct UTF-8 string |\n |\n3.5.1 fe = \"?\" |\n3.5.2 ff = \"?\" |\n3.5.3 fe fe ff ff = \"????\" |\n |\n4 Overlong sequences |\n |\nThe following sequences are not malformed according to the letter of |\nthe Unicode 2.0 standard. However, they are longer then necessary and |\na correct UTF-8 encoder is not allowed to produce them. A \"safe UTF-8 |\ndecoder\" should reject them just like malformed sequences for two |\nreasons: (1) It helps to debug applications if overlong sequences are |\nnot treated as valid representations of characters, because this helps |\nto spot problems more quickly. (2) Overlong sequences provide |\nalternative representations of characters, that could maliciously be |\nused to bypass filters that check only for ASCII characters. For |\ninstance, a 2-byte encoded line feed (LF) would not be caught by a |\nline counter that counts only 0x0a bytes, but it would still be |\nprocessed as a line feed by an unsafe UTF-8 decoder later in the |\npipeline. From a security point of view, ASCII compatibility of UTF-8 |\nsequences means also, that ASCII characters are *only* allowed to be |\nrepresented by ASCII bytes in the range 0x00-0x7f. To ensure this |\naspect of ASCII compatibility, use only \"safe UTF-8 decoders\" that |\nreject overlong UTF-8 sequences for which a shorter encoding exists. |\n |\n4.1 Examples of an overlong ASCII character |\n |\nWith a safe UTF-8 decoder, all of the following five overlong |\nrepresentations of the ASCII character slash (\"/\") should be rejected |\nlike a malformed UTF-8 sequence, for instance by substituting it with |\na replacement character. If you see a slash below, you do not have a |\nsafe UTF-8 decoder! |\n |\n4.1.1 U+002F = c0 af = \"??\" |\n4.1.2 U+002F = e0 80 af = \"???\" |\n4.1.3 U+002F = f0 80 80 af = \"????\" |\n4.1.4 U+002F = f8 80 80 80 af = \"?????\" |\n4.1.5 U+002F = fc 80 80 80 80 af = \"??????\" |\n |\n4.2 Maximum overlong sequences |\n |\nBelow you see the highest Unicode value that is still resulting in an |\noverlong sequence if represented with the given number of bytes. This |\nis a boundary test for safe UTF-8 decoders. All five characters should |\nbe rejected like malformed UTF-8 sequences. |\n |\n4.2.1 U-0000007F = c1 bf = \"??\" |\n4.2.2 U-000007FF = e0 9f bf = \"???\" |\n4.2.3 U-0000FFFF = f0 8f bf bf = \"????\" |\n4.2.4 U-001FFFFF = f8 87 bf bf bf = \"?????\" |\n4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = \"??????\" |\n |\n4.3 Overlong representation of the NUL character |\n |\nThe following five sequences should also be rejected like malformed |\nUTF-8 sequences and should not be treated like the ASCII NUL |\ncharacter. |\n |\n4.3.1 U+0000 = c0 80 = \"??\" |\n4.3.2 U+0000 = e0 80 80 = \"???\" |\n4.3.3 U+0000 = f0 80 80 80 = \"????\" |\n4.3.4 U+0000 = f8 80 80 80 80 = \"?????\" |\n4.3.5 U+0000 = fc 80 80 80 80 80 = \"??????\" |\n |\n5 Illegal code positions |\n |\nThe following UTF-8 sequences should be rejected like malformed |\nsequences, because they never represent valid ISO 10646 characters and |\na UTF-8 decoder that accepts them might introduce security problems |\ncomparable to overlong UTF-8 sequences. |\n |\n5.1 Single UTF-16 surrogates |\n |\n5.1.1 U+D800 = ed a0 80 = \"???\" |\n5.1.2 U+DB7F = ed ad bf = \"???\" |\n5.1.3 U+DB80 = ed ae 80 = \"???\" |\n5.1.4 U+DBFF = ed af bf = \"???\" |\n5.1.5 U+DC00 = ed b0 80 = \"???\" |\n5.1.6 U+DF80 = ed be 80 = \"???\" |\n5.1.7 U+DFFF = ed bf bf = \"???\" |\n |\n5.2 Paired UTF-16 surrogates |\n |\n5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = \"??????\" |\n5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = \"??????\" |\n5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = \"??????\" |\n5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = \"??????\" |\n5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = \"??????\" |\n5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = \"??????\" |\n5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = \"??????\" |\n5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = \"??????\" |\n |\n5.3 Other illegal code positions |\n |\n5.3.1 U+FFFE = ef bf be = \"?\" |\n5.3.2 U+FFFF = ef bf bf = \"?\" |\n |\nTHE END |\n"] |
---|