DRIVER=serial !l=JJj"T=p,F 0%lze67}LZ7 qs?/* utfebcdic.h * * Copyright (C) 2001, 2002, 2003, 2005, 2006, 2007, 2009, * 2010, 2011 by Larry Wall, Nick Ing-Simmons, and others * * You may distribute under the terms of either the GNU General Public * License or the Artistic License, as specified in the README file. * * Macros to implement UTF-EBCDIC as perl's internal encoding * Adapted from version 7.1 of Unicode Technical Report #16: * http://www.unicode.org/unicode/reports/tr16 * * To summarize, the way it works is: * To convert an EBCDIC code point to UTF-EBCDIC: * 1) convert to Unicode. No conversion is necesary for code points above * 255, as Unicode and EBCDIC are identical in this range. For smaller * code points, the conversion is done by lookup in the PL_e2a table (with * inverse PL_a2e) in the generated file 'ebcdic_tables.h'. The 'a' * stands for ASCII platform, meaning 0-255 Unicode. * 2) convert that to a utf8-like string called I8 ('I' stands for * intermediate) with variant characters occupying multiple bytes. This * step is similar to the utf8-creating step from Unicode, but the details * are different. This transformation is called UTF8-Mod. There is a * chart about the bit patterns in a comment later in this file. But * essentially here are the differences: * UTF8 I8 * invariant byte starts with 0 starts with 0 or 100 * continuation byte starts with 10 starts with 101 * start byte same in both: if the code point requires N bytes, * then the leading N bits are 1, followed by a 0. If * all 8 bits in the first byte are 1, the code point * will occupy 14 bytes (compared to 13 in Perl's * extended UTF-8). This is incompatible with what * tr16 implies should be the representation of code * points 2**30 and above, but allows Perl to be able * to represent all code points that fit in a 64-bit * word in either our extended UTF-EBCDIC or UTF-8. * 3) Use the algorithm in tr16 to convert each byte from step 2 into * final UTF-EBCDIC. This is done by table lookup from a table * constructed from the algorithm, reproduced in ebcdic_tables.h as * PL_utf2e, with its inverse being PL_e2utf. They are constructed so that * all EBCDIC invariants remain invariant, but no others do, and the first * byte of a variant will always have its upper bit set. But note that * the upper bit of some invariants is also 1. The table also is designed * so that lexically comparing two UTF-EBCDIC-variant characters yields * the Unicode code point order. (To get native code point order, one has * to convert the latin1-range characters to their native code point * value.) * * For example, the ordinal value of 'A' is 193 in EBCDIC, and also is 193 in * UTF-EBCDIC. Step 1) converts it to 65, Step 2 leaves it at 65, and Step 3 * converts it back to 193. As an example of how a variant character works, * take LATIN SMALL LETTER Y WITH DIAERESIS, which is typically 0xDF in * EBCDIC. Step 1 converts it to the Unicode value, 0xFF. Step 2 converts * that to two bytes = 11000111 10111111 = C7 BF, and Step 3 converts those to * 0x8B 0x73. * * If you're starting from Unicode, skip step 1. For UTF-EBCDIC to straight * EBCDIC, reverse the steps. * * The EBCDIC invariants have been chosen to be those characters whose Unicode * equivalents have ordinal numbers less than 160, that is the same characters * that are expressible in ASCII, plus the C1 controls. So there are 160 * invariants instead of the 128 in UTF-8. * * The purpose of Step 3 is to make the encoding be invariant for the chosen * characters. This messes up the convenient patterns found in step 2, so * generally, one has to undo step 3 into a temporary to use them. However, * one "shadow", or parallel table, PL_utf8skip, has been constructed that * doesn't require undoing things. It is such that for each byte, it says * how long th