Return to pcreunicode.3 CVS log

Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc

Annotation of embedaddon/pcre/doc/pcreunicode.3, revision 1.1.1.1

1.1       misho       1: .TH PCREUNICODE 3
                      2: .SH NAME
                      3: PCRE - Perl-compatible regular expressions
                      4: .SH "UTF-8 AND UNICODE PROPERTY SUPPORT"
                      5: .rs
                      6: .sp
                      7: In order process UTF-8 strings, you must build PCRE to include UTF-8 support in
                      8: the code, and, in addition, you must call
                      9: .\" HREF
                     10: \fBpcre_compile()\fP
                     11: .\"
                     12: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
                     13: (*UTF8). When either of these is the case, both the pattern and any subject
                     14: strings that are matched against it are treated as UTF-8 strings instead of
                     15: strings of 1-byte characters. PCRE does not support any other formats (in
                     16: particular, it does not support UTF-16).
                     17: .P
                     18: If you compile PCRE with UTF-8 support, but do not use it at run time, the
                     19: library will be a bit bigger, but the additional run time overhead is limited
                     20: to testing the PCRE_UTF8 flag occasionally, so should not be very big.
                     21: .P
                     22: If PCRE is built with Unicode character property support (which implies UTF-8
                     23: support), the escape sequences \ep{..}, \eP{..}, and \eX are supported.
                     24: The available properties that can be tested are limited to the general
                     25: category properties such as Lu for an upper case letter or Nd for a decimal
                     26: number, the Unicode script names such as Arabic or Han, and the derived
                     27: properties Any and L&. A full list is given in the
                     28: .\" HREF
                     29: \fBpcrepattern\fP
                     30: .\"
                     31: documentation. Only the short names for properties are supported. For example,
                     32: \ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported.
                     33: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
                     34: compatibility with Perl 5.6. PCRE does not support this.
                     35: .
                     36: .
                     37: .\" HTML <a name="utf8strings"></a>
                     38: .SS "Validity of UTF-8 strings"
                     39: .rs
                     40: .sp
                     41: When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects
                     42: are (by default) checked for validity on entry to the relevant functions. From
                     43: release 7.3 of PCRE, the check is according the rules of RFC 3629, which are
                     44: themselves derived from the Unicode specification. Earlier releases of PCRE
                     45: followed the rules of RFC 2279, which allows the full range of 31-bit values (0
                     46: to 0x7FFFFFFF). The current check allows only values in the range U+0 to
                     47: U+10FFFF, excluding U+D800 to U+DFFF.
                     48: .P
                     49: The excluded code points are the "Low Surrogate Area" of Unicode, of which the
                     50: Unicode Standard says this: "The Low Surrogate Area does not contain any
                     51: character assignments, consequently no character code charts or namelists are
                     52: provided for this area. Surrogates are reserved for use with UTF-16 and then
                     53: must be used in pairs." The code points that are encoded by UTF-16 pairs are
                     54: available as independent code points in the UTF-8 encoding. (In other words,
                     55: the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up
                     56: UTF-8.)
                     57: .P
                     58: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
                     59: compile time, the only additional information is the offset to the first byte
                     60: of the failing character. The runtime functions \fBpcre_exec()\fP and
                     61: \fBpcre_dfa_exec()\fP also pass back this information, as well as a more
                     62: detailed reason code if the caller has provided memory in which to do this.
                     63: .P
                     64: In some situations, you may already know that your strings are valid, and
                     65: therefore want to skip these checks in order to improve performance. If you set
                     66: the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that
                     67: the pattern or subject it is given (respectively) contains only valid UTF-8
                     68: codes. In this case, it does not diagnose an invalid UTF-8 string.
                     69: .P
                     70: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
                     71: happens depends on why the string is invalid. If the string conforms to the
                     72: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
                     73: in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted
                     74: version of \fBpcre_exec()\fP. In other words, apart from the initial validity
                     75: test, these functions (when in UTF-8 mode) handle strings according to the more
                     76: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
                     77: \fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or
                     78: if the string does not even conform to RFC 2279, the result is undefined. Your
                     79: program may crash.
                     80: .P
                     81: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
                     82: encoded in a UTF-8-like manner as per the old RFC, you can set
                     83: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
                     84: situation, you will have to apply your own validity check, and avoid the use of
                     85: JIT optimization.
                     86: .
                     87: .
                     88: .SS "General comments about UTF-8 mode"
                     89: .rs
                     90: .sp
                     91: 1. An unbraced hexadecimal escape sequence (such as \exb3) matches a two-byte
                     92: UTF-8 character if the value is greater than 127.
                     93: .P
                     94: 2. Octal numbers up to \e777 are recognized, and match two-byte UTF-8
                     95: characters for values greater than \e177.
                     96: .P
                     97: 3. Repeat quantifiers apply to complete UTF-8 characters, not to individual
                     98: bytes, for example: \ex{100}{3}.
                     99: .P
                    100: 4. The dot metacharacter matches one UTF-8 character instead of a single byte.
                    101: .P
                    102: 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode,
                    103: but its use can lead to some strange effects because it breaks up multibyte
                    104: characters (see the description of \eC in the
                    105: .\" HREF
                    106: \fBpcrepattern\fP
                    107: .\"
                    108: documentation). The use of \eC is not supported in the alternative matching
                    109: function \fBpcre_dfa_exec()\fP, nor is it supported in UTF-8 mode by the JIT
                    110: optimization of \fBpcre_exec()\fP. If JIT optimization is requested for a UTF-8
                    111: pattern that contains \eC, it will not succeed, and so the matching will be
                    112: carried out by the normal interpretive function.
                    113: .P
                    114: 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
                    115: test characters of any code value, but, by default, the characters that PCRE
                    116: recognizes as digits, spaces, or word characters remain the same set as before,
                    117: all with values less than 256. This remains true even when PCRE is built to
                    118: include Unicode property support, because to do otherwise would slow down PCRE
                    119: in many common cases. Note in particular that this applies to \eb and \eB,
                    120: because they are defined in terms of \ew and \eW. If you really want to test
                    121: for a wider sense of, say, "digit", you can use explicit Unicode property tests
                    122: such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option, the way that
                    123: the character escapes work is changed so that Unicode properties are used to
                    124: determine which characters match. There are more details in the section on
                    125: .\" HTML <a href="pcrepattern.html#genericchartypes">
                    126: .\" </a>
                    127: generic character types
                    128: .\"
                    129: in the
                    130: .\" HREF
                    131: \fBpcrepattern\fP
                    132: .\"
                    133: documentation.
                    134: .P
                    135: 7. Similarly, characters that match the POSIX named character classes are all
                    136: low-valued characters, unless the PCRE_UCP option is set.
                    137: .P
                    138: 8. However, the horizontal and vertical whitespace matching escapes (\eh, \eH,
                    139: \ev, and \eV) do match all the appropriate Unicode characters, whether or not
                    140: PCRE_UCP is set.
                    141: .P
                    142: 9. Case-insensitive matching applies only to characters whose values are less
                    143: than 128, unless PCRE is built with Unicode property support. Even when Unicode
                    144: property support is available, PCRE still uses its own character tables when
                    145: checking the case of low-valued characters, so as not to degrade performance.
                    146: The Unicode property information is used only for characters with higher
                    147: values. Furthermore, PCRE supports case-insensitive matching only when there is
                    148: a one-to-one mapping between a letter's cases. There are a small number of
                    149: many-to-one mappings in Unicode; these are not supported by PCRE.
                    150: .
                    151: .
                    152: .SH AUTHOR
                    153: .rs
                    154: .sp
                    155: .nf
                    156: Philip Hazel
                    157: University Computing Service
                    158: Cambridge CB2 3QH, England.
                    159: .fi
                    160: .
                    161: .
                    162: .SH REVISION
                    163: .rs
                    164: .sp
                    165: .nf
                    166: Last updated: 19 October 2011
                    167: Copyright (c) 1997-2011 University of Cambridge.
                    168: .fi