Annotation of embedaddon/pcre/doc/pcreunicode.3, revision 1.1
1.1 ! misho 1: .TH PCREUNICODE 3
! 2: .SH NAME
! 3: PCRE - Perl-compatible regular expressions
! 4: .SH "UTF-8 AND UNICODE PROPERTY SUPPORT"
! 5: .rs
! 6: .sp
! 7: In order process UTF-8 strings, you must build PCRE to include UTF-8 support in
! 8: the code, and, in addition, you must call
! 9: .\" HREF
! 10: \fBpcre_compile()\fP
! 11: .\"
! 12: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
! 13: (*UTF8). When either of these is the case, both the pattern and any subject
! 14: strings that are matched against it are treated as UTF-8 strings instead of
! 15: strings of 1-byte characters. PCRE does not support any other formats (in
! 16: particular, it does not support UTF-16).
! 17: .P
! 18: If you compile PCRE with UTF-8 support, but do not use it at run time, the
! 19: library will be a bit bigger, but the additional run time overhead is limited
! 20: to testing the PCRE_UTF8 flag occasionally, so should not be very big.
! 21: .P
! 22: If PCRE is built with Unicode character property support (which implies UTF-8
! 23: support), the escape sequences \ep{..}, \eP{..}, and \eX are supported.
! 24: The available properties that can be tested are limited to the general
! 25: category properties such as Lu for an upper case letter or Nd for a decimal
! 26: number, the Unicode script names such as Arabic or Han, and the derived
! 27: properties Any and L&. A full list is given in the
! 28: .\" HREF
! 29: \fBpcrepattern\fP
! 30: .\"
! 31: documentation. Only the short names for properties are supported. For example,
! 32: \ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported.
! 33: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
! 34: compatibility with Perl 5.6. PCRE does not support this.
! 35: .
! 36: .
! 37: .\" HTML <a name="utf8strings"></a>
! 38: .SS "Validity of UTF-8 strings"
! 39: .rs
! 40: .sp
! 41: When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects
! 42: are (by default) checked for validity on entry to the relevant functions. From
! 43: release 7.3 of PCRE, the check is according the rules of RFC 3629, which are
! 44: themselves derived from the Unicode specification. Earlier releases of PCRE
! 45: followed the rules of RFC 2279, which allows the full range of 31-bit values (0
! 46: to 0x7FFFFFFF). The current check allows only values in the range U+0 to
! 47: U+10FFFF, excluding U+D800 to U+DFFF.
! 48: .P
! 49: The excluded code points are the "Low Surrogate Area" of Unicode, of which the
! 50: Unicode Standard says this: "The Low Surrogate Area does not contain any
! 51: character assignments, consequently no character code charts or namelists are
! 52: provided for this area. Surrogates are reserved for use with UTF-16 and then
! 53: must be used in pairs." The code points that are encoded by UTF-16 pairs are
! 54: available as independent code points in the UTF-8 encoding. (In other words,
! 55: the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up
! 56: UTF-8.)
! 57: .P
! 58: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
! 59: compile time, the only additional information is the offset to the first byte
! 60: of the failing character. The runtime functions \fBpcre_exec()\fP and
! 61: \fBpcre_dfa_exec()\fP also pass back this information, as well as a more
! 62: detailed reason code if the caller has provided memory in which to do this.
! 63: .P
! 64: In some situations, you may already know that your strings are valid, and
! 65: therefore want to skip these checks in order to improve performance. If you set
! 66: the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that
! 67: the pattern or subject it is given (respectively) contains only valid UTF-8
! 68: codes. In this case, it does not diagnose an invalid UTF-8 string.
! 69: .P
! 70: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
! 71: happens depends on why the string is invalid. If the string conforms to the
! 72: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
! 73: in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted
! 74: version of \fBpcre_exec()\fP. In other words, apart from the initial validity
! 75: test, these functions (when in UTF-8 mode) handle strings according to the more
! 76: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
! 77: \fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or
! 78: if the string does not even conform to RFC 2279, the result is undefined. Your
! 79: program may crash.
! 80: .P
! 81: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
! 82: encoded in a UTF-8-like manner as per the old RFC, you can set
! 83: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
! 84: situation, you will have to apply your own validity check, and avoid the use of
! 85: JIT optimization.
! 86: .
! 87: .
! 88: .SS "General comments about UTF-8 mode"
! 89: .rs
! 90: .sp
! 91: 1. An unbraced hexadecimal escape sequence (such as \exb3) matches a two-byte
! 92: UTF-8 character if the value is greater than 127.
! 93: .P
! 94: 2. Octal numbers up to \e777 are recognized, and match two-byte UTF-8
! 95: characters for values greater than \e177.
! 96: .P
! 97: 3. Repeat quantifiers apply to complete UTF-8 characters, not to individual
! 98: bytes, for example: \ex{100}{3}.
! 99: .P
! 100: 4. The dot metacharacter matches one UTF-8 character instead of a single byte.
! 101: .P
! 102: 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode,
! 103: but its use can lead to some strange effects because it breaks up multibyte
! 104: characters (see the description of \eC in the
! 105: .\" HREF
! 106: \fBpcrepattern\fP
! 107: .\"
! 108: documentation). The use of \eC is not supported in the alternative matching
! 109: function \fBpcre_dfa_exec()\fP, nor is it supported in UTF-8 mode by the JIT
! 110: optimization of \fBpcre_exec()\fP. If JIT optimization is requested for a UTF-8
! 111: pattern that contains \eC, it will not succeed, and so the matching will be
! 112: carried out by the normal interpretive function.
! 113: .P
! 114: 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
! 115: test characters of any code value, but, by default, the characters that PCRE
! 116: recognizes as digits, spaces, or word characters remain the same set as before,
! 117: all with values less than 256. This remains true even when PCRE is built to
! 118: include Unicode property support, because to do otherwise would slow down PCRE
! 119: in many common cases. Note in particular that this applies to \eb and \eB,
! 120: because they are defined in terms of \ew and \eW. If you really want to test
! 121: for a wider sense of, say, "digit", you can use explicit Unicode property tests
! 122: such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option, the way that
! 123: the character escapes work is changed so that Unicode properties are used to
! 124: determine which characters match. There are more details in the section on
! 125: .\" HTML <a href="pcrepattern.html#genericchartypes">
! 126: .\" </a>
! 127: generic character types
! 128: .\"
! 129: in the
! 130: .\" HREF
! 131: \fBpcrepattern\fP
! 132: .\"
! 133: documentation.
! 134: .P
! 135: 7. Similarly, characters that match the POSIX named character classes are all
! 136: low-valued characters, unless the PCRE_UCP option is set.
! 137: .P
! 138: 8. However, the horizontal and vertical whitespace matching escapes (\eh, \eH,
! 139: \ev, and \eV) do match all the appropriate Unicode characters, whether or not
! 140: PCRE_UCP is set.
! 141: .P
! 142: 9. Case-insensitive matching applies only to characters whose values are less
! 143: than 128, unless PCRE is built with Unicode property support. Even when Unicode
! 144: property support is available, PCRE still uses its own character tables when
! 145: checking the case of low-valued characters, so as not to degrade performance.
! 146: The Unicode property information is used only for characters with higher
! 147: values. Furthermore, PCRE supports case-insensitive matching only when there is
! 148: a one-to-one mapping between a letter's cases. There are a small number of
! 149: many-to-one mappings in Unicode; these are not supported by PCRE.
! 150: .
! 151: .
! 152: .SH AUTHOR
! 153: .rs
! 154: .sp
! 155: .nf
! 156: Philip Hazel
! 157: University Computing Service
! 158: Cambridge CB2 3QH, England.
! 159: .fi
! 160: .
! 161: .
! 162: .SH REVISION
! 163: .rs
! 164: .sp
! 165: .nf
! 166: Last updated: 19 October 2011
! 167: Copyright (c) 1997-2011 University of Cambridge.
! 168: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc