Annotation of embedaddon/pcre/doc/html/pcreunicode.html, revision 1.1
1.1 ! misho 1: <html>
! 2: <head>
! 3: <title>pcreunicode specification</title>
! 4: </head>
! 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
! 6: <h1>pcreunicode man page</h1>
! 7: <p>
! 8: Return to the <a href="index.html">PCRE index page</a>.
! 9: </p>
! 10: <p>
! 11: This page is part of the PCRE HTML documentation. It was generated automatically
! 12: from the original man page. If there is any nonsense in it, please consult the
! 13: man page, in case the conversion went wrong.
! 14: <br>
! 15: <br><b>
! 16: UTF-8 AND UNICODE PROPERTY SUPPORT
! 17: </b><br>
! 18: <P>
! 19: In order process UTF-8 strings, you must build PCRE to include UTF-8 support in
! 20: the code, and, in addition, you must call
! 21: <a href="pcre_compile.html"><b>pcre_compile()</b></a>
! 22: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
! 23: (*UTF8). When either of these is the case, both the pattern and any subject
! 24: strings that are matched against it are treated as UTF-8 strings instead of
! 25: strings of 1-byte characters. PCRE does not support any other formats (in
! 26: particular, it does not support UTF-16).
! 27: </P>
! 28: <P>
! 29: If you compile PCRE with UTF-8 support, but do not use it at run time, the
! 30: library will be a bit bigger, but the additional run time overhead is limited
! 31: to testing the PCRE_UTF8 flag occasionally, so should not be very big.
! 32: </P>
! 33: <P>
! 34: If PCRE is built with Unicode character property support (which implies UTF-8
! 35: support), the escape sequences \p{..}, \P{..}, and \X are supported.
! 36: The available properties that can be tested are limited to the general
! 37: category properties such as Lu for an upper case letter or Nd for a decimal
! 38: number, the Unicode script names such as Arabic or Han, and the derived
! 39: properties Any and L&. A full list is given in the
! 40: <a href="pcrepattern.html"><b>pcrepattern</b></a>
! 41: documentation. Only the short names for properties are supported. For example,
! 42: \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported.
! 43: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
! 44: compatibility with Perl 5.6. PCRE does not support this.
! 45: <a name="utf8strings"></a></P>
! 46: <br><b>
! 47: Validity of UTF-8 strings
! 48: </b><br>
! 49: <P>
! 50: When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects
! 51: are (by default) checked for validity on entry to the relevant functions. From
! 52: release 7.3 of PCRE, the check is according the rules of RFC 3629, which are
! 53: themselves derived from the Unicode specification. Earlier releases of PCRE
! 54: followed the rules of RFC 2279, which allows the full range of 31-bit values (0
! 55: to 0x7FFFFFFF). The current check allows only values in the range U+0 to
! 56: U+10FFFF, excluding U+D800 to U+DFFF.
! 57: </P>
! 58: <P>
! 59: The excluded code points are the "Low Surrogate Area" of Unicode, of which the
! 60: Unicode Standard says this: "The Low Surrogate Area does not contain any
! 61: character assignments, consequently no character code charts or namelists are
! 62: provided for this area. Surrogates are reserved for use with UTF-16 and then
! 63: must be used in pairs." The code points that are encoded by UTF-16 pairs are
! 64: available as independent code points in the UTF-8 encoding. (In other words,
! 65: the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up
! 66: UTF-8.)
! 67: </P>
! 68: <P>
! 69: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
! 70: compile time, the only additional information is the offset to the first byte
! 71: of the failing character. The runtime functions <b>pcre_exec()</b> and
! 72: <b>pcre_dfa_exec()</b> also pass back this information, as well as a more
! 73: detailed reason code if the caller has provided memory in which to do this.
! 74: </P>
! 75: <P>
! 76: In some situations, you may already know that your strings are valid, and
! 77: therefore want to skip these checks in order to improve performance. If you set
! 78: the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that
! 79: the pattern or subject it is given (respectively) contains only valid UTF-8
! 80: codes. In this case, it does not diagnose an invalid UTF-8 string.
! 81: </P>
! 82: <P>
! 83: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
! 84: happens depends on why the string is invalid. If the string conforms to the
! 85: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
! 86: in the range 0 to 0x7FFFFFFF by <b>pcre_dfa_exec()</b> and the interpreted
! 87: version of <b>pcre_exec()</b>. In other words, apart from the initial validity
! 88: test, these functions (when in UTF-8 mode) handle strings according to the more
! 89: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
! 90: <b>pcre_exec()</b> supports only RFC 3629. If you are using JIT optimization, or
! 91: if the string does not even conform to RFC 2279, the result is undefined. Your
! 92: program may crash.
! 93: </P>
! 94: <P>
! 95: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
! 96: encoded in a UTF-8-like manner as per the old RFC, you can set
! 97: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
! 98: situation, you will have to apply your own validity check, and avoid the use of
! 99: JIT optimization.
! 100: </P>
! 101: <br><b>
! 102: General comments about UTF-8 mode
! 103: </b><br>
! 104: <P>
! 105: 1. An unbraced hexadecimal escape sequence (such as \xb3) matches a two-byte
! 106: UTF-8 character if the value is greater than 127.
! 107: </P>
! 108: <P>
! 109: 2. Octal numbers up to \777 are recognized, and match two-byte UTF-8
! 110: characters for values greater than \177.
! 111: </P>
! 112: <P>
! 113: 3. Repeat quantifiers apply to complete UTF-8 characters, not to individual
! 114: bytes, for example: \x{100}{3}.
! 115: </P>
! 116: <P>
! 117: 4. The dot metacharacter matches one UTF-8 character instead of a single byte.
! 118: </P>
! 119: <P>
! 120: 5. The escape sequence \C can be used to match a single byte in UTF-8 mode,
! 121: but its use can lead to some strange effects because it breaks up multibyte
! 122: characters (see the description of \C in the
! 123: <a href="pcrepattern.html"><b>pcrepattern</b></a>
! 124: documentation). The use of \C is not supported in the alternative matching
! 125: function <b>pcre_dfa_exec()</b>, nor is it supported in UTF-8 mode by the JIT
! 126: optimization of <b>pcre_exec()</b>. If JIT optimization is requested for a UTF-8
! 127: pattern that contains \C, it will not succeed, and so the matching will be
! 128: carried out by the normal interpretive function.
! 129: </P>
! 130: <P>
! 131: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
! 132: test characters of any code value, but, by default, the characters that PCRE
! 133: recognizes as digits, spaces, or word characters remain the same set as before,
! 134: all with values less than 256. This remains true even when PCRE is built to
! 135: include Unicode property support, because to do otherwise would slow down PCRE
! 136: in many common cases. Note in particular that this applies to \b and \B,
! 137: because they are defined in terms of \w and \W. If you really want to test
! 138: for a wider sense of, say, "digit", you can use explicit Unicode property tests
! 139: such as \p{Nd}. Alternatively, if you set the PCRE_UCP option, the way that
! 140: the character escapes work is changed so that Unicode properties are used to
! 141: determine which characters match. There are more details in the section on
! 142: <a href="pcrepattern.html#genericchartypes">generic character types</a>
! 143: in the
! 144: <a href="pcrepattern.html"><b>pcrepattern</b></a>
! 145: documentation.
! 146: </P>
! 147: <P>
! 148: 7. Similarly, characters that match the POSIX named character classes are all
! 149: low-valued characters, unless the PCRE_UCP option is set.
! 150: </P>
! 151: <P>
! 152: 8. However, the horizontal and vertical whitespace matching escapes (\h, \H,
! 153: \v, and \V) do match all the appropriate Unicode characters, whether or not
! 154: PCRE_UCP is set.
! 155: </P>
! 156: <P>
! 157: 9. Case-insensitive matching applies only to characters whose values are less
! 158: than 128, unless PCRE is built with Unicode property support. Even when Unicode
! 159: property support is available, PCRE still uses its own character tables when
! 160: checking the case of low-valued characters, so as not to degrade performance.
! 161: The Unicode property information is used only for characters with higher
! 162: values. Furthermore, PCRE supports case-insensitive matching only when there is
! 163: a one-to-one mapping between a letter's cases. There are a small number of
! 164: many-to-one mappings in Unicode; these are not supported by PCRE.
! 165: </P>
! 166: <br><b>
! 167: AUTHOR
! 168: </b><br>
! 169: <P>
! 170: Philip Hazel
! 171: <br>
! 172: University Computing Service
! 173: <br>
! 174: Cambridge CB2 3QH, England.
! 175: <br>
! 176: </P>
! 177: <br><b>
! 178: REVISION
! 179: </b><br>
! 180: <P>
! 181: Last updated: 19 October 2011
! 182: <br>
! 183: Copyright © 1997-2011 University of Cambridge.
! 184: <br>
! 185: <p>
! 186: Return to the <a href="index.html">PCRE index page</a>.
! 187: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html