Annotation of embedaddon/pcre/doc/html/pcreunicode.html, revision 1.1.1.2
1.1 misho 1: <html>
2: <head>
3: <title>pcreunicode specification</title>
4: </head>
5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
6: <h1>pcreunicode man page</h1>
7: <p>
8: Return to the <a href="index.html">PCRE index page</a>.
9: </p>
10: <p>
11: This page is part of the PCRE HTML documentation. It was generated automatically
12: from the original man page. If there is any nonsense in it, please consult the
13: man page, in case the conversion went wrong.
14: <br>
15: <br><b>
1.1.1.2 ! misho 16: UTF-8, UTF-16, AND UNICODE PROPERTY SUPPORT
1.1 misho 17: </b><br>
18: <P>
1.1.1.2 ! misho 19: From Release 8.30, in addition to its previous UTF-8 support, PCRE also
! 20: supports UTF-16 by means of a separate 16-bit library. This can be built as
! 21: well as, or instead of, the 8-bit library.
! 22: </P>
! 23: <br><b>
! 24: UTF-8 SUPPORT
! 25: </b><br>
! 26: <P>
! 27: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
! 28: support, and, in addition, you must call
1.1 misho 29: <a href="pcre_compile.html"><b>pcre_compile()</b></a>
30: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
31: (*UTF8). When either of these is the case, both the pattern and any subject
32: strings that are matched against it are treated as UTF-8 strings instead of
1.1.1.2 ! misho 33: strings of 1-byte characters.
1.1 misho 34: </P>
1.1.1.2 ! misho 35: <br><b>
! 36: UTF-16 SUPPORT
! 37: </b><br>
1.1 misho 38: <P>
1.1.1.2 ! misho 39: In order process UTF-16 strings, you must build PCRE's 16-bit library with UTF
! 40: support, and, in addition, you must call
! 41: <a href="pcre_compile.html"><b>pcre16_compile()</b></a>
! 42: with the PCRE_UTF16 option flag, or the pattern must start with the sequence
! 43: (*UTF16). When either of these is the case, both the pattern and any subject
! 44: strings that are matched against it are treated as UTF-16 strings instead of
! 45: strings of 16-bit characters.
! 46: </P>
! 47: <br><b>
! 48: UTF SUPPORT OVERHEAD
! 49: </b><br>
! 50: <P>
! 51: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 52: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.2 ! misho 53: to testing the PCRE_UTF8/16 flag occasionally, so should not be very big.
1.1 misho 54: </P>
1.1.1.2 ! misho 55: <br><b>
! 56: UNICODE PROPERTY SUPPORT
! 57: </b><br>
1.1 misho 58: <P>
1.1.1.2 ! misho 59: If PCRE is built with Unicode character property support (which implies UTF
! 60: support), the escape sequences \p{..}, \P{..}, and \X can be used.
1.1 misho 61: The available properties that can be tested are limited to the general
62: category properties such as Lu for an upper case letter or Nd for a decimal
63: number, the Unicode script names such as Arabic or Han, and the derived
64: properties Any and L&. A full list is given in the
65: <a href="pcrepattern.html"><b>pcrepattern</b></a>
66: documentation. Only the short names for properties are supported. For example,
67: \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported.
68: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
69: compatibility with Perl 5.6. PCRE does not support this.
70: <a name="utf8strings"></a></P>
71: <br><b>
72: Validity of UTF-8 strings
73: </b><br>
74: <P>
1.1.1.2 ! misho 75: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
! 76: subjects are (by default) checked for validity on entry to the relevant
! 77: functions. From release 7.3 of PCRE, the check is according the rules of RFC
! 78: 3629, which are themselves derived from the Unicode specification. Earlier
! 79: releases of PCRE followed the rules of RFC 2279, which allows the full range of
! 80: 31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the
! 81: range U+0 to U+10FFFF, excluding U+D800 to U+DFFF.
! 82: </P>
! 83: <P>
! 84: The excluded code points are the "Surrogate Area" of Unicode. They are reserved
! 85: for use by UTF-16, where they are used in pairs to encode codepoints with
! 86: values greater than 0xFFFF. The code points that are encoded by UTF-16 pairs
! 87: are available independently in the UTF-8 encoding. (In other words, the whole
! 88: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
1.1 misho 89: </P>
90: <P>
91: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
92: compile time, the only additional information is the offset to the first byte
93: of the failing character. The runtime functions <b>pcre_exec()</b> and
94: <b>pcre_dfa_exec()</b> also pass back this information, as well as a more
95: detailed reason code if the caller has provided memory in which to do this.
96: </P>
97: <P>
98: In some situations, you may already know that your strings are valid, and
99: therefore want to skip these checks in order to improve performance. If you set
100: the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that
101: the pattern or subject it is given (respectively) contains only valid UTF-8
102: codes. In this case, it does not diagnose an invalid UTF-8 string.
103: </P>
104: <P>
105: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
106: happens depends on why the string is invalid. If the string conforms to the
107: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
108: in the range 0 to 0x7FFFFFFF by <b>pcre_dfa_exec()</b> and the interpreted
109: version of <b>pcre_exec()</b>. In other words, apart from the initial validity
110: test, these functions (when in UTF-8 mode) handle strings according to the more
111: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
112: <b>pcre_exec()</b> supports only RFC 3629. If you are using JIT optimization, or
113: if the string does not even conform to RFC 2279, the result is undefined. Your
114: program may crash.
115: </P>
116: <P>
117: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
118: encoded in a UTF-8-like manner as per the old RFC, you can set
119: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
120: situation, you will have to apply your own validity check, and avoid the use of
121: JIT optimization.
1.1.1.2 ! misho 122: <a name="utf16strings"></a></P>
! 123: <br><b>
! 124: Validity of UTF-16 strings
! 125: </b><br>
! 126: <P>
! 127: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
! 128: passed as patterns and subjects are (by default) checked for validity on entry
! 129: to the relevant functions. Values other than those in the surrogate range
! 130: U+D800 to U+DFFF are independent code points. Values in the surrogate range
! 131: must be used in pairs in the correct manner.
! 132: </P>
! 133: <P>
! 134: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
! 135: compile time, the only additional information is the offset to the first data
! 136: unit of the failing character. The runtime functions <b>pcre16_exec()</b> and
! 137: <b>pcre16_dfa_exec()</b> also pass back this information, as well as a more
! 138: detailed reason code if the caller has provided memory in which to do this.
! 139: </P>
! 140: <P>
! 141: In some situations, you may already know that your strings are valid, and
! 142: therefore want to skip these checks in order to improve performance. If you set
! 143: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
! 144: the pattern or subject it is given (respectively) contains only valid UTF-16
! 145: sequences. In this case, it does not diagnose an invalid UTF-16 string.
1.1 misho 146: </P>
147: <br><b>
1.1.1.2 ! misho 148: General comments about UTF modes
1.1 misho 149: </b><br>
150: <P>
1.1.1.2 ! misho 151: 1. Codepoints less than 256 can be specified by either braced or unbraced
! 152: hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger values
! 153: have to use braced sequences.
1.1 misho 154: </P>
155: <P>
1.1.1.2 ! misho 156: 2. Octal numbers up to \777 are recognized, and in UTF-8 mode, they match
! 157: two-byte characters for values greater than \177.
1.1 misho 158: </P>
159: <P>
1.1.1.2 ! misho 160: 3. Repeat quantifiers apply to complete UTF characters, not to individual
! 161: data units, for example: \x{100}{3}.
1.1 misho 162: </P>
163: <P>
1.1.1.2 ! misho 164: 4. The dot metacharacter matches one UTF character instead of a single data
! 165: unit.
1.1 misho 166: </P>
167: <P>
1.1.1.2 ! misho 168: 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or
! 169: a single 16-bit data unit in UTF-16 mode, but its use can lead to some strange
! 170: effects because it breaks up multi-unit characters (see the description of \C
! 171: in the
1.1 misho 172: <a href="pcrepattern.html"><b>pcrepattern</b></a>
173: documentation). The use of \C is not supported in the alternative matching
1.1.1.2 ! misho 174: function <b>pcre[16]_dfa_exec()</b>, nor is it supported in UTF mode by the JIT
! 175: optimization of <b>pcre[16]_exec()</b>. If JIT optimization is requested for a
! 176: UTF pattern that contains \C, it will not succeed, and so the matching will
! 177: be carried out by the normal interpretive function.
1.1 misho 178: </P>
179: <P>
180: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
181: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 ! misho 182: recognizes as digits, spaces, or word characters remain the same set as in
! 183: non-UTF mode, all with values less than 256. This remains true even when PCRE
! 184: is built to include Unicode property support, because to do otherwise would
! 185: slow down PCRE in many common cases. Note in particular that this applies to
! 186: \b and \B, because they are defined in terms of \w and \W. If you really
! 187: want to test for a wider sense of, say, "digit", you can use explicit Unicode
! 188: property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option,
! 189: the way that the character escapes work is changed so that Unicode properties
! 190: are used to determine which characters match. There are more details in the
! 191: section on
1.1 misho 192: <a href="pcrepattern.html#genericchartypes">generic character types</a>
193: in the
194: <a href="pcrepattern.html"><b>pcrepattern</b></a>
195: documentation.
196: </P>
197: <P>
198: 7. Similarly, characters that match the POSIX named character classes are all
199: low-valued characters, unless the PCRE_UCP option is set.
200: </P>
201: <P>
202: 8. However, the horizontal and vertical whitespace matching escapes (\h, \H,
203: \v, and \V) do match all the appropriate Unicode characters, whether or not
204: PCRE_UCP is set.
205: </P>
206: <P>
207: 9. Case-insensitive matching applies only to characters whose values are less
208: than 128, unless PCRE is built with Unicode property support. Even when Unicode
209: property support is available, PCRE still uses its own character tables when
210: checking the case of low-valued characters, so as not to degrade performance.
211: The Unicode property information is used only for characters with higher
212: values. Furthermore, PCRE supports case-insensitive matching only when there is
213: a one-to-one mapping between a letter's cases. There are a small number of
214: many-to-one mappings in Unicode; these are not supported by PCRE.
215: </P>
216: <br><b>
217: AUTHOR
218: </b><br>
219: <P>
220: Philip Hazel
221: <br>
222: University Computing Service
223: <br>
224: Cambridge CB2 3QH, England.
225: <br>
226: </P>
227: <br><b>
228: REVISION
229: </b><br>
230: <P>
1.1.1.2 ! misho 231: Last updated: 13 January 2012
1.1 misho 232: <br>
1.1.1.2 ! misho 233: Copyright © 1997-2012 University of Cambridge.
1.1 misho 234: <br>
235: <p>
236: Return to the <a href="index.html">PCRE index page</a>.
237: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html