Annotation of embedaddon/pcre/doc/html/pcreunicode.html, revision 1.1.1.3
1.1 misho 1: <html>
2: <head>
3: <title>pcreunicode specification</title>
4: </head>
5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
6: <h1>pcreunicode man page</h1>
7: <p>
8: Return to the <a href="index.html">PCRE index page</a>.
9: </p>
10: <p>
11: This page is part of the PCRE HTML documentation. It was generated automatically
12: from the original man page. If there is any nonsense in it, please consult the
13: man page, in case the conversion went wrong.
14: <br>
15: <br><b>
1.1.1.2 misho 16: UTF-8, UTF-16, AND UNICODE PROPERTY SUPPORT
1.1 misho 17: </b><br>
18: <P>
1.1.1.2 misho 19: From Release 8.30, in addition to its previous UTF-8 support, PCRE also
20: supports UTF-16 by means of a separate 16-bit library. This can be built as
21: well as, or instead of, the 8-bit library.
22: </P>
23: <br><b>
24: UTF-8 SUPPORT
25: </b><br>
26: <P>
27: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
28: support, and, in addition, you must call
1.1 misho 29: <a href="pcre_compile.html"><b>pcre_compile()</b></a>
30: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
31: (*UTF8). When either of these is the case, both the pattern and any subject
32: strings that are matched against it are treated as UTF-8 strings instead of
1.1.1.2 misho 33: strings of 1-byte characters.
1.1 misho 34: </P>
1.1.1.2 misho 35: <br><b>
36: UTF-16 SUPPORT
37: </b><br>
1.1 misho 38: <P>
1.1.1.2 misho 39: In order process UTF-16 strings, you must build PCRE's 16-bit library with UTF
40: support, and, in addition, you must call
41: <a href="pcre_compile.html"><b>pcre16_compile()</b></a>
42: with the PCRE_UTF16 option flag, or the pattern must start with the sequence
43: (*UTF16). When either of these is the case, both the pattern and any subject
44: strings that are matched against it are treated as UTF-16 strings instead of
45: strings of 16-bit characters.
46: </P>
47: <br><b>
48: UTF SUPPORT OVERHEAD
49: </b><br>
50: <P>
51: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 52: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.2 misho 53: to testing the PCRE_UTF8/16 flag occasionally, so should not be very big.
1.1 misho 54: </P>
1.1.1.2 misho 55: <br><b>
56: UNICODE PROPERTY SUPPORT
57: </b><br>
1.1 misho 58: <P>
1.1.1.2 misho 59: If PCRE is built with Unicode character property support (which implies UTF
60: support), the escape sequences \p{..}, \P{..}, and \X can be used.
1.1 misho 61: The available properties that can be tested are limited to the general
62: category properties such as Lu for an upper case letter or Nd for a decimal
63: number, the Unicode script names such as Arabic or Han, and the derived
64: properties Any and L&. A full list is given in the
65: <a href="pcrepattern.html"><b>pcrepattern</b></a>
66: documentation. Only the short names for properties are supported. For example,
67: \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported.
68: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
69: compatibility with Perl 5.6. PCRE does not support this.
70: <a name="utf8strings"></a></P>
71: <br><b>
72: Validity of UTF-8 strings
73: </b><br>
74: <P>
1.1.1.2 misho 75: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
76: subjects are (by default) checked for validity on entry to the relevant
1.1.1.3 ! misho 77: functions. The entire string is checked before any other processing takes
! 78: place. From release 7.3 of PCRE, the check is according the rules of RFC 3629,
! 79: which are themselves derived from the Unicode specification. Earlier releases
! 80: of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit
! 81: values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0
! 82: to U+10FFFF, excluding U+D800 to U+DFFF.
1.1.1.2 misho 83: </P>
84: <P>
85: The excluded code points are the "Surrogate Area" of Unicode. They are reserved
86: for use by UTF-16, where they are used in pairs to encode codepoints with
87: values greater than 0xFFFF. The code points that are encoded by UTF-16 pairs
88: are available independently in the UTF-8 encoding. (In other words, the whole
89: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
1.1 misho 90: </P>
91: <P>
92: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
93: compile time, the only additional information is the offset to the first byte
1.1.1.3 ! misho 94: of the failing character. The run-time functions <b>pcre_exec()</b> and
1.1 misho 95: <b>pcre_dfa_exec()</b> also pass back this information, as well as a more
96: detailed reason code if the caller has provided memory in which to do this.
97: </P>
98: <P>
99: In some situations, you may already know that your strings are valid, and
1.1.1.3 ! misho 100: therefore want to skip these checks in order to improve performance, for
! 101: example in the case of a long subject string that is being scanned repeatedly
! 102: with different patterns. If you set the PCRE_NO_UTF8_CHECK flag at compile time
! 103: or at run time, PCRE assumes that the pattern or subject it is given
! 104: (respectively) contains only valid UTF-8 codes. In this case, it does not
! 105: diagnose an invalid UTF-8 string.
1.1 misho 106: </P>
107: <P>
108: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
109: happens depends on why the string is invalid. If the string conforms to the
110: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
111: in the range 0 to 0x7FFFFFFF by <b>pcre_dfa_exec()</b> and the interpreted
112: version of <b>pcre_exec()</b>. In other words, apart from the initial validity
113: test, these functions (when in UTF-8 mode) handle strings according to the more
114: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
115: <b>pcre_exec()</b> supports only RFC 3629. If you are using JIT optimization, or
116: if the string does not even conform to RFC 2279, the result is undefined. Your
117: program may crash.
118: </P>
119: <P>
120: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
121: encoded in a UTF-8-like manner as per the old RFC, you can set
122: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
123: situation, you will have to apply your own validity check, and avoid the use of
124: JIT optimization.
1.1.1.2 misho 125: <a name="utf16strings"></a></P>
126: <br><b>
127: Validity of UTF-16 strings
128: </b><br>
129: <P>
130: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
131: passed as patterns and subjects are (by default) checked for validity on entry
132: to the relevant functions. Values other than those in the surrogate range
133: U+D800 to U+DFFF are independent code points. Values in the surrogate range
134: must be used in pairs in the correct manner.
135: </P>
136: <P>
137: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
138: compile time, the only additional information is the offset to the first data
1.1.1.3 ! misho 139: unit of the failing character. The run-time functions <b>pcre16_exec()</b> and
1.1.1.2 misho 140: <b>pcre16_dfa_exec()</b> also pass back this information, as well as a more
141: detailed reason code if the caller has provided memory in which to do this.
142: </P>
143: <P>
144: In some situations, you may already know that your strings are valid, and
145: therefore want to skip these checks in order to improve performance. If you set
146: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
147: the pattern or subject it is given (respectively) contains only valid UTF-16
148: sequences. In this case, it does not diagnose an invalid UTF-16 string.
1.1 misho 149: </P>
150: <br><b>
1.1.1.2 misho 151: General comments about UTF modes
1.1 misho 152: </b><br>
153: <P>
1.1.1.2 misho 154: 1. Codepoints less than 256 can be specified by either braced or unbraced
155: hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger values
156: have to use braced sequences.
1.1 misho 157: </P>
158: <P>
1.1.1.2 misho 159: 2. Octal numbers up to \777 are recognized, and in UTF-8 mode, they match
160: two-byte characters for values greater than \177.
1.1 misho 161: </P>
162: <P>
1.1.1.2 misho 163: 3. Repeat quantifiers apply to complete UTF characters, not to individual
164: data units, for example: \x{100}{3}.
1.1 misho 165: </P>
166: <P>
1.1.1.2 misho 167: 4. The dot metacharacter matches one UTF character instead of a single data
168: unit.
1.1 misho 169: </P>
170: <P>
1.1.1.2 misho 171: 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or
172: a single 16-bit data unit in UTF-16 mode, but its use can lead to some strange
173: effects because it breaks up multi-unit characters (see the description of \C
174: in the
1.1 misho 175: <a href="pcrepattern.html"><b>pcrepattern</b></a>
176: documentation). The use of \C is not supported in the alternative matching
1.1.1.2 misho 177: function <b>pcre[16]_dfa_exec()</b>, nor is it supported in UTF mode by the JIT
178: optimization of <b>pcre[16]_exec()</b>. If JIT optimization is requested for a
179: UTF pattern that contains \C, it will not succeed, and so the matching will
180: be carried out by the normal interpretive function.
1.1 misho 181: </P>
182: <P>
183: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
184: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 misho 185: recognizes as digits, spaces, or word characters remain the same set as in
186: non-UTF mode, all with values less than 256. This remains true even when PCRE
187: is built to include Unicode property support, because to do otherwise would
188: slow down PCRE in many common cases. Note in particular that this applies to
189: \b and \B, because they are defined in terms of \w and \W. If you really
190: want to test for a wider sense of, say, "digit", you can use explicit Unicode
191: property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option,
192: the way that the character escapes work is changed so that Unicode properties
193: are used to determine which characters match. There are more details in the
194: section on
1.1 misho 195: <a href="pcrepattern.html#genericchartypes">generic character types</a>
196: in the
197: <a href="pcrepattern.html"><b>pcrepattern</b></a>
198: documentation.
199: </P>
200: <P>
201: 7. Similarly, characters that match the POSIX named character classes are all
202: low-valued characters, unless the PCRE_UCP option is set.
203: </P>
204: <P>
1.1.1.3 ! misho 205: 8. However, the horizontal and vertical white space matching escapes (\h, \H,
1.1 misho 206: \v, and \V) do match all the appropriate Unicode characters, whether or not
207: PCRE_UCP is set.
208: </P>
209: <P>
210: 9. Case-insensitive matching applies only to characters whose values are less
211: than 128, unless PCRE is built with Unicode property support. Even when Unicode
212: property support is available, PCRE still uses its own character tables when
213: checking the case of low-valued characters, so as not to degrade performance.
214: The Unicode property information is used only for characters with higher
215: values. Furthermore, PCRE supports case-insensitive matching only when there is
216: a one-to-one mapping between a letter's cases. There are a small number of
217: many-to-one mappings in Unicode; these are not supported by PCRE.
218: </P>
219: <br><b>
220: AUTHOR
221: </b><br>
222: <P>
223: Philip Hazel
224: <br>
225: University Computing Service
226: <br>
227: Cambridge CB2 3QH, England.
228: <br>
229: </P>
230: <br><b>
231: REVISION
232: </b><br>
233: <P>
1.1.1.3 ! misho 234: Last updated: 14 April 2012
1.1 misho 235: <br>
1.1.1.2 misho 236: Copyright © 1997-2012 University of Cambridge.
1.1 misho 237: <br>
238: <p>
239: Return to the <a href="index.html">PCRE index page</a>.
240: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html