Annotation of embedaddon/pcre/doc/html/pcreunicode.html, revision 1.1.1.4
1.1 misho 1: <html>
2: <head>
3: <title>pcreunicode specification</title>
4: </head>
5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
6: <h1>pcreunicode man page</h1>
7: <p>
8: Return to the <a href="index.html">PCRE index page</a>.
9: </p>
10: <p>
11: This page is part of the PCRE HTML documentation. It was generated automatically
12: from the original man page. If there is any nonsense in it, please consult the
13: man page, in case the conversion went wrong.
14: <br>
15: <br><b>
1.1.1.4 ! misho 16: UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT
1.1 misho 17: </b><br>
18: <P>
1.1.1.4 ! misho 19: As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and
! 20: UTF-32 (from release 8.32), by means of two additional libraries. They can be
! 21: built as well as, or instead of, the 8-bit library.
1.1.1.2 misho 22: </P>
23: <br><b>
24: UTF-8 SUPPORT
25: </b><br>
26: <P>
27: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
28: support, and, in addition, you must call
1.1 misho 29: <a href="pcre_compile.html"><b>pcre_compile()</b></a>
30: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
1.1.1.4 ! misho 31: (*UTF8) or (*UTF). When either of these is the case, both the pattern and any
! 32: subject strings that are matched against it are treated as UTF-8 strings
! 33: instead of strings of individual 1-byte characters.
! 34: </P>
! 35: <br><b>
! 36: UTF-16 AND UTF-32 SUPPORT
! 37: </b><br>
! 38: <P>
! 39: In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or
! 40: 32-bit library with UTF support, and, in addition, you must call
! 41: <a href="pcre16_compile.html"><b>pcre16_compile()</b></a>
! 42: or
! 43: <a href="pcre32_compile.html"><b>pcre32_compile()</b></a>
! 44: with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively,
! 45: the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or
! 46: (*UTF), which can be used with either library. When UTF mode is set, both the
! 47: pattern and any subject strings that are matched against it are treated as
! 48: UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit
! 49: characters.
1.1.1.2 misho 50: </P>
51: <br><b>
52: UTF SUPPORT OVERHEAD
53: </b><br>
54: <P>
55: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 56: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.4 ! misho 57: to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big.
1.1 misho 58: </P>
1.1.1.2 misho 59: <br><b>
60: UNICODE PROPERTY SUPPORT
61: </b><br>
1.1 misho 62: <P>
1.1.1.2 misho 63: If PCRE is built with Unicode character property support (which implies UTF
64: support), the escape sequences \p{..}, \P{..}, and \X can be used.
1.1 misho 65: The available properties that can be tested are limited to the general
66: category properties such as Lu for an upper case letter or Nd for a decimal
67: number, the Unicode script names such as Arabic or Han, and the derived
1.1.1.4 ! misho 68: properties Any and L&. Full lists is given in the
1.1 misho 69: <a href="pcrepattern.html"><b>pcrepattern</b></a>
1.1.1.4 ! misho 70: and
! 71: <a href="pcresyntax.html"><b>pcresyntax</b></a>
1.1 misho 72: documentation. Only the short names for properties are supported. For example,
73: \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported.
74: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
75: compatibility with Perl 5.6. PCRE does not support this.
76: <a name="utf8strings"></a></P>
77: <br><b>
78: Validity of UTF-8 strings
79: </b><br>
80: <P>
1.1.1.2 misho 81: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
82: subjects are (by default) checked for validity on entry to the relevant
1.1.1.3 misho 83: functions. The entire string is checked before any other processing takes
84: place. From release 7.3 of PCRE, the check is according the rules of RFC 3629,
85: which are themselves derived from the Unicode specification. Earlier releases
86: of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit
87: values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0
1.1.1.4 ! misho 88: to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called
! 89: "non-character" code points are no longer excluded because Unicode corrigendum
! 90: #9 makes it clear that they should not be.)
1.1.1.2 misho 91: </P>
92: <P>
1.1.1.4 ! misho 93: Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16,
! 94: where they are used in pairs to encode codepoints with values greater than
! 95: 0xFFFF. The code points that are encoded by UTF-16 pairs are available
! 96: independently in the UTF-8 and UTF-32 encodings. (In other words, the whole
! 97: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and
! 98: UTF-32.)
1.1 misho 99: </P>
100: <P>
101: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
102: compile time, the only additional information is the offset to the first byte
1.1.1.3 misho 103: of the failing character. The run-time functions <b>pcre_exec()</b> and
1.1 misho 104: <b>pcre_dfa_exec()</b> also pass back this information, as well as a more
105: detailed reason code if the caller has provided memory in which to do this.
106: </P>
107: <P>
108: In some situations, you may already know that your strings are valid, and
1.1.1.3 misho 109: therefore want to skip these checks in order to improve performance, for
1.1.1.4 ! misho 110: example in the case of a long subject string that is being scanned repeatedly.
! 111: If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE
! 112: assumes that the pattern or subject it is given (respectively) contains only
! 113: valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string.
! 114: </P>
! 115: <P>
! 116: Note that passing PCRE_NO_UTF8_CHECK to <b>pcre_compile()</b> just disables the
! 117: check for the pattern; it does not also apply to subject strings. If you want
! 118: to disable the check for a subject string you must pass this option to
! 119: <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b>.
! 120: </P>
! 121: <P>
! 122: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result
! 123: is undefined and your program may crash.
1.1.1.2 misho 124: <a name="utf16strings"></a></P>
125: <br><b>
126: Validity of UTF-16 strings
127: </b><br>
128: <P>
129: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
130: passed as patterns and subjects are (by default) checked for validity on entry
131: to the relevant functions. Values other than those in the surrogate range
132: U+D800 to U+DFFF are independent code points. Values in the surrogate range
133: must be used in pairs in the correct manner.
134: </P>
135: <P>
136: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
137: compile time, the only additional information is the offset to the first data
1.1.1.3 misho 138: unit of the failing character. The run-time functions <b>pcre16_exec()</b> and
1.1.1.2 misho 139: <b>pcre16_dfa_exec()</b> also pass back this information, as well as a more
140: detailed reason code if the caller has provided memory in which to do this.
141: </P>
142: <P>
143: In some situations, you may already know that your strings are valid, and
144: therefore want to skip these checks in order to improve performance. If you set
145: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
146: the pattern or subject it is given (respectively) contains only valid UTF-16
147: sequences. In this case, it does not diagnose an invalid UTF-16 string.
1.1.1.4 ! misho 148: However, if an invalid string is passed, the result is undefined.
! 149: <a name="utf32strings"></a></P>
! 150: <br><b>
! 151: Validity of UTF-32 strings
! 152: </b><br>
! 153: <P>
! 154: When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are
! 155: passed as patterns and subjects are (by default) checked for validity on entry
! 156: to the relevant functions. This check allows only values in the range U+0
! 157: to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF.
! 158: </P>
! 159: <P>
! 160: If an invalid UTF-32 string is passed to PCRE, an error return is given. At
! 161: compile time, the only additional information is the offset to the first data
! 162: unit of the failing character. The run-time functions <b>pcre32_exec()</b> and
! 163: <b>pcre32_dfa_exec()</b> also pass back this information, as well as a more
! 164: detailed reason code if the caller has provided memory in which to do this.
! 165: </P>
! 166: <P>
! 167: In some situations, you may already know that your strings are valid, and
! 168: therefore want to skip these checks in order to improve performance. If you set
! 169: the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that
! 170: the pattern or subject it is given (respectively) contains only valid UTF-32
! 171: sequences. In this case, it does not diagnose an invalid UTF-32 string.
! 172: However, if an invalid string is passed, the result is undefined.
1.1 misho 173: </P>
174: <br><b>
1.1.1.2 misho 175: General comments about UTF modes
1.1 misho 176: </b><br>
177: <P>
1.1.1.4 ! misho 178: 1. Codepoints less than 256 can be specified in patterns by either braced or
! 179: unbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger
! 180: values have to use braced sequences.
1.1 misho 181: </P>
182: <P>
1.1.1.4 ! misho 183: 2. Octal numbers up to \777 are recognized, and in UTF-8 mode they match
1.1.1.2 misho 184: two-byte characters for values greater than \177.
1.1 misho 185: </P>
186: <P>
1.1.1.2 misho 187: 3. Repeat quantifiers apply to complete UTF characters, not to individual
188: data units, for example: \x{100}{3}.
1.1 misho 189: </P>
190: <P>
1.1.1.2 misho 191: 4. The dot metacharacter matches one UTF character instead of a single data
192: unit.
1.1 misho 193: </P>
194: <P>
1.1.1.2 misho 195: 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or
1.1.1.4 ! misho 196: a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in
! 197: UTF-32 mode, but its use can lead to some strange effects because it breaks up
! 198: multi-unit characters (see the description of \C in the
1.1 misho 199: <a href="pcrepattern.html"><b>pcrepattern</b></a>
200: documentation). The use of \C is not supported in the alternative matching
1.1.1.4 ! misho 201: function <b>pcre[16|32]_dfa_exec()</b>, nor is it supported in UTF mode by the
! 202: JIT optimization of <b>pcre[16|32]_exec()</b>. If JIT optimization is requested
! 203: for a UTF pattern that contains \C, it will not succeed, and so the matching
! 204: will be carried out by the normal interpretive function.
1.1 misho 205: </P>
206: <P>
207: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
208: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 misho 209: recognizes as digits, spaces, or word characters remain the same set as in
210: non-UTF mode, all with values less than 256. This remains true even when PCRE
211: is built to include Unicode property support, because to do otherwise would
212: slow down PCRE in many common cases. Note in particular that this applies to
213: \b and \B, because they are defined in terms of \w and \W. If you really
214: want to test for a wider sense of, say, "digit", you can use explicit Unicode
215: property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option,
216: the way that the character escapes work is changed so that Unicode properties
217: are used to determine which characters match. There are more details in the
218: section on
1.1 misho 219: <a href="pcrepattern.html#genericchartypes">generic character types</a>
220: in the
221: <a href="pcrepattern.html"><b>pcrepattern</b></a>
222: documentation.
223: </P>
224: <P>
225: 7. Similarly, characters that match the POSIX named character classes are all
226: low-valued characters, unless the PCRE_UCP option is set.
227: </P>
228: <P>
1.1.1.3 misho 229: 8. However, the horizontal and vertical white space matching escapes (\h, \H,
1.1 misho 230: \v, and \V) do match all the appropriate Unicode characters, whether or not
231: PCRE_UCP is set.
232: </P>
233: <P>
234: 9. Case-insensitive matching applies only to characters whose values are less
1.1.1.4 ! misho 235: than 128, unless PCRE is built with Unicode property support. A few Unicode
! 236: characters such as Greek sigma have more than two codepoints that are
! 237: case-equivalent. Up to and including PCRE release 8.31, only one-to-one case
! 238: mappings were supported, but later releases (with Unicode property support) do
! 239: treat as case-equivalent all versions of characters such as Greek sigma.
1.1 misho 240: </P>
241: <br><b>
242: AUTHOR
243: </b><br>
244: <P>
245: Philip Hazel
246: <br>
247: University Computing Service
248: <br>
249: Cambridge CB2 3QH, England.
250: <br>
251: </P>
252: <br><b>
253: REVISION
254: </b><br>
255: <P>
1.1.1.4 ! misho 256: Last updated: 27 February 2013
1.1 misho 257: <br>
1.1.1.4 ! misho 258: Copyright © 1997-2013 University of Cambridge.
1.1 misho 259: <br>
260: <p>
261: Return to the <a href="index.html">PCRE index page</a>.
262: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html