Annotation of embedaddon/pcre/doc/pcreunicode.3, revision 1.1.1.4
1.1.1.4 ! misho 1: .TH PCREUNICODE 3 "27 February 2013" "PCRE 8.33"
1.1 misho 2: .SH NAME
3: PCRE - Perl-compatible regular expressions
1.1.1.4 ! misho 4: .SH "UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT"
1.1 misho 5: .rs
6: .sp
1.1.1.4 ! misho 7: As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and
! 8: UTF-32 (from release 8.32), by means of two additional libraries. They can be
! 9: built as well as, or instead of, the 8-bit library.
1.1.1.2 misho 10: .
11: .
12: .SH "UTF-8 SUPPORT"
13: .rs
14: .sp
15: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
16: support, and, in addition, you must call
1.1 misho 17: .\" HREF
18: \fBpcre_compile()\fP
19: .\"
20: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
1.1.1.4 ! misho 21: (*UTF8) or (*UTF). When either of these is the case, both the pattern and any
! 22: subject strings that are matched against it are treated as UTF-8 strings
! 23: instead of strings of individual 1-byte characters.
1.1.1.2 misho 24: .
25: .
1.1.1.4 ! misho 26: .SH "UTF-16 AND UTF-32 SUPPORT"
1.1.1.2 misho 27: .rs
28: .sp
1.1.1.4 ! misho 29: In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or
! 30: 32-bit library with UTF support, and, in addition, you must call
! 31: .\" HREF
1.1.1.2 misho 32: \fBpcre16_compile()\fP
33: .\"
1.1.1.4 ! misho 34: or
! 35: .\" HREF
! 36: \fBpcre32_compile()\fP
! 37: .\"
! 38: with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively,
! 39: the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or
! 40: (*UTF), which can be used with either library. When UTF mode is set, both the
! 41: pattern and any subject strings that are matched against it are treated as
! 42: UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit
! 43: characters.
1.1.1.2 misho 44: .
45: .
46: .SH "UTF SUPPORT OVERHEAD"
47: .rs
48: .sp
49: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 50: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.4 ! misho 51: to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big.
1.1.1.2 misho 52: .
53: .
54: .SH "UNICODE PROPERTY SUPPORT"
55: .rs
56: .sp
57: If PCRE is built with Unicode character property support (which implies UTF
58: support), the escape sequences \ep{..}, \eP{..}, and \eX can be used.
1.1 misho 59: The available properties that can be tested are limited to the general
60: category properties such as Lu for an upper case letter or Nd for a decimal
61: number, the Unicode script names such as Arabic or Han, and the derived
1.1.1.4 ! misho 62: properties Any and L&. Full lists is given in the
1.1 misho 63: .\" HREF
64: \fBpcrepattern\fP
65: .\"
1.1.1.4 ! misho 66: and
! 67: .\" HREF
! 68: \fBpcresyntax\fP
! 69: .\"
1.1 misho 70: documentation. Only the short names for properties are supported. For example,
71: \ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported.
72: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
73: compatibility with Perl 5.6. PCRE does not support this.
74: .
75: .
76: .\" HTML <a name="utf8strings"></a>
77: .SS "Validity of UTF-8 strings"
78: .rs
79: .sp
1.1.1.2 misho 80: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
81: subjects are (by default) checked for validity on entry to the relevant
1.1.1.3 misho 82: functions. The entire string is checked before any other processing takes
83: place. From release 7.3 of PCRE, the check is according the rules of RFC 3629,
84: which are themselves derived from the Unicode specification. Earlier releases
85: of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit
86: values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0
1.1.1.4 ! misho 87: to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called
! 88: "non-character" code points are no longer excluded because Unicode corrigendum
! 89: #9 makes it clear that they should not be.)
! 90: .P
! 91: Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16,
! 92: where they are used in pairs to encode codepoints with values greater than
! 93: 0xFFFF. The code points that are encoded by UTF-16 pairs are available
! 94: independently in the UTF-8 and UTF-32 encodings. (In other words, the whole
! 95: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and
! 96: UTF-32.)
1.1 misho 97: .P
98: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
99: compile time, the only additional information is the offset to the first byte
1.1.1.3 misho 100: of the failing character. The run-time functions \fBpcre_exec()\fP and
1.1 misho 101: \fBpcre_dfa_exec()\fP also pass back this information, as well as a more
102: detailed reason code if the caller has provided memory in which to do this.
103: .P
104: In some situations, you may already know that your strings are valid, and
1.1.1.3 misho 105: therefore want to skip these checks in order to improve performance, for
1.1.1.4 ! misho 106: example in the case of a long subject string that is being scanned repeatedly.
! 107: If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE
! 108: assumes that the pattern or subject it is given (respectively) contains only
! 109: valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string.
! 110: .P
! 111: Note that passing PCRE_NO_UTF8_CHECK to \fBpcre_compile()\fP just disables the
! 112: check for the pattern; it does not also apply to subject strings. If you want
! 113: to disable the check for a subject string you must pass this option to
! 114: \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP.
! 115: .P
! 116: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result
! 117: is undefined and your program may crash.
1.1 misho 118: .
119: .
1.1.1.2 misho 120: .\" HTML <a name="utf16strings"></a>
121: .SS "Validity of UTF-16 strings"
1.1 misho 122: .rs
123: .sp
1.1.1.2 misho 124: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
125: passed as patterns and subjects are (by default) checked for validity on entry
126: to the relevant functions. Values other than those in the surrogate range
127: U+D800 to U+DFFF are independent code points. Values in the surrogate range
128: must be used in pairs in the correct manner.
129: .P
130: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
131: compile time, the only additional information is the offset to the first data
1.1.1.3 misho 132: unit of the failing character. The run-time functions \fBpcre16_exec()\fP and
1.1.1.2 misho 133: \fBpcre16_dfa_exec()\fP also pass back this information, as well as a more
134: detailed reason code if the caller has provided memory in which to do this.
1.1 misho 135: .P
1.1.1.2 misho 136: In some situations, you may already know that your strings are valid, and
137: therefore want to skip these checks in order to improve performance. If you set
138: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
139: the pattern or subject it is given (respectively) contains only valid UTF-16
140: sequences. In this case, it does not diagnose an invalid UTF-16 string.
1.1.1.4 ! misho 141: However, if an invalid string is passed, the result is undefined.
! 142: .
! 143: .
! 144: .\" HTML <a name="utf32strings"></a>
! 145: .SS "Validity of UTF-32 strings"
! 146: .rs
! 147: .sp
! 148: When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are
! 149: passed as patterns and subjects are (by default) checked for validity on entry
! 150: to the relevant functions. This check allows only values in the range U+0
! 151: to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF.
! 152: .P
! 153: If an invalid UTF-32 string is passed to PCRE, an error return is given. At
! 154: compile time, the only additional information is the offset to the first data
! 155: unit of the failing character. The run-time functions \fBpcre32_exec()\fP and
! 156: \fBpcre32_dfa_exec()\fP also pass back this information, as well as a more
! 157: detailed reason code if the caller has provided memory in which to do this.
! 158: .P
! 159: In some situations, you may already know that your strings are valid, and
! 160: therefore want to skip these checks in order to improve performance. If you set
! 161: the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that
! 162: the pattern or subject it is given (respectively) contains only valid UTF-32
! 163: sequences. In this case, it does not diagnose an invalid UTF-32 string.
! 164: However, if an invalid string is passed, the result is undefined.
1.1.1.2 misho 165: .
166: .
167: .SS "General comments about UTF modes"
168: .rs
169: .sp
1.1.1.4 ! misho 170: 1. Codepoints less than 256 can be specified in patterns by either braced or
! 171: unbraced hexadecimal escape sequences (for example, \ex{b3} or \exb3). Larger
! 172: values have to use braced sequences.
1.1.1.2 misho 173: .P
1.1.1.4 ! misho 174: 2. Octal numbers up to \e777 are recognized, and in UTF-8 mode they match
1.1.1.2 misho 175: two-byte characters for values greater than \e177.
176: .P
177: 3. Repeat quantifiers apply to complete UTF characters, not to individual
178: data units, for example: \ex{100}{3}.
179: .P
180: 4. The dot metacharacter matches one UTF character instead of a single data
181: unit.
182: .P
183: 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, or
1.1.1.4 ! misho 184: a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in
! 185: UTF-32 mode, but its use can lead to some strange effects because it breaks up
! 186: multi-unit characters (see the description of \eC in the
1.1 misho 187: .\" HREF
188: \fBpcrepattern\fP
189: .\"
190: documentation). The use of \eC is not supported in the alternative matching
1.1.1.4 ! misho 191: function \fBpcre[16|32]_dfa_exec()\fP, nor is it supported in UTF mode by the
! 192: JIT optimization of \fBpcre[16|32]_exec()\fP. If JIT optimization is requested
! 193: for a UTF pattern that contains \eC, it will not succeed, and so the matching
! 194: will be carried out by the normal interpretive function.
1.1 misho 195: .P
196: 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
197: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 misho 198: recognizes as digits, spaces, or word characters remain the same set as in
199: non-UTF mode, all with values less than 256. This remains true even when PCRE
200: is built to include Unicode property support, because to do otherwise would
201: slow down PCRE in many common cases. Note in particular that this applies to
202: \eb and \eB, because they are defined in terms of \ew and \eW. If you really
203: want to test for a wider sense of, say, "digit", you can use explicit Unicode
204: property tests such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option,
205: the way that the character escapes work is changed so that Unicode properties
206: are used to determine which characters match. There are more details in the
207: section on
1.1 misho 208: .\" HTML <a href="pcrepattern.html#genericchartypes">
209: .\" </a>
210: generic character types
211: .\"
212: in the
213: .\" HREF
214: \fBpcrepattern\fP
215: .\"
216: documentation.
217: .P
218: 7. Similarly, characters that match the POSIX named character classes are all
219: low-valued characters, unless the PCRE_UCP option is set.
220: .P
1.1.1.3 misho 221: 8. However, the horizontal and vertical white space matching escapes (\eh, \eH,
1.1 misho 222: \ev, and \eV) do match all the appropriate Unicode characters, whether or not
223: PCRE_UCP is set.
224: .P
225: 9. Case-insensitive matching applies only to characters whose values are less
1.1.1.4 ! misho 226: than 128, unless PCRE is built with Unicode property support. A few Unicode
! 227: characters such as Greek sigma have more than two codepoints that are
! 228: case-equivalent. Up to and including PCRE release 8.31, only one-to-one case
! 229: mappings were supported, but later releases (with Unicode property support) do
! 230: treat as case-equivalent all versions of characters such as Greek sigma.
1.1 misho 231: .
232: .
233: .SH AUTHOR
234: .rs
235: .sp
236: .nf
237: Philip Hazel
238: University Computing Service
239: Cambridge CB2 3QH, England.
240: .fi
241: .
242: .
243: .SH REVISION
244: .rs
245: .sp
246: .nf
1.1.1.4 ! misho 247: Last updated: 27 February 2013
! 248: Copyright (c) 1997-2013 University of Cambridge.
1.1 misho 249: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc