Annotation of embedaddon/pcre/doc/pcreunicode.3, revision 1.1.1.3
1.1.1.3 ! misho 1: .TH PCREUNICODE 3 "14 April 2012" "PCRE 8.30"
1.1 misho 2: .SH NAME
3: PCRE - Perl-compatible regular expressions
1.1.1.2 misho 4: .SH "UTF-8, UTF-16, AND UNICODE PROPERTY SUPPORT"
1.1 misho 5: .rs
6: .sp
1.1.1.2 misho 7: From Release 8.30, in addition to its previous UTF-8 support, PCRE also
8: supports UTF-16 by means of a separate 16-bit library. This can be built as
9: well as, or instead of, the 8-bit library.
10: .
11: .
12: .SH "UTF-8 SUPPORT"
13: .rs
14: .sp
15: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
16: support, and, in addition, you must call
1.1 misho 17: .\" HREF
18: \fBpcre_compile()\fP
19: .\"
20: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
21: (*UTF8). When either of these is the case, both the pattern and any subject
22: strings that are matched against it are treated as UTF-8 strings instead of
1.1.1.2 misho 23: strings of 1-byte characters.
24: .
25: .
26: .SH "UTF-16 SUPPORT"
27: .rs
28: .sp
29: In order process UTF-16 strings, you must build PCRE's 16-bit library with UTF
30: support, and, in addition, you must call
31: .\" HTML <a href="pcre_compile.html">
32: .\" </a>
33: \fBpcre16_compile()\fP
34: .\"
35: with the PCRE_UTF16 option flag, or the pattern must start with the sequence
36: (*UTF16). When either of these is the case, both the pattern and any subject
37: strings that are matched against it are treated as UTF-16 strings instead of
38: strings of 16-bit characters.
39: .
40: .
41: .SH "UTF SUPPORT OVERHEAD"
42: .rs
43: .sp
44: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 45: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.2 misho 46: to testing the PCRE_UTF8/16 flag occasionally, so should not be very big.
47: .
48: .
49: .SH "UNICODE PROPERTY SUPPORT"
50: .rs
51: .sp
52: If PCRE is built with Unicode character property support (which implies UTF
53: support), the escape sequences \ep{..}, \eP{..}, and \eX can be used.
1.1 misho 54: The available properties that can be tested are limited to the general
55: category properties such as Lu for an upper case letter or Nd for a decimal
56: number, the Unicode script names such as Arabic or Han, and the derived
57: properties Any and L&. A full list is given in the
58: .\" HREF
59: \fBpcrepattern\fP
60: .\"
61: documentation. Only the short names for properties are supported. For example,
62: \ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported.
63: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
64: compatibility with Perl 5.6. PCRE does not support this.
65: .
66: .
67: .\" HTML <a name="utf8strings"></a>
68: .SS "Validity of UTF-8 strings"
69: .rs
70: .sp
1.1.1.2 misho 71: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
72: subjects are (by default) checked for validity on entry to the relevant
1.1.1.3 ! misho 73: functions. The entire string is checked before any other processing takes
! 74: place. From release 7.3 of PCRE, the check is according the rules of RFC 3629,
! 75: which are themselves derived from the Unicode specification. Earlier releases
! 76: of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit
! 77: values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0
! 78: to U+10FFFF, excluding U+D800 to U+DFFF.
1.1.1.2 misho 79: .P
80: The excluded code points are the "Surrogate Area" of Unicode. They are reserved
81: for use by UTF-16, where they are used in pairs to encode codepoints with
82: values greater than 0xFFFF. The code points that are encoded by UTF-16 pairs
83: are available independently in the UTF-8 encoding. (In other words, the whole
84: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
1.1 misho 85: .P
86: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
87: compile time, the only additional information is the offset to the first byte
1.1.1.3 ! misho 88: of the failing character. The run-time functions \fBpcre_exec()\fP and
1.1 misho 89: \fBpcre_dfa_exec()\fP also pass back this information, as well as a more
90: detailed reason code if the caller has provided memory in which to do this.
91: .P
92: In some situations, you may already know that your strings are valid, and
1.1.1.3 ! misho 93: therefore want to skip these checks in order to improve performance, for
! 94: example in the case of a long subject string that is being scanned repeatedly
! 95: with different patterns. If you set the PCRE_NO_UTF8_CHECK flag at compile time
! 96: or at run time, PCRE assumes that the pattern or subject it is given
! 97: (respectively) contains only valid UTF-8 codes. In this case, it does not
! 98: diagnose an invalid UTF-8 string.
1.1 misho 99: .P
100: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
101: happens depends on why the string is invalid. If the string conforms to the
102: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
103: in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted
104: version of \fBpcre_exec()\fP. In other words, apart from the initial validity
105: test, these functions (when in UTF-8 mode) handle strings according to the more
106: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
107: \fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or
108: if the string does not even conform to RFC 2279, the result is undefined. Your
109: program may crash.
110: .P
111: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
112: encoded in a UTF-8-like manner as per the old RFC, you can set
113: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
114: situation, you will have to apply your own validity check, and avoid the use of
115: JIT optimization.
116: .
117: .
1.1.1.2 misho 118: .\" HTML <a name="utf16strings"></a>
119: .SS "Validity of UTF-16 strings"
1.1 misho 120: .rs
121: .sp
1.1.1.2 misho 122: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
123: passed as patterns and subjects are (by default) checked for validity on entry
124: to the relevant functions. Values other than those in the surrogate range
125: U+D800 to U+DFFF are independent code points. Values in the surrogate range
126: must be used in pairs in the correct manner.
127: .P
128: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
129: compile time, the only additional information is the offset to the first data
1.1.1.3 ! misho 130: unit of the failing character. The run-time functions \fBpcre16_exec()\fP and
1.1.1.2 misho 131: \fBpcre16_dfa_exec()\fP also pass back this information, as well as a more
132: detailed reason code if the caller has provided memory in which to do this.
1.1 misho 133: .P
1.1.1.2 misho 134: In some situations, you may already know that your strings are valid, and
135: therefore want to skip these checks in order to improve performance. If you set
136: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
137: the pattern or subject it is given (respectively) contains only valid UTF-16
138: sequences. In this case, it does not diagnose an invalid UTF-16 string.
139: .
140: .
141: .SS "General comments about UTF modes"
142: .rs
143: .sp
144: 1. Codepoints less than 256 can be specified by either braced or unbraced
145: hexadecimal escape sequences (for example, \ex{b3} or \exb3). Larger values
146: have to use braced sequences.
147: .P
148: 2. Octal numbers up to \e777 are recognized, and in UTF-8 mode, they match
149: two-byte characters for values greater than \e177.
150: .P
151: 3. Repeat quantifiers apply to complete UTF characters, not to individual
152: data units, for example: \ex{100}{3}.
153: .P
154: 4. The dot metacharacter matches one UTF character instead of a single data
155: unit.
156: .P
157: 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, or
158: a single 16-bit data unit in UTF-16 mode, but its use can lead to some strange
159: effects because it breaks up multi-unit characters (see the description of \eC
160: in the
1.1 misho 161: .\" HREF
162: \fBpcrepattern\fP
163: .\"
164: documentation). The use of \eC is not supported in the alternative matching
1.1.1.2 misho 165: function \fBpcre[16]_dfa_exec()\fP, nor is it supported in UTF mode by the JIT
166: optimization of \fBpcre[16]_exec()\fP. If JIT optimization is requested for a
167: UTF pattern that contains \eC, it will not succeed, and so the matching will
168: be carried out by the normal interpretive function.
1.1 misho 169: .P
170: 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
171: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 misho 172: recognizes as digits, spaces, or word characters remain the same set as in
173: non-UTF mode, all with values less than 256. This remains true even when PCRE
174: is built to include Unicode property support, because to do otherwise would
175: slow down PCRE in many common cases. Note in particular that this applies to
176: \eb and \eB, because they are defined in terms of \ew and \eW. If you really
177: want to test for a wider sense of, say, "digit", you can use explicit Unicode
178: property tests such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option,
179: the way that the character escapes work is changed so that Unicode properties
180: are used to determine which characters match. There are more details in the
181: section on
1.1 misho 182: .\" HTML <a href="pcrepattern.html#genericchartypes">
183: .\" </a>
184: generic character types
185: .\"
186: in the
187: .\" HREF
188: \fBpcrepattern\fP
189: .\"
190: documentation.
191: .P
192: 7. Similarly, characters that match the POSIX named character classes are all
193: low-valued characters, unless the PCRE_UCP option is set.
194: .P
1.1.1.3 ! misho 195: 8. However, the horizontal and vertical white space matching escapes (\eh, \eH,
1.1 misho 196: \ev, and \eV) do match all the appropriate Unicode characters, whether or not
197: PCRE_UCP is set.
198: .P
199: 9. Case-insensitive matching applies only to characters whose values are less
200: than 128, unless PCRE is built with Unicode property support. Even when Unicode
201: property support is available, PCRE still uses its own character tables when
202: checking the case of low-valued characters, so as not to degrade performance.
203: The Unicode property information is used only for characters with higher
204: values. Furthermore, PCRE supports case-insensitive matching only when there is
205: a one-to-one mapping between a letter's cases. There are a small number of
206: many-to-one mappings in Unicode; these are not supported by PCRE.
207: .
208: .
209: .SH AUTHOR
210: .rs
211: .sp
212: .nf
213: Philip Hazel
214: University Computing Service
215: Cambridge CB2 3QH, England.
216: .fi
217: .
218: .
219: .SH REVISION
220: .rs
221: .sp
222: .nf
1.1.1.3 ! misho 223: Last updated: 14 April 2012
1.1.1.2 misho 224: Copyright (c) 1997-2012 University of Cambridge.
1.1 misho 225: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc