Annotation of embedaddon/pcre/doc/pcreunicode.3, revision 1.1.1.2
1.1 misho 1: .TH PCREUNICODE 3
2: .SH NAME
3: PCRE - Perl-compatible regular expressions
1.1.1.2 ! misho 4: .SH "UTF-8, UTF-16, AND UNICODE PROPERTY SUPPORT"
1.1 misho 5: .rs
6: .sp
1.1.1.2 ! misho 7: From Release 8.30, in addition to its previous UTF-8 support, PCRE also
! 8: supports UTF-16 by means of a separate 16-bit library. This can be built as
! 9: well as, or instead of, the 8-bit library.
! 10: .
! 11: .
! 12: .SH "UTF-8 SUPPORT"
! 13: .rs
! 14: .sp
! 15: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF
! 16: support, and, in addition, you must call
1.1 misho 17: .\" HREF
18: \fBpcre_compile()\fP
19: .\"
20: with the PCRE_UTF8 option flag, or the pattern must start with the sequence
21: (*UTF8). When either of these is the case, both the pattern and any subject
22: strings that are matched against it are treated as UTF-8 strings instead of
1.1.1.2 ! misho 23: strings of 1-byte characters.
! 24: .
! 25: .
! 26: .SH "UTF-16 SUPPORT"
! 27: .rs
! 28: .sp
! 29: In order process UTF-16 strings, you must build PCRE's 16-bit library with UTF
! 30: support, and, in addition, you must call
! 31: .\" HTML <a href="pcre_compile.html">
! 32: .\" </a>
! 33: \fBpcre16_compile()\fP
! 34: .\"
! 35: with the PCRE_UTF16 option flag, or the pattern must start with the sequence
! 36: (*UTF16). When either of these is the case, both the pattern and any subject
! 37: strings that are matched against it are treated as UTF-16 strings instead of
! 38: strings of 16-bit characters.
! 39: .
! 40: .
! 41: .SH "UTF SUPPORT OVERHEAD"
! 42: .rs
! 43: .sp
! 44: If you compile PCRE with UTF support, but do not use it at run time, the
1.1 misho 45: library will be a bit bigger, but the additional run time overhead is limited
1.1.1.2 ! misho 46: to testing the PCRE_UTF8/16 flag occasionally, so should not be very big.
! 47: .
! 48: .
! 49: .SH "UNICODE PROPERTY SUPPORT"
! 50: .rs
! 51: .sp
! 52: If PCRE is built with Unicode character property support (which implies UTF
! 53: support), the escape sequences \ep{..}, \eP{..}, and \eX can be used.
1.1 misho 54: The available properties that can be tested are limited to the general
55: category properties such as Lu for an upper case letter or Nd for a decimal
56: number, the Unicode script names such as Arabic or Han, and the derived
57: properties Any and L&. A full list is given in the
58: .\" HREF
59: \fBpcrepattern\fP
60: .\"
61: documentation. Only the short names for properties are supported. For example,
62: \ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported.
63: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for
64: compatibility with Perl 5.6. PCRE does not support this.
65: .
66: .
67: .\" HTML <a name="utf8strings"></a>
68: .SS "Validity of UTF-8 strings"
69: .rs
70: .sp
1.1.1.2 ! misho 71: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and
! 72: subjects are (by default) checked for validity on entry to the relevant
! 73: functions. From release 7.3 of PCRE, the check is according the rules of RFC
! 74: 3629, which are themselves derived from the Unicode specification. Earlier
! 75: releases of PCRE followed the rules of RFC 2279, which allows the full range of
! 76: 31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the
! 77: range U+0 to U+10FFFF, excluding U+D800 to U+DFFF.
! 78: .P
! 79: The excluded code points are the "Surrogate Area" of Unicode. They are reserved
! 80: for use by UTF-16, where they are used in pairs to encode codepoints with
! 81: values greater than 0xFFFF. The code points that are encoded by UTF-16 pairs
! 82: are available independently in the UTF-8 encoding. (In other words, the whole
! 83: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.)
1.1 misho 84: .P
85: If an invalid UTF-8 string is passed to PCRE, an error return is given. At
86: compile time, the only additional information is the offset to the first byte
87: of the failing character. The runtime functions \fBpcre_exec()\fP and
88: \fBpcre_dfa_exec()\fP also pass back this information, as well as a more
89: detailed reason code if the caller has provided memory in which to do this.
90: .P
91: In some situations, you may already know that your strings are valid, and
92: therefore want to skip these checks in order to improve performance. If you set
93: the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that
94: the pattern or subject it is given (respectively) contains only valid UTF-8
95: codes. In this case, it does not diagnose an invalid UTF-8 string.
96: .P
97: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what
98: happens depends on why the string is invalid. If the string conforms to the
99: "old" definition of UTF-8 (RFC 2279), it is processed as a string of characters
100: in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted
101: version of \fBpcre_exec()\fP. In other words, apart from the initial validity
102: test, these functions (when in UTF-8 mode) handle strings according to the more
103: liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for
104: \fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or
105: if the string does not even conform to RFC 2279, the result is undefined. Your
106: program may crash.
107: .P
108: If you want to process strings of values in the full range 0 to 0x7FFFFFFF,
109: encoded in a UTF-8-like manner as per the old RFC, you can set
110: PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this
111: situation, you will have to apply your own validity check, and avoid the use of
112: JIT optimization.
113: .
114: .
1.1.1.2 ! misho 115: .\" HTML <a name="utf16strings"></a>
! 116: .SS "Validity of UTF-16 strings"
1.1 misho 117: .rs
118: .sp
1.1.1.2 ! misho 119: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are
! 120: passed as patterns and subjects are (by default) checked for validity on entry
! 121: to the relevant functions. Values other than those in the surrogate range
! 122: U+D800 to U+DFFF are independent code points. Values in the surrogate range
! 123: must be used in pairs in the correct manner.
! 124: .P
! 125: If an invalid UTF-16 string is passed to PCRE, an error return is given. At
! 126: compile time, the only additional information is the offset to the first data
! 127: unit of the failing character. The runtime functions \fBpcre16_exec()\fP and
! 128: \fBpcre16_dfa_exec()\fP also pass back this information, as well as a more
! 129: detailed reason code if the caller has provided memory in which to do this.
1.1 misho 130: .P
1.1.1.2 ! misho 131: In some situations, you may already know that your strings are valid, and
! 132: therefore want to skip these checks in order to improve performance. If you set
! 133: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that
! 134: the pattern or subject it is given (respectively) contains only valid UTF-16
! 135: sequences. In this case, it does not diagnose an invalid UTF-16 string.
! 136: .
! 137: .
! 138: .SS "General comments about UTF modes"
! 139: .rs
! 140: .sp
! 141: 1. Codepoints less than 256 can be specified by either braced or unbraced
! 142: hexadecimal escape sequences (for example, \ex{b3} or \exb3). Larger values
! 143: have to use braced sequences.
! 144: .P
! 145: 2. Octal numbers up to \e777 are recognized, and in UTF-8 mode, they match
! 146: two-byte characters for values greater than \e177.
! 147: .P
! 148: 3. Repeat quantifiers apply to complete UTF characters, not to individual
! 149: data units, for example: \ex{100}{3}.
! 150: .P
! 151: 4. The dot metacharacter matches one UTF character instead of a single data
! 152: unit.
! 153: .P
! 154: 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, or
! 155: a single 16-bit data unit in UTF-16 mode, but its use can lead to some strange
! 156: effects because it breaks up multi-unit characters (see the description of \eC
! 157: in the
1.1 misho 158: .\" HREF
159: \fBpcrepattern\fP
160: .\"
161: documentation). The use of \eC is not supported in the alternative matching
1.1.1.2 ! misho 162: function \fBpcre[16]_dfa_exec()\fP, nor is it supported in UTF mode by the JIT
! 163: optimization of \fBpcre[16]_exec()\fP. If JIT optimization is requested for a
! 164: UTF pattern that contains \eC, it will not succeed, and so the matching will
! 165: be carried out by the normal interpretive function.
1.1 misho 166: .P
167: 6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly
168: test characters of any code value, but, by default, the characters that PCRE
1.1.1.2 ! misho 169: recognizes as digits, spaces, or word characters remain the same set as in
! 170: non-UTF mode, all with values less than 256. This remains true even when PCRE
! 171: is built to include Unicode property support, because to do otherwise would
! 172: slow down PCRE in many common cases. Note in particular that this applies to
! 173: \eb and \eB, because they are defined in terms of \ew and \eW. If you really
! 174: want to test for a wider sense of, say, "digit", you can use explicit Unicode
! 175: property tests such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option,
! 176: the way that the character escapes work is changed so that Unicode properties
! 177: are used to determine which characters match. There are more details in the
! 178: section on
1.1 misho 179: .\" HTML <a href="pcrepattern.html#genericchartypes">
180: .\" </a>
181: generic character types
182: .\"
183: in the
184: .\" HREF
185: \fBpcrepattern\fP
186: .\"
187: documentation.
188: .P
189: 7. Similarly, characters that match the POSIX named character classes are all
190: low-valued characters, unless the PCRE_UCP option is set.
191: .P
192: 8. However, the horizontal and vertical whitespace matching escapes (\eh, \eH,
193: \ev, and \eV) do match all the appropriate Unicode characters, whether or not
194: PCRE_UCP is set.
195: .P
196: 9. Case-insensitive matching applies only to characters whose values are less
197: than 128, unless PCRE is built with Unicode property support. Even when Unicode
198: property support is available, PCRE still uses its own character tables when
199: checking the case of low-valued characters, so as not to degrade performance.
200: The Unicode property information is used only for characters with higher
201: values. Furthermore, PCRE supports case-insensitive matching only when there is
202: a one-to-one mapping between a letter's cases. There are a small number of
203: many-to-one mappings in Unicode; these are not supported by PCRE.
204: .
205: .
206: .SH AUTHOR
207: .rs
208: .sp
209: .nf
210: Philip Hazel
211: University Computing Service
212: Cambridge CB2 3QH, England.
213: .fi
214: .
215: .
216: .SH REVISION
217: .rs
218: .sp
219: .nf
1.1.1.2 ! misho 220: Last updated: 13 January 2012
! 221: Copyright (c) 1997-2012 University of Cambridge.
1.1 misho 222: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcreunicode.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc