version 1.1, 2012/02/21 23:05:52
|
version 1.1.1.4, 2013/07/22 08:25:56
|
Line 1
|
Line 1
|
.TH PCREUNICODE 3 | .TH PCREUNICODE 3 "27 February 2013" "PCRE 8.33" |
.SH NAME |
.SH NAME |
PCRE - Perl-compatible regular expressions |
PCRE - Perl-compatible regular expressions |
.SH "UTF-8 AND UNICODE PROPERTY SUPPORT" | .SH "UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT" |
.rs |
.rs |
.sp |
.sp |
In order process UTF-8 strings, you must build PCRE to include UTF-8 support in | As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and |
the code, and, in addition, you must call | UTF-32 (from release 8.32), by means of two additional libraries. They can be |
| built as well as, or instead of, the 8-bit library. |
| . |
| . |
| .SH "UTF-8 SUPPORT" |
| .rs |
| .sp |
| In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF |
| support, and, in addition, you must call |
.\" HREF |
.\" HREF |
\fBpcre_compile()\fP |
\fBpcre_compile()\fP |
.\" |
.\" |
with the PCRE_UTF8 option flag, or the pattern must start with the sequence |
with the PCRE_UTF8 option flag, or the pattern must start with the sequence |
(*UTF8). When either of these is the case, both the pattern and any subject | (*UTF8) or (*UTF). When either of these is the case, both the pattern and any |
strings that are matched against it are treated as UTF-8 strings instead of | subject strings that are matched against it are treated as UTF-8 strings |
strings of 1-byte characters. PCRE does not support any other formats (in | instead of strings of individual 1-byte characters. |
particular, it does not support UTF-16). | . |
.P | . |
If you compile PCRE with UTF-8 support, but do not use it at run time, the | .SH "UTF-16 AND UTF-32 SUPPORT" |
| .rs |
| .sp |
| In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or |
| 32-bit library with UTF support, and, in addition, you must call |
| .\" HREF |
| \fBpcre16_compile()\fP |
| .\" |
| or |
| .\" HREF |
| \fBpcre32_compile()\fP |
| .\" |
| with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively, |
| the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or |
| (*UTF), which can be used with either library. When UTF mode is set, both the |
| pattern and any subject strings that are matched against it are treated as |
| UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit |
| characters. |
| . |
| . |
| .SH "UTF SUPPORT OVERHEAD" |
| .rs |
| .sp |
| If you compile PCRE with UTF support, but do not use it at run time, the |
library will be a bit bigger, but the additional run time overhead is limited |
library will be a bit bigger, but the additional run time overhead is limited |
to testing the PCRE_UTF8 flag occasionally, so should not be very big. | to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big. |
.P | . |
If PCRE is built with Unicode character property support (which implies UTF-8 | . |
support), the escape sequences \ep{..}, \eP{..}, and \eX are supported. | .SH "UNICODE PROPERTY SUPPORT" |
| .rs |
| .sp |
| If PCRE is built with Unicode character property support (which implies UTF |
| support), the escape sequences \ep{..}, \eP{..}, and \eX can be used. |
The available properties that can be tested are limited to the general |
The available properties that can be tested are limited to the general |
category properties such as Lu for an upper case letter or Nd for a decimal |
category properties such as Lu for an upper case letter or Nd for a decimal |
number, the Unicode script names such as Arabic or Han, and the derived |
number, the Unicode script names such as Arabic or Han, and the derived |
properties Any and L&. A full list is given in the | properties Any and L&. Full lists is given in the |
.\" HREF |
.\" HREF |
\fBpcrepattern\fP |
\fBpcrepattern\fP |
.\" |
.\" |
|
and |
|
.\" HREF |
|
\fBpcresyntax\fP |
|
.\" |
documentation. Only the short names for properties are supported. For example, |
documentation. Only the short names for properties are supported. For example, |
\ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported. |
\ep{L} matches a letter. Its Perl synonym, \ep{Letter}, is not supported. |
Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
Furthermore, in Perl, many properties may optionally be prefixed by "Is", for |
Line 38 compatibility with Perl 5.6. PCRE does not support thi
|
Line 77 compatibility with Perl 5.6. PCRE does not support thi
|
.SS "Validity of UTF-8 strings" |
.SS "Validity of UTF-8 strings" |
.rs |
.rs |
.sp |
.sp |
When you set the PCRE_UTF8 flag, the strings passed as patterns and subjects | When you set the PCRE_UTF8 flag, the byte strings passed as patterns and |
are (by default) checked for validity on entry to the relevant functions. From | subjects are (by default) checked for validity on entry to the relevant |
release 7.3 of PCRE, the check is according the rules of RFC 3629, which are | functions. The entire string is checked before any other processing takes |
themselves derived from the Unicode specification. Earlier releases of PCRE | place. From release 7.3 of PCRE, the check is according the rules of RFC 3629, |
followed the rules of RFC 2279, which allows the full range of 31-bit values (0 | which are themselves derived from the Unicode specification. Earlier releases |
to 0x7FFFFFFF). The current check allows only values in the range U+0 to | of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit |
U+10FFFF, excluding U+D800 to U+DFFF. | values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 |
| to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called |
| "non-character" code points are no longer excluded because Unicode corrigendum |
| #9 makes it clear that they should not be.) |
.P |
.P |
The excluded code points are the "Low Surrogate Area" of Unicode, of which the | Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, |
Unicode Standard says this: "The Low Surrogate Area does not contain any | where they are used in pairs to encode codepoints with values greater than |
character assignments, consequently no character code charts or namelists are | 0xFFFF. The code points that are encoded by UTF-16 pairs are available |
provided for this area. Surrogates are reserved for use with UTF-16 and then | independently in the UTF-8 and UTF-32 encodings. (In other words, the whole |
must be used in pairs." The code points that are encoded by UTF-16 pairs are | surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and |
available as independent code points in the UTF-8 encoding. (In other words, | UTF-32.) |
the whole surrogate thing is a fudge for UTF-16 which unfortunately messes up | |
UTF-8.) | |
.P |
.P |
If an invalid UTF-8 string is passed to PCRE, an error return is given. At |
If an invalid UTF-8 string is passed to PCRE, an error return is given. At |
compile time, the only additional information is the offset to the first byte |
compile time, the only additional information is the offset to the first byte |
of the failing character. The runtime functions \fBpcre_exec()\fP and | of the failing character. The run-time functions \fBpcre_exec()\fP and |
\fBpcre_dfa_exec()\fP also pass back this information, as well as a more |
\fBpcre_dfa_exec()\fP also pass back this information, as well as a more |
detailed reason code if the caller has provided memory in which to do this. |
detailed reason code if the caller has provided memory in which to do this. |
.P |
.P |
In some situations, you may already know that your strings are valid, and |
In some situations, you may already know that your strings are valid, and |
|
therefore want to skip these checks in order to improve performance, for |
|
example in the case of a long subject string that is being scanned repeatedly. |
|
If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE |
|
assumes that the pattern or subject it is given (respectively) contains only |
|
valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string. |
|
.P |
|
Note that passing PCRE_NO_UTF8_CHECK to \fBpcre_compile()\fP just disables the |
|
check for the pattern; it does not also apply to subject strings. If you want |
|
to disable the check for a subject string you must pass this option to |
|
\fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP. |
|
.P |
|
If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result |
|
is undefined and your program may crash. |
|
. |
|
. |
|
.\" HTML <a name="utf16strings"></a> |
|
.SS "Validity of UTF-16 strings" |
|
.rs |
|
.sp |
|
When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are |
|
passed as patterns and subjects are (by default) checked for validity on entry |
|
to the relevant functions. Values other than those in the surrogate range |
|
U+D800 to U+DFFF are independent code points. Values in the surrogate range |
|
must be used in pairs in the correct manner. |
|
.P |
|
If an invalid UTF-16 string is passed to PCRE, an error return is given. At |
|
compile time, the only additional information is the offset to the first data |
|
unit of the failing character. The run-time functions \fBpcre16_exec()\fP and |
|
\fBpcre16_dfa_exec()\fP also pass back this information, as well as a more |
|
detailed reason code if the caller has provided memory in which to do this. |
|
.P |
|
In some situations, you may already know that your strings are valid, and |
therefore want to skip these checks in order to improve performance. If you set |
therefore want to skip these checks in order to improve performance. If you set |
the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that | the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that |
the pattern or subject it is given (respectively) contains only valid UTF-8 | the pattern or subject it is given (respectively) contains only valid UTF-16 |
codes. In this case, it does not diagnose an invalid UTF-8 string. | sequences. In this case, it does not diagnose an invalid UTF-16 string. |
| However, if an invalid string is passed, the result is undefined. |
| . |
| . |
| .\" HTML <a name="utf32strings"></a> |
| .SS "Validity of UTF-32 strings" |
| .rs |
| .sp |
| When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are |
| passed as patterns and subjects are (by default) checked for validity on entry |
| to the relevant functions. This check allows only values in the range U+0 |
| to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF. |
.P |
.P |
If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what | If an invalid UTF-32 string is passed to PCRE, an error return is given. At |
happens depends on why the string is invalid. If the string conforms to the | compile time, the only additional information is the offset to the first data |
"old" definition of UTF-8 (RFC 2279), it is processed as a string of characters | unit of the failing character. The run-time functions \fBpcre32_exec()\fP and |
in the range 0 to 0x7FFFFFFF by \fBpcre_dfa_exec()\fP and the interpreted | \fBpcre32_dfa_exec()\fP also pass back this information, as well as a more |
version of \fBpcre_exec()\fP. In other words, apart from the initial validity | detailed reason code if the caller has provided memory in which to do this. |
test, these functions (when in UTF-8 mode) handle strings according to the more | |
liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for | |
\fBpcre_exec()\fP supports only RFC 3629. If you are using JIT optimization, or | |
if the string does not even conform to RFC 2279, the result is undefined. Your | |
program may crash. | |
.P |
.P |
If you want to process strings of values in the full range 0 to 0x7FFFFFFF, | In some situations, you may already know that your strings are valid, and |
encoded in a UTF-8-like manner as per the old RFC, you can set | therefore want to skip these checks in order to improve performance. If you set |
PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this | the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that |
situation, you will have to apply your own validity check, and avoid the use of | the pattern or subject it is given (respectively) contains only valid UTF-32 |
JIT optimization. | sequences. In this case, it does not diagnose an invalid UTF-32 string. |
| However, if an invalid string is passed, the result is undefined. |
. |
. |
. |
. |
.SS "General comments about UTF-8 mode" | .SS "General comments about UTF modes" |
.rs |
.rs |
.sp |
.sp |
1. An unbraced hexadecimal escape sequence (such as \exb3) matches a two-byte | 1. Codepoints less than 256 can be specified in patterns by either braced or |
UTF-8 character if the value is greater than 127. | unbraced hexadecimal escape sequences (for example, \ex{b3} or \exb3). Larger |
| values have to use braced sequences. |
.P |
.P |
2. Octal numbers up to \e777 are recognized, and match two-byte UTF-8 | 2. Octal numbers up to \e777 are recognized, and in UTF-8 mode they match |
characters for values greater than \e177. | two-byte characters for values greater than \e177. |
.P |
.P |
3. Repeat quantifiers apply to complete UTF-8 characters, not to individual | 3. Repeat quantifiers apply to complete UTF characters, not to individual |
bytes, for example: \ex{100}{3}. | data units, for example: \ex{100}{3}. |
.P |
.P |
4. The dot metacharacter matches one UTF-8 character instead of a single byte. | 4. The dot metacharacter matches one UTF character instead of a single data |
| unit. |
.P |
.P |
5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, | 5. The escape sequence \eC can be used to match a single byte in UTF-8 mode, or |
but its use can lead to some strange effects because it breaks up multibyte | a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in |
characters (see the description of \eC in the | UTF-32 mode, but its use can lead to some strange effects because it breaks up |
| multi-unit characters (see the description of \eC in the |
.\" HREF |
.\" HREF |
\fBpcrepattern\fP |
\fBpcrepattern\fP |
.\" |
.\" |
documentation). The use of \eC is not supported in the alternative matching |
documentation). The use of \eC is not supported in the alternative matching |
function \fBpcre_dfa_exec()\fP, nor is it supported in UTF-8 mode by the JIT | function \fBpcre[16|32]_dfa_exec()\fP, nor is it supported in UTF mode by the |
optimization of \fBpcre_exec()\fP. If JIT optimization is requested for a UTF-8 | JIT optimization of \fBpcre[16|32]_exec()\fP. If JIT optimization is requested |
pattern that contains \eC, it will not succeed, and so the matching will be | for a UTF pattern that contains \eC, it will not succeed, and so the matching |
carried out by the normal interpretive function. | will be carried out by the normal interpretive function. |
.P |
.P |
6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly |
6. The character escapes \eb, \eB, \ed, \eD, \es, \eS, \ew, and \eW correctly |
test characters of any code value, but, by default, the characters that PCRE |
test characters of any code value, but, by default, the characters that PCRE |
recognizes as digits, spaces, or word characters remain the same set as before, | recognizes as digits, spaces, or word characters remain the same set as in |
all with values less than 256. This remains true even when PCRE is built to | non-UTF mode, all with values less than 256. This remains true even when PCRE |
include Unicode property support, because to do otherwise would slow down PCRE | is built to include Unicode property support, because to do otherwise would |
in many common cases. Note in particular that this applies to \eb and \eB, | slow down PCRE in many common cases. Note in particular that this applies to |
because they are defined in terms of \ew and \eW. If you really want to test | \eb and \eB, because they are defined in terms of \ew and \eW. If you really |
for a wider sense of, say, "digit", you can use explicit Unicode property tests | want to test for a wider sense of, say, "digit", you can use explicit Unicode |
such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option, the way that | property tests such as \ep{Nd}. Alternatively, if you set the PCRE_UCP option, |
the character escapes work is changed so that Unicode properties are used to | the way that the character escapes work is changed so that Unicode properties |
determine which characters match. There are more details in the section on | are used to determine which characters match. There are more details in the |
| section on |
.\" HTML <a href="pcrepattern.html#genericchartypes"> |
.\" HTML <a href="pcrepattern.html#genericchartypes"> |
.\" </a> |
.\" </a> |
generic character types |
generic character types |
Line 135 documentation.
|
Line 218 documentation.
|
7. Similarly, characters that match the POSIX named character classes are all |
7. Similarly, characters that match the POSIX named character classes are all |
low-valued characters, unless the PCRE_UCP option is set. |
low-valued characters, unless the PCRE_UCP option is set. |
.P |
.P |
8. However, the horizontal and vertical whitespace matching escapes (\eh, \eH, | 8. However, the horizontal and vertical white space matching escapes (\eh, \eH, |
\ev, and \eV) do match all the appropriate Unicode characters, whether or not |
\ev, and \eV) do match all the appropriate Unicode characters, whether or not |
PCRE_UCP is set. |
PCRE_UCP is set. |
.P |
.P |
9. Case-insensitive matching applies only to characters whose values are less |
9. Case-insensitive matching applies only to characters whose values are less |
than 128, unless PCRE is built with Unicode property support. Even when Unicode | than 128, unless PCRE is built with Unicode property support. A few Unicode |
property support is available, PCRE still uses its own character tables when | characters such as Greek sigma have more than two codepoints that are |
checking the case of low-valued characters, so as not to degrade performance. | case-equivalent. Up to and including PCRE release 8.31, only one-to-one case |
The Unicode property information is used only for characters with higher | mappings were supported, but later releases (with Unicode property support) do |
values. Furthermore, PCRE supports case-insensitive matching only when there is | treat as case-equivalent all versions of characters such as Greek sigma. |
a one-to-one mapping between a letter's cases. There are a small number of | |
many-to-one mappings in Unicode; these are not supported by PCRE. | |
. |
. |
. |
. |
.SH AUTHOR |
.SH AUTHOR |
Line 163 Cambridge CB2 3QH, England.
|
Line 244 Cambridge CB2 3QH, England.
|
.rs |
.rs |
.sp |
.sp |
.nf |
.nf |
Last updated: 19 October 2011 | Last updated: 27 February 2013 |
Copyright (c) 1997-2011 University of Cambridge. | Copyright (c) 1997-2013 University of Cambridge. |
.fi |
.fi |