-From Release 8.30, in addition to its previous UTF-8 support, PCRE also -supports UTF-16 by means of a separate 16-bit library. This can be built as -well as, or instead of, the 8-bit library. +As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and +UTF-32 (from release 8.32), by means of two additional libraries. They can be +built as well as, or instead of, the 8-bit library.
-In order process UTF-16 strings, you must build PCRE's 16-bit library with UTF -support, and, in addition, you must call -pcre16_compile() -with the PCRE_UTF16 option flag, or the pattern must start with the sequence -(*UTF16). When either of these is the case, both the pattern and any subject -strings that are matched against it are treated as UTF-16 strings instead of -strings of 16-bit characters. +In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or +32-bit library with UTF support, and, in addition, you must call +pcre16_compile() +or +pcre32_compile() +with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively, +the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or +(*UTF), which can be used with either library. When UTF mode is set, both the +pattern and any subject strings that are matched against it are treated as +UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit +characters.
If you compile PCRE with UTF support, but do not use it at run time, the library will be a bit bigger, but the additional run time overhead is limited -to testing the PCRE_UTF8/16 flag occasionally, so should not be very big. +to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big.
When you set the PCRE_UTF8 flag, the byte strings passed as patterns and subjects are (by default) checked for validity on entry to the relevant -functions. From release 7.3 of PCRE, the check is according the rules of RFC -3629, which are themselves derived from the Unicode specification. Earlier -releases of PCRE followed the rules of RFC 2279, which allows the full range of -31-bit values (0 to 0x7FFFFFFF). The current check allows only values in the -range U+0 to U+10FFFF, excluding U+D800 to U+DFFF. +functions. The entire string is checked before any other processing takes +place. From release 7.3 of PCRE, the check is according the rules of RFC 3629, +which are themselves derived from the Unicode specification. Earlier releases +of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit +values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 +to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called +"non-character" code points are no longer excluded because Unicode corrigendum +#9 makes it clear that they should not be.)
-The excluded code points are the "Surrogate Area" of Unicode. They are reserved -for use by UTF-16, where they are used in pairs to encode codepoints with -values greater than 0xFFFF. The code points that are encoded by UTF-16 pairs -are available independently in the UTF-8 encoding. (In other words, the whole -surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8.) +Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, +where they are used in pairs to encode codepoints with values greater than +0xFFFF. The code points that are encoded by UTF-16 pairs are available +independently in the UTF-8 and UTF-32 encodings. (In other words, the whole +surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and +UTF-32.)
If an invalid UTF-8 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first byte -of the failing character. The runtime functions pcre_exec() and +of the failing character. The run-time functions pcre_exec() and pcre_dfa_exec() also pass back this information, as well as a more detailed reason code if the caller has provided memory in which to do this.
In some situations, you may already know that your strings are valid, and -therefore want to skip these checks in order to improve performance. If you set -the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE assumes that -the pattern or subject it is given (respectively) contains only valid UTF-8 -codes. In this case, it does not diagnose an invalid UTF-8 string. +therefore want to skip these checks in order to improve performance, for +example in the case of a long subject string that is being scanned repeatedly. +If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE +assumes that the pattern or subject it is given (respectively) contains only +valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string.
-If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, what -happens depends on why the string is invalid. If the string conforms to the -"old" definition of UTF-8 (RFC 2279), it is processed as a string of characters -in the range 0 to 0x7FFFFFFF by pcre_dfa_exec() and the interpreted -version of pcre_exec(). In other words, apart from the initial validity -test, these functions (when in UTF-8 mode) handle strings according to the more -liberal rules of RFC 2279. However, the just-in-time (JIT) optimization for -pcre_exec() supports only RFC 3629. If you are using JIT optimization, or -if the string does not even conform to RFC 2279, the result is undefined. Your -program may crash. +Note that passing PCRE_NO_UTF8_CHECK to pcre_compile() just disables the +check for the pattern; it does not also apply to subject strings. If you want +to disable the check for a subject string you must pass this option to +pcre_exec() or pcre_dfa_exec().
-If you want to process strings of values in the full range 0 to 0x7FFFFFFF, -encoded in a UTF-8-like manner as per the old RFC, you can set -PCRE_NO_UTF8_CHECK to bypass the more restrictive test. However, in this -situation, you will have to apply your own validity check, and avoid the use of -JIT optimization. +If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result +is undefined and your program may crash.
If an invalid UTF-16 string is passed to PCRE, an error return is given. At compile time, the only additional information is the offset to the first data -unit of the failing character. The runtime functions pcre16_exec() and +unit of the failing character. The run-time functions pcre16_exec() and pcre16_dfa_exec() also pass back this information, as well as a more detailed reason code if the caller has provided memory in which to do this.
@@ -143,17 +145,42 @@ therefore want to skip these checks in order to improv the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that the pattern or subject it is given (respectively) contains only valid UTF-16 sequences. In this case, it does not diagnose an invalid UTF-16 string. +However, if an invalid string is passed, the result is undefined. + ++When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are +passed as patterns and subjects are (by default) checked for validity on entry +to the relevant functions. This check allows only values in the range U+0 +to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF.
++If an invalid UTF-32 string is passed to PCRE, an error return is given. At +compile time, the only additional information is the offset to the first data +unit of the failing character. The run-time functions pcre32_exec() and +pcre32_dfa_exec() also pass back this information, as well as a more +detailed reason code if the caller has provided memory in which to do this. +
++In some situations, you may already know that your strings are valid, and +therefore want to skip these checks in order to improve performance. If you set +the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that +the pattern or subject it is given (respectively) contains only valid UTF-32 +sequences. In this case, it does not diagnose an invalid UTF-32 string. +However, if an invalid string is passed, the result is undefined. +
-1. Codepoints less than 256 can be specified by either braced or unbraced -hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger values -have to use braced sequences. +1. Codepoints less than 256 can be specified in patterns by either braced or +unbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger +values have to use braced sequences.
-2. Octal numbers up to \777 are recognized, and in UTF-8 mode, they match +2. Octal numbers up to \777 are recognized, and in UTF-8 mode they match two-byte characters for values greater than \177.
@@ -166,15 +193,15 @@ unit.
5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or -a single 16-bit data unit in UTF-16 mode, but its use can lead to some strange -effects because it breaks up multi-unit characters (see the description of \C -in the +a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in +UTF-32 mode, but its use can lead to some strange effects because it breaks up +multi-unit characters (see the description of \C in the pcrepattern documentation). The use of \C is not supported in the alternative matching -function pcre[16]_dfa_exec(), nor is it supported in UTF mode by the JIT -optimization of pcre[16]_exec(). If JIT optimization is requested for a -UTF pattern that contains \C, it will not succeed, and so the matching will -be carried out by the normal interpretive function. +function pcre[16|32]_dfa_exec(), nor is it supported in UTF mode by the +JIT optimization of pcre[16|32]_exec(). If JIT optimization is requested +for a UTF pattern that contains \C, it will not succeed, and so the matching +will be carried out by the normal interpretive function.
6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly @@ -199,19 +226,17 @@ documentation. low-valued characters, unless the PCRE_UCP option is set.
-8. However, the horizontal and vertical whitespace matching escapes (\h, \H, +8. However, the horizontal and vertical white space matching escapes (\h, \H, \v, and \V) do match all the appropriate Unicode characters, whether or not PCRE_UCP is set.
9. Case-insensitive matching applies only to characters whose values are less -than 128, unless PCRE is built with Unicode property support. Even when Unicode -property support is available, PCRE still uses its own character tables when -checking the case of low-valued characters, so as not to degrade performance. -The Unicode property information is used only for characters with higher -values. Furthermore, PCRE supports case-insensitive matching only when there is -a one-to-one mapping between a letter's cases. There are a small number of -many-to-one mappings in Unicode; these are not supported by PCRE. +than 128, unless PCRE is built with Unicode property support. A few Unicode +characters such as Greek sigma have more than two codepoints that are +case-equivalent. Up to and including PCRE release 8.31, only one-to-one case +mappings were supported, but later releases (with Unicode property support) do +treat as case-equivalent all versions of characters such as Greek sigma.
-Last updated: 13 January 2012
+Last updated: 27 February 2013
-Copyright © 1997-2012 University of Cambridge.
+Copyright © 1997-2013 University of Cambridge.
Return to the PCRE index page.