embedaddon/pcre/doc/html/pcreunicode.html - view

File: [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html / pcreunicode.html
Revision 1.1.1.4 (vendor branch): download - view: text, annotated - select for diffs - revision graph
Mon Jul 22 08:25:57 2013 UTC (11 years, 8 months ago) by misho
Branches: pcre, MAIN
CVS tags: v8_34, v8_33, HEAD

8.33

1: <html> 2: <head> 3: <title>pcreunicode specification</title> 4: </head> 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6: <h1>pcreunicode man page</h1> 7:  8: Return to the <a href="index.html">PCRE index page</a>. 9:  10:  11: This page is part of the PCRE HTML documentation. It was generated automatically 12: from the original man page. If there is any nonsense in it, please consult the 13: man page, in case the conversion went wrong. 14:   15:   16: UTF-8, UTF-16, UTF-32, AND UNICODE PROPERTY SUPPORT 17:   18:  19: As well as UTF-8 support, PCRE also supports UTF-16 (from release 8.30) and 20: UTF-32 (from release 8.32), by means of two additional libraries. They can be 21: built as well as, or instead of, the 8-bit library. 22:  23:   24: UTF-8 SUPPORT 25:   26:  27: In order process UTF-8 strings, you must build PCRE's 8-bit library with UTF 28: support, and, in addition, you must call 29: <a href="pcre_compile.html">pcre_compile()</a> 30: with the PCRE_UTF8 option flag, or the pattern must start with the sequence 31: (*UTF8) or (*UTF). When either of these is the case, both the pattern and any 32: subject strings that are matched against it are treated as UTF-8 strings 33: instead of strings of individual 1-byte characters. 34:  35:   36: UTF-16 AND UTF-32 SUPPORT 37:   38:  39: In order process UTF-16 or UTF-32 strings, you must build PCRE's 16-bit or 40: 32-bit library with UTF support, and, in addition, you must call 41: <a href="pcre16_compile.html">pcre16_compile()</a> 42: or 43: <a href="pcre32_compile.html">pcre32_compile()</a> 44: with the PCRE_UTF16 or PCRE_UTF32 option flag, as appropriate. Alternatively, 45: the pattern must start with the sequence (*UTF16), (*UTF32), as appropriate, or 46: (*UTF), which can be used with either library. When UTF mode is set, both the 47: pattern and any subject strings that are matched against it are treated as 48: UTF-16 or UTF-32 strings instead of strings of individual 16-bit or 32-bit 49: characters. 50:  51:   52: UTF SUPPORT OVERHEAD 53:   54:  55: If you compile PCRE with UTF support, but do not use it at run time, the 56: library will be a bit bigger, but the additional run time overhead is limited 57: to testing the PCRE_UTF[8|16|32] flag occasionally, so should not be very big. 58:  59:   60: UNICODE PROPERTY SUPPORT 61:   62:  63: If PCRE is built with Unicode character property support (which implies UTF 64: support), the escape sequences \p{..}, \P{..}, and \X can be used. 65: The available properties that can be tested are limited to the general 66: category properties such as Lu for an upper case letter or Nd for a decimal 67: number, the Unicode script names such as Arabic or Han, and the derived 68: properties Any and L&. Full lists is given in the 69: <a href="pcrepattern.html">pcrepattern</a> 70: and 71: <a href="pcresyntax.html">pcresyntax</a> 72: documentation. Only the short names for properties are supported. For example, 73: \p{L} matches a letter. Its Perl synonym, \p{Letter}, is not supported. 74: Furthermore, in Perl, many properties may optionally be prefixed by "Is", for 75: compatibility with Perl 5.6. PCRE does not support this. 76: <a name="utf8strings"></a> 77: 78: Validity of UTF-8 strings 79: 80: 81: When you set the PCRE_UTF8 flag, the byte strings passed as patterns and 82: subjects are (by default) checked for validity on entry to the relevant 83: functions. The entire string is checked before any other processing takes 84: place. From release 7.3 of PCRE, the check is according the rules of RFC 3629, 85: which are themselves derived from the Unicode specification. Earlier releases 86: of PCRE followed the rules of RFC 2279, which allows the full range of 31-bit 87: values (0 to 0x7FFFFFFF). The current check allows only values in the range U+0 88: to U+10FFFF, excluding the surrogate area. (From release 8.33 the so-called 89: "non-character" code points are no longer excluded because Unicode corrigendum 90: #9 makes it clear that they should not be.) 91: 92: 93: Characters in the "Surrogate Area" of Unicode are reserved for use by UTF-16, 94: where they are used in pairs to encode codepoints with values greater than 95: 0xFFFF. The code points that are encoded by UTF-16 pairs are available 96: independently in the UTF-8 and UTF-32 encodings. (In other words, the whole 97: surrogate thing is a fudge for UTF-16 which unfortunately messes up UTF-8 and 98: UTF-32.) 99: 100: 101: If an invalid UTF-8 string is passed to PCRE, an error return is given. At 102: compile time, the only additional information is the offset to the first byte 103: of the failing character. The run-time functions pcre_exec() and 104: pcre_dfa_exec() also pass back this information, as well as a more 105: detailed reason code if the caller has provided memory in which to do this. 106: 107: 108: In some situations, you may already know that your strings are valid, and 109: therefore want to skip these checks in order to improve performance, for 110: example in the case of a long subject string that is being scanned repeatedly. 111: If you set the PCRE_NO_UTF8_CHECK flag at compile time or at run time, PCRE 112: assumes that the pattern or subject it is given (respectively) contains only 113: valid UTF-8 codes. In this case, it does not diagnose an invalid UTF-8 string. 114: 115: 116: Note that passing PCRE_NO_UTF8_CHECK to pcre_compile() just disables the 117: check for the pattern; it does not also apply to subject strings. If you want 118: to disable the check for a subject string you must pass this option to 119: pcre_exec() or pcre_dfa_exec(). 120:  121:  122: If you pass an invalid UTF-8 string when PCRE_NO_UTF8_CHECK is set, the result 123: is undefined and your program may crash. 124: <a name="utf16strings"></a> 125:   126: Validity of UTF-16 strings 127:   128:  129: When you set the PCRE_UTF16 flag, the strings of 16-bit data units that are 130: passed as patterns and subjects are (by default) checked for validity on entry 131: to the relevant functions. Values other than those in the surrogate range 132: U+D800 to U+DFFF are independent code points. Values in the surrogate range 133: must be used in pairs in the correct manner. 134:  135:  136: If an invalid UTF-16 string is passed to PCRE, an error return is given. At 137: compile time, the only additional information is the offset to the first data 138: unit of the failing character. The run-time functions pcre16_exec() and 139: pcre16_dfa_exec() also pass back this information, as well as a more 140: detailed reason code if the caller has provided memory in which to do this. 141:  142:  143: In some situations, you may already know that your strings are valid, and 144: therefore want to skip these checks in order to improve performance. If you set 145: the PCRE_NO_UTF16_CHECK flag at compile time or at run time, PCRE assumes that 146: the pattern or subject it is given (respectively) contains only valid UTF-16 147: sequences. In this case, it does not diagnose an invalid UTF-16 string. 148: However, if an invalid string is passed, the result is undefined. 149: <a name="utf32strings"></a> 150:   151: Validity of UTF-32 strings 152:   153:  154: When you set the PCRE_UTF32 flag, the strings of 32-bit data units that are 155: passed as patterns and subjects are (by default) checked for validity on entry 156: to the relevant functions. This check allows only values in the range U+0 157: to U+10FFFF, excluding the surrogate area U+D800 to U+DFFF. 158:  159:  160: If an invalid UTF-32 string is passed to PCRE, an error return is given. At 161: compile time, the only additional information is the offset to the first data 162: unit of the failing character. The run-time functions pcre32_exec() and 163: pcre32_dfa_exec() also pass back this information, as well as a more 164: detailed reason code if the caller has provided memory in which to do this. 165:  166:  167: In some situations, you may already know that your strings are valid, and 168: therefore want to skip these checks in order to improve performance. If you set 169: the PCRE_NO_UTF32_CHECK flag at compile time or at run time, PCRE assumes that 170: the pattern or subject it is given (respectively) contains only valid UTF-32 171: sequences. In this case, it does not diagnose an invalid UTF-32 string. 172: However, if an invalid string is passed, the result is undefined. 173:  174:   175: General comments about UTF modes 176:   177:  178: 1. Codepoints less than 256 can be specified in patterns by either braced or 179: unbraced hexadecimal escape sequences (for example, \x{b3} or \xb3). Larger 180: values have to use braced sequences. 181:  182:  183: 2. Octal numbers up to \777 are recognized, and in UTF-8 mode they match 184: two-byte characters for values greater than \177. 185:  186:  187: 3. Repeat quantifiers apply to complete UTF characters, not to individual 188: data units, for example: \x{100}{3}. 189:  190:  191: 4. The dot metacharacter matches one UTF character instead of a single data 192: unit. 193:  194:  195: 5. The escape sequence \C can be used to match a single byte in UTF-8 mode, or 196: a single 16-bit data unit in UTF-16 mode, or a single 32-bit data unit in 197: UTF-32 mode, but its use can lead to some strange effects because it breaks up 198: multi-unit characters (see the description of \C in the 199: <a href="pcrepattern.html">pcrepattern</a> 200: documentation). The use of \C is not supported in the alternative matching 201: function pcre[16|32]_dfa_exec(), nor is it supported in UTF mode by the 202: JIT optimization of pcre[16|32]_exec(). If JIT optimization is requested 203: for a UTF pattern that contains \C, it will not succeed, and so the matching 204: will be carried out by the normal interpretive function. 205:  206:  207: 6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly 208: test characters of any code value, but, by default, the characters that PCRE 209: recognizes as digits, spaces, or word characters remain the same set as in 210: non-UTF mode, all with values less than 256. This remains true even when PCRE 211: is built to include Unicode property support, because to do otherwise would 212: slow down PCRE in many common cases. Note in particular that this applies to 213: \b and \B, because they are defined in terms of \w and \W. If you really 214: want to test for a wider sense of, say, "digit", you can use explicit Unicode 215: property tests such as \p{Nd}. Alternatively, if you set the PCRE_UCP option, 216: the way that the character escapes work is changed so that Unicode properties 217: are used to determine which characters match. There are more details in the 218: section on 219: <a href="pcrepattern.html#genericchartypes">generic character types</a> 220: in the 221: <a href="pcrepattern.html">pcrepattern</a> 222: documentation. 223:  224:  225: 7. Similarly, characters that match the POSIX named character classes are all 226: low-valued characters, unless the PCRE_UCP option is set. 227:  228:  229: 8. However, the horizontal and vertical white space matching escapes (\h, \H, 230: \v, and \V) do match all the appropriate Unicode characters, whether or not 231: PCRE_UCP is set. 232:  233:  234: 9. Case-insensitive matching applies only to characters whose values are less 235: than 128, unless PCRE is built with Unicode property support. A few Unicode 236: characters such as Greek sigma have more than two codepoints that are 237: case-equivalent. Up to and including PCRE release 8.31, only one-to-one case 238: mappings were supported, but later releases (with Unicode property support) do 239: treat as case-equivalent all versions of characters such as Greek sigma. 240:  241:   242: AUTHOR 243:   244:  245: Philip Hazel 246:   247: University Computing Service 248:   249: Cambridge CB2 3QH, England. 250:   251:  252:   253: REVISION 254:   255:  256: Last updated: 27 February 2013 257:   258: Copyright © 1997-2013 University of Cambridge. 259:   260:  261: Return to the <a href="index.html">PCRE index page</a>. 262: 