1: .TH PCREPATTERN 3 "03 December 2013" "PCRE 8.34"
2: .SH NAME
3: PCRE - Perl-compatible regular expressions
4: .SH "PCRE REGULAR EXPRESSION DETAILS"
5: .rs
6: .sp
7: The syntax and semantics of the regular expressions that are supported by PCRE
8: are described in detail below. There is a quick-reference syntax summary in the
9: .\" HREF
10: \fBpcresyntax\fP
11: .\"
12: page. PCRE tries to match Perl syntax and semantics as closely as it can. PCRE
13: also supports some alternative regular expression syntax (which does not
14: conflict with the Perl syntax) in order to provide some compatibility with
15: regular expressions in Python, .NET, and Oniguruma.
16: .P
17: Perl's regular expressions are described in its own documentation, and
18: regular expressions in general are covered in a number of books, some of which
19: have copious examples. Jeffrey Friedl's "Mastering Regular Expressions",
20: published by O'Reilly, covers regular expressions in great detail. This
21: description of PCRE's regular expressions is intended as reference material.
22: .P
23: This document discusses the patterns that are supported by PCRE when one its
24: main matching functions, \fBpcre_exec()\fP (8-bit) or \fBpcre[16|32]_exec()\fP
25: (16- or 32-bit), is used. PCRE also has alternative matching functions,
26: \fBpcre_dfa_exec()\fP and \fBpcre[16|32_dfa_exec()\fP, which match using a
27: different algorithm that is not Perl-compatible. Some of the features discussed
28: below are not available when DFA matching is used. The advantages and
29: disadvantages of the alternative functions, and how they differ from the normal
30: functions, are discussed in the
31: .\" HREF
32: \fBpcrematching\fP
33: .\"
34: page.
35: .
36: .
37: .SH "SPECIAL START-OF-PATTERN ITEMS"
38: .rs
39: .sp
40: A number of options that can be passed to \fBpcre_compile()\fP can also be set
41: by special items at the start of a pattern. These are not Perl-compatible, but
42: are provided to make these options accessible to pattern writers who are not
43: able to change the program that processes the pattern. Any number of these
44: items may appear, but they must all be together right at the start of the
45: pattern string, and the letters must be in upper case.
46: .
47: .
48: .SS "UTF support"
49: .rs
50: .sp
51: The original operation of PCRE was on strings of one-byte characters. However,
52: there is now also support for UTF-8 strings in the original library, an
53: extra library that supports 16-bit and UTF-16 character strings, and a
54: third library that supports 32-bit and UTF-32 character strings. To use these
55: features, PCRE must be built to include appropriate support. When using UTF
56: strings you must either call the compiling function with the PCRE_UTF8,
57: PCRE_UTF16, or PCRE_UTF32 option, or the pattern must start with one of
58: these special sequences:
59: .sp
60: (*UTF8)
61: (*UTF16)
62: (*UTF32)
63: (*UTF)
64: .sp
65: (*UTF) is a generic sequence that can be used with any of the libraries.
66: Starting a pattern with such a sequence is equivalent to setting the relevant
67: option. How setting a UTF mode affects pattern matching is mentioned in several
68: places below. There is also a summary of features in the
69: .\" HREF
70: \fBpcreunicode\fP
71: .\"
72: page.
73: .P
74: Some applications that allow their users to supply patterns may wish to
75: restrict them to non-UTF data for security reasons. If the PCRE_NEVER_UTF
76: option is set at compile time, (*UTF) etc. are not allowed, and their
77: appearance causes an error.
78: .
79: .
80: .SS "Unicode property support"
81: .rs
82: .sp
83: Another special sequence that may appear at the start of a pattern is (*UCP).
84: This has the same effect as setting the PCRE_UCP option: it causes sequences
85: such as \ed and \ew to use Unicode properties to determine character types,
86: instead of recognizing only characters with codes less than 128 via a lookup
87: table.
88: .
89: .
90: .SS "Disabling auto-possessification"
91: .rs
92: .sp
93: If a pattern starts with (*NO_AUTO_POSSESS), it has the same effect as setting
94: the PCRE_NO_AUTO_POSSESS option at compile time. This stops PCRE from making
95: quantifiers possessive when what follows cannot match the repeated item. For
96: example, by default a+b is treated as a++b. For more details, see the
97: .\" HREF
98: \fBpcreapi\fP
99: .\"
100: documentation.
101: .
102: .
103: .SS "Disabling start-up optimizations"
104: .rs
105: .sp
106: If a pattern starts with (*NO_START_OPT), it has the same effect as setting the
107: PCRE_NO_START_OPTIMIZE option either at compile or matching time. This disables
108: several optimizations for quickly reaching "no match" results. For more
109: details, see the
110: .\" HREF
111: \fBpcreapi\fP
112: .\"
113: documentation.
114: .
115: .
116: .\" HTML <a name="newlines"></a>
117: .SS "Newline conventions"
118: .rs
119: .sp
120: PCRE supports five different conventions for indicating line breaks in
121: strings: a single CR (carriage return) character, a single LF (linefeed)
122: character, the two-character sequence CRLF, any of the three preceding, or any
123: Unicode newline sequence. The
124: .\" HREF
125: \fBpcreapi\fP
126: .\"
127: page has
128: .\" HTML <a href="pcreapi.html#newlines">
129: .\" </a>
130: further discussion
131: .\"
132: about newlines, and shows how to set the newline convention in the
133: \fIoptions\fP arguments for the compiling and matching functions.
134: .P
135: It is also possible to specify a newline convention by starting a pattern
136: string with one of the following five sequences:
137: .sp
138: (*CR) carriage return
139: (*LF) linefeed
140: (*CRLF) carriage return, followed by linefeed
141: (*ANYCRLF) any of the three above
142: (*ANY) all Unicode newline sequences
143: .sp
144: These override the default and the options given to the compiling function. For
145: example, on a Unix system where LF is the default newline sequence, the pattern
146: .sp
147: (*CR)a.b
148: .sp
149: changes the convention to CR. That pattern matches "a\enb" because LF is no
150: longer a newline. If more than one of these settings is present, the last one
151: is used.
152: .P
153: The newline convention affects where the circumflex and dollar assertions are
154: true. It also affects the interpretation of the dot metacharacter when
155: PCRE_DOTALL is not set, and the behaviour of \eN. However, it does not affect
156: what the \eR escape sequence matches. By default, this is any Unicode newline
157: sequence, for Perl compatibility. However, this can be changed; see the
158: description of \eR in the section entitled
159: .\" HTML <a href="#newlineseq">
160: .\" </a>
161: "Newline sequences"
162: .\"
163: below. A change of \eR setting can be combined with a change of newline
164: convention.
165: .
166: .
167: .SS "Setting match and recursion limits"
168: .rs
169: .sp
170: The caller of \fBpcre_exec()\fP can set a limit on the number of times the
171: internal \fBmatch()\fP function is called and on the maximum depth of
172: recursive calls. These facilities are provided to catch runaway matches that
173: are provoked by patterns with huge matching trees (a typical example is a
174: pattern with nested unlimited repeats) and to avoid running out of system stack
175: by too much recursion. When one of these limits is reached, \fBpcre_exec()\fP
176: gives an error return. The limits can also be set by items at the start of the
177: pattern of the form
178: .sp
179: (*LIMIT_MATCH=d)
180: (*LIMIT_RECURSION=d)
181: .sp
182: where d is any number of decimal digits. However, the value of the setting must
183: be less than the value set (or defaulted) by the caller of \fBpcre_exec()\fP
184: for it to have any effect. In other words, the pattern writer can lower the
185: limits set by the programmer, but not raise them. If there is more than one
186: setting of one of these limits, the lower value is used.
187: .
188: .
189: .SH "EBCDIC CHARACTER CODES"
190: .rs
191: .sp
192: PCRE can be compiled to run in an environment that uses EBCDIC as its character
193: code rather than ASCII or Unicode (typically a mainframe system). In the
194: sections below, character code values are ASCII or Unicode; in an EBCDIC
195: environment these characters may have different code values, and there are no
196: code points greater than 255.
197: .
198: .
199: .SH "CHARACTERS AND METACHARACTERS"
200: .rs
201: .sp
202: A regular expression is a pattern that is matched against a subject string from
203: left to right. Most characters stand for themselves in a pattern, and match the
204: corresponding characters in the subject. As a trivial example, the pattern
205: .sp
206: The quick brown fox
207: .sp
208: matches a portion of a subject string that is identical to itself. When
209: caseless matching is specified (the PCRE_CASELESS option), letters are matched
210: independently of case. In a UTF mode, PCRE always understands the concept of
211: case for characters whose values are less than 128, so caseless matching is
212: always possible. For characters with higher values, the concept of case is
213: supported if PCRE is compiled with Unicode property support, but not otherwise.
214: If you want to use caseless matching for characters 128 and above, you must
215: ensure that PCRE is compiled with Unicode property support as well as with
216: UTF support.
217: .P
218: The power of regular expressions comes from the ability to include alternatives
219: and repetitions in the pattern. These are encoded in the pattern by the use of
220: \fImetacharacters\fP, which do not stand for themselves but instead are
221: interpreted in some special way.
222: .P
223: There are two different sets of metacharacters: those that are recognized
224: anywhere in the pattern except within square brackets, and those that are
225: recognized within square brackets. Outside square brackets, the metacharacters
226: are as follows:
227: .sp
228: \e general escape character with several uses
229: ^ assert start of string (or line, in multiline mode)
230: $ assert end of string (or line, in multiline mode)
231: . match any character except newline (by default)
232: [ start character class definition
233: | start of alternative branch
234: ( start subpattern
235: ) end subpattern
236: ? extends the meaning of (
237: also 0 or 1 quantifier
238: also quantifier minimizer
239: * 0 or more quantifier
240: + 1 or more quantifier
241: also "possessive quantifier"
242: { start min/max quantifier
243: .sp
244: Part of a pattern that is in square brackets is called a "character class". In
245: a character class the only metacharacters are:
246: .sp
247: \e general escape character
248: ^ negate the class, but only if the first character
249: - indicates character range
250: .\" JOIN
251: [ POSIX character class (only if followed by POSIX
252: syntax)
253: ] terminates the character class
254: .sp
255: The following sections describe the use of each of the metacharacters.
256: .
257: .
258: .SH BACKSLASH
259: .rs
260: .sp
261: The backslash character has several uses. Firstly, if it is followed by a
262: character that is not a number or a letter, it takes away any special meaning
263: that character may have. This use of backslash as an escape character applies
264: both inside and outside character classes.
265: .P
266: For example, if you want to match a * character, you write \e* in the pattern.
267: This escaping action applies whether or not the following character would
268: otherwise be interpreted as a metacharacter, so it is always safe to precede a
269: non-alphanumeric with backslash to specify that it stands for itself. In
270: particular, if you want to match a backslash, you write \e\e.
271: .P
272: In a UTF mode, only ASCII numbers and letters have any special meaning after a
273: backslash. All other characters (in particular, those whose codepoints are
274: greater than 127) are treated as literals.
275: .P
276: If a pattern is compiled with the PCRE_EXTENDED option, most white space in the
277: pattern (other than in a character class), and characters between a # outside a
278: character class and the next newline, inclusive, are ignored. An escaping
279: backslash can be used to include a white space or # character as part of the
280: pattern.
281: .P
282: If you want to remove the special meaning from a sequence of characters, you
283: can do so by putting them between \eQ and \eE. This is different from Perl in
284: that $ and @ are handled as literals in \eQ...\eE sequences in PCRE, whereas in
285: Perl, $ and @ cause variable interpolation. Note the following examples:
286: .sp
287: Pattern PCRE matches Perl matches
288: .sp
289: .\" JOIN
290: \eQabc$xyz\eE abc$xyz abc followed by the
291: contents of $xyz
292: \eQabc\e$xyz\eE abc\e$xyz abc\e$xyz
293: \eQabc\eE\e$\eQxyz\eE abc$xyz abc$xyz
294: .sp
295: The \eQ...\eE sequence is recognized both inside and outside character classes.
296: An isolated \eE that is not preceded by \eQ is ignored. If \eQ is not followed
297: by \eE later in the pattern, the literal interpretation continues to the end of
298: the pattern (that is, \eE is assumed at the end). If the isolated \eQ is inside
299: a character class, this causes an error, because the character class is not
300: terminated.
301: .
302: .
303: .\" HTML <a name="digitsafterbackslash"></a>
304: .SS "Non-printing characters"
305: .rs
306: .sp
307: A second use of backslash provides a way of encoding non-printing characters
308: in patterns in a visible manner. There is no restriction on the appearance of
309: non-printing characters, apart from the binary zero that terminates a pattern,
310: but when a pattern is being prepared by text editing, it is often easier to use
311: one of the following escape sequences than the binary character it represents:
312: .sp
313: \ea alarm, that is, the BEL character (hex 07)
314: \ecx "control-x", where x is any ASCII character
315: \ee escape (hex 1B)
316: \ef form feed (hex 0C)
317: \en linefeed (hex 0A)
318: \er carriage return (hex 0D)
319: \et tab (hex 09)
320: \e0dd character with octal code 0dd
321: \eddd character with octal code ddd, or back reference
322: \eo{ddd..} character with octal code ddd..
323: \exhh character with hex code hh
324: \ex{hhh..} character with hex code hhh.. (non-JavaScript mode)
325: \euhhhh character with hex code hhhh (JavaScript mode only)
326: .sp
327: The precise effect of \ecx on ASCII characters is as follows: if x is a lower
328: case letter, it is converted to upper case. Then bit 6 of the character (hex
329: 40) is inverted. Thus \ecA to \ecZ become hex 01 to hex 1A (A is 41, Z is 5A),
330: but \ec{ becomes hex 3B ({ is 7B), and \ec; becomes hex 7B (; is 3B). If the
331: data item (byte or 16-bit value) following \ec has a value greater than 127, a
332: compile-time error occurs. This locks out non-ASCII characters in all modes.
333: .P
334: The \ec facility was designed for use with ASCII characters, but with the
335: extension to Unicode it is even less useful than it once was. It is, however,
336: recognized when PCRE is compiled in EBCDIC mode, where data items are always
337: bytes. In this mode, all values are valid after \ec. If the next character is a
338: lower case letter, it is converted to upper case. Then the 0xc0 bits of the
339: byte are inverted. Thus \ecA becomes hex 01, as in ASCII (A is C1), but because
340: the EBCDIC letters are disjoint, \ecZ becomes hex 29 (Z is E9), and other
341: characters also generate different values.
342: .P
343: After \e0 up to two further octal digits are read. If there are fewer than two
344: digits, just those that are present are used. Thus the sequence \e0\ex\e07
345: specifies two binary zeros followed by a BEL character (code value 7). Make
346: sure you supply two digits after the initial zero if the pattern character that
347: follows is itself an octal digit.
348: .P
349: The escape \eo must be followed by a sequence of octal digits, enclosed in
350: braces. An error occurs if this is not the case. This escape is a recent
351: addition to Perl; it provides way of specifying character code points as octal
352: numbers greater than 0777, and it also allows octal numbers and back references
353: to be unambiguously specified.
354: .P
355: For greater clarity and unambiguity, it is best to avoid following \e by a
356: digit greater than zero. Instead, use \eo{} or \ex{} to specify character
357: numbers, and \eg{} to specify back references. The following paragraphs
358: describe the old, ambiguous syntax.
359: .P
360: The handling of a backslash followed by a digit other than 0 is complicated,
361: and Perl has changed in recent releases, causing PCRE also to change. Outside a
362: character class, PCRE reads the digit and any following digits as a decimal
363: number. If the number is less than 8, or if there have been at least that many
364: previous capturing left parentheses in the expression, the entire sequence is
365: taken as a \fIback reference\fP. A description of how this works is given
366: .\" HTML <a href="#backreferences">
367: .\" </a>
368: later,
369: .\"
370: following the discussion of
371: .\" HTML <a href="#subpattern">
372: .\" </a>
373: parenthesized subpatterns.
374: .\"
375: .P
376: Inside a character class, or if the decimal number following \e is greater than
377: 7 and there have not been that many capturing subpatterns, PCRE handles \e8 and
378: \e9 as the literal characters "8" and "9", and otherwise re-reads up to three
379: octal digits following the backslash, using them to generate a data character.
380: Any subsequent digits stand for themselves. For example:
381: .sp
382: \e040 is another way of writing an ASCII space
383: .\" JOIN
384: \e40 is the same, provided there are fewer than 40
385: previous capturing subpatterns
386: \e7 is always a back reference
387: .\" JOIN
388: \e11 might be a back reference, or another way of
389: writing a tab
390: \e011 is always a tab
391: \e0113 is a tab followed by the character "3"
392: .\" JOIN
393: \e113 might be a back reference, otherwise the
394: character with octal code 113
395: .\" JOIN
396: \e377 might be a back reference, otherwise
397: the value 255 (decimal)
398: .\" JOIN
399: \e81 is either a back reference, or the two
400: characters "8" and "1"
401: .sp
402: Note that octal values of 100 or greater that are specified using this syntax
403: must not be introduced by a leading zero, because no more than three octal
404: digits are ever read.
405: .P
406: By default, after \ex that is not followed by {, from zero to two hexadecimal
407: digits are read (letters can be in upper or lower case). Any number of
408: hexadecimal digits may appear between \ex{ and }. If a character other than
409: a hexadecimal digit appears between \ex{ and }, or if there is no terminating
410: }, an error occurs.
411: .P
412: If the PCRE_JAVASCRIPT_COMPAT option is set, the interpretation of \ex is
413: as just described only when it is followed by two hexadecimal digits.
414: Otherwise, it matches a literal "x" character. In JavaScript mode, support for
415: code points greater than 256 is provided by \eu, which must be followed by
416: four hexadecimal digits; otherwise it matches a literal "u" character.
417: .P
418: Characters whose value is less than 256 can be defined by either of the two
419: syntaxes for \ex (or by \eu in JavaScript mode). There is no difference in the
420: way they are handled. For example, \exdc is exactly the same as \ex{dc} (or
421: \eu00dc in JavaScript mode).
422: .
423: .
424: .SS "Constraints on character values"
425: .rs
426: .sp
427: Characters that are specified using octal or hexadecimal numbers are
428: limited to certain values, as follows:
429: .sp
430: 8-bit non-UTF mode less than 0x100
431: 8-bit UTF-8 mode less than 0x10ffff and a valid codepoint
432: 16-bit non-UTF mode less than 0x10000
433: 16-bit UTF-16 mode less than 0x10ffff and a valid codepoint
434: 32-bit non-UTF mode less than 0x100000000
435: 32-bit UTF-32 mode less than 0x10ffff and a valid codepoint
436: .sp
437: Invalid Unicode codepoints are the range 0xd800 to 0xdfff (the so-called
438: "surrogate" codepoints), and 0xffef.
439: .
440: .
441: .SS "Escape sequences in character classes"
442: .rs
443: .sp
444: All the sequences that define a single character value can be used both inside
445: and outside character classes. In addition, inside a character class, \eb is
446: interpreted as the backspace character (hex 08).
447: .P
448: \eN is not allowed in a character class. \eB, \eR, and \eX are not special
449: inside a character class. Like other unrecognized escape sequences, they are
450: treated as the literal characters "B", "R", and "X" by default, but cause an
451: error if the PCRE_EXTRA option is set. Outside a character class, these
452: sequences have different meanings.
453: .
454: .
455: .SS "Unsupported escape sequences"
456: .rs
457: .sp
458: In Perl, the sequences \el, \eL, \eu, and \eU are recognized by its string
459: handler and used to modify the case of following characters. By default, PCRE
460: does not support these escape sequences. However, if the PCRE_JAVASCRIPT_COMPAT
461: option is set, \eU matches a "U" character, and \eu can be used to define a
462: character by code point, as described in the previous section.
463: .
464: .
465: .SS "Absolute and relative back references"
466: .rs
467: .sp
468: The sequence \eg followed by an unsigned or a negative number, optionally
469: enclosed in braces, is an absolute or relative back reference. A named back
470: reference can be coded as \eg{name}. Back references are discussed
471: .\" HTML <a href="#backreferences">
472: .\" </a>
473: later,
474: .\"
475: following the discussion of
476: .\" HTML <a href="#subpattern">
477: .\" </a>
478: parenthesized subpatterns.
479: .\"
480: .
481: .
482: .SS "Absolute and relative subroutine calls"
483: .rs
484: .sp
485: For compatibility with Oniguruma, the non-Perl syntax \eg followed by a name or
486: a number enclosed either in angle brackets or single quotes, is an alternative
487: syntax for referencing a subpattern as a "subroutine". Details are discussed
488: .\" HTML <a href="#onigurumasubroutines">
489: .\" </a>
490: later.
491: .\"
492: Note that \eg{...} (Perl syntax) and \eg<...> (Oniguruma syntax) are \fInot\fP
493: synonymous. The former is a back reference; the latter is a
494: .\" HTML <a href="#subpatternsassubroutines">
495: .\" </a>
496: subroutine
497: .\"
498: call.
499: .
500: .
501: .\" HTML <a name="genericchartypes"></a>
502: .SS "Generic character types"
503: .rs
504: .sp
505: Another use of backslash is for specifying generic character types:
506: .sp
507: \ed any decimal digit
508: \eD any character that is not a decimal digit
509: \eh any horizontal white space character
510: \eH any character that is not a horizontal white space character
511: \es any white space character
512: \eS any character that is not a white space character
513: \ev any vertical white space character
514: \eV any character that is not a vertical white space character
515: \ew any "word" character
516: \eW any "non-word" character
517: .sp
518: There is also the single sequence \eN, which matches a non-newline character.
519: This is the same as
520: .\" HTML <a href="#fullstopdot">
521: .\" </a>
522: the "." metacharacter
523: .\"
524: when PCRE_DOTALL is not set. Perl also uses \eN to match characters by name;
525: PCRE does not support this.
526: .P
527: Each pair of lower and upper case escape sequences partitions the complete set
528: of characters into two disjoint sets. Any given character matches one, and only
529: one, of each pair. The sequences can appear both inside and outside character
530: classes. They each match one character of the appropriate type. If the current
531: matching point is at the end of the subject string, all of them fail, because
532: there is no character to match.
533: .P
534: For compatibility with Perl, \es did not used to match the VT character (code
535: 11), which made it different from the the POSIX "space" class. However, Perl
536: added VT at release 5.18, and PCRE followed suit at release 8.34. The default
537: \es characters are now HT (9), LF (10), VT (11), FF (12), CR (13), and space
538: (32), which are defined as white space in the "C" locale. This list may vary if
539: locale-specific matching is taking place. For example, in some locales the
540: "non-breaking space" character (\exA0) is recognized as white space, and in
541: others the VT character is not.
542: .P
543: A "word" character is an underscore or any character that is a letter or digit.
544: By default, the definition of letters and digits is controlled by PCRE's
545: low-valued character tables, and may vary if locale-specific matching is taking
546: place (see
547: .\" HTML <a href="pcreapi.html#localesupport">
548: .\" </a>
549: "Locale support"
550: .\"
551: in the
552: .\" HREF
553: \fBpcreapi\fP
554: .\"
555: page). For example, in a French locale such as "fr_FR" in Unix-like systems,
556: or "french" in Windows, some character codes greater than 127 are used for
557: accented letters, and these are then matched by \ew. The use of locales with
558: Unicode is discouraged.
559: .P
560: By default, characters whose code points are greater than 127 never match \ed,
561: \es, or \ew, and always match \eD, \eS, and \eW, although this may vary for
562: characters in the range 128-255 when locale-specific matching is happening.
563: These escape sequences retain their original meanings from before Unicode
564: support was available, mainly for efficiency reasons. If PCRE is compiled with
565: Unicode property support, and the PCRE_UCP option is set, the behaviour is
566: changed so that Unicode properties are used to determine character types, as
567: follows:
568: .sp
569: \ed any character that matches \ep{Nd} (decimal digit)
570: \es any character that matches \ep{Z} or \eh or \ev
571: \ew any character that matches \ep{L} or \ep{N}, plus underscore
572: .sp
573: The upper case escapes match the inverse sets of characters. Note that \ed
574: matches only decimal digits, whereas \ew matches any Unicode digit, as well as
575: any Unicode letter, and underscore. Note also that PCRE_UCP affects \eb, and
576: \eB because they are defined in terms of \ew and \eW. Matching these sequences
577: is noticeably slower when PCRE_UCP is set.
578: .P
579: The sequences \eh, \eH, \ev, and \eV are features that were added to Perl at
580: release 5.10. In contrast to the other sequences, which match only ASCII
581: characters by default, these always match certain high-valued code points,
582: whether or not PCRE_UCP is set. The horizontal space characters are:
583: .sp
584: U+0009 Horizontal tab (HT)
585: U+0020 Space
586: U+00A0 Non-break space
587: U+1680 Ogham space mark
588: U+180E Mongolian vowel separator
589: U+2000 En quad
590: U+2001 Em quad
591: U+2002 En space
592: U+2003 Em space
593: U+2004 Three-per-em space
594: U+2005 Four-per-em space
595: U+2006 Six-per-em space
596: U+2007 Figure space
597: U+2008 Punctuation space
598: U+2009 Thin space
599: U+200A Hair space
600: U+202F Narrow no-break space
601: U+205F Medium mathematical space
602: U+3000 Ideographic space
603: .sp
604: The vertical space characters are:
605: .sp
606: U+000A Linefeed (LF)
607: U+000B Vertical tab (VT)
608: U+000C Form feed (FF)
609: U+000D Carriage return (CR)
610: U+0085 Next line (NEL)
611: U+2028 Line separator
612: U+2029 Paragraph separator
613: .sp
614: In 8-bit, non-UTF-8 mode, only the characters with codepoints less than 256 are
615: relevant.
616: .
617: .
618: .\" HTML <a name="newlineseq"></a>
619: .SS "Newline sequences"
620: .rs
621: .sp
622: Outside a character class, by default, the escape sequence \eR matches any
623: Unicode newline sequence. In 8-bit non-UTF-8 mode \eR is equivalent to the
624: following:
625: .sp
626: (?>\er\en|\en|\ex0b|\ef|\er|\ex85)
627: .sp
628: This is an example of an "atomic group", details of which are given
629: .\" HTML <a href="#atomicgroup">
630: .\" </a>
631: below.
632: .\"
633: This particular group matches either the two-character sequence CR followed by
634: LF, or one of the single characters LF (linefeed, U+000A), VT (vertical tab,
635: U+000B), FF (form feed, U+000C), CR (carriage return, U+000D), or NEL (next
636: line, U+0085). The two-character sequence is treated as a single unit that
637: cannot be split.
638: .P
639: In other modes, two additional characters whose codepoints are greater than 255
640: are added: LS (line separator, U+2028) and PS (paragraph separator, U+2029).
641: Unicode character property support is not needed for these characters to be
642: recognized.
643: .P
644: It is possible to restrict \eR to match only CR, LF, or CRLF (instead of the
645: complete set of Unicode line endings) by setting the option PCRE_BSR_ANYCRLF
646: either at compile time or when the pattern is matched. (BSR is an abbrevation
647: for "backslash R".) This can be made the default when PCRE is built; if this is
648: the case, the other behaviour can be requested via the PCRE_BSR_UNICODE option.
649: It is also possible to specify these settings by starting a pattern string with
650: one of the following sequences:
651: .sp
652: (*BSR_ANYCRLF) CR, LF, or CRLF only
653: (*BSR_UNICODE) any Unicode newline sequence
654: .sp
655: These override the default and the options given to the compiling function, but
656: they can themselves be overridden by options given to a matching function. Note
657: that these special settings, which are not Perl-compatible, are recognized only
658: at the very start of a pattern, and that they must be in upper case. If more
659: than one of them is present, the last one is used. They can be combined with a
660: change of newline convention; for example, a pattern can start with:
661: .sp
662: (*ANY)(*BSR_ANYCRLF)
663: .sp
664: They can also be combined with the (*UTF8), (*UTF16), (*UTF32), (*UTF) or
665: (*UCP) special sequences. Inside a character class, \eR is treated as an
666: unrecognized escape sequence, and so matches the letter "R" by default, but
667: causes an error if PCRE_EXTRA is set.
668: .
669: .
670: .\" HTML <a name="uniextseq"></a>
671: .SS Unicode character properties
672: .rs
673: .sp
674: When PCRE is built with Unicode character property support, three additional
675: escape sequences that match characters with specific properties are available.
676: When in 8-bit non-UTF-8 mode, these sequences are of course limited to testing
677: characters whose codepoints are less than 256, but they do work in this mode.
678: The extra escape sequences are:
679: .sp
680: \ep{\fIxx\fP} a character with the \fIxx\fP property
681: \eP{\fIxx\fP} a character without the \fIxx\fP property
682: \eX a Unicode extended grapheme cluster
683: .sp
684: The property names represented by \fIxx\fP above are limited to the Unicode
685: script names, the general category properties, "Any", which matches any
686: character (including newline), and some special PCRE properties (described
687: in the
688: .\" HTML <a href="#extraprops">
689: .\" </a>
690: next section).
691: .\"
692: Other Perl properties such as "InMusicalSymbols" are not currently supported by
693: PCRE. Note that \eP{Any} does not match any characters, so always causes a
694: match failure.
695: .P
696: Sets of Unicode characters are defined as belonging to certain scripts. A
697: character from one of these sets can be matched using a script name. For
698: example:
699: .sp
700: \ep{Greek}
701: \eP{Han}
702: .sp
703: Those that are not part of an identified script are lumped together as
704: "Common". The current list of scripts is:
705: .P
706: Arabic,
707: Armenian,
708: Avestan,
709: Balinese,
710: Bamum,
711: Batak,
712: Bengali,
713: Bopomofo,
714: Brahmi,
715: Braille,
716: Buginese,
717: Buhid,
718: Canadian_Aboriginal,
719: Carian,
720: Chakma,
721: Cham,
722: Cherokee,
723: Common,
724: Coptic,
725: Cuneiform,
726: Cypriot,
727: Cyrillic,
728: Deseret,
729: Devanagari,
730: Egyptian_Hieroglyphs,
731: Ethiopic,
732: Georgian,
733: Glagolitic,
734: Gothic,
735: Greek,
736: Gujarati,
737: Gurmukhi,
738: Han,
739: Hangul,
740: Hanunoo,
741: Hebrew,
742: Hiragana,
743: Imperial_Aramaic,
744: Inherited,
745: Inscriptional_Pahlavi,
746: Inscriptional_Parthian,
747: Javanese,
748: Kaithi,
749: Kannada,
750: Katakana,
751: Kayah_Li,
752: Kharoshthi,
753: Khmer,
754: Lao,
755: Latin,
756: Lepcha,
757: Limbu,
758: Linear_B,
759: Lisu,
760: Lycian,
761: Lydian,
762: Malayalam,
763: Mandaic,
764: Meetei_Mayek,
765: Meroitic_Cursive,
766: Meroitic_Hieroglyphs,
767: Miao,
768: Mongolian,
769: Myanmar,
770: New_Tai_Lue,
771: Nko,
772: Ogham,
773: Old_Italic,
774: Old_Persian,
775: Old_South_Arabian,
776: Old_Turkic,
777: Ol_Chiki,
778: Oriya,
779: Osmanya,
780: Phags_Pa,
781: Phoenician,
782: Rejang,
783: Runic,
784: Samaritan,
785: Saurashtra,
786: Sharada,
787: Shavian,
788: Sinhala,
789: Sora_Sompeng,
790: Sundanese,
791: Syloti_Nagri,
792: Syriac,
793: Tagalog,
794: Tagbanwa,
795: Tai_Le,
796: Tai_Tham,
797: Tai_Viet,
798: Takri,
799: Tamil,
800: Telugu,
801: Thaana,
802: Thai,
803: Tibetan,
804: Tifinagh,
805: Ugaritic,
806: Vai,
807: Yi.
808: .P
809: Each character has exactly one Unicode general category property, specified by
810: a two-letter abbreviation. For compatibility with Perl, negation can be
811: specified by including a circumflex between the opening brace and the property
812: name. For example, \ep{^Lu} is the same as \eP{Lu}.
813: .P
814: If only one letter is specified with \ep or \eP, it includes all the general
815: category properties that start with that letter. In this case, in the absence
816: of negation, the curly brackets in the escape sequence are optional; these two
817: examples have the same effect:
818: .sp
819: \ep{L}
820: \epL
821: .sp
822: The following general category property codes are supported:
823: .sp
824: C Other
825: Cc Control
826: Cf Format
827: Cn Unassigned
828: Co Private use
829: Cs Surrogate
830: .sp
831: L Letter
832: Ll Lower case letter
833: Lm Modifier letter
834: Lo Other letter
835: Lt Title case letter
836: Lu Upper case letter
837: .sp
838: M Mark
839: Mc Spacing mark
840: Me Enclosing mark
841: Mn Non-spacing mark
842: .sp
843: N Number
844: Nd Decimal number
845: Nl Letter number
846: No Other number
847: .sp
848: P Punctuation
849: Pc Connector punctuation
850: Pd Dash punctuation
851: Pe Close punctuation
852: Pf Final punctuation
853: Pi Initial punctuation
854: Po Other punctuation
855: Ps Open punctuation
856: .sp
857: S Symbol
858: Sc Currency symbol
859: Sk Modifier symbol
860: Sm Mathematical symbol
861: So Other symbol
862: .sp
863: Z Separator
864: Zl Line separator
865: Zp Paragraph separator
866: Zs Space separator
867: .sp
868: The special property L& is also supported: it matches a character that has
869: the Lu, Ll, or Lt property, in other words, a letter that is not classified as
870: a modifier or "other".
871: .P
872: The Cs (Surrogate) property applies only to characters in the range U+D800 to
873: U+DFFF. Such characters are not valid in Unicode strings and so
874: cannot be tested by PCRE, unless UTF validity checking has been turned off
875: (see the discussion of PCRE_NO_UTF8_CHECK, PCRE_NO_UTF16_CHECK and
876: PCRE_NO_UTF32_CHECK in the
877: .\" HREF
878: \fBpcreapi\fP
879: .\"
880: page). Perl does not support the Cs property.
881: .P
882: The long synonyms for property names that Perl supports (such as \ep{Letter})
883: are not supported by PCRE, nor is it permitted to prefix any of these
884: properties with "Is".
885: .P
886: No character that is in the Unicode table has the Cn (unassigned) property.
887: Instead, this property is assumed for any code point that is not in the
888: Unicode table.
889: .P
890: Specifying caseless matching does not affect these escape sequences. For
891: example, \ep{Lu} always matches only upper case letters. This is different from
892: the behaviour of current versions of Perl.
893: .P
894: Matching characters by Unicode property is not fast, because PCRE has to do a
895: multistage table lookup in order to find a character's property. That is why
896: the traditional escape sequences such as \ed and \ew do not use Unicode
897: properties in PCRE by default, though you can make them do so by setting the
898: PCRE_UCP option or by starting the pattern with (*UCP).
899: .
900: .
901: .SS Extended grapheme clusters
902: .rs
903: .sp
904: The \eX escape matches any number of Unicode characters that form an "extended
905: grapheme cluster", and treats the sequence as an atomic group
906: .\" HTML <a href="#atomicgroup">
907: .\" </a>
908: (see below).
909: .\"
910: Up to and including release 8.31, PCRE matched an earlier, simpler definition
911: that was equivalent to
912: .sp
913: (?>\ePM\epM*)
914: .sp
915: That is, it matched a character without the "mark" property, followed by zero
916: or more characters with the "mark" property. Characters with the "mark"
917: property are typically non-spacing accents that affect the preceding character.
918: .P
919: This simple definition was extended in Unicode to include more complicated
920: kinds of composite character by giving each character a grapheme breaking
921: property, and creating rules that use these properties to define the boundaries
922: of extended grapheme clusters. In releases of PCRE later than 8.31, \eX matches
923: one of these clusters.
924: .P
925: \eX always matches at least one character. Then it decides whether to add
926: additional characters according to the following rules for ending a cluster:
927: .P
928: 1. End at the end of the subject string.
929: .P
930: 2. Do not end between CR and LF; otherwise end after any control character.
931: .P
932: 3. Do not break Hangul (a Korean script) syllable sequences. Hangul characters
933: are of five types: L, V, T, LV, and LVT. An L character may be followed by an
934: L, V, LV, or LVT character; an LV or V character may be followed by a V or T
935: character; an LVT or T character may be follwed only by a T character.
936: .P
937: 4. Do not end before extending characters or spacing marks. Characters with
938: the "mark" property always have the "extend" grapheme breaking property.
939: .P
940: 5. Do not end after prepend characters.
941: .P
942: 6. Otherwise, end the cluster.
943: .
944: .
945: .\" HTML <a name="extraprops"></a>
946: .SS PCRE's additional properties
947: .rs
948: .sp
949: As well as the standard Unicode properties described above, PCRE supports four
950: more that make it possible to convert traditional escape sequences such as \ew
951: and \es to use Unicode properties. PCRE uses these non-standard, non-Perl
952: properties internally when PCRE_UCP is set. However, they may also be used
953: explicitly. These properties are:
954: .sp
955: Xan Any alphanumeric character
956: Xps Any POSIX space character
957: Xsp Any Perl space character
958: Xwd Any Perl "word" character
959: .sp
960: Xan matches characters that have either the L (letter) or the N (number)
961: property. Xps matches the characters tab, linefeed, vertical tab, form feed, or
962: carriage return, and any other character that has the Z (separator) property.
963: Xsp is the same as Xps; it used to exclude vertical tab, for Perl
964: compatibility, but Perl changed, and so PCRE followed at release 8.34. Xwd
965: matches the same characters as Xan, plus underscore.
966: .P
967: There is another non-standard property, Xuc, which matches any character that
968: can be represented by a Universal Character Name in C++ and other programming
969: languages. These are the characters $, @, ` (grave accent), and all characters
970: with Unicode code points greater than or equal to U+00A0, except for the
971: surrogates U+D800 to U+DFFF. Note that most base (ASCII) characters are
972: excluded. (Universal Character Names are of the form \euHHHH or \eUHHHHHHHH
973: where H is a hexadecimal digit. Note that the Xuc property does not match these
974: sequences but the characters that they represent.)
975: .
976: .
977: .\" HTML <a name="resetmatchstart"></a>
978: .SS "Resetting the match start"
979: .rs
980: .sp
981: The escape sequence \eK causes any previously matched characters not to be
982: included in the final matched sequence. For example, the pattern:
983: .sp
984: foo\eKbar
985: .sp
986: matches "foobar", but reports that it has matched "bar". This feature is
987: similar to a lookbehind assertion
988: .\" HTML <a href="#lookbehind">
989: .\" </a>
990: (described below).
991: .\"
992: However, in this case, the part of the subject before the real match does not
993: have to be of fixed length, as lookbehind assertions do. The use of \eK does
994: not interfere with the setting of
995: .\" HTML <a href="#subpattern">
996: .\" </a>
997: captured substrings.
998: .\"
999: For example, when the pattern
1000: .sp
1001: (foo)\eKbar
1002: .sp
1003: matches "foobar", the first substring is still set to "foo".
1004: .P
1005: Perl documents that the use of \eK within assertions is "not well defined". In
1006: PCRE, \eK is acted upon when it occurs inside positive assertions, but is
1007: ignored in negative assertions.
1008: .
1009: .
1010: .\" HTML <a name="smallassertions"></a>
1011: .SS "Simple assertions"
1012: .rs
1013: .sp
1014: The final use of backslash is for certain simple assertions. An assertion
1015: specifies a condition that has to be met at a particular point in a match,
1016: without consuming any characters from the subject string. The use of
1017: subpatterns for more complicated assertions is described
1018: .\" HTML <a href="#bigassertions">
1019: .\" </a>
1020: below.
1021: .\"
1022: The backslashed assertions are:
1023: .sp
1024: \eb matches at a word boundary
1025: \eB matches when not at a word boundary
1026: \eA matches at the start of the subject
1027: \eZ matches at the end of the subject
1028: also matches before a newline at the end of the subject
1029: \ez matches only at the end of the subject
1030: \eG matches at the first matching position in the subject
1031: .sp
1032: Inside a character class, \eb has a different meaning; it matches the backspace
1033: character. If any other of these assertions appears in a character class, by
1034: default it matches the corresponding literal character (for example, \eB
1035: matches the letter B). However, if the PCRE_EXTRA option is set, an "invalid
1036: escape sequence" error is generated instead.
1037: .P
1038: A word boundary is a position in the subject string where the current character
1039: and the previous character do not both match \ew or \eW (i.e. one matches
1040: \ew and the other matches \eW), or the start or end of the string if the
1041: first or last character matches \ew, respectively. In a UTF mode, the meanings
1042: of \ew and \eW can be changed by setting the PCRE_UCP option. When this is
1043: done, it also affects \eb and \eB. Neither PCRE nor Perl has a separate "start
1044: of word" or "end of word" metasequence. However, whatever follows \eb normally
1045: determines which it is. For example, the fragment \eba matches "a" at the start
1046: of a word.
1047: .P
1048: The \eA, \eZ, and \ez assertions differ from the traditional circumflex and
1049: dollar (described in the next section) in that they only ever match at the very
1050: start and end of the subject string, whatever options are set. Thus, they are
1051: independent of multiline mode. These three assertions are not affected by the
1052: PCRE_NOTBOL or PCRE_NOTEOL options, which affect only the behaviour of the
1053: circumflex and dollar metacharacters. However, if the \fIstartoffset\fP
1054: argument of \fBpcre_exec()\fP is non-zero, indicating that matching is to start
1055: at a point other than the beginning of the subject, \eA can never match. The
1056: difference between \eZ and \ez is that \eZ matches before a newline at the end
1057: of the string as well as at the very end, whereas \ez matches only at the end.
1058: .P
1059: The \eG assertion is true only when the current matching position is at the
1060: start point of the match, as specified by the \fIstartoffset\fP argument of
1061: \fBpcre_exec()\fP. It differs from \eA when the value of \fIstartoffset\fP is
1062: non-zero. By calling \fBpcre_exec()\fP multiple times with appropriate
1063: arguments, you can mimic Perl's /g option, and it is in this kind of
1064: implementation where \eG can be useful.
1065: .P
1066: Note, however, that PCRE's interpretation of \eG, as the start of the current
1067: match, is subtly different from Perl's, which defines it as the end of the
1068: previous match. In Perl, these can be different when the previously matched
1069: string was empty. Because PCRE does just one match at a time, it cannot
1070: reproduce this behaviour.
1071: .P
1072: If all the alternatives of a pattern begin with \eG, the expression is anchored
1073: to the starting match position, and the "anchored" flag is set in the compiled
1074: regular expression.
1075: .
1076: .
1077: .SH "CIRCUMFLEX AND DOLLAR"
1078: .rs
1079: .sp
1080: The circumflex and dollar metacharacters are zero-width assertions. That is,
1081: they test for a particular condition being true without consuming any
1082: characters from the subject string.
1083: .P
1084: Outside a character class, in the default matching mode, the circumflex
1085: character is an assertion that is true only if the current matching point is at
1086: the start of the subject string. If the \fIstartoffset\fP argument of
1087: \fBpcre_exec()\fP is non-zero, circumflex can never match if the PCRE_MULTILINE
1088: option is unset. Inside a character class, circumflex has an entirely different
1089: meaning
1090: .\" HTML <a href="#characterclass">
1091: .\" </a>
1092: (see below).
1093: .\"
1094: .P
1095: Circumflex need not be the first character of the pattern if a number of
1096: alternatives are involved, but it should be the first thing in each alternative
1097: in which it appears if the pattern is ever to match that branch. If all
1098: possible alternatives start with a circumflex, that is, if the pattern is
1099: constrained to match only at the start of the subject, it is said to be an
1100: "anchored" pattern. (There are also other constructs that can cause a pattern
1101: to be anchored.)
1102: .P
1103: The dollar character is an assertion that is true only if the current matching
1104: point is at the end of the subject string, or immediately before a newline at
1105: the end of the string (by default). Note, however, that it does not actually
1106: match the newline. Dollar need not be the last character of the pattern if a
1107: number of alternatives are involved, but it should be the last item in any
1108: branch in which it appears. Dollar has no special meaning in a character class.
1109: .P
1110: The meaning of dollar can be changed so that it matches only at the very end of
1111: the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This
1112: does not affect the \eZ assertion.
1113: .P
1114: The meanings of the circumflex and dollar characters are changed if the
1115: PCRE_MULTILINE option is set. When this is the case, a circumflex matches
1116: immediately after internal newlines as well as at the start of the subject
1117: string. It does not match after a newline that ends the string. A dollar
1118: matches before any newlines in the string, as well as at the very end, when
1119: PCRE_MULTILINE is set. When newline is specified as the two-character
1120: sequence CRLF, isolated CR and LF characters do not indicate newlines.
1121: .P
1122: For example, the pattern /^abc$/ matches the subject string "def\enabc" (where
1123: \en represents a newline) in multiline mode, but not otherwise. Consequently,
1124: patterns that are anchored in single line mode because all branches start with
1125: ^ are not anchored in multiline mode, and a match for circumflex is possible
1126: when the \fIstartoffset\fP argument of \fBpcre_exec()\fP is non-zero. The
1127: PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set.
1128: .P
1129: Note that the sequences \eA, \eZ, and \ez can be used to match the start and
1130: end of the subject in both modes, and if all branches of a pattern start with
1131: \eA it is always anchored, whether or not PCRE_MULTILINE is set.
1132: .
1133: .
1134: .\" HTML <a name="fullstopdot"></a>
1135: .SH "FULL STOP (PERIOD, DOT) AND \eN"
1136: .rs
1137: .sp
1138: Outside a character class, a dot in the pattern matches any one character in
1139: the subject string except (by default) a character that signifies the end of a
1140: line.
1141: .P
1142: When a line ending is defined as a single character, dot never matches that
1143: character; when the two-character sequence CRLF is used, dot does not match CR
1144: if it is immediately followed by LF, but otherwise it matches all characters
1145: (including isolated CRs and LFs). When any Unicode line endings are being
1146: recognized, dot does not match CR or LF or any of the other line ending
1147: characters.
1148: .P
1149: The behaviour of dot with regard to newlines can be changed. If the PCRE_DOTALL
1150: option is set, a dot matches any one character, without exception. If the
1151: two-character sequence CRLF is present in the subject string, it takes two dots
1152: to match it.
1153: .P
1154: The handling of dot is entirely independent of the handling of circumflex and
1155: dollar, the only relationship being that they both involve newlines. Dot has no
1156: special meaning in a character class.
1157: .P
1158: The escape sequence \eN behaves like a dot, except that it is not affected by
1159: the PCRE_DOTALL option. In other words, it matches any character except one
1160: that signifies the end of a line. Perl also uses \eN to match characters by
1161: name; PCRE does not support this.
1162: .
1163: .
1164: .SH "MATCHING A SINGLE DATA UNIT"
1165: .rs
1166: .sp
1167: Outside a character class, the escape sequence \eC matches any one data unit,
1168: whether or not a UTF mode is set. In the 8-bit library, one data unit is one
1169: byte; in the 16-bit library it is a 16-bit unit; in the 32-bit library it is
1170: a 32-bit unit. Unlike a dot, \eC always
1171: matches line-ending characters. The feature is provided in Perl in order to
1172: match individual bytes in UTF-8 mode, but it is unclear how it can usefully be
1173: used. Because \eC breaks up characters into individual data units, matching one
1174: unit with \eC in a UTF mode means that the rest of the string may start with a
1175: malformed UTF character. This has undefined results, because PCRE assumes that
1176: it is dealing with valid UTF strings (and by default it checks this at the
1177: start of processing unless the PCRE_NO_UTF8_CHECK, PCRE_NO_UTF16_CHECK or
1178: PCRE_NO_UTF32_CHECK option is used).
1179: .P
1180: PCRE does not allow \eC to appear in lookbehind assertions
1181: .\" HTML <a href="#lookbehind">
1182: .\" </a>
1183: (described below)
1184: .\"
1185: in a UTF mode, because this would make it impossible to calculate the length of
1186: the lookbehind.
1187: .P
1188: In general, the \eC escape sequence is best avoided. However, one
1189: way of using it that avoids the problem of malformed UTF characters is to use a
1190: lookahead to check the length of the next character, as in this pattern, which
1191: could be used with a UTF-8 string (ignore white space and line breaks):
1192: .sp
1193: (?| (?=[\ex00-\ex7f])(\eC) |
1194: (?=[\ex80-\ex{7ff}])(\eC)(\eC) |
1195: (?=[\ex{800}-\ex{ffff}])(\eC)(\eC)(\eC) |
1196: (?=[\ex{10000}-\ex{1fffff}])(\eC)(\eC)(\eC)(\eC))
1197: .sp
1198: A group that starts with (?| resets the capturing parentheses numbers in each
1199: alternative (see
1200: .\" HTML <a href="#dupsubpatternnumber">
1201: .\" </a>
1202: "Duplicate Subpattern Numbers"
1203: .\"
1204: below). The assertions at the start of each branch check the next UTF-8
1205: character for values whose encoding uses 1, 2, 3, or 4 bytes, respectively. The
1206: character's individual bytes are then captured by the appropriate number of
1207: groups.
1208: .
1209: .
1210: .\" HTML <a name="characterclass"></a>
1211: .SH "SQUARE BRACKETS AND CHARACTER CLASSES"
1212: .rs
1213: .sp
1214: An opening square bracket introduces a character class, terminated by a closing
1215: square bracket. A closing square bracket on its own is not special by default.
1216: However, if the PCRE_JAVASCRIPT_COMPAT option is set, a lone closing square
1217: bracket causes a compile-time error. If a closing square bracket is required as
1218: a member of the class, it should be the first data character in the class
1219: (after an initial circumflex, if present) or escaped with a backslash.
1220: .P
1221: A character class matches a single character in the subject. In a UTF mode, the
1222: character may be more than one data unit long. A matched character must be in
1223: the set of characters defined by the class, unless the first character in the
1224: class definition is a circumflex, in which case the subject character must not
1225: be in the set defined by the class. If a circumflex is actually required as a
1226: member of the class, ensure it is not the first character, or escape it with a
1227: backslash.
1228: .P
1229: For example, the character class [aeiou] matches any lower case vowel, while
1230: [^aeiou] matches any character that is not a lower case vowel. Note that a
1231: circumflex is just a convenient notation for specifying the characters that
1232: are in the class by enumerating those that are not. A class that starts with a
1233: circumflex is not an assertion; it still consumes a character from the subject
1234: string, and therefore it fails if the current pointer is at the end of the
1235: string.
1236: .P
1237: In UTF-8 (UTF-16, UTF-32) mode, characters with values greater than 255 (0xffff)
1238: can be included in a class as a literal string of data units, or by using the
1239: \ex{ escaping mechanism.
1240: .P
1241: When caseless matching is set, any letters in a class represent both their
1242: upper case and lower case versions, so for example, a caseless [aeiou] matches
1243: "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a
1244: caseful version would. In a UTF mode, PCRE always understands the concept of
1245: case for characters whose values are less than 128, so caseless matching is
1246: always possible. For characters with higher values, the concept of case is
1247: supported if PCRE is compiled with Unicode property support, but not otherwise.
1248: If you want to use caseless matching in a UTF mode for characters 128 and
1249: above, you must ensure that PCRE is compiled with Unicode property support as
1250: well as with UTF support.
1251: .P
1252: Characters that might indicate line breaks are never treated in any special way
1253: when matching character classes, whatever line-ending sequence is in use, and
1254: whatever setting of the PCRE_DOTALL and PCRE_MULTILINE options is used. A class
1255: such as [^a] always matches one of these characters.
1256: .P
1257: The minus (hyphen) character can be used to specify a range of characters in a
1258: character class. For example, [d-m] matches any letter between d and m,
1259: inclusive. If a minus character is required in a class, it must be escaped with
1260: a backslash or appear in a position where it cannot be interpreted as
1261: indicating a range, typically as the first or last character in the class, or
1262: immediately after a range. For example, [b-d-z] matches letters in the range b
1263: to d, a hyphen character, or z.
1264: .P
1265: It is not possible to have the literal character "]" as the end character of a
1266: range. A pattern such as [W-]46] is interpreted as a class of two characters
1267: ("W" and "-") followed by a literal string "46]", so it would match "W46]" or
1268: "-46]". However, if the "]" is escaped with a backslash it is interpreted as
1269: the end of range, so [W-\e]46] is interpreted as a class containing a range
1270: followed by two other characters. The octal or hexadecimal representation of
1271: "]" can also be used to end a range.
1272: .P
1273: An error is generated if a POSIX character class (see below) or an escape
1274: sequence other than one that defines a single character appears at a point
1275: where a range ending character is expected. For example, [z-\exff] is valid,
1276: but [A-\ed] and [A-[:digit:]] are not.
1277: .P
1278: Ranges operate in the collating sequence of character values. They can also be
1279: used for characters specified numerically, for example [\e000-\e037]. Ranges
1280: can include any characters that are valid for the current mode.
1281: .P
1282: If a range that includes letters is used when caseless matching is set, it
1283: matches the letters in either case. For example, [W-c] is equivalent to
1284: [][\e\e^_`wxyzabc], matched caselessly, and in a non-UTF mode, if character
1285: tables for a French locale are in use, [\exc8-\excb] matches accented E
1286: characters in both cases. In UTF modes, PCRE supports the concept of case for
1287: characters with values greater than 128 only when it is compiled with Unicode
1288: property support.
1289: .P
1290: The character escape sequences \ed, \eD, \eh, \eH, \ep, \eP, \es, \eS, \ev,
1291: \eV, \ew, and \eW may appear in a character class, and add the characters that
1292: they match to the class. For example, [\edABCDEF] matches any hexadecimal
1293: digit. In UTF modes, the PCRE_UCP option affects the meanings of \ed, \es, \ew
1294: and their upper case partners, just as it does when they appear outside a
1295: character class, as described in the section entitled
1296: .\" HTML <a href="#genericchartypes">
1297: .\" </a>
1298: "Generic character types"
1299: .\"
1300: above. The escape sequence \eb has a different meaning inside a character
1301: class; it matches the backspace character. The sequences \eB, \eN, \eR, and \eX
1302: are not special inside a character class. Like any other unrecognized escape
1303: sequences, they are treated as the literal characters "B", "N", "R", and "X" by
1304: default, but cause an error if the PCRE_EXTRA option is set.
1305: .P
1306: A circumflex can conveniently be used with the upper case character types to
1307: specify a more restricted set of characters than the matching lower case type.
1308: For example, the class [^\eW_] matches any letter or digit, but not underscore,
1309: whereas [\ew] includes underscore. A positive character class should be read as
1310: "something OR something OR ..." and a negative class as "NOT something AND NOT
1311: something AND NOT ...".
1312: .P
1313: The only metacharacters that are recognized in character classes are backslash,
1314: hyphen (only where it can be interpreted as specifying a range), circumflex
1315: (only at the start), opening square bracket (only when it can be interpreted as
1316: introducing a POSIX class name, or for a special compatibility feature - see
1317: the next two sections), and the terminating closing square bracket. However,
1318: escaping other non-alphanumeric characters does no harm.
1319: .
1320: .
1321: .SH "POSIX CHARACTER CLASSES"
1322: .rs
1323: .sp
1324: Perl supports the POSIX notation for character classes. This uses names
1325: enclosed by [: and :] within the enclosing square brackets. PCRE also supports
1326: this notation. For example,
1327: .sp
1328: [01[:alpha:]%]
1329: .sp
1330: matches "0", "1", any alphabetic character, or "%". The supported class names
1331: are:
1332: .sp
1333: alnum letters and digits
1334: alpha letters
1335: ascii character codes 0 - 127
1336: blank space or tab only
1337: cntrl control characters
1338: digit decimal digits (same as \ed)
1339: graph printing characters, excluding space
1340: lower lower case letters
1341: print printing characters, including space
1342: punct printing characters, excluding letters and digits and space
1343: space white space (the same as \es from PCRE 8.34)
1344: upper upper case letters
1345: word "word" characters (same as \ew)
1346: xdigit hexadecimal digits
1347: .sp
1348: The default "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13),
1349: and space (32). If locale-specific matching is taking place, the list of space
1350: characters may be different; there may be fewer or more of them. "Space" used
1351: to be different to \es, which did not include VT, for Perl compatibility.
1352: However, Perl changed at release 5.18, and PCRE followed at release 8.34.
1353: "Space" and \es now match the same set of characters.
1354: .P
1355: The name "word" is a Perl extension, and "blank" is a GNU extension from Perl
1356: 5.8. Another Perl extension is negation, which is indicated by a ^ character
1357: after the colon. For example,
1358: .sp
1359: [12[:^digit:]]
1360: .sp
1361: matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX
1362: syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not
1363: supported, and an error is given if they are encountered.
1364: .P
1365: By default, characters with values greater than 128 do not match any of the
1366: POSIX character classes. However, if the PCRE_UCP option is passed to
1367: \fBpcre_compile()\fP, some of the classes are changed so that Unicode character
1368: properties are used. This is achieved by replacing certain POSIX classes by
1369: other sequences, as follows:
1370: .sp
1371: [:alnum:] becomes \ep{Xan}
1372: [:alpha:] becomes \ep{L}
1373: [:blank:] becomes \eh
1374: [:digit:] becomes \ep{Nd}
1375: [:lower:] becomes \ep{Ll}
1376: [:space:] becomes \ep{Xps}
1377: [:upper:] becomes \ep{Lu}
1378: [:word:] becomes \ep{Xwd}
1379: .sp
1380: Negated versions, such as [:^alpha:] use \eP instead of \ep. Three other POSIX
1381: classes are handled specially in UCP mode:
1382: .TP 10
1383: [:graph:]
1384: This matches characters that have glyphs that mark the page when printed. In
1385: Unicode property terms, it matches all characters with the L, M, N, P, S, or Cf
1386: properties, except for:
1387: .sp
1388: U+061C Arabic Letter Mark
1389: U+180E Mongolian Vowel Separator
1390: U+2066 - U+2069 Various "isolate"s
1391: .sp
1392: .TP 10
1393: [:print:]
1394: This matches the same characters as [:graph:] plus space characters that are
1395: not controls, that is, characters with the Zs property.
1396: .TP 10
1397: [:punct:]
1398: This matches all characters that have the Unicode P (punctuation) property,
1399: plus those characters whose code points are less than 128 that have the S
1400: (Symbol) property.
1401: .P
1402: The other POSIX classes are unchanged, and match only characters with code
1403: points less than 128.
1404: .
1405: .
1406: .SH "COMPATIBILITY FEATURE FOR WORD BOUNDARIES"
1407: .rs
1408: .sp
1409: In the POSIX.2 compliant library that was included in 4.4BSD Unix, the ugly
1410: syntax [[:<:]] and [[:>:]] is used for matching "start of word" and "end of
1411: word". PCRE treats these items as follows:
1412: .sp
1413: [[:<:]] is converted to \eb(?=\ew)
1414: [[:>:]] is converted to \eb(?<=\ew)
1415: .sp
1416: Only these exact character sequences are recognized. A sequence such as
1417: [a[:<:]b] provokes error for an unrecognized POSIX class name. This support is
1418: not compatible with Perl. It is provided to help migrations from other
1419: environments, and is best not used in any new patterns. Note that \eb matches
1420: at the start and the end of a word (see
1421: .\" HTML <a href="#smallassertions">
1422: .\" </a>
1423: "Simple assertions"
1424: .\"
1425: above), and in a Perl-style pattern the preceding or following character
1426: normally shows which is wanted, without the need for the assertions that are
1427: used above in order to give exactly the POSIX behaviour.
1428: .
1429: .
1430: .SH "VERTICAL BAR"
1431: .rs
1432: .sp
1433: Vertical bar characters are used to separate alternative patterns. For example,
1434: the pattern
1435: .sp
1436: gilbert|sullivan
1437: .sp
1438: matches either "gilbert" or "sullivan". Any number of alternatives may appear,
1439: and an empty alternative is permitted (matching the empty string). The matching
1440: process tries each alternative in turn, from left to right, and the first one
1441: that succeeds is used. If the alternatives are within a subpattern
1442: .\" HTML <a href="#subpattern">
1443: .\" </a>
1444: (defined below),
1445: .\"
1446: "succeeds" means matching the rest of the main pattern as well as the
1447: alternative in the subpattern.
1448: .
1449: .
1450: .SH "INTERNAL OPTION SETTING"
1451: .rs
1452: .sp
1453: The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and
1454: PCRE_EXTENDED options (which are Perl-compatible) can be changed from within
1455: the pattern by a sequence of Perl option letters enclosed between "(?" and ")".
1456: The option letters are
1457: .sp
1458: i for PCRE_CASELESS
1459: m for PCRE_MULTILINE
1460: s for PCRE_DOTALL
1461: x for PCRE_EXTENDED
1462: .sp
1463: For example, (?im) sets caseless, multiline matching. It is also possible to
1464: unset these options by preceding the letter with a hyphen, and a combined
1465: setting and unsetting such as (?im-sx), which sets PCRE_CASELESS and
1466: PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, is also
1467: permitted. If a letter appears both before and after the hyphen, the option is
1468: unset.
1469: .P
1470: The PCRE-specific options PCRE_DUPNAMES, PCRE_UNGREEDY, and PCRE_EXTRA can be
1471: changed in the same way as the Perl-compatible options by using the characters
1472: J, U and X respectively.
1473: .P
1474: When one of these option changes occurs at top level (that is, not inside
1475: subpattern parentheses), the change applies to the remainder of the pattern
1476: that follows. If the change is placed right at the start of a pattern, PCRE
1477: extracts it into the global options (and it will therefore show up in data
1478: extracted by the \fBpcre_fullinfo()\fP function).
1479: .P
1480: An option change within a subpattern (see below for a description of
1481: subpatterns) affects only that part of the subpattern that follows it, so
1482: .sp
1483: (a(?i)b)c
1484: .sp
1485: matches abc and aBc and no other strings (assuming PCRE_CASELESS is not used).
1486: By this means, options can be made to have different settings in different
1487: parts of the pattern. Any changes made in one alternative do carry on
1488: into subsequent branches within the same subpattern. For example,
1489: .sp
1490: (a(?i)b|c)
1491: .sp
1492: matches "ab", "aB", "c", and "C", even though when matching "C" the first
1493: branch is abandoned before the option setting. This is because the effects of
1494: option settings happen at compile time. There would be some very weird
1495: behaviour otherwise.
1496: .P
1497: \fBNote:\fP There are other PCRE-specific options that can be set by the
1498: application when the compiling or matching functions are called. In some cases
1499: the pattern can contain special leading sequences such as (*CRLF) to override
1500: what the application has set or what has been defaulted. Details are given in
1501: the section entitled
1502: .\" HTML <a href="#newlineseq">
1503: .\" </a>
1504: "Newline sequences"
1505: .\"
1506: above. There are also the (*UTF8), (*UTF16),(*UTF32), and (*UCP) leading
1507: sequences that can be used to set UTF and Unicode property modes; they are
1508: equivalent to setting the PCRE_UTF8, PCRE_UTF16, PCRE_UTF32 and the PCRE_UCP
1509: options, respectively. The (*UTF) sequence is a generic version that can be
1510: used with any of the libraries. However, the application can set the
1511: PCRE_NEVER_UTF option, which locks out the use of the (*UTF) sequences.
1512: .
1513: .
1514: .\" HTML <a name="subpattern"></a>
1515: .SH SUBPATTERNS
1516: .rs
1517: .sp
1518: Subpatterns are delimited by parentheses (round brackets), which can be nested.
1519: Turning part of a pattern into a subpattern does two things:
1520: .sp
1521: 1. It localizes a set of alternatives. For example, the pattern
1522: .sp
1523: cat(aract|erpillar|)
1524: .sp
1525: matches "cataract", "caterpillar", or "cat". Without the parentheses, it would
1526: match "cataract", "erpillar" or an empty string.
1527: .sp
1528: 2. It sets up the subpattern as a capturing subpattern. This means that, when
1529: the whole pattern matches, that portion of the subject string that matched the
1530: subpattern is passed back to the caller via the \fIovector\fP argument of the
1531: matching function. (This applies only to the traditional matching functions;
1532: the DFA matching functions do not support capturing.)
1533: .P
1534: Opening parentheses are counted from left to right (starting from 1) to obtain
1535: numbers for the capturing subpatterns. For example, if the string "the red
1536: king" is matched against the pattern
1537: .sp
1538: the ((red|white) (king|queen))
1539: .sp
1540: the captured substrings are "red king", "red", and "king", and are numbered 1,
1541: 2, and 3, respectively.
1542: .P
1543: The fact that plain parentheses fulfil two functions is not always helpful.
1544: There are often times when a grouping subpattern is required without a
1545: capturing requirement. If an opening parenthesis is followed by a question mark
1546: and a colon, the subpattern does not do any capturing, and is not counted when
1547: computing the number of any subsequent capturing subpatterns. For example, if
1548: the string "the white queen" is matched against the pattern
1549: .sp
1550: the ((?:red|white) (king|queen))
1551: .sp
1552: the captured substrings are "white queen" and "queen", and are numbered 1 and
1553: 2. The maximum number of capturing subpatterns is 65535.
1554: .P
1555: As a convenient shorthand, if any option settings are required at the start of
1556: a non-capturing subpattern, the option letters may appear between the "?" and
1557: the ":". Thus the two patterns
1558: .sp
1559: (?i:saturday|sunday)
1560: (?:(?i)saturday|sunday)
1561: .sp
1562: match exactly the same set of strings. Because alternative branches are tried
1563: from left to right, and options are not reset until the end of the subpattern
1564: is reached, an option setting in one branch does affect subsequent branches, so
1565: the above patterns match "SUNDAY" as well as "Saturday".
1566: .
1567: .
1568: .\" HTML <a name="dupsubpatternnumber"></a>
1569: .SH "DUPLICATE SUBPATTERN NUMBERS"
1570: .rs
1571: .sp
1572: Perl 5.10 introduced a feature whereby each alternative in a subpattern uses
1573: the same numbers for its capturing parentheses. Such a subpattern starts with
1574: (?| and is itself a non-capturing subpattern. For example, consider this
1575: pattern:
1576: .sp
1577: (?|(Sat)ur|(Sun))day
1578: .sp
1579: Because the two alternatives are inside a (?| group, both sets of capturing
1580: parentheses are numbered one. Thus, when the pattern matches, you can look
1581: at captured substring number one, whichever alternative matched. This construct
1582: is useful when you want to capture part, but not all, of one of a number of
1583: alternatives. Inside a (?| group, parentheses are numbered as usual, but the
1584: number is reset at the start of each branch. The numbers of any capturing
1585: parentheses that follow the subpattern start after the highest number used in
1586: any branch. The following example is taken from the Perl documentation. The
1587: numbers underneath show in which buffer the captured content will be stored.
1588: .sp
1589: # before ---------------branch-reset----------- after
1590: / ( a ) (?| x ( y ) z | (p (q) r) | (t) u (v) ) ( z ) /x
1591: # 1 2 2 3 2 3 4
1592: .sp
1593: A back reference to a numbered subpattern uses the most recent value that is
1594: set for that number by any subpattern. The following pattern matches "abcabc"
1595: or "defdef":
1596: .sp
1597: /(?|(abc)|(def))\e1/
1598: .sp
1599: In contrast, a subroutine call to a numbered subpattern always refers to the
1600: first one in the pattern with the given number. The following pattern matches
1601: "abcabc" or "defabc":
1602: .sp
1603: /(?|(abc)|(def))(?1)/
1604: .sp
1605: If a
1606: .\" HTML <a href="#conditions">
1607: .\" </a>
1608: condition test
1609: .\"
1610: for a subpattern's having matched refers to a non-unique number, the test is
1611: true if any of the subpatterns of that number have matched.
1612: .P
1613: An alternative approach to using this "branch reset" feature is to use
1614: duplicate named subpatterns, as described in the next section.
1615: .
1616: .
1617: .SH "NAMED SUBPATTERNS"
1618: .rs
1619: .sp
1620: Identifying capturing parentheses by number is simple, but it can be very hard
1621: to keep track of the numbers in complicated regular expressions. Furthermore,
1622: if an expression is modified, the numbers may change. To help with this
1623: difficulty, PCRE supports the naming of subpatterns. This feature was not
1624: added to Perl until release 5.10. Python had the feature earlier, and PCRE
1625: introduced it at release 4.0, using the Python syntax. PCRE now supports both
1626: the Perl and the Python syntax. Perl allows identically numbered subpatterns to
1627: have different names, but PCRE does not.
1628: .P
1629: In PCRE, a subpattern can be named in one of three ways: (?<name>...) or
1630: (?'name'...) as in Perl, or (?P<name>...) as in Python. References to capturing
1631: parentheses from other parts of the pattern, such as
1632: .\" HTML <a href="#backreferences">
1633: .\" </a>
1634: back references,
1635: .\"
1636: .\" HTML <a href="#recursion">
1637: .\" </a>
1638: recursion,
1639: .\"
1640: and
1641: .\" HTML <a href="#conditions">
1642: .\" </a>
1643: conditions,
1644: .\"
1645: can be made by name as well as by number.
1646: .P
1647: Names consist of up to 32 alphanumeric characters and underscores, but must
1648: start with a non-digit. Named capturing parentheses are still allocated numbers
1649: as well as names, exactly as if the names were not present. The PCRE API
1650: provides function calls for extracting the name-to-number translation table
1651: from a compiled pattern. There is also a convenience function for extracting a
1652: captured substring by name.
1653: .P
1654: By default, a name must be unique within a pattern, but it is possible to relax
1655: this constraint by setting the PCRE_DUPNAMES option at compile time. (Duplicate
1656: names are also always permitted for subpatterns with the same number, set up as
1657: described in the previous section.) Duplicate names can be useful for patterns
1658: where only one instance of the named parentheses can match. Suppose you want to
1659: match the name of a weekday, either as a 3-letter abbreviation or as the full
1660: name, and in both cases you want to extract the abbreviation. This pattern
1661: (ignoring the line breaks) does the job:
1662: .sp
1663: (?<DN>Mon|Fri|Sun)(?:day)?|
1664: (?<DN>Tue)(?:sday)?|
1665: (?<DN>Wed)(?:nesday)?|
1666: (?<DN>Thu)(?:rsday)?|
1667: (?<DN>Sat)(?:urday)?
1668: .sp
1669: There are five capturing substrings, but only one is ever set after a match.
1670: (An alternative way of solving this problem is to use a "branch reset"
1671: subpattern, as described in the previous section.)
1672: .P
1673: The convenience function for extracting the data by name returns the substring
1674: for the first (and in this example, the only) subpattern of that name that
1675: matched. This saves searching to find which numbered subpattern it was.
1676: .P
1677: If you make a back reference to a non-unique named subpattern from elsewhere in
1678: the pattern, the subpatterns to which the name refers are checked in the order
1679: in which they appear in the overall pattern. The first one that is set is used
1680: for the reference. For example, this pattern matches both "foofoo" and
1681: "barbar" but not "foobar" or "barfoo":
1682: .sp
1683: (?:(?<n>foo)|(?<n>bar))\ek<n>
1684: .sp
1685: .P
1686: If you make a subroutine call to a non-unique named subpattern, the one that
1687: corresponds to the first occurrence of the name is used. In the absence of
1688: duplicate numbers (see the previous section) this is the one with the lowest
1689: number.
1690: .P
1691: If you use a named reference in a condition
1692: test (see the
1693: .\"
1694: .\" HTML <a href="#conditions">
1695: .\" </a>
1696: section about conditions
1697: .\"
1698: below), either to check whether a subpattern has matched, or to check for
1699: recursion, all subpatterns with the same name are tested. If the condition is
1700: true for any one of them, the overall condition is true. This is the same
1701: behaviour as testing by number. For further details of the interfaces for
1702: handling named subpatterns, see the
1703: .\" HREF
1704: \fBpcreapi\fP
1705: .\"
1706: documentation.
1707: .P
1708: \fBWarning:\fP You cannot use different names to distinguish between two
1709: subpatterns with the same number because PCRE uses only the numbers when
1710: matching. For this reason, an error is given at compile time if different names
1711: are given to subpatterns with the same number. However, you can always give the
1712: same name to subpatterns with the same number, even when PCRE_DUPNAMES is not
1713: set.
1714: .
1715: .
1716: .SH REPETITION
1717: .rs
1718: .sp
1719: Repetition is specified by quantifiers, which can follow any of the following
1720: items:
1721: .sp
1722: a literal data character
1723: the dot metacharacter
1724: the \eC escape sequence
1725: the \eX escape sequence
1726: the \eR escape sequence
1727: an escape such as \ed or \epL that matches a single character
1728: a character class
1729: a back reference (see next section)
1730: a parenthesized subpattern (including assertions)
1731: a subroutine call to a subpattern (recursive or otherwise)
1732: .sp
1733: The general repetition quantifier specifies a minimum and maximum number of
1734: permitted matches, by giving the two numbers in curly brackets (braces),
1735: separated by a comma. The numbers must be less than 65536, and the first must
1736: be less than or equal to the second. For example:
1737: .sp
1738: z{2,4}
1739: .sp
1740: matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special
1741: character. If the second number is omitted, but the comma is present, there is
1742: no upper limit; if the second number and the comma are both omitted, the
1743: quantifier specifies an exact number of required matches. Thus
1744: .sp
1745: [aeiou]{3,}
1746: .sp
1747: matches at least 3 successive vowels, but may match many more, while
1748: .sp
1749: \ed{8}
1750: .sp
1751: matches exactly 8 digits. An opening curly bracket that appears in a position
1752: where a quantifier is not allowed, or one that does not match the syntax of a
1753: quantifier, is taken as a literal character. For example, {,6} is not a
1754: quantifier, but a literal string of four characters.
1755: .P
1756: In UTF modes, quantifiers apply to characters rather than to individual data
1757: units. Thus, for example, \ex{100}{2} matches two characters, each of
1758: which is represented by a two-byte sequence in a UTF-8 string. Similarly,
1759: \eX{3} matches three Unicode extended grapheme clusters, each of which may be
1760: several data units long (and they may be of different lengths).
1761: .P
1762: The quantifier {0} is permitted, causing the expression to behave as if the
1763: previous item and the quantifier were not present. This may be useful for
1764: subpatterns that are referenced as
1765: .\" HTML <a href="#subpatternsassubroutines">
1766: .\" </a>
1767: subroutines
1768: .\"
1769: from elsewhere in the pattern (but see also the section entitled
1770: .\" HTML <a href="#subdefine">
1771: .\" </a>
1772: "Defining subpatterns for use by reference only"
1773: .\"
1774: below). Items other than subpatterns that have a {0} quantifier are omitted
1775: from the compiled pattern.
1776: .P
1777: For convenience, the three most common quantifiers have single-character
1778: abbreviations:
1779: .sp
1780: * is equivalent to {0,}
1781: + is equivalent to {1,}
1782: ? is equivalent to {0,1}
1783: .sp
1784: It is possible to construct infinite loops by following a subpattern that can
1785: match no characters with a quantifier that has no upper limit, for example:
1786: .sp
1787: (a?)*
1788: .sp
1789: Earlier versions of Perl and PCRE used to give an error at compile time for
1790: such patterns. However, because there are cases where this can be useful, such
1791: patterns are now accepted, but if any repetition of the subpattern does in fact
1792: match no characters, the loop is forcibly broken.
1793: .P
1794: By default, the quantifiers are "greedy", that is, they match as much as
1795: possible (up to the maximum number of permitted times), without causing the
1796: rest of the pattern to fail. The classic example of where this gives problems
1797: is in trying to match comments in C programs. These appear between /* and */
1798: and within the comment, individual * and / characters may appear. An attempt to
1799: match C comments by applying the pattern
1800: .sp
1801: /\e*.*\e*/
1802: .sp
1803: to the string
1804: .sp
1805: /* first comment */ not comment /* second comment */
1806: .sp
1807: fails, because it matches the entire string owing to the greediness of the .*
1808: item.
1809: .P
1810: However, if a quantifier is followed by a question mark, it ceases to be
1811: greedy, and instead matches the minimum number of times possible, so the
1812: pattern
1813: .sp
1814: /\e*.*?\e*/
1815: .sp
1816: does the right thing with the C comments. The meaning of the various
1817: quantifiers is not otherwise changed, just the preferred number of matches.
1818: Do not confuse this use of question mark with its use as a quantifier in its
1819: own right. Because it has two uses, it can sometimes appear doubled, as in
1820: .sp
1821: \ed??\ed
1822: .sp
1823: which matches one digit by preference, but can match two if that is the only
1824: way the rest of the pattern matches.
1825: .P
1826: If the PCRE_UNGREEDY option is set (an option that is not available in Perl),
1827: the quantifiers are not greedy by default, but individual ones can be made
1828: greedy by following them with a question mark. In other words, it inverts the
1829: default behaviour.
1830: .P
1831: When a parenthesized subpattern is quantified with a minimum repeat count that
1832: is greater than 1 or with a limited maximum, more memory is required for the
1833: compiled pattern, in proportion to the size of the minimum or maximum.
1834: .P
1835: If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equivalent
1836: to Perl's /s) is set, thus allowing the dot to match newlines, the pattern is
1837: implicitly anchored, because whatever follows will be tried against every
1838: character position in the subject string, so there is no point in retrying the
1839: overall match at any position after the first. PCRE normally treats such a
1840: pattern as though it were preceded by \eA.
1841: .P
1842: In cases where it is known that the subject string contains no newlines, it is
1843: worth setting PCRE_DOTALL in order to obtain this optimization, or
1844: alternatively using ^ to indicate anchoring explicitly.
1845: .P
1846: However, there are some cases where the optimization cannot be used. When .*
1847: is inside capturing parentheses that are the subject of a back reference
1848: elsewhere in the pattern, a match at the start may fail where a later one
1849: succeeds. Consider, for example:
1850: .sp
1851: (.*)abc\e1
1852: .sp
1853: If the subject is "xyz123abc123" the match point is the fourth character. For
1854: this reason, such a pattern is not implicitly anchored.
1855: .P
1856: Another case where implicit anchoring is not applied is when the leading .* is
1857: inside an atomic group. Once again, a match at the start may fail where a later
1858: one succeeds. Consider this pattern:
1859: .sp
1860: (?>.*?a)b
1861: .sp
1862: It matches "ab" in the subject "aab". The use of the backtracking control verbs
1863: (*PRUNE) and (*SKIP) also disable this optimization.
1864: .P
1865: When a capturing subpattern is repeated, the value captured is the substring
1866: that matched the final iteration. For example, after
1867: .sp
1868: (tweedle[dume]{3}\es*)+
1869: .sp
1870: has matched "tweedledum tweedledee" the value of the captured substring is
1871: "tweedledee". However, if there are nested capturing subpatterns, the
1872: corresponding captured values may have been set in previous iterations. For
1873: example, after
1874: .sp
1875: /(a|(b))+/
1876: .sp
1877: matches "aba" the value of the second captured substring is "b".
1878: .
1879: .
1880: .\" HTML <a name="atomicgroup"></a>
1881: .SH "ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS"
1882: .rs
1883: .sp
1884: With both maximizing ("greedy") and minimizing ("ungreedy" or "lazy")
1885: repetition, failure of what follows normally causes the repeated item to be
1886: re-evaluated to see if a different number of repeats allows the rest of the
1887: pattern to match. Sometimes it is useful to prevent this, either to change the
1888: nature of the match, or to cause it fail earlier than it otherwise might, when
1889: the author of the pattern knows there is no point in carrying on.
1890: .P
1891: Consider, for example, the pattern \ed+foo when applied to the subject line
1892: .sp
1893: 123456bar
1894: .sp
1895: After matching all 6 digits and then failing to match "foo", the normal
1896: action of the matcher is to try again with only 5 digits matching the \ed+
1897: item, and then with 4, and so on, before ultimately failing. "Atomic grouping"
1898: (a term taken from Jeffrey Friedl's book) provides the means for specifying
1899: that once a subpattern has matched, it is not to be re-evaluated in this way.
1900: .P
1901: If we use atomic grouping for the previous example, the matcher gives up
1902: immediately on failing to match "foo" the first time. The notation is a kind of
1903: special parenthesis, starting with (?> as in this example:
1904: .sp
1905: (?>\ed+)foo
1906: .sp
1907: This kind of parenthesis "locks up" the part of the pattern it contains once
1908: it has matched, and a failure further into the pattern is prevented from
1909: backtracking into it. Backtracking past it to previous items, however, works as
1910: normal.
1911: .P
1912: An alternative description is that a subpattern of this type matches the string
1913: of characters that an identical standalone pattern would match, if anchored at
1914: the current point in the subject string.
1915: .P
1916: Atomic grouping subpatterns are not capturing subpatterns. Simple cases such as
1917: the above example can be thought of as a maximizing repeat that must swallow
1918: everything it can. So, while both \ed+ and \ed+? are prepared to adjust the
1919: number of digits they match in order to make the rest of the pattern match,
1920: (?>\ed+) can only match an entire sequence of digits.
1921: .P
1922: Atomic groups in general can of course contain arbitrarily complicated
1923: subpatterns, and can be nested. However, when the subpattern for an atomic
1924: group is just a single repeated item, as in the example above, a simpler
1925: notation, called a "possessive quantifier" can be used. This consists of an
1926: additional + character following a quantifier. Using this notation, the
1927: previous example can be rewritten as
1928: .sp
1929: \ed++foo
1930: .sp
1931: Note that a possessive quantifier can be used with an entire group, for
1932: example:
1933: .sp
1934: (abc|xyz){2,3}+
1935: .sp
1936: Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY
1937: option is ignored. They are a convenient notation for the simpler forms of
1938: atomic group. However, there is no difference in the meaning of a possessive
1939: quantifier and the equivalent atomic group, though there may be a performance
1940: difference; possessive quantifiers should be slightly faster.
1941: .P
1942: The possessive quantifier syntax is an extension to the Perl 5.8 syntax.
1943: Jeffrey Friedl originated the idea (and the name) in the first edition of his
1944: book. Mike McCloskey liked it, so implemented it when he built Sun's Java
1945: package, and PCRE copied it from there. It ultimately found its way into Perl
1946: at release 5.10.
1947: .P
1948: PCRE has an optimization that automatically "possessifies" certain simple
1949: pattern constructs. For example, the sequence A+B is treated as A++B because
1950: there is no point in backtracking into a sequence of A's when B must follow.
1951: .P
1952: When a pattern contains an unlimited repeat inside a subpattern that can itself
1953: be repeated an unlimited number of times, the use of an atomic group is the
1954: only way to avoid some failing matches taking a very long time indeed. The
1955: pattern
1956: .sp
1957: (\eD+|<\ed+>)*[!?]
1958: .sp
1959: matches an unlimited number of substrings that either consist of non-digits, or
1960: digits enclosed in <>, followed by either ! or ?. When it matches, it runs
1961: quickly. However, if it is applied to
1962: .sp
1963: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
1964: .sp
1965: it takes a long time before reporting failure. This is because the string can
1966: be divided between the internal \eD+ repeat and the external * repeat in a
1967: large number of ways, and all have to be tried. (The example uses [!?] rather
1968: than a single character at the end, because both PCRE and Perl have an
1969: optimization that allows for fast failure when a single character is used. They
1970: remember the last single character that is required for a match, and fail early
1971: if it is not present in the string.) If the pattern is changed so that it uses
1972: an atomic group, like this:
1973: .sp
1974: ((?>\eD+)|<\ed+>)*[!?]
1975: .sp
1976: sequences of non-digits cannot be broken, and failure happens quickly.
1977: .
1978: .
1979: .\" HTML <a name="backreferences"></a>
1980: .SH "BACK REFERENCES"
1981: .rs
1982: .sp
1983: Outside a character class, a backslash followed by a digit greater than 0 (and
1984: possibly further digits) is a back reference to a capturing subpattern earlier
1985: (that is, to its left) in the pattern, provided there have been that many
1986: previous capturing left parentheses.
1987: .P
1988: However, if the decimal number following the backslash is less than 10, it is
1989: always taken as a back reference, and causes an error only if there are not
1990: that many capturing left parentheses in the entire pattern. In other words, the
1991: parentheses that are referenced need not be to the left of the reference for
1992: numbers less than 10. A "forward back reference" of this type can make sense
1993: when a repetition is involved and the subpattern to the right has participated
1994: in an earlier iteration.
1995: .P
1996: It is not possible to have a numerical "forward back reference" to a subpattern
1997: whose number is 10 or more using this syntax because a sequence such as \e50 is
1998: interpreted as a character defined in octal. See the subsection entitled
1999: "Non-printing characters"
2000: .\" HTML <a href="#digitsafterbackslash">
2001: .\" </a>
2002: above
2003: .\"
2004: for further details of the handling of digits following a backslash. There is
2005: no such problem when named parentheses are used. A back reference to any
2006: subpattern is possible using named parentheses (see below).
2007: .P
2008: Another way of avoiding the ambiguity inherent in the use of digits following a
2009: backslash is to use the \eg escape sequence. This escape must be followed by an
2010: unsigned number or a negative number, optionally enclosed in braces. These
2011: examples are all identical:
2012: .sp
2013: (ring), \e1
2014: (ring), \eg1
2015: (ring), \eg{1}
2016: .sp
2017: An unsigned number specifies an absolute reference without the ambiguity that
2018: is present in the older syntax. It is also useful when literal digits follow
2019: the reference. A negative number is a relative reference. Consider this
2020: example:
2021: .sp
2022: (abc(def)ghi)\eg{-1}
2023: .sp
2024: The sequence \eg{-1} is a reference to the most recently started capturing
2025: subpattern before \eg, that is, is it equivalent to \e2 in this example.
2026: Similarly, \eg{-2} would be equivalent to \e1. The use of relative references
2027: can be helpful in long patterns, and also in patterns that are created by
2028: joining together fragments that contain references within themselves.
2029: .P
2030: A back reference matches whatever actually matched the capturing subpattern in
2031: the current subject string, rather than anything matching the subpattern
2032: itself (see
2033: .\" HTML <a href="#subpatternsassubroutines">
2034: .\" </a>
2035: "Subpatterns as subroutines"
2036: .\"
2037: below for a way of doing that). So the pattern
2038: .sp
2039: (sens|respons)e and \e1ibility
2040: .sp
2041: matches "sense and sensibility" and "response and responsibility", but not
2042: "sense and responsibility". If caseful matching is in force at the time of the
2043: back reference, the case of letters is relevant. For example,
2044: .sp
2045: ((?i)rah)\es+\e1
2046: .sp
2047: matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original
2048: capturing subpattern is matched caselessly.
2049: .P
2050: There are several different ways of writing back references to named
2051: subpatterns. The .NET syntax \ek{name} and the Perl syntax \ek<name> or
2052: \ek'name' are supported, as is the Python syntax (?P=name). Perl 5.10's unified
2053: back reference syntax, in which \eg can be used for both numeric and named
2054: references, is also supported. We could rewrite the above example in any of
2055: the following ways:
2056: .sp
2057: (?<p1>(?i)rah)\es+\ek<p1>
2058: (?'p1'(?i)rah)\es+\ek{p1}
2059: (?P<p1>(?i)rah)\es+(?P=p1)
2060: (?<p1>(?i)rah)\es+\eg{p1}
2061: .sp
2062: A subpattern that is referenced by name may appear in the pattern before or
2063: after the reference.
2064: .P
2065: There may be more than one back reference to the same subpattern. If a
2066: subpattern has not actually been used in a particular match, any back
2067: references to it always fail by default. For example, the pattern
2068: .sp
2069: (a|(bc))\e2
2070: .sp
2071: always fails if it starts to match "a" rather than "bc". However, if the
2072: PCRE_JAVASCRIPT_COMPAT option is set at compile time, a back reference to an
2073: unset value matches an empty string.
2074: .P
2075: Because there may be many capturing parentheses in a pattern, all digits
2076: following a backslash are taken as part of a potential back reference number.
2077: If the pattern continues with a digit character, some delimiter must be used to
2078: terminate the back reference. If the PCRE_EXTENDED option is set, this can be
2079: white space. Otherwise, the \eg{ syntax or an empty comment (see
2080: .\" HTML <a href="#comments">
2081: .\" </a>
2082: "Comments"
2083: .\"
2084: below) can be used.
2085: .
2086: .SS "Recursive back references"
2087: .rs
2088: .sp
2089: A back reference that occurs inside the parentheses to which it refers fails
2090: when the subpattern is first used, so, for example, (a\e1) never matches.
2091: However, such references can be useful inside repeated subpatterns. For
2092: example, the pattern
2093: .sp
2094: (a|b\e1)+
2095: .sp
2096: matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of
2097: the subpattern, the back reference matches the character string corresponding
2098: to the previous iteration. In order for this to work, the pattern must be such
2099: that the first iteration does not need to match the back reference. This can be
2100: done using alternation, as in the example above, or by a quantifier with a
2101: minimum of zero.
2102: .P
2103: Back references of this type cause the group that they reference to be treated
2104: as an
2105: .\" HTML <a href="#atomicgroup">
2106: .\" </a>
2107: atomic group.
2108: .\"
2109: Once the whole group has been matched, a subsequent matching failure cannot
2110: cause backtracking into the middle of the group.
2111: .
2112: .
2113: .\" HTML <a name="bigassertions"></a>
2114: .SH ASSERTIONS
2115: .rs
2116: .sp
2117: An assertion is a test on the characters following or preceding the current
2118: matching point that does not actually consume any characters. The simple
2119: assertions coded as \eb, \eB, \eA, \eG, \eZ, \ez, ^ and $ are described
2120: .\" HTML <a href="#smallassertions">
2121: .\" </a>
2122: above.
2123: .\"
2124: .P
2125: More complicated assertions are coded as subpatterns. There are two kinds:
2126: those that look ahead of the current position in the subject string, and those
2127: that look behind it. An assertion subpattern is matched in the normal way,
2128: except that it does not cause the current matching position to be changed.
2129: .P
2130: Assertion subpatterns are not capturing subpatterns. If such an assertion
2131: contains capturing subpatterns within it, these are counted for the purposes of
2132: numbering the capturing subpatterns in the whole pattern. However, substring
2133: capturing is carried out only for positive assertions. (Perl sometimes, but not
2134: always, does do capturing in negative assertions.)
2135: .P
2136: For compatibility with Perl, assertion subpatterns may be repeated; though
2137: it makes no sense to assert the same thing several times, the side effect of
2138: capturing parentheses may occasionally be useful. In practice, there only three
2139: cases:
2140: .sp
2141: (1) If the quantifier is {0}, the assertion is never obeyed during matching.
2142: However, it may contain internal capturing parenthesized groups that are called
2143: from elsewhere via the
2144: .\" HTML <a href="#subpatternsassubroutines">
2145: .\" </a>
2146: subroutine mechanism.
2147: .\"
2148: .sp
2149: (2) If quantifier is {0,n} where n is greater than zero, it is treated as if it
2150: were {0,1}. At run time, the rest of the pattern match is tried with and
2151: without the assertion, the order depending on the greediness of the quantifier.
2152: .sp
2153: (3) If the minimum repetition is greater than zero, the quantifier is ignored.
2154: The assertion is obeyed just once when encountered during matching.
2155: .
2156: .
2157: .SS "Lookahead assertions"
2158: .rs
2159: .sp
2160: Lookahead assertions start with (?= for positive assertions and (?! for
2161: negative assertions. For example,
2162: .sp
2163: \ew+(?=;)
2164: .sp
2165: matches a word followed by a semicolon, but does not include the semicolon in
2166: the match, and
2167: .sp
2168: foo(?!bar)
2169: .sp
2170: matches any occurrence of "foo" that is not followed by "bar". Note that the
2171: apparently similar pattern
2172: .sp
2173: (?!foo)bar
2174: .sp
2175: does not find an occurrence of "bar" that is preceded by something other than
2176: "foo"; it finds any occurrence of "bar" whatsoever, because the assertion
2177: (?!foo) is always true when the next three characters are "bar". A
2178: lookbehind assertion is needed to achieve the other effect.
2179: .P
2180: If you want to force a matching failure at some point in a pattern, the most
2181: convenient way to do it is with (?!) because an empty string always matches, so
2182: an assertion that requires there not to be an empty string must always fail.
2183: The backtracking control verb (*FAIL) or (*F) is a synonym for (?!).
2184: .
2185: .
2186: .\" HTML <a name="lookbehind"></a>
2187: .SS "Lookbehind assertions"
2188: .rs
2189: .sp
2190: Lookbehind assertions start with (?<= for positive assertions and (?<! for
2191: negative assertions. For example,
2192: .sp
2193: (?<!foo)bar
2194: .sp
2195: does find an occurrence of "bar" that is not preceded by "foo". The contents of
2196: a lookbehind assertion are restricted such that all the strings it matches must
2197: have a fixed length. However, if there are several top-level alternatives, they
2198: do not all have to have the same fixed length. Thus
2199: .sp
2200: (?<=bullock|donkey)
2201: .sp
2202: is permitted, but
2203: .sp
2204: (?<!dogs?|cats?)
2205: .sp
2206: causes an error at compile time. Branches that match different length strings
2207: are permitted only at the top level of a lookbehind assertion. This is an
2208: extension compared with Perl, which requires all branches to match the same
2209: length of string. An assertion such as
2210: .sp
2211: (?<=ab(c|de))
2212: .sp
2213: is not permitted, because its single top-level branch can match two different
2214: lengths, but it is acceptable to PCRE if rewritten to use two top-level
2215: branches:
2216: .sp
2217: (?<=abc|abde)
2218: .sp
2219: In some cases, the escape sequence \eK
2220: .\" HTML <a href="#resetmatchstart">
2221: .\" </a>
2222: (see above)
2223: .\"
2224: can be used instead of a lookbehind assertion to get round the fixed-length
2225: restriction.
2226: .P
2227: The implementation of lookbehind assertions is, for each alternative, to
2228: temporarily move the current position back by the fixed length and then try to
2229: match. If there are insufficient characters before the current position, the
2230: assertion fails.
2231: .P
2232: In a UTF mode, PCRE does not allow the \eC escape (which matches a single data
2233: unit even in a UTF mode) to appear in lookbehind assertions, because it makes
2234: it impossible to calculate the length of the lookbehind. The \eX and \eR
2235: escapes, which can match different numbers of data units, are also not
2236: permitted.
2237: .P
2238: .\" HTML <a href="#subpatternsassubroutines">
2239: .\" </a>
2240: "Subroutine"
2241: .\"
2242: calls (see below) such as (?2) or (?&X) are permitted in lookbehinds, as long
2243: as the subpattern matches a fixed-length string.
2244: .\" HTML <a href="#recursion">
2245: .\" </a>
2246: Recursion,
2247: .\"
2248: however, is not supported.
2249: .P
2250: Possessive quantifiers can be used in conjunction with lookbehind assertions to
2251: specify efficient matching of fixed-length strings at the end of subject
2252: strings. Consider a simple pattern such as
2253: .sp
2254: abcd$
2255: .sp
2256: when applied to a long string that does not match. Because matching proceeds
2257: from left to right, PCRE will look for each "a" in the subject and then see if
2258: what follows matches the rest of the pattern. If the pattern is specified as
2259: .sp
2260: ^.*abcd$
2261: .sp
2262: the initial .* matches the entire string at first, but when this fails (because
2263: there is no following "a"), it backtracks to match all but the last character,
2264: then all but the last two characters, and so on. Once again the search for "a"
2265: covers the entire string, from right to left, so we are no better off. However,
2266: if the pattern is written as
2267: .sp
2268: ^.*+(?<=abcd)
2269: .sp
2270: there can be no backtracking for the .*+ item; it can match only the entire
2271: string. The subsequent lookbehind assertion does a single test on the last four
2272: characters. If it fails, the match fails immediately. For long strings, this
2273: approach makes a significant difference to the processing time.
2274: .
2275: .
2276: .SS "Using multiple assertions"
2277: .rs
2278: .sp
2279: Several assertions (of any sort) may occur in succession. For example,
2280: .sp
2281: (?<=\ed{3})(?<!999)foo
2282: .sp
2283: matches "foo" preceded by three digits that are not "999". Notice that each of
2284: the assertions is applied independently at the same point in the subject
2285: string. First there is a check that the previous three characters are all
2286: digits, and then there is a check that the same three characters are not "999".
2287: This pattern does \fInot\fP match "foo" preceded by six characters, the first
2288: of which are digits and the last three of which are not "999". For example, it
2289: doesn't match "123abcfoo". A pattern to do that is
2290: .sp
2291: (?<=\ed{3}...)(?<!999)foo
2292: .sp
2293: This time the first assertion looks at the preceding six characters, checking
2294: that the first three are digits, and then the second assertion checks that the
2295: preceding three characters are not "999".
2296: .P
2297: Assertions can be nested in any combination. For example,
2298: .sp
2299: (?<=(?<!foo)bar)baz
2300: .sp
2301: matches an occurrence of "baz" that is preceded by "bar" which in turn is not
2302: preceded by "foo", while
2303: .sp
2304: (?<=\ed{3}(?!999)...)foo
2305: .sp
2306: is another pattern that matches "foo" preceded by three digits and any three
2307: characters that are not "999".
2308: .
2309: .
2310: .\" HTML <a name="conditions"></a>
2311: .SH "CONDITIONAL SUBPATTERNS"
2312: .rs
2313: .sp
2314: It is possible to cause the matching process to obey a subpattern
2315: conditionally or to choose between two alternative subpatterns, depending on
2316: the result of an assertion, or whether a specific capturing subpattern has
2317: already been matched. The two possible forms of conditional subpattern are:
2318: .sp
2319: (?(condition)yes-pattern)
2320: (?(condition)yes-pattern|no-pattern)
2321: .sp
2322: If the condition is satisfied, the yes-pattern is used; otherwise the
2323: no-pattern (if present) is used. If there are more than two alternatives in the
2324: subpattern, a compile-time error occurs. Each of the two alternatives may
2325: itself contain nested subpatterns of any form, including conditional
2326: subpatterns; the restriction to two alternatives applies only at the level of
2327: the condition. This pattern fragment is an example where the alternatives are
2328: complex:
2329: .sp
2330: (?(1) (A|B|C) | (D | (?(2)E|F) | E) )
2331: .sp
2332: .P
2333: There are four kinds of condition: references to subpatterns, references to
2334: recursion, a pseudo-condition called DEFINE, and assertions.
2335: .
2336: .SS "Checking for a used subpattern by number"
2337: .rs
2338: .sp
2339: If the text between the parentheses consists of a sequence of digits, the
2340: condition is true if a capturing subpattern of that number has previously
2341: matched. If there is more than one capturing subpattern with the same number
2342: (see the earlier
2343: .\"
2344: .\" HTML <a href="#recursion">
2345: .\" </a>
2346: section about duplicate subpattern numbers),
2347: .\"
2348: the condition is true if any of them have matched. An alternative notation is
2349: to precede the digits with a plus or minus sign. In this case, the subpattern
2350: number is relative rather than absolute. The most recently opened parentheses
2351: can be referenced by (?(-1), the next most recent by (?(-2), and so on. Inside
2352: loops it can also make sense to refer to subsequent groups. The next
2353: parentheses to be opened can be referenced as (?(+1), and so on. (The value
2354: zero in any of these forms is not used; it provokes a compile-time error.)
2355: .P
2356: Consider the following pattern, which contains non-significant white space to
2357: make it more readable (assume the PCRE_EXTENDED option) and to divide it into
2358: three parts for ease of discussion:
2359: .sp
2360: ( \e( )? [^()]+ (?(1) \e) )
2361: .sp
2362: The first part matches an optional opening parenthesis, and if that
2363: character is present, sets it as the first captured substring. The second part
2364: matches one or more characters that are not parentheses. The third part is a
2365: conditional subpattern that tests whether or not the first set of parentheses
2366: matched. If they did, that is, if subject started with an opening parenthesis,
2367: the condition is true, and so the yes-pattern is executed and a closing
2368: parenthesis is required. Otherwise, since no-pattern is not present, the
2369: subpattern matches nothing. In other words, this pattern matches a sequence of
2370: non-parentheses, optionally enclosed in parentheses.
2371: .P
2372: If you were embedding this pattern in a larger one, you could use a relative
2373: reference:
2374: .sp
2375: ...other stuff... ( \e( )? [^()]+ (?(-1) \e) ) ...
2376: .sp
2377: This makes the fragment independent of the parentheses in the larger pattern.
2378: .
2379: .SS "Checking for a used subpattern by name"
2380: .rs
2381: .sp
2382: Perl uses the syntax (?(<name>)...) or (?('name')...) to test for a used
2383: subpattern by name. For compatibility with earlier versions of PCRE, which had
2384: this facility before Perl, the syntax (?(name)...) is also recognized.
2385: .P
2386: Rewriting the above example to use a named subpattern gives this:
2387: .sp
2388: (?<OPEN> \e( )? [^()]+ (?(<OPEN>) \e) )
2389: .sp
2390: If the name used in a condition of this kind is a duplicate, the test is
2391: applied to all subpatterns of the same name, and is true if any one of them has
2392: matched.
2393: .
2394: .SS "Checking for pattern recursion"
2395: .rs
2396: .sp
2397: If the condition is the string (R), and there is no subpattern with the name R,
2398: the condition is true if a recursive call to the whole pattern or any
2399: subpattern has been made. If digits or a name preceded by ampersand follow the
2400: letter R, for example:
2401: .sp
2402: (?(R3)...) or (?(R&name)...)
2403: .sp
2404: the condition is true if the most recent recursion is into a subpattern whose
2405: number or name is given. This condition does not check the entire recursion
2406: stack. If the name used in a condition of this kind is a duplicate, the test is
2407: applied to all subpatterns of the same name, and is true if any one of them is
2408: the most recent recursion.
2409: .P
2410: At "top level", all these recursion test conditions are false.
2411: .\" HTML <a href="#recursion">
2412: .\" </a>
2413: The syntax for recursive patterns
2414: .\"
2415: is described below.
2416: .
2417: .\" HTML <a name="subdefine"></a>
2418: .SS "Defining subpatterns for use by reference only"
2419: .rs
2420: .sp
2421: If the condition is the string (DEFINE), and there is no subpattern with the
2422: name DEFINE, the condition is always false. In this case, there may be only one
2423: alternative in the subpattern. It is always skipped if control reaches this
2424: point in the pattern; the idea of DEFINE is that it can be used to define
2425: subroutines that can be referenced from elsewhere. (The use of
2426: .\" HTML <a href="#subpatternsassubroutines">
2427: .\" </a>
2428: subroutines
2429: .\"
2430: is described below.) For example, a pattern to match an IPv4 address such as
2431: "192.168.23.245" could be written like this (ignore white space and line
2432: breaks):
2433: .sp
2434: (?(DEFINE) (?<byte> 2[0-4]\ed | 25[0-5] | 1\ed\ed | [1-9]?\ed) )
2435: \eb (?&byte) (\e.(?&byte)){3} \eb
2436: .sp
2437: The first part of the pattern is a DEFINE group inside which a another group
2438: named "byte" is defined. This matches an individual component of an IPv4
2439: address (a number less than 256). When matching takes place, this part of the
2440: pattern is skipped because DEFINE acts like a false condition. The rest of the
2441: pattern uses references to the named group to match the four dot-separated
2442: components of an IPv4 address, insisting on a word boundary at each end.
2443: .
2444: .SS "Assertion conditions"
2445: .rs
2446: .sp
2447: If the condition is not in any of the above formats, it must be an assertion.
2448: This may be a positive or negative lookahead or lookbehind assertion. Consider
2449: this pattern, again containing non-significant white space, and with the two
2450: alternatives on the second line:
2451: .sp
2452: (?(?=[^a-z]*[a-z])
2453: \ed{2}-[a-z]{3}-\ed{2} | \ed{2}-\ed{2}-\ed{2} )
2454: .sp
2455: The condition is a positive lookahead assertion that matches an optional
2456: sequence of non-letters followed by a letter. In other words, it tests for the
2457: presence of at least one letter in the subject. If a letter is found, the
2458: subject is matched against the first alternative; otherwise it is matched
2459: against the second. This pattern matches strings in one of the two forms
2460: dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits.
2461: .
2462: .
2463: .\" HTML <a name="comments"></a>
2464: .SH COMMENTS
2465: .rs
2466: .sp
2467: There are two ways of including comments in patterns that are processed by
2468: PCRE. In both cases, the start of the comment must not be in a character class,
2469: nor in the middle of any other sequence of related characters such as (?: or a
2470: subpattern name or number. The characters that make up a comment play no part
2471: in the pattern matching.
2472: .P
2473: The sequence (?# marks the start of a comment that continues up to the next
2474: closing parenthesis. Nested parentheses are not permitted. If the PCRE_EXTENDED
2475: option is set, an unescaped # character also introduces a comment, which in
2476: this case continues to immediately after the next newline character or
2477: character sequence in the pattern. Which characters are interpreted as newlines
2478: is controlled by the options passed to a compiling function or by a special
2479: sequence at the start of the pattern, as described in the section entitled
2480: .\" HTML <a href="#newlines">
2481: .\" </a>
2482: "Newline conventions"
2483: .\"
2484: above. Note that the end of this type of comment is a literal newline sequence
2485: in the pattern; escape sequences that happen to represent a newline do not
2486: count. For example, consider this pattern when PCRE_EXTENDED is set, and the
2487: default newline convention is in force:
2488: .sp
2489: abc #comment \en still comment
2490: .sp
2491: On encountering the # character, \fBpcre_compile()\fP skips along, looking for
2492: a newline in the pattern. The sequence \en is still literal at this stage, so
2493: it does not terminate the comment. Only an actual character with the code value
2494: 0x0a (the default newline) does so.
2495: .
2496: .
2497: .\" HTML <a name="recursion"></a>
2498: .SH "RECURSIVE PATTERNS"
2499: .rs
2500: .sp
2501: Consider the problem of matching a string in parentheses, allowing for
2502: unlimited nested parentheses. Without the use of recursion, the best that can
2503: be done is to use a pattern that matches up to some fixed depth of nesting. It
2504: is not possible to handle an arbitrary nesting depth.
2505: .P
2506: For some time, Perl has provided a facility that allows regular expressions to
2507: recurse (amongst other things). It does this by interpolating Perl code in the
2508: expression at run time, and the code can refer to the expression itself. A Perl
2509: pattern using code interpolation to solve the parentheses problem can be
2510: created like this:
2511: .sp
2512: $re = qr{\e( (?: (?>[^()]+) | (?p{$re}) )* \e)}x;
2513: .sp
2514: The (?p{...}) item interpolates Perl code at run time, and in this case refers
2515: recursively to the pattern in which it appears.
2516: .P
2517: Obviously, PCRE cannot support the interpolation of Perl code. Instead, it
2518: supports special syntax for recursion of the entire pattern, and also for
2519: individual subpattern recursion. After its introduction in PCRE and Python,
2520: this kind of recursion was subsequently introduced into Perl at release 5.10.
2521: .P
2522: A special item that consists of (? followed by a number greater than zero and a
2523: closing parenthesis is a recursive subroutine call of the subpattern of the
2524: given number, provided that it occurs inside that subpattern. (If not, it is a
2525: .\" HTML <a href="#subpatternsassubroutines">
2526: .\" </a>
2527: non-recursive subroutine
2528: .\"
2529: call, which is described in the next section.) The special item (?R) or (?0) is
2530: a recursive call of the entire regular expression.
2531: .P
2532: This PCRE pattern solves the nested parentheses problem (assume the
2533: PCRE_EXTENDED option is set so that white space is ignored):
2534: .sp
2535: \e( ( [^()]++ | (?R) )* \e)
2536: .sp
2537: First it matches an opening parenthesis. Then it matches any number of
2538: substrings which can either be a sequence of non-parentheses, or a recursive
2539: match of the pattern itself (that is, a correctly parenthesized substring).
2540: Finally there is a closing parenthesis. Note the use of a possessive quantifier
2541: to avoid backtracking into sequences of non-parentheses.
2542: .P
2543: If this were part of a larger pattern, you would not want to recurse the entire
2544: pattern, so instead you could use this:
2545: .sp
2546: ( \e( ( [^()]++ | (?1) )* \e) )
2547: .sp
2548: We have put the pattern into parentheses, and caused the recursion to refer to
2549: them instead of the whole pattern.
2550: .P
2551: In a larger pattern, keeping track of parenthesis numbers can be tricky. This
2552: is made easier by the use of relative references. Instead of (?1) in the
2553: pattern above you can write (?-2) to refer to the second most recently opened
2554: parentheses preceding the recursion. In other words, a negative number counts
2555: capturing parentheses leftwards from the point at which it is encountered.
2556: .P
2557: It is also possible to refer to subsequently opened parentheses, by writing
2558: references such as (?+2). However, these cannot be recursive because the
2559: reference is not inside the parentheses that are referenced. They are always
2560: .\" HTML <a href="#subpatternsassubroutines">
2561: .\" </a>
2562: non-recursive subroutine
2563: .\"
2564: calls, as described in the next section.
2565: .P
2566: An alternative approach is to use named parentheses instead. The Perl syntax
2567: for this is (?&name); PCRE's earlier syntax (?P>name) is also supported. We
2568: could rewrite the above example as follows:
2569: .sp
2570: (?<pn> \e( ( [^()]++ | (?&pn) )* \e) )
2571: .sp
2572: If there is more than one subpattern with the same name, the earliest one is
2573: used.
2574: .P
2575: This particular example pattern that we have been looking at contains nested
2576: unlimited repeats, and so the use of a possessive quantifier for matching
2577: strings of non-parentheses is important when applying the pattern to strings
2578: that do not match. For example, when this pattern is applied to
2579: .sp
2580: (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()
2581: .sp
2582: it yields "no match" quickly. However, if a possessive quantifier is not used,
2583: the match runs for a very long time indeed because there are so many different
2584: ways the + and * repeats can carve up the subject, and all have to be tested
2585: before failure can be reported.
2586: .P
2587: At the end of a match, the values of capturing parentheses are those from
2588: the outermost level. If you want to obtain intermediate values, a callout
2589: function can be used (see below and the
2590: .\" HREF
2591: \fBpcrecallout\fP
2592: .\"
2593: documentation). If the pattern above is matched against
2594: .sp
2595: (ab(cd)ef)
2596: .sp
2597: the value for the inner capturing parentheses (numbered 2) is "ef", which is
2598: the last value taken on at the top level. If a capturing subpattern is not
2599: matched at the top level, its final captured value is unset, even if it was
2600: (temporarily) set at a deeper level during the matching process.
2601: .P
2602: If there are more than 15 capturing parentheses in a pattern, PCRE has to
2603: obtain extra memory to store data during a recursion, which it does by using
2604: \fBpcre_malloc\fP, freeing it via \fBpcre_free\fP afterwards. If no memory can
2605: be obtained, the match fails with the PCRE_ERROR_NOMEMORY error.
2606: .P
2607: Do not confuse the (?R) item with the condition (R), which tests for recursion.
2608: Consider this pattern, which matches text in angle brackets, allowing for
2609: arbitrary nesting. Only digits are allowed in nested brackets (that is, when
2610: recursing), whereas any characters are permitted at the outer level.
2611: .sp
2612: < (?: (?(R) \ed++ | [^<>]*+) | (?R)) * >
2613: .sp
2614: In this pattern, (?(R) is the start of a conditional subpattern, with two
2615: different alternatives for the recursive and non-recursive cases. The (?R) item
2616: is the actual recursive call.
2617: .
2618: .
2619: .\" HTML <a name="recursiondifference"></a>
2620: .SS "Differences in recursion processing between PCRE and Perl"
2621: .rs
2622: .sp
2623: Recursion processing in PCRE differs from Perl in two important ways. In PCRE
2624: (like Python, but unlike Perl), a recursive subpattern call is always treated
2625: as an atomic group. That is, once it has matched some of the subject string, it
2626: is never re-entered, even if it contains untried alternatives and there is a
2627: subsequent matching failure. This can be illustrated by the following pattern,
2628: which purports to match a palindromic string that contains an odd number of
2629: characters (for example, "a", "aba", "abcba", "abcdcba"):
2630: .sp
2631: ^(.|(.)(?1)\e2)$
2632: .sp
2633: The idea is that it either matches a single character, or two identical
2634: characters surrounding a sub-palindrome. In Perl, this pattern works; in PCRE
2635: it does not if the pattern is longer than three characters. Consider the
2636: subject string "abcba":
2637: .P
2638: At the top level, the first character is matched, but as it is not at the end
2639: of the string, the first alternative fails; the second alternative is taken
2640: and the recursion kicks in. The recursive call to subpattern 1 successfully
2641: matches the next character ("b"). (Note that the beginning and end of line
2642: tests are not part of the recursion).
2643: .P
2644: Back at the top level, the next character ("c") is compared with what
2645: subpattern 2 matched, which was "a". This fails. Because the recursion is
2646: treated as an atomic group, there are now no backtracking points, and so the
2647: entire match fails. (Perl is able, at this point, to re-enter the recursion and
2648: try the second alternative.) However, if the pattern is written with the
2649: alternatives in the other order, things are different:
2650: .sp
2651: ^((.)(?1)\e2|.)$
2652: .sp
2653: This time, the recursing alternative is tried first, and continues to recurse
2654: until it runs out of characters, at which point the recursion fails. But this
2655: time we do have another alternative to try at the higher level. That is the big
2656: difference: in the previous case the remaining alternative is at a deeper
2657: recursion level, which PCRE cannot use.
2658: .P
2659: To change the pattern so that it matches all palindromic strings, not just
2660: those with an odd number of characters, it is tempting to change the pattern to
2661: this:
2662: .sp
2663: ^((.)(?1)\e2|.?)$
2664: .sp
2665: Again, this works in Perl, but not in PCRE, and for the same reason. When a
2666: deeper recursion has matched a single character, it cannot be entered again in
2667: order to match an empty string. The solution is to separate the two cases, and
2668: write out the odd and even cases as alternatives at the higher level:
2669: .sp
2670: ^(?:((.)(?1)\e2|)|((.)(?3)\e4|.))
2671: .sp
2672: If you want to match typical palindromic phrases, the pattern has to ignore all
2673: non-word characters, which can be done like this:
2674: .sp
2675: ^\eW*+(?:((.)\eW*+(?1)\eW*+\e2|)|((.)\eW*+(?3)\eW*+\e4|\eW*+.\eW*+))\eW*+$
2676: .sp
2677: If run with the PCRE_CASELESS option, this pattern matches phrases such as "A
2678: man, a plan, a canal: Panama!" and it works well in both PCRE and Perl. Note
2679: the use of the possessive quantifier *+ to avoid backtracking into sequences of
2680: non-word characters. Without this, PCRE takes a great deal longer (ten times or
2681: more) to match typical phrases, and Perl takes so long that you think it has
2682: gone into a loop.
2683: .P
2684: \fBWARNING\fP: The palindrome-matching patterns above work only if the subject
2685: string does not start with a palindrome that is shorter than the entire string.
2686: For example, although "abcba" is correctly matched, if the subject is "ababa",
2687: PCRE finds the palindrome "aba" at the start, then fails at top level because
2688: the end of the string does not follow. Once again, it cannot jump back into the
2689: recursion to try other alternatives, so the entire match fails.
2690: .P
2691: The second way in which PCRE and Perl differ in their recursion processing is
2692: in the handling of captured values. In Perl, when a subpattern is called
2693: recursively or as a subpattern (see the next section), it has no access to any
2694: values that were captured outside the recursion, whereas in PCRE these values
2695: can be referenced. Consider this pattern:
2696: .sp
2697: ^(.)(\e1|a(?2))
2698: .sp
2699: In PCRE, this pattern matches "bab". The first capturing parentheses match "b",
2700: then in the second group, when the back reference \e1 fails to match "b", the
2701: second alternative matches "a" and then recurses. In the recursion, \e1 does
2702: now match "b" and so the whole match succeeds. In Perl, the pattern fails to
2703: match because inside the recursive call \e1 cannot access the externally set
2704: value.
2705: .
2706: .
2707: .\" HTML <a name="subpatternsassubroutines"></a>
2708: .SH "SUBPATTERNS AS SUBROUTINES"
2709: .rs
2710: .sp
2711: If the syntax for a recursive subpattern call (either by number or by
2712: name) is used outside the parentheses to which it refers, it operates like a
2713: subroutine in a programming language. The called subpattern may be defined
2714: before or after the reference. A numbered reference can be absolute or
2715: relative, as in these examples:
2716: .sp
2717: (...(absolute)...)...(?2)...
2718: (...(relative)...)...(?-1)...
2719: (...(?+1)...(relative)...
2720: .sp
2721: An earlier example pointed out that the pattern
2722: .sp
2723: (sens|respons)e and \e1ibility
2724: .sp
2725: matches "sense and sensibility" and "response and responsibility", but not
2726: "sense and responsibility". If instead the pattern
2727: .sp
2728: (sens|respons)e and (?1)ibility
2729: .sp
2730: is used, it does match "sense and responsibility" as well as the other two
2731: strings. Another example is given in the discussion of DEFINE above.
2732: .P
2733: All subroutine calls, whether recursive or not, are always treated as atomic
2734: groups. That is, once a subroutine has matched some of the subject string, it
2735: is never re-entered, even if it contains untried alternatives and there is a
2736: subsequent matching failure. Any capturing parentheses that are set during the
2737: subroutine call revert to their previous values afterwards.
2738: .P
2739: Processing options such as case-independence are fixed when a subpattern is
2740: defined, so if it is used as a subroutine, such options cannot be changed for
2741: different calls. For example, consider this pattern:
2742: .sp
2743: (abc)(?i:(?-1))
2744: .sp
2745: It matches "abcabc". It does not match "abcABC" because the change of
2746: processing option does not affect the called subpattern.
2747: .
2748: .
2749: .\" HTML <a name="onigurumasubroutines"></a>
2750: .SH "ONIGURUMA SUBROUTINE SYNTAX"
2751: .rs
2752: .sp
2753: For compatibility with Oniguruma, the non-Perl syntax \eg followed by a name or
2754: a number enclosed either in angle brackets or single quotes, is an alternative
2755: syntax for referencing a subpattern as a subroutine, possibly recursively. Here
2756: are two of the examples used above, rewritten using this syntax:
2757: .sp
2758: (?<pn> \e( ( (?>[^()]+) | \eg<pn> )* \e) )
2759: (sens|respons)e and \eg'1'ibility
2760: .sp
2761: PCRE supports an extension to Oniguruma: if a number is preceded by a
2762: plus or a minus sign it is taken as a relative reference. For example:
2763: .sp
2764: (abc)(?i:\eg<-1>)
2765: .sp
2766: Note that \eg{...} (Perl syntax) and \eg<...> (Oniguruma syntax) are \fInot\fP
2767: synonymous. The former is a back reference; the latter is a subroutine call.
2768: .
2769: .
2770: .SH CALLOUTS
2771: .rs
2772: .sp
2773: Perl has a feature whereby using the sequence (?{...}) causes arbitrary Perl
2774: code to be obeyed in the middle of matching a regular expression. This makes it
2775: possible, amongst other things, to extract different substrings that match the
2776: same pair of parentheses when there is a repetition.
2777: .P
2778: PCRE provides a similar feature, but of course it cannot obey arbitrary Perl
2779: code. The feature is called "callout". The caller of PCRE provides an external
2780: function by putting its entry point in the global variable \fIpcre_callout\fP
2781: (8-bit library) or \fIpcre[16|32]_callout\fP (16-bit or 32-bit library).
2782: By default, this variable contains NULL, which disables all calling out.
2783: .P
2784: Within a regular expression, (?C) indicates the points at which the external
2785: function is to be called. If you want to identify different callout points, you
2786: can put a number less than 256 after the letter C. The default value is zero.
2787: For example, this pattern has two callout points:
2788: .sp
2789: (?C1)abc(?C2)def
2790: .sp
2791: If the PCRE_AUTO_CALLOUT flag is passed to a compiling function, callouts are
2792: automatically installed before each item in the pattern. They are all numbered
2793: 255. If there is a conditional group in the pattern whose condition is an
2794: assertion, an additional callout is inserted just before the condition. An
2795: explicit callout may also be set at this position, as in this example:
2796: .sp
2797: (?(?C9)(?=a)abc|def)
2798: .sp
2799: Note that this applies only to assertion conditions, not to other types of
2800: condition.
2801: .P
2802: During matching, when PCRE reaches a callout point, the external function is
2803: called. It is provided with the number of the callout, the position in the
2804: pattern, and, optionally, one item of data originally supplied by the caller of
2805: the matching function. The callout function may cause matching to proceed, to
2806: backtrack, or to fail altogether.
2807: .P
2808: By default, PCRE implements a number of optimizations at compile time and
2809: matching time, and one side-effect is that sometimes callouts are skipped. If
2810: you need all possible callouts to happen, you need to set options that disable
2811: the relevant optimizations. More details, and a complete description of the
2812: interface to the callout function, are given in the
2813: .\" HREF
2814: \fBpcrecallout\fP
2815: .\"
2816: documentation.
2817: .
2818: .
2819: .\" HTML <a name="backtrackcontrol"></a>
2820: .SH "BACKTRACKING CONTROL"
2821: .rs
2822: .sp
2823: Perl 5.10 introduced a number of "Special Backtracking Control Verbs", which
2824: are still described in the Perl documentation as "experimental and subject to
2825: change or removal in a future version of Perl". It goes on to say: "Their usage
2826: in production code should be noted to avoid problems during upgrades." The same
2827: remarks apply to the PCRE features described in this section.
2828: .P
2829: The new verbs make use of what was previously invalid syntax: an opening
2830: parenthesis followed by an asterisk. They are generally of the form
2831: (*VERB) or (*VERB:NAME). Some may take either form, possibly behaving
2832: differently depending on whether or not a name is present. A name is any
2833: sequence of characters that does not include a closing parenthesis. The maximum
2834: length of name is 255 in the 8-bit library and 65535 in the 16-bit and 32-bit
2835: libraries. If the name is empty, that is, if the closing parenthesis
2836: immediately follows the colon, the effect is as if the colon were not there.
2837: Any number of these verbs may occur in a pattern.
2838: .P
2839: Since these verbs are specifically related to backtracking, most of them can be
2840: used only when the pattern is to be matched using one of the traditional
2841: matching functions, because these use a backtracking algorithm. With the
2842: exception of (*FAIL), which behaves like a failing negative assertion, the
2843: backtracking control verbs cause an error if encountered by a DFA matching
2844: function.
2845: .P
2846: The behaviour of these verbs in
2847: .\" HTML <a href="#btrepeat">
2848: .\" </a>
2849: repeated groups,
2850: .\"
2851: .\" HTML <a href="#btassert">
2852: .\" </a>
2853: assertions,
2854: .\"
2855: and in
2856: .\" HTML <a href="#btsub">
2857: .\" </a>
2858: subpatterns called as subroutines
2859: .\"
2860: (whether or not recursively) is documented below.
2861: .
2862: .
2863: .\" HTML <a name="nooptimize"></a>
2864: .SS "Optimizations that affect backtracking verbs"
2865: .rs
2866: .sp
2867: PCRE contains some optimizations that are used to speed up matching by running
2868: some checks at the start of each match attempt. For example, it may know the
2869: minimum length of matching subject, or that a particular character must be
2870: present. When one of these optimizations bypasses the running of a match, any
2871: included backtracking verbs will not, of course, be processed. You can suppress
2872: the start-of-match optimizations by setting the PCRE_NO_START_OPTIMIZE option
2873: when calling \fBpcre_compile()\fP or \fBpcre_exec()\fP, or by starting the
2874: pattern with (*NO_START_OPT). There is more discussion of this option in the
2875: section entitled
2876: .\" HTML <a href="pcreapi.html#execoptions">
2877: .\" </a>
2878: "Option bits for \fBpcre_exec()\fP"
2879: .\"
2880: in the
2881: .\" HREF
2882: \fBpcreapi\fP
2883: .\"
2884: documentation.
2885: .P
2886: Experiments with Perl suggest that it too has similar optimizations, sometimes
2887: leading to anomalous results.
2888: .
2889: .
2890: .SS "Verbs that act immediately"
2891: .rs
2892: .sp
2893: The following verbs act as soon as they are encountered. They may not be
2894: followed by a name.
2895: .sp
2896: (*ACCEPT)
2897: .sp
2898: This verb causes the match to end successfully, skipping the remainder of the
2899: pattern. However, when it is inside a subpattern that is called as a
2900: subroutine, only that subpattern is ended successfully. Matching then continues
2901: at the outer level. If (*ACCEPT) in triggered in a positive assertion, the
2902: assertion succeeds; in a negative assertion, the assertion fails.
2903: .P
2904: If (*ACCEPT) is inside capturing parentheses, the data so far is captured. For
2905: example:
2906: .sp
2907: A((?:A|B(*ACCEPT)|C)D)
2908: .sp
2909: This matches "AB", "AAD", or "ACD"; when it matches "AB", "B" is captured by
2910: the outer parentheses.
2911: .sp
2912: (*FAIL) or (*F)
2913: .sp
2914: This verb causes a matching failure, forcing backtracking to occur. It is
2915: equivalent to (?!) but easier to read. The Perl documentation notes that it is
2916: probably useful only when combined with (?{}) or (??{}). Those are, of course,
2917: Perl features that are not present in PCRE. The nearest equivalent is the
2918: callout feature, as for example in this pattern:
2919: .sp
2920: a+(?C)(*FAIL)
2921: .sp
2922: A match with the string "aaaa" always fails, but the callout is taken before
2923: each backtrack happens (in this example, 10 times).
2924: .
2925: .
2926: .SS "Recording which path was taken"
2927: .rs
2928: .sp
2929: There is one verb whose main purpose is to track how a match was arrived at,
2930: though it also has a secondary use in conjunction with advancing the match
2931: starting point (see (*SKIP) below).
2932: .sp
2933: (*MARK:NAME) or (*:NAME)
2934: .sp
2935: A name is always required with this verb. There may be as many instances of
2936: (*MARK) as you like in a pattern, and their names do not have to be unique.
2937: .P
2938: When a match succeeds, the name of the last-encountered (*MARK:NAME),
2939: (*PRUNE:NAME), or (*THEN:NAME) on the matching path is passed back to the
2940: caller as described in the section entitled
2941: .\" HTML <a href="pcreapi.html#extradata">
2942: .\" </a>
2943: "Extra data for \fBpcre_exec()\fP"
2944: .\"
2945: in the
2946: .\" HREF
2947: \fBpcreapi\fP
2948: .\"
2949: documentation. Here is an example of \fBpcretest\fP output, where the /K
2950: modifier requests the retrieval and outputting of (*MARK) data:
2951: .sp
2952: re> /X(*MARK:A)Y|X(*MARK:B)Z/K
2953: data> XY
2954: 0: XY
2955: MK: A
2956: XZ
2957: 0: XZ
2958: MK: B
2959: .sp
2960: The (*MARK) name is tagged with "MK:" in this output, and in this example it
2961: indicates which of the two alternatives matched. This is a more efficient way
2962: of obtaining this information than putting each alternative in its own
2963: capturing parentheses.
2964: .P
2965: If a verb with a name is encountered in a positive assertion that is true, the
2966: name is recorded and passed back if it is the last-encountered. This does not
2967: happen for negative assertions or failing positive assertions.
2968: .P
2969: After a partial match or a failed match, the last encountered name in the
2970: entire match process is returned. For example:
2971: .sp
2972: re> /X(*MARK:A)Y|X(*MARK:B)Z/K
2973: data> XP
2974: No match, mark = B
2975: .sp
2976: Note that in this unanchored example the mark is retained from the match
2977: attempt that started at the letter "X" in the subject. Subsequent match
2978: attempts starting at "P" and then with an empty string do not get as far as the
2979: (*MARK) item, but nevertheless do not reset it.
2980: .P
2981: If you are interested in (*MARK) values after failed matches, you should
2982: probably set the PCRE_NO_START_OPTIMIZE option
2983: .\" HTML <a href="#nooptimize">
2984: .\" </a>
2985: (see above)
2986: .\"
2987: to ensure that the match is always attempted.
2988: .
2989: .
2990: .SS "Verbs that act after backtracking"
2991: .rs
2992: .sp
2993: The following verbs do nothing when they are encountered. Matching continues
2994: with what follows, but if there is no subsequent match, causing a backtrack to
2995: the verb, a failure is forced. That is, backtracking cannot pass to the left of
2996: the verb. However, when one of these verbs appears inside an atomic group or an
2997: assertion that is true, its effect is confined to that group, because once the
2998: group has been matched, there is never any backtracking into it. In this
2999: situation, backtracking can "jump back" to the left of the entire atomic group
3000: or assertion. (Remember also, as stated above, that this localization also
3001: applies in subroutine calls.)
3002: .P
3003: These verbs differ in exactly what kind of failure occurs when backtracking
3004: reaches them. The behaviour described below is what happens when the verb is
3005: not in a subroutine or an assertion. Subsequent sections cover these special
3006: cases.
3007: .sp
3008: (*COMMIT)
3009: .sp
3010: This verb, which may not be followed by a name, causes the whole match to fail
3011: outright if there is a later matching failure that causes backtracking to reach
3012: it. Even if the pattern is unanchored, no further attempts to find a match by
3013: advancing the starting point take place. If (*COMMIT) is the only backtracking
3014: verb that is encountered, once it has been passed \fBpcre_exec()\fP is
3015: committed to finding a match at the current starting point, or not at all. For
3016: example:
3017: .sp
3018: a+(*COMMIT)b
3019: .sp
3020: This matches "xxaab" but not "aacaab". It can be thought of as a kind of
3021: dynamic anchor, or "I've started, so I must finish." The name of the most
3022: recently passed (*MARK) in the path is passed back when (*COMMIT) forces a
3023: match failure.
3024: .P
3025: If there is more than one backtracking verb in a pattern, a different one that
3026: follows (*COMMIT) may be triggered first, so merely passing (*COMMIT) during a
3027: match does not always guarantee that a match must be at this starting point.
3028: .P
3029: Note that (*COMMIT) at the start of a pattern is not the same as an anchor,
3030: unless PCRE's start-of-match optimizations are turned off, as shown in this
3031: \fBpcretest\fP example:
3032: .sp
3033: re> /(*COMMIT)abc/
3034: data> xyzabc
3035: 0: abc
3036: xyzabc\eY
3037: No match
3038: .sp
3039: PCRE knows that any match must start with "a", so the optimization skips along
3040: the subject to "a" before running the first match attempt, which succeeds. When
3041: the optimization is disabled by the \eY escape in the second subject, the match
3042: starts at "x" and so the (*COMMIT) causes it to fail without trying any other
3043: starting points.
3044: .sp
3045: (*PRUNE) or (*PRUNE:NAME)
3046: .sp
3047: This verb causes the match to fail at the current starting position in the
3048: subject if there is a later matching failure that causes backtracking to reach
3049: it. If the pattern is unanchored, the normal "bumpalong" advance to the next
3050: starting character then happens. Backtracking can occur as usual to the left of
3051: (*PRUNE), before it is reached, or when matching to the right of (*PRUNE), but
3052: if there is no match to the right, backtracking cannot cross (*PRUNE). In
3053: simple cases, the use of (*PRUNE) is just an alternative to an atomic group or
3054: possessive quantifier, but there are some uses of (*PRUNE) that cannot be
3055: expressed in any other way. In an anchored pattern (*PRUNE) has the same effect
3056: as (*COMMIT).
3057: .P
3058: The behaviour of (*PRUNE:NAME) is the not the same as (*MARK:NAME)(*PRUNE).
3059: It is like (*MARK:NAME) in that the name is remembered for passing back to the
3060: caller. However, (*SKIP:NAME) searches only for names set with (*MARK).
3061: .sp
3062: (*SKIP)
3063: .sp
3064: This verb, when given without a name, is like (*PRUNE), except that if the
3065: pattern is unanchored, the "bumpalong" advance is not to the next character,
3066: but to the position in the subject where (*SKIP) was encountered. (*SKIP)
3067: signifies that whatever text was matched leading up to it cannot be part of a
3068: successful match. Consider:
3069: .sp
3070: a+(*SKIP)b
3071: .sp
3072: If the subject is "aaaac...", after the first match attempt fails (starting at
3073: the first character in the string), the starting point skips on to start the
3074: next attempt at "c". Note that a possessive quantifer does not have the same
3075: effect as this example; although it would suppress backtracking during the
3076: first match attempt, the second attempt would start at the second character
3077: instead of skipping on to "c".
3078: .sp
3079: (*SKIP:NAME)
3080: .sp
3081: When (*SKIP) has an associated name, its behaviour is modified. When it is
3082: triggered, the previous path through the pattern is searched for the most
3083: recent (*MARK) that has the same name. If one is found, the "bumpalong" advance
3084: is to the subject position that corresponds to that (*MARK) instead of to where
3085: (*SKIP) was encountered. If no (*MARK) with a matching name is found, the
3086: (*SKIP) is ignored.
3087: .P
3088: Note that (*SKIP:NAME) searches only for names set by (*MARK:NAME). It ignores
3089: names that are set by (*PRUNE:NAME) or (*THEN:NAME).
3090: .sp
3091: (*THEN) or (*THEN:NAME)
3092: .sp
3093: This verb causes a skip to the next innermost alternative when backtracking
3094: reaches it. That is, it cancels any further backtracking within the current
3095: alternative. Its name comes from the observation that it can be used for a
3096: pattern-based if-then-else block:
3097: .sp
3098: ( COND1 (*THEN) FOO | COND2 (*THEN) BAR | COND3 (*THEN) BAZ ) ...
3099: .sp
3100: If the COND1 pattern matches, FOO is tried (and possibly further items after
3101: the end of the group if FOO succeeds); on failure, the matcher skips to the
3102: second alternative and tries COND2, without backtracking into COND1. If that
3103: succeeds and BAR fails, COND3 is tried. If subsequently BAZ fails, there are no
3104: more alternatives, so there is a backtrack to whatever came before the entire
3105: group. If (*THEN) is not inside an alternation, it acts like (*PRUNE).
3106: .P
3107: The behaviour of (*THEN:NAME) is the not the same as (*MARK:NAME)(*THEN).
3108: It is like (*MARK:NAME) in that the name is remembered for passing back to the
3109: caller. However, (*SKIP:NAME) searches only for names set with (*MARK).
3110: .P
3111: A subpattern that does not contain a | character is just a part of the
3112: enclosing alternative; it is not a nested alternation with only one
3113: alternative. The effect of (*THEN) extends beyond such a subpattern to the
3114: enclosing alternative. Consider this pattern, where A, B, etc. are complex
3115: pattern fragments that do not contain any | characters at this level:
3116: .sp
3117: A (B(*THEN)C) | D
3118: .sp
3119: If A and B are matched, but there is a failure in C, matching does not
3120: backtrack into A; instead it moves to the next alternative, that is, D.
3121: However, if the subpattern containing (*THEN) is given an alternative, it
3122: behaves differently:
3123: .sp
3124: A (B(*THEN)C | (*FAIL)) | D
3125: .sp
3126: The effect of (*THEN) is now confined to the inner subpattern. After a failure
3127: in C, matching moves to (*FAIL), which causes the whole subpattern to fail
3128: because there are no more alternatives to try. In this case, matching does now
3129: backtrack into A.
3130: .P
3131: Note that a conditional subpattern is not considered as having two
3132: alternatives, because only one is ever used. In other words, the | character in
3133: a conditional subpattern has a different meaning. Ignoring white space,
3134: consider:
3135: .sp
3136: ^.*? (?(?=a) a | b(*THEN)c )
3137: .sp
3138: If the subject is "ba", this pattern does not match. Because .*? is ungreedy,
3139: it initially matches zero characters. The condition (?=a) then fails, the
3140: character "b" is matched, but "c" is not. At this point, matching does not
3141: backtrack to .*? as might perhaps be expected from the presence of the |
3142: character. The conditional subpattern is part of the single alternative that
3143: comprises the whole pattern, and so the match fails. (If there was a backtrack
3144: into .*?, allowing it to match "b", the match would succeed.)
3145: .P
3146: The verbs just described provide four different "strengths" of control when
3147: subsequent matching fails. (*THEN) is the weakest, carrying on the match at the
3148: next alternative. (*PRUNE) comes next, failing the match at the current
3149: starting position, but allowing an advance to the next character (for an
3150: unanchored pattern). (*SKIP) is similar, except that the advance may be more
3151: than one character. (*COMMIT) is the strongest, causing the entire match to
3152: fail.
3153: .
3154: .
3155: .SS "More than one backtracking verb"
3156: .rs
3157: .sp
3158: If more than one backtracking verb is present in a pattern, the one that is
3159: backtracked onto first acts. For example, consider this pattern, where A, B,
3160: etc. are complex pattern fragments:
3161: .sp
3162: (A(*COMMIT)B(*THEN)C|ABD)
3163: .sp
3164: If A matches but B fails, the backtrack to (*COMMIT) causes the entire match to
3165: fail. However, if A and B match, but C fails, the backtrack to (*THEN) causes
3166: the next alternative (ABD) to be tried. This behaviour is consistent, but is
3167: not always the same as Perl's. It means that if two or more backtracking verbs
3168: appear in succession, all the the last of them has no effect. Consider this
3169: example:
3170: .sp
3171: ...(*COMMIT)(*PRUNE)...
3172: .sp
3173: If there is a matching failure to the right, backtracking onto (*PRUNE) causes
3174: it to be triggered, and its action is taken. There can never be a backtrack
3175: onto (*COMMIT).
3176: .
3177: .
3178: .\" HTML <a name="btrepeat"></a>
3179: .SS "Backtracking verbs in repeated groups"
3180: .rs
3181: .sp
3182: PCRE differs from Perl in its handling of backtracking verbs in repeated
3183: groups. For example, consider:
3184: .sp
3185: /(a(*COMMIT)b)+ac/
3186: .sp
3187: If the subject is "abac", Perl matches, but PCRE fails because the (*COMMIT) in
3188: the second repeat of the group acts.
3189: .
3190: .
3191: .\" HTML <a name="btassert"></a>
3192: .SS "Backtracking verbs in assertions"
3193: .rs
3194: .sp
3195: (*FAIL) in an assertion has its normal effect: it forces an immediate backtrack.
3196: .P
3197: (*ACCEPT) in a positive assertion causes the assertion to succeed without any
3198: further processing. In a negative assertion, (*ACCEPT) causes the assertion to
3199: fail without any further processing.
3200: .P
3201: The other backtracking verbs are not treated specially if they appear in a
3202: positive assertion. In particular, (*THEN) skips to the next alternative in the
3203: innermost enclosing group that has alternations, whether or not this is within
3204: the assertion.
3205: .P
3206: Negative assertions are, however, different, in order to ensure that changing a
3207: positive assertion into a negative assertion changes its result. Backtracking
3208: into (*COMMIT), (*SKIP), or (*PRUNE) causes a negative assertion to be true,
3209: without considering any further alternative branches in the assertion.
3210: Backtracking into (*THEN) causes it to skip to the next enclosing alternative
3211: within the assertion (the normal behaviour), but if the assertion does not have
3212: such an alternative, (*THEN) behaves like (*PRUNE).
3213: .
3214: .
3215: .\" HTML <a name="btsub"></a>
3216: .SS "Backtracking verbs in subroutines"
3217: .rs
3218: .sp
3219: These behaviours occur whether or not the subpattern is called recursively.
3220: Perl's treatment of subroutines is different in some cases.
3221: .P
3222: (*FAIL) in a subpattern called as a subroutine has its normal effect: it forces
3223: an immediate backtrack.
3224: .P
3225: (*ACCEPT) in a subpattern called as a subroutine causes the subroutine match to
3226: succeed without any further processing. Matching then continues after the
3227: subroutine call.
3228: .P
3229: (*COMMIT), (*SKIP), and (*PRUNE) in a subpattern called as a subroutine cause
3230: the subroutine match to fail.
3231: .P
3232: (*THEN) skips to the next alternative in the innermost enclosing group within
3233: the subpattern that has alternatives. If there is no such group within the
3234: subpattern, (*THEN) causes the subroutine match to fail.
3235: .
3236: .
3237: .SH "SEE ALSO"
3238: .rs
3239: .sp
3240: \fBpcreapi\fP(3), \fBpcrecallout\fP(3), \fBpcrematching\fP(3),
3241: \fBpcresyntax\fP(3), \fBpcre\fP(3), \fBpcre16(3)\fP, \fBpcre32(3)\fP.
3242: .
3243: .
3244: .SH AUTHOR
3245: .rs
3246: .sp
3247: .nf
3248: Philip Hazel
3249: University Computing Service
3250: Cambridge CB2 3QH, England.
3251: .fi
3252: .
3253: .
3254: .SH REVISION
3255: .rs
3256: .sp
3257: .nf
3258: Last updated: 03 December 2013
3259: Copyright (c) 1997-2013 University of Cambridge.
3260: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrepattern.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc