Annotation of embedaddon/pcre/doc/html/pcrepartial.html, revision 1.1
1.1 ! misho 1: <html>
! 2: <head>
! 3: <title>pcrepartial specification</title>
! 4: </head>
! 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
! 6: <h1>pcrepartial man page</h1>
! 7: <p>
! 8: Return to the <a href="index.html">PCRE index page</a>.
! 9: </p>
! 10: <p>
! 11: This page is part of the PCRE HTML documentation. It was generated automatically
! 12: from the original man page. If there is any nonsense in it, please consult the
! 13: man page, in case the conversion went wrong.
! 14: <br>
! 15: <ul>
! 16: <li><a name="TOC1" href="#SEC1">PARTIAL MATCHING IN PCRE</a>
! 17: <li><a name="TOC2" href="#SEC2">PARTIAL MATCHING USING pcre_exec()</a>
! 18: <li><a name="TOC3" href="#SEC3">PARTIAL MATCHING USING pcre_dfa_exec()</a>
! 19: <li><a name="TOC4" href="#SEC4">PARTIAL MATCHING AND WORD BOUNDARIES</a>
! 20: <li><a name="TOC5" href="#SEC5">FORMERLY RESTRICTED PATTERNS</a>
! 21: <li><a name="TOC6" href="#SEC6">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a>
! 22: <li><a name="TOC7" href="#SEC7">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()</a>
! 23: <li><a name="TOC8" href="#SEC8">MULTI-SEGMENT MATCHING WITH pcre_exec()</a>
! 24: <li><a name="TOC9" href="#SEC9">ISSUES WITH MULTI-SEGMENT MATCHING</a>
! 25: <li><a name="TOC10" href="#SEC10">AUTHOR</a>
! 26: <li><a name="TOC11" href="#SEC11">REVISION</a>
! 27: </ul>
! 28: <br><a name="SEC1" href="#TOC1">PARTIAL MATCHING IN PCRE</a><br>
! 29: <P>
! 30: In normal use of PCRE, if the subject string that is passed to
! 31: <b>pcre_exec()</b> or <b>pcre_dfa_exec()</b> matches as far as it goes, but is
! 32: too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
! 33: are circumstances where it might be helpful to distinguish this case from other
! 34: cases in which there is no match.
! 35: </P>
! 36: <P>
! 37: Consider, for example, an application where a human is required to type in data
! 38: for a field with specific formatting requirements. An example might be a date
! 39: in the form <i>ddmmmyy</i>, defined by this pattern:
! 40: <pre>
! 41: ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$
! 42: </pre>
! 43: If the application sees the user's keystrokes one by one, and can check that
! 44: what has been typed so far is potentially valid, it is able to raise an error
! 45: as soon as a mistake is made, by beeping and not reflecting the character that
! 46: has been typed, for example. This immediate feedback is likely to be a better
! 47: user interface than a check that is delayed until the entire string has been
! 48: entered. Partial matching can also be useful when the subject string is very
! 49: long and is not all available at once.
! 50: </P>
! 51: <P>
! 52: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
! 53: PCRE_PARTIAL_HARD options, which can be set when calling <b>pcre_exec()</b> or
! 54: <b>pcre_dfa_exec()</b>. For backwards compatibility, PCRE_PARTIAL is a synonym
! 55: for PCRE_PARTIAL_SOFT. The essential difference between the two options is
! 56: whether or not a partial match is preferred to an alternative complete match,
! 57: though the details differ between the two matching functions. If both options
! 58: are set, PCRE_PARTIAL_HARD takes precedence.
! 59: </P>
! 60: <P>
! 61: Setting a partial matching option for <b>pcre_exec()</b> disables the use of any
! 62: just-in-time code that was set up by calling <b>pcre_study()</b> with the
! 63: PCRE_STUDY_JIT_COMPILE option. It also disables two of PCRE's standard
! 64: optimizations. PCRE remembers the last literal byte in a pattern, and abandons
! 65: matching immediately if such a byte is not present in the subject string. This
! 66: optimization cannot be used for a subject string that might match only
! 67: partially. If the pattern was studied, PCRE knows the minimum length of a
! 68: matching string, and does not bother to run the matching function on shorter
! 69: strings. This optimization is also disabled for partial matching.
! 70: </P>
! 71: <br><a name="SEC2" href="#TOC1">PARTIAL MATCHING USING pcre_exec()</a><br>
! 72: <P>
! 73: A partial match occurs during a call to <b>pcre_exec()</b> when the end of the
! 74: subject string is reached successfully, but matching cannot continue because
! 75: more characters are needed. However, at least one character in the subject must
! 76: have been inspected. This character need not form part of the final matched
! 77: string; lookbehind assertions and the \K escape sequence provide ways of
! 78: inspecting characters before the start of a matched substring. The requirement
! 79: for inspecting at least one character exists because an empty string can always
! 80: be matched; without such a restriction there would always be a partial match of
! 81: an empty string at the end of the subject.
! 82: </P>
! 83: <P>
! 84: If there are at least two slots in the offsets vector when <b>pcre_exec()</b>
! 85: returns with a partial match, the first slot is set to the offset of the
! 86: earliest character that was inspected when the partial match was found. For
! 87: convenience, the second offset points to the end of the subject so that a
! 88: substring can easily be identified.
! 89: </P>
! 90: <P>
! 91: For the majority of patterns, the first offset identifies the start of the
! 92: partially matched string. However, for patterns that contain lookbehind
! 93: assertions, or \K, or begin with \b or \B, earlier characters have been
! 94: inspected while carrying out the match. For example:
! 95: <pre>
! 96: /(?<=abc)123/
! 97: </pre>
! 98: This pattern matches "123", but only if it is preceded by "abc". If the subject
! 99: string is "xyzabc12", the offsets after a partial match are for the substring
! 100: "abc12", because all these characters are needed if another match is tried
! 101: with extra characters added to the subject.
! 102: </P>
! 103: <P>
! 104: What happens when a partial match is identified depends on which of the two
! 105: partial matching options are set.
! 106: </P>
! 107: <br><b>
! 108: PCRE_PARTIAL_SOFT with pcre_exec()
! 109: </b><br>
! 110: <P>
! 111: If PCRE_PARTIAL_SOFT is set when <b>pcre_exec()</b> identifies a partial match,
! 112: the partial match is remembered, but matching continues as normal, and other
! 113: alternatives in the pattern are tried. If no complete match can be found,
! 114: <b>pcre_exec()</b> returns PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH.
! 115: </P>
! 116: <P>
! 117: This option is "soft" because it prefers a complete match over a partial match.
! 118: All the various matching items in a pattern behave as if the subject string is
! 119: potentially complete. For example, \z, \Z, and $ match at the end of the
! 120: subject, as normal, and for \b and \B the end of the subject is treated as a
! 121: non-alphanumeric.
! 122: </P>
! 123: <P>
! 124: If there is more than one partial match, the first one that was found provides
! 125: the data that is returned. Consider this pattern:
! 126: <pre>
! 127: /123\w+X|dogY/
! 128: </pre>
! 129: If this is matched against the subject string "abc123dog", both
! 130: alternatives fail to match, but the end of the subject is reached during
! 131: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9,
! 132: identifying "123dog" as the first partial match that was found. (In this
! 133: example, there are two partial matches, because "dog" on its own partially
! 134: matches the second alternative.)
! 135: </P>
! 136: <br><b>
! 137: PCRE_PARTIAL_HARD with pcre_exec()
! 138: </b><br>
! 139: <P>
! 140: If PCRE_PARTIAL_HARD is set for <b>pcre_exec()</b>, it returns
! 141: PCRE_ERROR_PARTIAL as soon as a partial match is found, without continuing to
! 142: search for possible complete matches. This option is "hard" because it prefers
! 143: an earlier partial match over a later complete match. For this reason, the
! 144: assumption is made that the end of the supplied subject string may not be the
! 145: true end of the available data, and so, if \z, \Z, \b, \B, or $ are
! 146: encountered at the end of the subject, the result is PCRE_ERROR_PARTIAL.
! 147: </P>
! 148: <P>
! 149: Setting PCRE_PARTIAL_HARD also affects the way <b>pcre_exec()</b> checks UTF-8
! 150: subject strings for validity. Normally, an invalid UTF-8 sequence causes the
! 151: error PCRE_ERROR_BADUTF8. However, in the special case of a truncated UTF-8
! 152: character at the end of the subject, PCRE_ERROR_SHORTUTF8 is returned when
! 153: PCRE_PARTIAL_HARD is set.
! 154: </P>
! 155: <br><b>
! 156: Comparing hard and soft partial matching
! 157: </b><br>
! 158: <P>
! 159: The difference between the two partial matching options can be illustrated by a
! 160: pattern such as:
! 161: <pre>
! 162: /dog(sbody)?/
! 163: </pre>
! 164: This matches either "dog" or "dogsbody", greedily (that is, it prefers the
! 165: longer string if possible). If it is matched against the string "dog" with
! 166: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
! 167: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
! 168: if the pattern is made ungreedy the result is different:
! 169: <pre>
! 170: /dog(sbody)??/
! 171: </pre>
! 172: In this case the result is always a complete match because <b>pcre_exec()</b>
! 173: finds that first, and it never continues after finding a match. It might be
! 174: easier to follow this explanation by thinking of the two patterns like this:
! 175: <pre>
! 176: /dog(sbody)?/ is the same as /dogsbody|dog/
! 177: /dog(sbody)??/ is the same as /dog|dogsbody/
! 178: </pre>
! 179: The second pattern will never match "dogsbody" when <b>pcre_exec()</b> is
! 180: used, because it will always find the shorter match first.
! 181: </P>
! 182: <br><a name="SEC3" href="#TOC1">PARTIAL MATCHING USING pcre_dfa_exec()</a><br>
! 183: <P>
! 184: The <b>pcre_dfa_exec()</b> function moves along the subject string character by
! 185: character, without backtracking, searching for all possible matches
! 186: simultaneously. If the end of the subject is reached before the end of the
! 187: pattern, there is the possibility of a partial match, again provided that at
! 188: least one character has been inspected.
! 189: </P>
! 190: <P>
! 191: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
! 192: have been no complete matches. Otherwise, the complete matches are returned.
! 193: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
! 194: complete matches. The portion of the string that was inspected when the longest
! 195: partial match was found is set as the first matching string, provided there are
! 196: at least two slots in the offsets vector.
! 197: </P>
! 198: <P>
! 199: Because <b>pcre_dfa_exec()</b> always searches for all possible matches, and
! 200: there is no difference between greedy and ungreedy repetition, its behaviour is
! 201: different from <b>pcre_exec</b> when PCRE_PARTIAL_HARD is set. Consider the
! 202: string "dog" matched against the ungreedy pattern shown above:
! 203: <pre>
! 204: /dog(sbody)??/
! 205: </pre>
! 206: Whereas <b>pcre_exec()</b> stops as soon as it finds the complete match for
! 207: "dog", <b>pcre_dfa_exec()</b> also finds the partial match for "dogsbody", and
! 208: so returns that when PCRE_PARTIAL_HARD is set.
! 209: </P>
! 210: <br><a name="SEC4" href="#TOC1">PARTIAL MATCHING AND WORD BOUNDARIES</a><br>
! 211: <P>
! 212: If a pattern ends with one of sequences \b or \B, which test for word
! 213: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
! 214: results. Consider this pattern:
! 215: <pre>
! 216: /\bcat\b/
! 217: </pre>
! 218: This matches "cat", provided there is a word boundary at either end. If the
! 219: subject string is "the cat", the comparison of the final "t" with a following
! 220: character cannot take place, so a partial match is found. However,
! 221: <b>pcre_exec()</b> carries on with normal matching, which matches \b at the end
! 222: of the subject when the last character is a letter, thus finding a complete
! 223: match. The result, therefore, is <i>not</i> PCRE_ERROR_PARTIAL. The same thing
! 224: happens with <b>pcre_dfa_exec()</b>, because it also finds the complete match.
! 225: </P>
! 226: <P>
! 227: Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, because
! 228: then the partial match takes precedence.
! 229: </P>
! 230: <br><a name="SEC5" href="#TOC1">FORMERLY RESTRICTED PATTERNS</a><br>
! 231: <P>
! 232: For releases of PCRE prior to 8.00, because of the way certain internal
! 233: optimizations were implemented in the <b>pcre_exec()</b> function, the
! 234: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
! 235: all patterns. From release 8.00 onwards, the restrictions no longer apply, and
! 236: partial matching with <b>pcre_exec()</b> can be requested for any pattern.
! 237: </P>
! 238: <P>
! 239: Items that were formerly restricted were repeated single characters and
! 240: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
! 241: conform to the restrictions, <b>pcre_exec()</b> returned the error code
! 242: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
! 243: PCRE_INFO_OKPARTIAL call to <b>pcre_fullinfo()</b> to find out if a compiled
! 244: pattern can be used for partial matching now always returns 1.
! 245: </P>
! 246: <br><a name="SEC6" href="#TOC1">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a><br>
! 247: <P>
! 248: If the escape sequence \P is present in a <b>pcretest</b> data line, the
! 249: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of <b>pcretest</b>
! 250: that uses the date example quoted above:
! 251: <pre>
! 252: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
! 253: data> 25jun04\P
! 254: 0: 25jun04
! 255: 1: jun
! 256: data> 25dec3\P
! 257: Partial match: 23dec3
! 258: data> 3ju\P
! 259: Partial match: 3ju
! 260: data> 3juj\P
! 261: No match
! 262: data> j\P
! 263: No match
! 264: </pre>
! 265: The first data string is matched completely, so <b>pcretest</b> shows the
! 266: matched substrings. The remaining four strings do not match the complete
! 267: pattern, but the first two are partial matches. Similar output is obtained
! 268: when <b>pcre_dfa_exec()</b> is used.
! 269: </P>
! 270: <P>
! 271: If the escape sequence \P is present more than once in a <b>pcretest</b> data
! 272: line, the PCRE_PARTIAL_HARD option is set for the match.
! 273: </P>
! 274: <br><a name="SEC7" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()</a><br>
! 275: <P>
! 276: When a partial match has been found using <b>pcre_dfa_exec()</b>, it is possible
! 277: to continue the match by providing additional subject data and calling
! 278: <b>pcre_dfa_exec()</b> again with the same compiled regular expression, this
! 279: time setting the PCRE_DFA_RESTART option. You must pass the same working
! 280: space as before, because this is where details of the previous partial match
! 281: are stored. Here is an example using <b>pcretest</b>, using the \R escape
! 282: sequence to set the PCRE_DFA_RESTART option (\D specifies the use of
! 283: <b>pcre_dfa_exec()</b>):
! 284: <pre>
! 285: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/
! 286: data> 23ja\P\D
! 287: Partial match: 23ja
! 288: data> n05\R\D
! 289: 0: n05
! 290: </pre>
! 291: The first call has "23ja" as the subject, and requests partial matching; the
! 292: second call has "n05" as the subject for the continued (restarted) match.
! 293: Notice that when the match is complete, only the last part is shown; PCRE does
! 294: not retain the previously partially-matched string. It is up to the calling
! 295: program to do that if it needs to.
! 296: </P>
! 297: <P>
! 298: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
! 299: PCRE_DFA_RESTART to continue partial matching over multiple segments. This
! 300: facility can be used to pass very long subject strings to
! 301: <b>pcre_dfa_exec()</b>.
! 302: </P>
! 303: <br><a name="SEC8" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_exec()</a><br>
! 304: <P>
! 305: From release 8.00, <b>pcre_exec()</b> can also be used to do multi-segment
! 306: matching. Unlike <b>pcre_dfa_exec()</b>, it is not possible to restart the
! 307: previous match with a new segment of data. Instead, new data must be added to
! 308: the previous subject string, and the entire match re-run, starting from the
! 309: point where the partial match occurred. Earlier data can be discarded. It is
! 310: best to use PCRE_PARTIAL_HARD in this situation, because it does not treat the
! 311: end of a segment as the end of the subject when matching \z, \Z, \b, \B,
! 312: and $. Consider an unanchored pattern that matches dates:
! 313: <pre>
! 314: re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/
! 315: data> The date is 23ja\P\P
! 316: Partial match: 23ja
! 317: </pre>
! 318: At this stage, an application could discard the text preceding "23ja", add on
! 319: text from the next segment, and call <b>pcre_exec()</b> again. Unlike
! 320: <b>pcre_dfa_exec()</b>, the entire matching string must always be available, and
! 321: the complete matching process occurs for each call, so more memory and more
! 322: processing time is needed.
! 323: </P>
! 324: <P>
! 325: <b>Note:</b> If the pattern contains lookbehind assertions, or \K, or starts
! 326: with \b or \B, the string that is returned for a partial match will include
! 327: characters that precede the partially matched string itself, because these must
! 328: be retained when adding on more characters for a subsequent matching attempt.
! 329: </P>
! 330: <br><a name="SEC9" href="#TOC1">ISSUES WITH MULTI-SEGMENT MATCHING</a><br>
! 331: <P>
! 332: Certain types of pattern may give problems with multi-segment matching,
! 333: whichever matching function is used.
! 334: </P>
! 335: <P>
! 336: 1. If the pattern contains a test for the beginning of a line, you need to pass
! 337: the PCRE_NOTBOL option when the subject string for any call does start at the
! 338: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when
! 339: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which
! 340: includes the effect of PCRE_NOTEOL.
! 341: </P>
! 342: <P>
! 343: 2. Lookbehind assertions at the start of a pattern are catered for in the
! 344: offsets that are returned for a partial match. However, in theory, a lookbehind
! 345: assertion later in the pattern could require even earlier characters to be
! 346: inspected, and it might not have been reached when a partial match occurs. This
! 347: is probably an extremely unlikely case; you could guard against it to a certain
! 348: extent by always including extra characters at the start.
! 349: </P>
! 350: <P>
! 351: 3. Matching a subject string that is split into multiple segments may not
! 352: always produce exactly the same result as matching over one single long string,
! 353: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
! 354: Word Boundaries" above describes an issue that arises if the pattern ends with
! 355: \b or \B. Another kind of difference may occur when there are multiple
! 356: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result
! 357: is given only when there are no completed matches. This means that as soon as
! 358: the shortest match has been found, continuation to a new subject segment is no
! 359: longer possible. Consider again this <b>pcretest</b> example:
! 360: <pre>
! 361: re> /dog(sbody)?/
! 362: data> dogsb\P
! 363: 0: dog
! 364: data> do\P\D
! 365: Partial match: do
! 366: data> gsb\R\P\D
! 367: 0: g
! 368: data> dogsbody\D
! 369: 0: dogsbody
! 370: 1: dog
! 371: </pre>
! 372: The first data line passes the string "dogsb" to <b>pcre_exec()</b>, setting the
! 373: PCRE_PARTIAL_SOFT option. Although the string is a partial match for
! 374: "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter string
! 375: "dog" is a complete match. Similarly, when the subject is presented to
! 376: <b>pcre_dfa_exec()</b> in several parts ("do" and "gsb" being the first two) the
! 377: match stops when "dog" has been found, and it is not possible to continue. On
! 378: the other hand, if "dogsbody" is presented as a single string,
! 379: <b>pcre_dfa_exec()</b> finds both matches.
! 380: </P>
! 381: <P>
! 382: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching
! 383: multi-segment data. The example above then behaves differently:
! 384: <pre>
! 385: re> /dog(sbody)?/
! 386: data> dogsb\P\P
! 387: Partial match: dogsb
! 388: data> do\P\D
! 389: Partial match: do
! 390: data> gsb\R\P\P\D
! 391: Partial match: gsb
! 392: </pre>
! 393: 4. Patterns that contain alternatives at the top level which do not all
! 394: start with the same pattern item may not work as expected when
! 395: PCRE_DFA_RESTART is used with <b>pcre_dfa_exec()</b>. For example, consider this
! 396: pattern:
! 397: <pre>
! 398: 1234|3789
! 399: </pre>
! 400: If the first part of the subject is "ABC123", a partial match of the first
! 401: alternative is found at offset 3. There is no partial match for the second
! 402: alternative, because such a match does not start at the same point in the
! 403: subject string. Attempting to continue with the string "7890" does not yield a
! 404: match because only those alternatives that match at one point in the subject
! 405: are remembered. The problem arises because the start of the second alternative
! 406: matches within the first alternative. There is no problem with anchored
! 407: patterns or patterns such as:
! 408: <pre>
! 409: 1234|ABCD
! 410: </pre>
! 411: where no string can be a partial match for both alternatives. This is not a
! 412: problem if <b>pcre_exec()</b> is used, because the entire match has to be rerun
! 413: each time:
! 414: <pre>
! 415: re> /1234|3789/
! 416: data> ABC123\P\P
! 417: Partial match: 123
! 418: data> 1237890
! 419: 0: 3789
! 420: </pre>
! 421: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running
! 422: the entire match can also be used with <b>pcre_dfa_exec()</b>. Another
! 423: possibility is to work with two buffers. If a partial match at offset <i>n</i>
! 424: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on
! 425: the second buffer, you can then try a new match starting at offset <i>n+1</i> in
! 426: the first buffer.
! 427: </P>
! 428: <br><a name="SEC10" href="#TOC1">AUTHOR</a><br>
! 429: <P>
! 430: Philip Hazel
! 431: <br>
! 432: University Computing Service
! 433: <br>
! 434: Cambridge CB2 3QH, England.
! 435: <br>
! 436: </P>
! 437: <br><a name="SEC11" href="#TOC1">REVISION</a><br>
! 438: <P>
! 439: Last updated: 26 August 2011
! 440: <br>
! 441: Copyright © 1997-2011 University of Cambridge.
! 442: <br>
! 443: <p>
! 444: Return to the <a href="index.html">PCRE index page</a>.
! 445: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrepartial.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html