Annotation of embedaddon/pcre/doc/pcrepartial.3, revision 1.1
1.1 ! misho 1: .TH PCREPARTIAL 3
! 2: .SH NAME
! 3: PCRE - Perl-compatible regular expressions
! 4: .SH "PARTIAL MATCHING IN PCRE"
! 5: .rs
! 6: .sp
! 7: In normal use of PCRE, if the subject string that is passed to
! 8: \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is
! 9: too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
! 10: are circumstances where it might be helpful to distinguish this case from other
! 11: cases in which there is no match.
! 12: .P
! 13: Consider, for example, an application where a human is required to type in data
! 14: for a field with specific formatting requirements. An example might be a date
! 15: in the form \fIddmmmyy\fP, defined by this pattern:
! 16: .sp
! 17: ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
! 18: .sp
! 19: If the application sees the user's keystrokes one by one, and can check that
! 20: what has been typed so far is potentially valid, it is able to raise an error
! 21: as soon as a mistake is made, by beeping and not reflecting the character that
! 22: has been typed, for example. This immediate feedback is likely to be a better
! 23: user interface than a check that is delayed until the entire string has been
! 24: entered. Partial matching can also be useful when the subject string is very
! 25: long and is not all available at once.
! 26: .P
! 27: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
! 28: PCRE_PARTIAL_HARD options, which can be set when calling \fBpcre_exec()\fP or
! 29: \fBpcre_dfa_exec()\fP. For backwards compatibility, PCRE_PARTIAL is a synonym
! 30: for PCRE_PARTIAL_SOFT. The essential difference between the two options is
! 31: whether or not a partial match is preferred to an alternative complete match,
! 32: though the details differ between the two matching functions. If both options
! 33: are set, PCRE_PARTIAL_HARD takes precedence.
! 34: .P
! 35: Setting a partial matching option for \fBpcre_exec()\fP disables the use of any
! 36: just-in-time code that was set up by calling \fBpcre_study()\fP with the
! 37: PCRE_STUDY_JIT_COMPILE option. It also disables two of PCRE's standard
! 38: optimizations. PCRE remembers the last literal byte in a pattern, and abandons
! 39: matching immediately if such a byte is not present in the subject string. This
! 40: optimization cannot be used for a subject string that might match only
! 41: partially. If the pattern was studied, PCRE knows the minimum length of a
! 42: matching string, and does not bother to run the matching function on shorter
! 43: strings. This optimization is also disabled for partial matching.
! 44: .
! 45: .
! 46: .SH "PARTIAL MATCHING USING pcre_exec()"
! 47: .rs
! 48: .sp
! 49: A partial match occurs during a call to \fBpcre_exec()\fP when the end of the
! 50: subject string is reached successfully, but matching cannot continue because
! 51: more characters are needed. However, at least one character in the subject must
! 52: have been inspected. This character need not form part of the final matched
! 53: string; lookbehind assertions and the \eK escape sequence provide ways of
! 54: inspecting characters before the start of a matched substring. The requirement
! 55: for inspecting at least one character exists because an empty string can always
! 56: be matched; without such a restriction there would always be a partial match of
! 57: an empty string at the end of the subject.
! 58: .P
! 59: If there are at least two slots in the offsets vector when \fBpcre_exec()\fP
! 60: returns with a partial match, the first slot is set to the offset of the
! 61: earliest character that was inspected when the partial match was found. For
! 62: convenience, the second offset points to the end of the subject so that a
! 63: substring can easily be identified.
! 64: .P
! 65: For the majority of patterns, the first offset identifies the start of the
! 66: partially matched string. However, for patterns that contain lookbehind
! 67: assertions, or \eK, or begin with \eb or \eB, earlier characters have been
! 68: inspected while carrying out the match. For example:
! 69: .sp
! 70: /(?<=abc)123/
! 71: .sp
! 72: This pattern matches "123", but only if it is preceded by "abc". If the subject
! 73: string is "xyzabc12", the offsets after a partial match are for the substring
! 74: "abc12", because all these characters are needed if another match is tried
! 75: with extra characters added to the subject.
! 76: .P
! 77: What happens when a partial match is identified depends on which of the two
! 78: partial matching options are set.
! 79: .
! 80: .
! 81: .SS "PCRE_PARTIAL_SOFT with pcre_exec()"
! 82: .rs
! 83: .sp
! 84: If PCRE_PARTIAL_SOFT is set when \fBpcre_exec()\fP identifies a partial match,
! 85: the partial match is remembered, but matching continues as normal, and other
! 86: alternatives in the pattern are tried. If no complete match can be found,
! 87: \fBpcre_exec()\fP returns PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH.
! 88: .P
! 89: This option is "soft" because it prefers a complete match over a partial match.
! 90: All the various matching items in a pattern behave as if the subject string is
! 91: potentially complete. For example, \ez, \eZ, and $ match at the end of the
! 92: subject, as normal, and for \eb and \eB the end of the subject is treated as a
! 93: non-alphanumeric.
! 94: .P
! 95: If there is more than one partial match, the first one that was found provides
! 96: the data that is returned. Consider this pattern:
! 97: .sp
! 98: /123\ew+X|dogY/
! 99: .sp
! 100: If this is matched against the subject string "abc123dog", both
! 101: alternatives fail to match, but the end of the subject is reached during
! 102: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9,
! 103: identifying "123dog" as the first partial match that was found. (In this
! 104: example, there are two partial matches, because "dog" on its own partially
! 105: matches the second alternative.)
! 106: .
! 107: .
! 108: .SS "PCRE_PARTIAL_HARD with pcre_exec()"
! 109: .rs
! 110: .sp
! 111: If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP, it returns
! 112: PCRE_ERROR_PARTIAL as soon as a partial match is found, without continuing to
! 113: search for possible complete matches. This option is "hard" because it prefers
! 114: an earlier partial match over a later complete match. For this reason, the
! 115: assumption is made that the end of the supplied subject string may not be the
! 116: true end of the available data, and so, if \ez, \eZ, \eb, \eB, or $ are
! 117: encountered at the end of the subject, the result is PCRE_ERROR_PARTIAL.
! 118: .P
! 119: Setting PCRE_PARTIAL_HARD also affects the way \fBpcre_exec()\fP checks UTF-8
! 120: subject strings for validity. Normally, an invalid UTF-8 sequence causes the
! 121: error PCRE_ERROR_BADUTF8. However, in the special case of a truncated UTF-8
! 122: character at the end of the subject, PCRE_ERROR_SHORTUTF8 is returned when
! 123: PCRE_PARTIAL_HARD is set.
! 124: .
! 125: .
! 126: .SS "Comparing hard and soft partial matching"
! 127: .rs
! 128: .sp
! 129: The difference between the two partial matching options can be illustrated by a
! 130: pattern such as:
! 131: .sp
! 132: /dog(sbody)?/
! 133: .sp
! 134: This matches either "dog" or "dogsbody", greedily (that is, it prefers the
! 135: longer string if possible). If it is matched against the string "dog" with
! 136: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
! 137: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
! 138: if the pattern is made ungreedy the result is different:
! 139: .sp
! 140: /dog(sbody)??/
! 141: .sp
! 142: In this case the result is always a complete match because \fBpcre_exec()\fP
! 143: finds that first, and it never continues after finding a match. It might be
! 144: easier to follow this explanation by thinking of the two patterns like this:
! 145: .sp
! 146: /dog(sbody)?/ is the same as /dogsbody|dog/
! 147: /dog(sbody)??/ is the same as /dog|dogsbody/
! 148: .sp
! 149: The second pattern will never match "dogsbody" when \fBpcre_exec()\fP is
! 150: used, because it will always find the shorter match first.
! 151: .
! 152: .
! 153: .SH "PARTIAL MATCHING USING pcre_dfa_exec()"
! 154: .rs
! 155: .sp
! 156: The \fBpcre_dfa_exec()\fP function moves along the subject string character by
! 157: character, without backtracking, searching for all possible matches
! 158: simultaneously. If the end of the subject is reached before the end of the
! 159: pattern, there is the possibility of a partial match, again provided that at
! 160: least one character has been inspected.
! 161: .P
! 162: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
! 163: have been no complete matches. Otherwise, the complete matches are returned.
! 164: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
! 165: complete matches. The portion of the string that was inspected when the longest
! 166: partial match was found is set as the first matching string, provided there are
! 167: at least two slots in the offsets vector.
! 168: .P
! 169: Because \fBpcre_dfa_exec()\fP always searches for all possible matches, and
! 170: there is no difference between greedy and ungreedy repetition, its behaviour is
! 171: different from \fBpcre_exec\fP when PCRE_PARTIAL_HARD is set. Consider the
! 172: string "dog" matched against the ungreedy pattern shown above:
! 173: .sp
! 174: /dog(sbody)??/
! 175: .sp
! 176: Whereas \fBpcre_exec()\fP stops as soon as it finds the complete match for
! 177: "dog", \fBpcre_dfa_exec()\fP also finds the partial match for "dogsbody", and
! 178: so returns that when PCRE_PARTIAL_HARD is set.
! 179: .
! 180: .
! 181: .SH "PARTIAL MATCHING AND WORD BOUNDARIES"
! 182: .rs
! 183: .sp
! 184: If a pattern ends with one of sequences \eb or \eB, which test for word
! 185: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
! 186: results. Consider this pattern:
! 187: .sp
! 188: /\ebcat\eb/
! 189: .sp
! 190: This matches "cat", provided there is a word boundary at either end. If the
! 191: subject string is "the cat", the comparison of the final "t" with a following
! 192: character cannot take place, so a partial match is found. However,
! 193: \fBpcre_exec()\fP carries on with normal matching, which matches \eb at the end
! 194: of the subject when the last character is a letter, thus finding a complete
! 195: match. The result, therefore, is \fInot\fP PCRE_ERROR_PARTIAL. The same thing
! 196: happens with \fBpcre_dfa_exec()\fP, because it also finds the complete match.
! 197: .P
! 198: Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, because
! 199: then the partial match takes precedence.
! 200: .
! 201: .
! 202: .SH "FORMERLY RESTRICTED PATTERNS"
! 203: .rs
! 204: .sp
! 205: For releases of PCRE prior to 8.00, because of the way certain internal
! 206: optimizations were implemented in the \fBpcre_exec()\fP function, the
! 207: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
! 208: all patterns. From release 8.00 onwards, the restrictions no longer apply, and
! 209: partial matching with \fBpcre_exec()\fP can be requested for any pattern.
! 210: .P
! 211: Items that were formerly restricted were repeated single characters and
! 212: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
! 213: conform to the restrictions, \fBpcre_exec()\fP returned the error code
! 214: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
! 215: PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled
! 216: pattern can be used for partial matching now always returns 1.
! 217: .
! 218: .
! 219: .SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
! 220: .rs
! 221: .sp
! 222: If the escape sequence \eP is present in a \fBpcretest\fP data line, the
! 223: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP
! 224: that uses the date example quoted above:
! 225: .sp
! 226: re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
! 227: data> 25jun04\eP
! 228: 0: 25jun04
! 229: 1: jun
! 230: data> 25dec3\eP
! 231: Partial match: 23dec3
! 232: data> 3ju\eP
! 233: Partial match: 3ju
! 234: data> 3juj\eP
! 235: No match
! 236: data> j\eP
! 237: No match
! 238: .sp
! 239: The first data string is matched completely, so \fBpcretest\fP shows the
! 240: matched substrings. The remaining four strings do not match the complete
! 241: pattern, but the first two are partial matches. Similar output is obtained
! 242: when \fBpcre_dfa_exec()\fP is used.
! 243: .P
! 244: If the escape sequence \eP is present more than once in a \fBpcretest\fP data
! 245: line, the PCRE_PARTIAL_HARD option is set for the match.
! 246: .
! 247: .
! 248: .SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()"
! 249: .rs
! 250: .sp
! 251: When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible
! 252: to continue the match by providing additional subject data and calling
! 253: \fBpcre_dfa_exec()\fP again with the same compiled regular expression, this
! 254: time setting the PCRE_DFA_RESTART option. You must pass the same working
! 255: space as before, because this is where details of the previous partial match
! 256: are stored. Here is an example using \fBpcretest\fP, using the \eR escape
! 257: sequence to set the PCRE_DFA_RESTART option (\eD specifies the use of
! 258: \fBpcre_dfa_exec()\fP):
! 259: .sp
! 260: re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
! 261: data> 23ja\eP\eD
! 262: Partial match: 23ja
! 263: data> n05\eR\eD
! 264: 0: n05
! 265: .sp
! 266: The first call has "23ja" as the subject, and requests partial matching; the
! 267: second call has "n05" as the subject for the continued (restarted) match.
! 268: Notice that when the match is complete, only the last part is shown; PCRE does
! 269: not retain the previously partially-matched string. It is up to the calling
! 270: program to do that if it needs to.
! 271: .P
! 272: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
! 273: PCRE_DFA_RESTART to continue partial matching over multiple segments. This
! 274: facility can be used to pass very long subject strings to
! 275: \fBpcre_dfa_exec()\fP.
! 276: .
! 277: .
! 278: .SH "MULTI-SEGMENT MATCHING WITH pcre_exec()"
! 279: .rs
! 280: .sp
! 281: From release 8.00, \fBpcre_exec()\fP can also be used to do multi-segment
! 282: matching. Unlike \fBpcre_dfa_exec()\fP, it is not possible to restart the
! 283: previous match with a new segment of data. Instead, new data must be added to
! 284: the previous subject string, and the entire match re-run, starting from the
! 285: point where the partial match occurred. Earlier data can be discarded. It is
! 286: best to use PCRE_PARTIAL_HARD in this situation, because it does not treat the
! 287: end of a segment as the end of the subject when matching \ez, \eZ, \eb, \eB,
! 288: and $. Consider an unanchored pattern that matches dates:
! 289: .sp
! 290: re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/
! 291: data> The date is 23ja\eP\eP
! 292: Partial match: 23ja
! 293: .sp
! 294: At this stage, an application could discard the text preceding "23ja", add on
! 295: text from the next segment, and call \fBpcre_exec()\fP again. Unlike
! 296: \fBpcre_dfa_exec()\fP, the entire matching string must always be available, and
! 297: the complete matching process occurs for each call, so more memory and more
! 298: processing time is needed.
! 299: .P
! 300: \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
! 301: with \eb or \eB, the string that is returned for a partial match will include
! 302: characters that precede the partially matched string itself, because these must
! 303: be retained when adding on more characters for a subsequent matching attempt.
! 304: .
! 305: .
! 306: .SH "ISSUES WITH MULTI-SEGMENT MATCHING"
! 307: .rs
! 308: .sp
! 309: Certain types of pattern may give problems with multi-segment matching,
! 310: whichever matching function is used.
! 311: .P
! 312: 1. If the pattern contains a test for the beginning of a line, you need to pass
! 313: the PCRE_NOTBOL option when the subject string for any call does start at the
! 314: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when
! 315: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which
! 316: includes the effect of PCRE_NOTEOL.
! 317: .P
! 318: 2. Lookbehind assertions at the start of a pattern are catered for in the
! 319: offsets that are returned for a partial match. However, in theory, a lookbehind
! 320: assertion later in the pattern could require even earlier characters to be
! 321: inspected, and it might not have been reached when a partial match occurs. This
! 322: is probably an extremely unlikely case; you could guard against it to a certain
! 323: extent by always including extra characters at the start.
! 324: .P
! 325: 3. Matching a subject string that is split into multiple segments may not
! 326: always produce exactly the same result as matching over one single long string,
! 327: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
! 328: Word Boundaries" above describes an issue that arises if the pattern ends with
! 329: \eb or \eB. Another kind of difference may occur when there are multiple
! 330: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result
! 331: is given only when there are no completed matches. This means that as soon as
! 332: the shortest match has been found, continuation to a new subject segment is no
! 333: longer possible. Consider again this \fBpcretest\fP example:
! 334: .sp
! 335: re> /dog(sbody)?/
! 336: data> dogsb\eP
! 337: 0: dog
! 338: data> do\eP\eD
! 339: Partial match: do
! 340: data> gsb\eR\eP\eD
! 341: 0: g
! 342: data> dogsbody\eD
! 343: 0: dogsbody
! 344: 1: dog
! 345: .sp
! 346: The first data line passes the string "dogsb" to \fBpcre_exec()\fP, setting the
! 347: PCRE_PARTIAL_SOFT option. Although the string is a partial match for
! 348: "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter string
! 349: "dog" is a complete match. Similarly, when the subject is presented to
! 350: \fBpcre_dfa_exec()\fP in several parts ("do" and "gsb" being the first two) the
! 351: match stops when "dog" has been found, and it is not possible to continue. On
! 352: the other hand, if "dogsbody" is presented as a single string,
! 353: \fBpcre_dfa_exec()\fP finds both matches.
! 354: .P
! 355: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching
! 356: multi-segment data. The example above then behaves differently:
! 357: .sp
! 358: re> /dog(sbody)?/
! 359: data> dogsb\eP\eP
! 360: Partial match: dogsb
! 361: data> do\eP\eD
! 362: Partial match: do
! 363: data> gsb\eR\eP\eP\eD
! 364: Partial match: gsb
! 365: .sp
! 366: 4. Patterns that contain alternatives at the top level which do not all
! 367: start with the same pattern item may not work as expected when
! 368: PCRE_DFA_RESTART is used with \fBpcre_dfa_exec()\fP. For example, consider this
! 369: pattern:
! 370: .sp
! 371: 1234|3789
! 372: .sp
! 373: If the first part of the subject is "ABC123", a partial match of the first
! 374: alternative is found at offset 3. There is no partial match for the second
! 375: alternative, because such a match does not start at the same point in the
! 376: subject string. Attempting to continue with the string "7890" does not yield a
! 377: match because only those alternatives that match at one point in the subject
! 378: are remembered. The problem arises because the start of the second alternative
! 379: matches within the first alternative. There is no problem with anchored
! 380: patterns or patterns such as:
! 381: .sp
! 382: 1234|ABCD
! 383: .sp
! 384: where no string can be a partial match for both alternatives. This is not a
! 385: problem if \fBpcre_exec()\fP is used, because the entire match has to be rerun
! 386: each time:
! 387: .sp
! 388: re> /1234|3789/
! 389: data> ABC123\eP\eP
! 390: Partial match: 123
! 391: data> 1237890
! 392: 0: 3789
! 393: .sp
! 394: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running
! 395: the entire match can also be used with \fBpcre_dfa_exec()\fP. Another
! 396: possibility is to work with two buffers. If a partial match at offset \fIn\fP
! 397: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on
! 398: the second buffer, you can then try a new match starting at offset \fIn+1\fP in
! 399: the first buffer.
! 400: .
! 401: .
! 402: .SH AUTHOR
! 403: .rs
! 404: .sp
! 405: .nf
! 406: Philip Hazel
! 407: University Computing Service
! 408: Cambridge CB2 3QH, England.
! 409: .fi
! 410: .
! 411: .
! 412: .SH REVISION
! 413: .rs
! 414: .sp
! 415: .nf
! 416: Last updated: 26 August 2011
! 417: Copyright (c) 1997-2011 University of Cambridge.
! 418: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrepartial.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc