Return to pcrepartial.3 CVS log

Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc

pcre

    1: .TH PCREPARTIAL 3
    2: .SH NAME
    3: PCRE - Perl-compatible regular expressions
    4: .SH "PARTIAL MATCHING IN PCRE"
    5: .rs
    6: .sp
    7: In normal use of PCRE, if the subject string that is passed to
    8: \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP matches as far as it goes, but is
    9: too short to match the entire pattern, PCRE_ERROR_NOMATCH is returned. There
   10: are circumstances where it might be helpful to distinguish this case from other
   11: cases in which there is no match.
   12: .P
   13: Consider, for example, an application where a human is required to type in data
   14: for a field with specific formatting requirements. An example might be a date
   15: in the form \fIddmmmyy\fP, defined by this pattern:
   16: .sp
   17:   ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
   18: .sp
   19: If the application sees the user's keystrokes one by one, and can check that
   20: what has been typed so far is potentially valid, it is able to raise an error
   21: as soon as a mistake is made, by beeping and not reflecting the character that
   22: has been typed, for example. This immediate feedback is likely to be a better
   23: user interface than a check that is delayed until the entire string has been
   24: entered. Partial matching can also be useful when the subject string is very
   25: long and is not all available at once.
   26: .P
   27: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
   28: PCRE_PARTIAL_HARD options, which can be set when calling \fBpcre_exec()\fP or
   29: \fBpcre_dfa_exec()\fP. For backwards compatibility, PCRE_PARTIAL is a synonym
   30: for PCRE_PARTIAL_SOFT. The essential difference between the two options is
   31: whether or not a partial match is preferred to an alternative complete match,
   32: though the details differ between the two matching functions. If both options
   33: are set, PCRE_PARTIAL_HARD takes precedence.
   34: .P
   35: Setting a partial matching option for \fBpcre_exec()\fP disables the use of any
   36: just-in-time code that was set up by calling \fBpcre_study()\fP with the
   37: PCRE_STUDY_JIT_COMPILE option. It also disables two of PCRE's standard
   38: optimizations. PCRE remembers the last literal byte in a pattern, and abandons
   39: matching immediately if such a byte is not present in the subject string. This
   40: optimization cannot be used for a subject string that might match only
   41: partially. If the pattern was studied, PCRE knows the minimum length of a
   42: matching string, and does not bother to run the matching function on shorter
   43: strings. This optimization is also disabled for partial matching.
   44: .
   45: .
   46: .SH "PARTIAL MATCHING USING pcre_exec()"
   47: .rs
   48: .sp
   49: A partial match occurs during a call to \fBpcre_exec()\fP when the end of the
   50: subject string is reached successfully, but matching cannot continue because
   51: more characters are needed. However, at least one character in the subject must
   52: have been inspected. This character need not form part of the final matched
   53: string; lookbehind assertions and the \eK escape sequence provide ways of
   54: inspecting characters before the start of a matched substring. The requirement
   55: for inspecting at least one character exists because an empty string can always
   56: be matched; without such a restriction there would always be a partial match of
   57: an empty string at the end of the subject.
   58: .P
   59: If there are at least two slots in the offsets vector when \fBpcre_exec()\fP
   60: returns with a partial match, the first slot is set to the offset of the
   61: earliest character that was inspected when the partial match was found. For
   62: convenience, the second offset points to the end of the subject so that a
   63: substring can easily be identified.
   64: .P
   65: For the majority of patterns, the first offset identifies the start of the
   66: partially matched string. However, for patterns that contain lookbehind
   67: assertions, or \eK, or begin with \eb or \eB, earlier characters have been
   68: inspected while carrying out the match. For example:
   69: .sp
   70:   /(?<=abc)123/
   71: .sp
   72: This pattern matches "123", but only if it is preceded by "abc". If the subject
   73: string is "xyzabc12", the offsets after a partial match are for the substring
   74: "abc12", because all these characters are needed if another match is tried
   75: with extra characters added to the subject.
   76: .P
   77: What happens when a partial match is identified depends on which of the two
   78: partial matching options are set.
   79: .
   80: .
   81: .SS "PCRE_PARTIAL_SOFT with pcre_exec()"
   82: .rs
   83: .sp
   84: If PCRE_PARTIAL_SOFT is set when \fBpcre_exec()\fP identifies a partial match,
   85: the partial match is remembered, but matching continues as normal, and other
   86: alternatives in the pattern are tried. If no complete match can be found,
   87: \fBpcre_exec()\fP returns PCRE_ERROR_PARTIAL instead of PCRE_ERROR_NOMATCH.
   88: .P
   89: This option is "soft" because it prefers a complete match over a partial match.
   90: All the various matching items in a pattern behave as if the subject string is
   91: potentially complete. For example, \ez, \eZ, and $ match at the end of the
   92: subject, as normal, and for \eb and \eB the end of the subject is treated as a
   93: non-alphanumeric.
   94: .P
   95: If there is more than one partial match, the first one that was found provides
   96: the data that is returned. Consider this pattern:
   97: .sp
   98:   /123\ew+X|dogY/
   99: .sp
  100: If this is matched against the subject string "abc123dog", both
  101: alternatives fail to match, but the end of the subject is reached during
  102: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9,
  103: identifying "123dog" as the first partial match that was found. (In this
  104: example, there are two partial matches, because "dog" on its own partially
  105: matches the second alternative.)
  106: .
  107: .
  108: .SS "PCRE_PARTIAL_HARD with pcre_exec()"
  109: .rs
  110: .sp
  111: If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP, it returns
  112: PCRE_ERROR_PARTIAL as soon as a partial match is found, without continuing to
  113: search for possible complete matches. This option is "hard" because it prefers
  114: an earlier partial match over a later complete match. For this reason, the
  115: assumption is made that the end of the supplied subject string may not be the
  116: true end of the available data, and so, if \ez, \eZ, \eb, \eB, or $ are
  117: encountered at the end of the subject, the result is PCRE_ERROR_PARTIAL.
  118: .P
  119: Setting PCRE_PARTIAL_HARD also affects the way \fBpcre_exec()\fP checks UTF-8
  120: subject strings for validity. Normally, an invalid UTF-8 sequence causes the
  121: error PCRE_ERROR_BADUTF8. However, in the special case of a truncated UTF-8
  122: character at the end of the subject, PCRE_ERROR_SHORTUTF8 is returned when
  123: PCRE_PARTIAL_HARD is set.
  124: .
  125: .
  126: .SS "Comparing hard and soft partial matching"
  127: .rs
  128: .sp
  129: The difference between the two partial matching options can be illustrated by a
  130: pattern such as:
  131: .sp
  132:   /dog(sbody)?/
  133: .sp
  134: This matches either "dog" or "dogsbody", greedily (that is, it prefers the
  135: longer string if possible). If it is matched against the string "dog" with
  136: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
  137: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
  138: if the pattern is made ungreedy the result is different:
  139: .sp
  140:   /dog(sbody)??/
  141: .sp
  142: In this case the result is always a complete match because \fBpcre_exec()\fP
  143: finds that first, and it never continues after finding a match. It might be
  144: easier to follow this explanation by thinking of the two patterns like this:
  145: .sp
  146:   /dog(sbody)?/    is the same as  /dogsbody|dog/
  147:   /dog(sbody)??/   is the same as  /dog|dogsbody/
  148: .sp
  149: The second pattern will never match "dogsbody" when \fBpcre_exec()\fP is
  150: used, because it will always find the shorter match first.
  151: .
  152: .
  153: .SH "PARTIAL MATCHING USING pcre_dfa_exec()"
  154: .rs
  155: .sp
  156: The \fBpcre_dfa_exec()\fP function moves along the subject string character by
  157: character, without backtracking, searching for all possible matches
  158: simultaneously. If the end of the subject is reached before the end of the
  159: pattern, there is the possibility of a partial match, again provided that at
  160: least one character has been inspected.
  161: .P
  162: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
  163: have been no complete matches. Otherwise, the complete matches are returned.
  164: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
  165: complete matches. The portion of the string that was inspected when the longest
  166: partial match was found is set as the first matching string, provided there are
  167: at least two slots in the offsets vector.
  168: .P
  169: Because \fBpcre_dfa_exec()\fP always searches for all possible matches, and
  170: there is no difference between greedy and ungreedy repetition, its behaviour is
  171: different from \fBpcre_exec\fP when PCRE_PARTIAL_HARD is set. Consider the
  172: string "dog" matched against the ungreedy pattern shown above:
  173: .sp
  174:   /dog(sbody)??/
  175: .sp
  176: Whereas \fBpcre_exec()\fP stops as soon as it finds the complete match for
  177: "dog", \fBpcre_dfa_exec()\fP also finds the partial match for "dogsbody", and
  178: so returns that when PCRE_PARTIAL_HARD is set.
  179: .
  180: .
  181: .SH "PARTIAL MATCHING AND WORD BOUNDARIES"
  182: .rs
  183: .sp
  184: If a pattern ends with one of sequences \eb or \eB, which test for word
  185: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
  186: results. Consider this pattern:
  187: .sp
  188:   /\ebcat\eb/
  189: .sp
  190: This matches "cat", provided there is a word boundary at either end. If the
  191: subject string is "the cat", the comparison of the final "t" with a following
  192: character cannot take place, so a partial match is found. However,
  193: \fBpcre_exec()\fP carries on with normal matching, which matches \eb at the end
  194: of the subject when the last character is a letter, thus finding a complete
  195: match. The result, therefore, is \fInot\fP PCRE_ERROR_PARTIAL. The same thing
  196: happens with \fBpcre_dfa_exec()\fP, because it also finds the complete match.
  197: .P
  198: Using PCRE_PARTIAL_HARD in this case does yield PCRE_ERROR_PARTIAL, because
  199: then the partial match takes precedence.
  200: .
  201: .
  202: .SH "FORMERLY RESTRICTED PATTERNS"
  203: .rs
  204: .sp
  205: For releases of PCRE prior to 8.00, because of the way certain internal
  206: optimizations were implemented in the \fBpcre_exec()\fP function, the
  207: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
  208: all patterns. From release 8.00 onwards, the restrictions no longer apply, and
  209: partial matching with \fBpcre_exec()\fP can be requested for any pattern.
  210: .P
  211: Items that were formerly restricted were repeated single characters and
  212: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
  213: conform to the restrictions, \fBpcre_exec()\fP returned the error code
  214: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
  215: PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled
  216: pattern can be used for partial matching now always returns 1.
  217: .
  218: .
  219: .SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
  220: .rs
  221: .sp
  222: If the escape sequence \eP is present in a \fBpcretest\fP data line, the
  223: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP
  224: that uses the date example quoted above:
  225: .sp
  226:     re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
  227:   data> 25jun04\eP
  228:    0: 25jun04
  229:    1: jun
  230:   data> 25dec3\eP
  231:   Partial match: 23dec3
  232:   data> 3ju\eP
  233:   Partial match: 3ju
  234:   data> 3juj\eP
  235:   No match
  236:   data> j\eP
  237:   No match
  238: .sp
  239: The first data string is matched completely, so \fBpcretest\fP shows the
  240: matched substrings. The remaining four strings do not match the complete
  241: pattern, but the first two are partial matches. Similar output is obtained
  242: when \fBpcre_dfa_exec()\fP is used.
  243: .P
  244: If the escape sequence \eP is present more than once in a \fBpcretest\fP data
  245: line, the PCRE_PARTIAL_HARD option is set for the match.
  246: .
  247: .
  248: .SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec()"
  249: .rs
  250: .sp
  251: When a partial match has been found using \fBpcre_dfa_exec()\fP, it is possible
  252: to continue the match by providing additional subject data and calling
  253: \fBpcre_dfa_exec()\fP again with the same compiled regular expression, this
  254: time setting the PCRE_DFA_RESTART option. You must pass the same working
  255: space as before, because this is where details of the previous partial match
  256: are stored. Here is an example using \fBpcretest\fP, using the \eR escape
  257: sequence to set the PCRE_DFA_RESTART option (\eD specifies the use of
  258: \fBpcre_dfa_exec()\fP):
  259: .sp
  260:     re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
  261:   data> 23ja\eP\eD
  262:   Partial match: 23ja
  263:   data> n05\eR\eD
  264:    0: n05
  265: .sp
  266: The first call has "23ja" as the subject, and requests partial matching; the
  267: second call has "n05" as the subject for the continued (restarted) match.
  268: Notice that when the match is complete, only the last part is shown; PCRE does
  269: not retain the previously partially-matched string. It is up to the calling
  270: program to do that if it needs to.
  271: .P
  272: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
  273: PCRE_DFA_RESTART to continue partial matching over multiple segments. This
  274: facility can be used to pass very long subject strings to
  275: \fBpcre_dfa_exec()\fP.
  276: .
  277: .
  278: .SH "MULTI-SEGMENT MATCHING WITH pcre_exec()"
  279: .rs
  280: .sp
  281: From release 8.00, \fBpcre_exec()\fP can also be used to do multi-segment
  282: matching. Unlike \fBpcre_dfa_exec()\fP, it is not possible to restart the
  283: previous match with a new segment of data. Instead, new data must be added to
  284: the previous subject string, and the entire match re-run, starting from the
  285: point where the partial match occurred. Earlier data can be discarded. It is
  286: best to use PCRE_PARTIAL_HARD in this situation, because it does not treat the
  287: end of a segment as the end of the subject when matching \ez, \eZ, \eb, \eB,
  288: and $. Consider an unanchored pattern that matches dates:
  289: .sp
  290:     re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/
  291:   data> The date is 23ja\eP\eP
  292:   Partial match: 23ja
  293: .sp
  294: At this stage, an application could discard the text preceding "23ja", add on
  295: text from the next segment, and call \fBpcre_exec()\fP again. Unlike
  296: \fBpcre_dfa_exec()\fP, the entire matching string must always be available, and
  297: the complete matching process occurs for each call, so more memory and more
  298: processing time is needed.
  299: .P
  300: \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
  301: with \eb or \eB, the string that is returned for a partial match will include
  302: characters that precede the partially matched string itself, because these must
  303: be retained when adding on more characters for a subsequent matching attempt.
  304: .
  305: .
  306: .SH "ISSUES WITH MULTI-SEGMENT MATCHING"
  307: .rs
  308: .sp
  309: Certain types of pattern may give problems with multi-segment matching,
  310: whichever matching function is used.
  311: .P
  312: 1. If the pattern contains a test for the beginning of a line, you need to pass
  313: the PCRE_NOTBOL option when the subject string for any call does start at the
  314: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when
  315: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which
  316: includes the effect of PCRE_NOTEOL.
  317: .P
  318: 2. Lookbehind assertions at the start of a pattern are catered for in the
  319: offsets that are returned for a partial match. However, in theory, a lookbehind
  320: assertion later in the pattern could require even earlier characters to be
  321: inspected, and it might not have been reached when a partial match occurs. This
  322: is probably an extremely unlikely case; you could guard against it to a certain
  323: extent by always including extra characters at the start.
  324: .P
  325: 3. Matching a subject string that is split into multiple segments may not
  326: always produce exactly the same result as matching over one single long string,
  327: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
  328: Word Boundaries" above describes an issue that arises if the pattern ends with
  329: \eb or \eB. Another kind of difference may occur when there are multiple
  330: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result
  331: is given only when there are no completed matches. This means that as soon as
  332: the shortest match has been found, continuation to a new subject segment is no
  333: longer possible. Consider again this \fBpcretest\fP example:
  334: .sp
  335:     re> /dog(sbody)?/
  336:   data> dogsb\eP
  337:    0: dog
  338:   data> do\eP\eD
  339:   Partial match: do
  340:   data> gsb\eR\eP\eD
  341:    0: g
  342:   data> dogsbody\eD
  343:    0: dogsbody
  344:    1: dog
  345: .sp
  346: The first data line passes the string "dogsb" to \fBpcre_exec()\fP, setting the
  347: PCRE_PARTIAL_SOFT option. Although the string is a partial match for
  348: "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter string
  349: "dog" is a complete match. Similarly, when the subject is presented to
  350: \fBpcre_dfa_exec()\fP in several parts ("do" and "gsb" being the first two) the
  351: match stops when "dog" has been found, and it is not possible to continue. On
  352: the other hand, if "dogsbody" is presented as a single string,
  353: \fBpcre_dfa_exec()\fP finds both matches.
  354: .P
  355: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching
  356: multi-segment data. The example above then behaves differently:
  357: .sp
  358:     re> /dog(sbody)?/
  359:   data> dogsb\eP\eP
  360:   Partial match: dogsb
  361:   data> do\eP\eD
  362:   Partial match: do
  363:   data> gsb\eR\eP\eP\eD
  364:   Partial match: gsb
  365: .sp
  366: 4. Patterns that contain alternatives at the top level which do not all
  367: start with the same pattern item may not work as expected when
  368: PCRE_DFA_RESTART is used with \fBpcre_dfa_exec()\fP. For example, consider this
  369: pattern:
  370: .sp
  371:   1234|3789
  372: .sp
  373: If the first part of the subject is "ABC123", a partial match of the first
  374: alternative is found at offset 3. There is no partial match for the second
  375: alternative, because such a match does not start at the same point in the
  376: subject string. Attempting to continue with the string "7890" does not yield a
  377: match because only those alternatives that match at one point in the subject
  378: are remembered. The problem arises because the start of the second alternative
  379: matches within the first alternative. There is no problem with anchored
  380: patterns or patterns such as:
  381: .sp
  382:   1234|ABCD
  383: .sp
  384: where no string can be a partial match for both alternatives. This is not a
  385: problem if \fBpcre_exec()\fP is used, because the entire match has to be rerun
  386: each time:
  387: .sp
  388:     re> /1234|3789/
  389:   data> ABC123\eP\eP
  390:   Partial match: 123
  391:   data> 1237890
  392:    0: 3789
  393: .sp
  394: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running
  395: the entire match can also be used with \fBpcre_dfa_exec()\fP. Another
  396: possibility is to work with two buffers. If a partial match at offset \fIn\fP
  397: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on
  398: the second buffer, you can then try a new match starting at offset \fIn+1\fP in
  399: the first buffer.
  400: .
  401: .
  402: .SH AUTHOR
  403: .rs
  404: .sp
  405: .nf
  406: Philip Hazel
  407: University Computing Service
  408: Cambridge CB2 3QH, England.
  409: .fi
  410: .
  411: .
  412: .SH REVISION
  413: .rs
  414: .sp
  415: .nf
  416: Last updated: 26 August 2011
  417: Copyright (c) 1997-2011 University of Cambridge.
  418: .fi