Return to pcrepartial.3 CVS log

Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc

pcre

    1: .TH PCREPARTIAL 3
    2: .SH NAME
    3: PCRE - Perl-compatible regular expressions
    4: .SH "PARTIAL MATCHING IN PCRE"
    5: .rs
    6: .sp
    7: In normal use of PCRE, if the subject string that is passed to a matching
    8: function matches as far as it goes, but is too short to match the entire
    9: pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where it might
   10: be helpful to distinguish this case from other cases in which there is no
   11: match.
   12: .P
   13: Consider, for example, an application where a human is required to type in data
   14: for a field with specific formatting requirements. An example might be a date
   15: in the form \fIddmmmyy\fP, defined by this pattern:
   16: .sp
   17:   ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
   18: .sp
   19: If the application sees the user's keystrokes one by one, and can check that
   20: what has been typed so far is potentially valid, it is able to raise an error
   21: as soon as a mistake is made, by beeping and not reflecting the character that
   22: has been typed, for example. This immediate feedback is likely to be a better
   23: user interface than a check that is delayed until the entire string has been
   24: entered. Partial matching can also be useful when the subject string is very
   25: long and is not all available at once.
   26: .P
   27: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
   28: PCRE_PARTIAL_HARD options, which can be set when calling any of the matching
   29: functions. For backwards compatibility, PCRE_PARTIAL is a synonym for
   30: PCRE_PARTIAL_SOFT. The essential difference between the two options is whether
   31: or not a partial match is preferred to an alternative complete match, though
   32: the details differ between the two types of matching function. If both options
   33: are set, PCRE_PARTIAL_HARD takes precedence.
   34: .P
   35: Setting a partial matching option disables the use of any just-in-time code
   36: that was set up by studying the compiled pattern with the
   37: PCRE_STUDY_JIT_COMPILE option. It also disables two of PCRE's standard
   38: optimizations. PCRE remembers the last literal data unit in a pattern, and
   39: abandons matching immediately if it is not present in the subject string. This
   40: optimization cannot be used for a subject string that might match only
   41: partially. If the pattern was studied, PCRE knows the minimum length of a
   42: matching string, and does not bother to run the matching function on shorter
   43: strings. This optimization is also disabled for partial matching.
   44: .
   45: .
   46: .SH "PARTIAL MATCHING USING pcre_exec() OR pcre16_exec()"
   47: .rs
   48: .sp
   49: A partial match occurs during a call to \fBpcre_exec()\fP or
   50: \fBpcre16_exec()\fP when the end of the subject string is reached successfully,
   51: but matching cannot continue because more characters are needed. However, at
   52: least one character in the subject must have been inspected. This character
   53: need not form part of the final matched string; lookbehind assertions and the
   54: \eK escape sequence provide ways of inspecting characters before the start of a
   55: matched substring. The requirement for inspecting at least one character exists
   56: because an empty string can always be matched; without such a restriction there
   57: would always be a partial match of an empty string at the end of the subject.
   58: .P
   59: If there are at least two slots in the offsets vector when a partial match is
   60: returned, the first slot is set to the offset of the earliest character that
   61: was inspected. For convenience, the second offset points to the end of the
   62: subject so that a substring can easily be identified.
   63: .P
   64: For the majority of patterns, the first offset identifies the start of the
   65: partially matched string. However, for patterns that contain lookbehind
   66: assertions, or \eK, or begin with \eb or \eB, earlier characters have been
   67: inspected while carrying out the match. For example:
   68: .sp
   69:   /(?<=abc)123/
   70: .sp
   71: This pattern matches "123", but only if it is preceded by "abc". If the subject
   72: string is "xyzabc12", the offsets after a partial match are for the substring
   73: "abc12", because all these characters are needed if another match is tried
   74: with extra characters added to the subject.
   75: .P
   76: What happens when a partial match is identified depends on which of the two
   77: partial matching options are set.
   78: .
   79: .
   80: .SS "PCRE_PARTIAL_SOFT WITH pcre_exec() OR pcre16_exec()"
   81: .rs
   82: .sp
   83: If PCRE_PARTIAL_SOFT is set when \fBpcre_exec()\fP or \fBpcre16_exec()\fP
   84: identifies a partial match, the partial match is remembered, but matching
   85: continues as normal, and other alternatives in the pattern are tried. If no
   86: complete match can be found, PCRE_ERROR_PARTIAL is returned instead of
   87: PCRE_ERROR_NOMATCH.
   88: .P
   89: This option is "soft" because it prefers a complete match over a partial match.
   90: All the various matching items in a pattern behave as if the subject string is
   91: potentially complete. For example, \ez, \eZ, and $ match at the end of the
   92: subject, as normal, and for \eb and \eB the end of the subject is treated as a
   93: non-alphanumeric.
   94: .P
   95: If there is more than one partial match, the first one that was found provides
   96: the data that is returned. Consider this pattern:
   97: .sp
   98:   /123\ew+X|dogY/
   99: .sp
  100: If this is matched against the subject string "abc123dog", both
  101: alternatives fail to match, but the end of the subject is reached during
  102: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9,
  103: identifying "123dog" as the first partial match that was found. (In this
  104: example, there are two partial matches, because "dog" on its own partially
  105: matches the second alternative.)
  106: .
  107: .
  108: .SS "PCRE_PARTIAL_HARD WITH pcre_exec() OR pcre16_exec()"
  109: .rs
  110: .sp
  111: If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP or \fBpcre16_exec()\fP,
  112: PCRE_ERROR_PARTIAL is returned as soon as a partial match is found, without
  113: continuing to search for possible complete matches. This option is "hard"
  114: because it prefers an earlier partial match over a later complete match. For
  115: this reason, the assumption is made that the end of the supplied subject string
  116: may not be the true end of the available data, and so, if \ez, \eZ, \eb, \eB,
  117: or $ are encountered at the end of the subject, the result is
  118: PCRE_ERROR_PARTIAL, provided that at least one character in the subject has
  119: been inspected.
  120: .P
  121: Setting PCRE_PARTIAL_HARD also affects the way UTF-8 and UTF-16
  122: subject strings are checked for validity. Normally, an invalid sequence
  123: causes the error PCRE_ERROR_BADUTF8 or PCRE_ERROR_BADUTF16. However, in the
  124: special case of a truncated character at the end of the subject,
  125: PCRE_ERROR_SHORTUTF8 or PCRE_ERROR_SHORTUTF16 is returned when
  126: PCRE_PARTIAL_HARD is set.
  127: .
  128: .
  129: .SS "Comparing hard and soft partial matching"
  130: .rs
  131: .sp
  132: The difference between the two partial matching options can be illustrated by a
  133: pattern such as:
  134: .sp
  135:   /dog(sbody)?/
  136: .sp
  137: This matches either "dog" or "dogsbody", greedily (that is, it prefers the
  138: longer string if possible). If it is matched against the string "dog" with
  139: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
  140: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
  141: if the pattern is made ungreedy the result is different:
  142: .sp
  143:   /dog(sbody)??/
  144: .sp
  145: In this case the result is always a complete match because that is found first,
  146: and matching never continues after finding a complete match. It might be easier
  147: to follow this explanation by thinking of the two patterns like this:
  148: .sp
  149:   /dog(sbody)?/    is the same as  /dogsbody|dog/
  150:   /dog(sbody)??/   is the same as  /dog|dogsbody/
  151: .sp
  152: The second pattern will never match "dogsbody", because it will always find the
  153: shorter match first.
  154: .
  155: .
  156: .SH "PARTIAL MATCHING USING pcre_dfa_exec() OR pcre16_dfa_exec()"
  157: .rs
  158: .sp
  159: The DFA functions move along the subject string character by character, without
  160: backtracking, searching for all possible matches simultaneously. If the end of
  161: the subject is reached before the end of the pattern, there is the possibility
  162: of a partial match, again provided that at least one character has been
  163: inspected.
  164: .P
  165: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
  166: have been no complete matches. Otherwise, the complete matches are returned.
  167: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
  168: complete matches. The portion of the string that was inspected when the longest
  169: partial match was found is set as the first matching string, provided there are
  170: at least two slots in the offsets vector.
  171: .P
  172: Because the DFA functions always search for all possible matches, and there is
  173: no difference between greedy and ungreedy repetition, their behaviour is
  174: different from the standard functions when PCRE_PARTIAL_HARD is set. Consider
  175: the string "dog" matched against the ungreedy pattern shown above:
  176: .sp
  177:   /dog(sbody)??/
  178: .sp
  179: Whereas the standard functions stop as soon as they find the complete match for
  180: "dog", the DFA functions also find the partial match for "dogsbody", and so
  181: return that when PCRE_PARTIAL_HARD is set.
  182: .
  183: .
  184: .SH "PARTIAL MATCHING AND WORD BOUNDARIES"
  185: .rs
  186: .sp
  187: If a pattern ends with one of sequences \eb or \eB, which test for word
  188: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
  189: results. Consider this pattern:
  190: .sp
  191:   /\ebcat\eb/
  192: .sp
  193: This matches "cat", provided there is a word boundary at either end. If the
  194: subject string is "the cat", the comparison of the final "t" with a following
  195: character cannot take place, so a partial match is found. However, normal
  196: matching carries on, and \eb matches at the end of the subject when the last
  197: character is a letter, so a complete match is found. The result, therefore, is
  198: \fInot\fP PCRE_ERROR_PARTIAL. Using PCRE_PARTIAL_HARD in this case does yield
  199: PCRE_ERROR_PARTIAL, because then the partial match takes precedence.
  200: .
  201: .
  202: .SH "FORMERLY RESTRICTED PATTERNS"
  203: .rs
  204: .sp
  205: For releases of PCRE prior to 8.00, because of the way certain internal
  206: optimizations were implemented in the \fBpcre_exec()\fP function, the
  207: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
  208: all patterns. From release 8.00 onwards, the restrictions no longer apply, and
  209: partial matching with can be requested for any pattern.
  210: .P
  211: Items that were formerly restricted were repeated single characters and
  212: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
  213: conform to the restrictions, \fBpcre_exec()\fP returned the error code
  214: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
  215: PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled
  216: pattern can be used for partial matching now always returns 1.
  217: .
  218: .
  219: .SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
  220: .rs
  221: .sp
  222: If the escape sequence \eP is present in a \fBpcretest\fP data line, the
  223: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP
  224: that uses the date example quoted above:
  225: .sp
  226:     re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
  227:   data> 25jun04\eP
  228:    0: 25jun04
  229:    1: jun
  230:   data> 25dec3\eP
  231:   Partial match: 23dec3
  232:   data> 3ju\eP
  233:   Partial match: 3ju
  234:   data> 3juj\eP
  235:   No match
  236:   data> j\eP
  237:   No match
  238: .sp
  239: The first data string is matched completely, so \fBpcretest\fP shows the
  240: matched substrings. The remaining four strings do not match the complete
  241: pattern, but the first two are partial matches. Similar output is obtained
  242: if DFA matching is used.
  243: .P
  244: If the escape sequence \eP is present more than once in a \fBpcretest\fP data
  245: line, the PCRE_PARTIAL_HARD option is set for the match.
  246: .
  247: .
  248: .SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre16_dfa_exec()"
  249: .rs
  250: .sp
  251: When a partial match has been found using a DFA matching function, it is
  252: possible to continue the match by providing additional subject data and calling
  253: the function again with the same compiled regular expression, this time setting
  254: the PCRE_DFA_RESTART option. You must pass the same working space as before,
  255: because this is where details of the previous partial match are stored. Here is
  256: an example using \fBpcretest\fP, using the \eR escape sequence to set the
  257: PCRE_DFA_RESTART option (\eD specifies the use of the DFA matching function):
  258: .sp
  259:     re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
  260:   data> 23ja\eP\eD
  261:   Partial match: 23ja
  262:   data> n05\eR\eD
  263:    0: n05
  264: .sp
  265: The first call has "23ja" as the subject, and requests partial matching; the
  266: second call has "n05" as the subject for the continued (restarted) match.
  267: Notice that when the match is complete, only the last part is shown; PCRE does
  268: not retain the previously partially-matched string. It is up to the calling
  269: program to do that if it needs to.
  270: .P
  271: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
  272: PCRE_DFA_RESTART to continue partial matching over multiple segments. This
  273: facility can be used to pass very long subject strings to the DFA matching
  274: functions.
  275: .
  276: .
  277: .SH "MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre16_exec()"
  278: .rs
  279: .sp
  280: From release 8.00, the standard matching functions can also be used to do
  281: multi-segment matching. Unlike the DFA functions, it is not possible to
  282: restart the previous match with a new segment of data. Instead, new data must
  283: be added to the previous subject string, and the entire match re-run, starting
  284: from the point where the partial match occurred. Earlier data can be discarded.
  285: .P
  286: It is best to use PCRE_PARTIAL_HARD in this situation, because it does not
  287: treat the end of a segment as the end of the subject when matching \ez, \eZ,
  288: \eb, \eB, and $. Consider an unanchored pattern that matches dates:
  289: .sp
  290:     re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/
  291:   data> The date is 23ja\eP\eP
  292:   Partial match: 23ja
  293: .sp
  294: At this stage, an application could discard the text preceding "23ja", add on
  295: text from the next segment, and call the matching function again. Unlike the
  296: DFA matching functions the entire matching string must always be available, and
  297: the complete matching process occurs for each call, so more memory and more
  298: processing time is needed.
  299: .P
  300: \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
  301: with \eb or \eB, the string that is returned for a partial match includes
  302: characters that precede the partially matched string itself, because these must
  303: be retained when adding on more characters for a subsequent matching attempt.
  304: .
  305: .
  306: .SH "ISSUES WITH MULTI-SEGMENT MATCHING"
  307: .rs
  308: .sp
  309: Certain types of pattern may give problems with multi-segment matching,
  310: whichever matching function is used.
  311: .P
  312: 1. If the pattern contains a test for the beginning of a line, you need to pass
  313: the PCRE_NOTBOL option when the subject string for any call does start at the
  314: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when
  315: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which
  316: includes the effect of PCRE_NOTEOL.
  317: .P
  318: 2. Lookbehind assertions at the start of a pattern are catered for in the
  319: offsets that are returned for a partial match. However, in theory, a lookbehind
  320: assertion later in the pattern could require even earlier characters to be
  321: inspected, and it might not have been reached when a partial match occurs. This
  322: is probably an extremely unlikely case; you could guard against it to a certain
  323: extent by always including extra characters at the start.
  324: .P
  325: 3. Matching a subject string that is split into multiple segments may not
  326: always produce exactly the same result as matching over one single long string,
  327: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
  328: Word Boundaries" above describes an issue that arises if the pattern ends with
  329: \eb or \eB. Another kind of difference may occur when there are multiple
  330: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result
  331: is given only when there are no completed matches. This means that as soon as
  332: the shortest match has been found, continuation to a new subject segment is no
  333: longer possible. Consider again this \fBpcretest\fP example:
  334: .sp
  335:     re> /dog(sbody)?/
  336:   data> dogsb\eP
  337:    0: dog
  338:   data> do\eP\eD
  339:   Partial match: do
  340:   data> gsb\eR\eP\eD
  341:    0: g
  342:   data> dogsbody\eD
  343:    0: dogsbody
  344:    1: dog
  345: .sp
  346: The first data line passes the string "dogsb" to a standard matching function,
  347: setting the PCRE_PARTIAL_SOFT option. Although the string is a partial match
  348: for "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter
  349: string "dog" is a complete match. Similarly, when the subject is presented to
  350: a DFA matching function in several parts ("do" and "gsb" being the first two)
  351: the match stops when "dog" has been found, and it is not possible to continue.
  352: On the other hand, if "dogsbody" is presented as a single string, a DFA
  353: matching function finds both matches.
  354: .P
  355: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching
  356: multi-segment data. The example above then behaves differently:
  357: .sp
  358:     re> /dog(sbody)?/
  359:   data> dogsb\eP\eP
  360:   Partial match: dogsb
  361:   data> do\eP\eD
  362:   Partial match: do
  363:   data> gsb\eR\eP\eP\eD
  364:   Partial match: gsb
  365: .sp
  366: 4. Patterns that contain alternatives at the top level which do not all start
  367: with the same pattern item may not work as expected when PCRE_DFA_RESTART is
  368: used. For example, consider this pattern:
  369: .sp
  370:   1234|3789
  371: .sp
  372: If the first part of the subject is "ABC123", a partial match of the first
  373: alternative is found at offset 3. There is no partial match for the second
  374: alternative, because such a match does not start at the same point in the
  375: subject string. Attempting to continue with the string "7890" does not yield a
  376: match because only those alternatives that match at one point in the subject
  377: are remembered. The problem arises because the start of the second alternative
  378: matches within the first alternative. There is no problem with anchored
  379: patterns or patterns such as:
  380: .sp
  381:   1234|ABCD
  382: .sp
  383: where no string can be a partial match for both alternatives. This is not a
  384: problem if a standard matching function is used, because the entire match has
  385: to be rerun each time:
  386: .sp
  387:     re> /1234|3789/
  388:   data> ABC123\eP\eP
  389:   Partial match: 123
  390:   data> 1237890
  391:    0: 3789
  392: .sp
  393: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running
  394: the entire match can also be used with the DFA matching functions. Another
  395: possibility is to work with two buffers. If a partial match at offset \fIn\fP
  396: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on
  397: the second buffer, you can then try a new match starting at offset \fIn+1\fP in
  398: the first buffer.
  399: .
  400: .
  401: .SH AUTHOR
  402: .rs
  403: .sp
  404: .nf
  405: Philip Hazel
  406: University Computing Service
  407: Cambridge CB2 3QH, England.
  408: .fi
  409: .
  410: .
  411: .SH REVISION
  412: .rs
  413: .sp
  414: .nf
  415: Last updated: 21 January 2012
  416: Copyright (c) 1997-2012 University of Cambridge.
  417: .fi