Annotation of embedaddon/pcre/doc/pcrepartial.3, revision 1.1.1.4
1.1.1.4 ! misho 1: .TH PCREPARTIAL 3 "20 February 2013" "PCRE 8.33"
1.1 misho 2: .SH NAME
3: PCRE - Perl-compatible regular expressions
4: .SH "PARTIAL MATCHING IN PCRE"
5: .rs
6: .sp
1.1.1.2 misho 7: In normal use of PCRE, if the subject string that is passed to a matching
8: function matches as far as it goes, but is too short to match the entire
9: pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where it might
10: be helpful to distinguish this case from other cases in which there is no
11: match.
1.1 misho 12: .P
13: Consider, for example, an application where a human is required to type in data
14: for a field with specific formatting requirements. An example might be a date
15: in the form \fIddmmmyy\fP, defined by this pattern:
16: .sp
17: ^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$
18: .sp
19: If the application sees the user's keystrokes one by one, and can check that
20: what has been typed so far is potentially valid, it is able to raise an error
21: as soon as a mistake is made, by beeping and not reflecting the character that
22: has been typed, for example. This immediate feedback is likely to be a better
23: user interface than a check that is delayed until the entire string has been
24: entered. Partial matching can also be useful when the subject string is very
25: long and is not all available at once.
26: .P
27: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and
1.1.1.2 misho 28: PCRE_PARTIAL_HARD options, which can be set when calling any of the matching
29: functions. For backwards compatibility, PCRE_PARTIAL is a synonym for
30: PCRE_PARTIAL_SOFT. The essential difference between the two options is whether
31: or not a partial match is preferred to an alternative complete match, though
32: the details differ between the two types of matching function. If both options
1.1 misho 33: are set, PCRE_PARTIAL_HARD takes precedence.
34: .P
1.1.1.3 misho 35: If you want to use partial matching with just-in-time optimized code, you must
1.1.1.4 ! misho 36: call \fBpcre_study()\fP, \fBpcre16_study()\fP or \fBpcre32_study()\fP with one
! 37: or both of these options:
1.1.1.3 misho 38: .sp
39: PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE
40: PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE
41: .sp
42: PCRE_STUDY_JIT_COMPILE should also be set if you are going to run non-partial
43: matches on the same pattern. If the appropriate JIT study mode has not been set
44: for a match, the interpretive matching code is used.
45: .P
46: Setting a partial matching option disables two of PCRE's standard
1.1.1.2 misho 47: optimizations. PCRE remembers the last literal data unit in a pattern, and
48: abandons matching immediately if it is not present in the subject string. This
1.1 misho 49: optimization cannot be used for a subject string that might match only
50: partially. If the pattern was studied, PCRE knows the minimum length of a
51: matching string, and does not bother to run the matching function on shorter
52: strings. This optimization is also disabled for partial matching.
53: .
54: .
1.1.1.4 ! misho 55: .SH "PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec()"
1.1 misho 56: .rs
57: .sp
1.1.1.2 misho 58: A partial match occurs during a call to \fBpcre_exec()\fP or
1.1.1.4 ! misho 59: \fBpcre[16|32]_exec()\fP when the end of the subject string is reached
! 60: successfully, but matching cannot continue because more characters are needed.
! 61: However, at least one character in the subject must have been inspected. This
! 62: character need not form part of the final matched string; lookbehind assertions
! 63: and the \eK escape sequence provide ways of inspecting characters before the
! 64: start of a matched substring. The requirement for inspecting at least one
! 65: character exists because an empty string can always be matched; without such a
! 66: restriction there would always be a partial match of an empty string at the end
! 67: of the subject.
1.1.1.2 misho 68: .P
69: If there are at least two slots in the offsets vector when a partial match is
70: returned, the first slot is set to the offset of the earliest character that
71: was inspected. For convenience, the second offset points to the end of the
1.1.1.4 ! misho 72: subject so that a substring can easily be identified. If there are at least
! 73: three slots in the offsets vector, the third slot is set to the offset of the
! 74: character where matching started.
! 75: .P
! 76: For the majority of patterns, the contents of the first and third slots will be
! 77: the same. However, for patterns that contain lookbehind assertions, or begin
! 78: with \eb or \eB, characters before the one where matching started may have been
! 79: inspected while carrying out the match. For example, consider this pattern:
1.1 misho 80: .sp
81: /(?<=abc)123/
82: .sp
83: This pattern matches "123", but only if it is preceded by "abc". If the subject
1.1.1.4 ! misho 84: string is "xyzabc12", the first two offsets after a partial match are for the
! 85: substring "abc12", because all these characters were inspected. However, the
! 86: third offset is set to 6, because that is the offset where matching began.
1.1 misho 87: .P
88: What happens when a partial match is identified depends on which of the two
89: partial matching options are set.
90: .
91: .
1.1.1.4 ! misho 92: .SS "PCRE_PARTIAL_SOFT WITH pcre_exec() OR pcre[16|32]_exec()"
1.1 misho 93: .rs
94: .sp
1.1.1.4 ! misho 95: If PCRE_PARTIAL_SOFT is set when \fBpcre_exec()\fP or \fBpcre[16|32]_exec()\fP
1.1.1.2 misho 96: identifies a partial match, the partial match is remembered, but matching
97: continues as normal, and other alternatives in the pattern are tried. If no
98: complete match can be found, PCRE_ERROR_PARTIAL is returned instead of
99: PCRE_ERROR_NOMATCH.
1.1 misho 100: .P
101: This option is "soft" because it prefers a complete match over a partial match.
102: All the various matching items in a pattern behave as if the subject string is
103: potentially complete. For example, \ez, \eZ, and $ match at the end of the
104: subject, as normal, and for \eb and \eB the end of the subject is treated as a
105: non-alphanumeric.
106: .P
107: If there is more than one partial match, the first one that was found provides
108: the data that is returned. Consider this pattern:
109: .sp
110: /123\ew+X|dogY/
111: .sp
112: If this is matched against the subject string "abc123dog", both
113: alternatives fail to match, but the end of the subject is reached during
114: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9,
115: identifying "123dog" as the first partial match that was found. (In this
116: example, there are two partial matches, because "dog" on its own partially
117: matches the second alternative.)
118: .
119: .
1.1.1.4 ! misho 120: .SS "PCRE_PARTIAL_HARD WITH pcre_exec() OR pcre[16|32]_exec()"
1.1 misho 121: .rs
122: .sp
1.1.1.4 ! misho 123: If PCRE_PARTIAL_HARD is set for \fBpcre_exec()\fP or \fBpcre[16|32]_exec()\fP,
1.1.1.2 misho 124: PCRE_ERROR_PARTIAL is returned as soon as a partial match is found, without
125: continuing to search for possible complete matches. This option is "hard"
126: because it prefers an earlier partial match over a later complete match. For
127: this reason, the assumption is made that the end of the supplied subject string
128: may not be the true end of the available data, and so, if \ez, \eZ, \eb, \eB,
129: or $ are encountered at the end of the subject, the result is
130: PCRE_ERROR_PARTIAL, provided that at least one character in the subject has
131: been inspected.
132: .P
133: Setting PCRE_PARTIAL_HARD also affects the way UTF-8 and UTF-16
134: subject strings are checked for validity. Normally, an invalid sequence
135: causes the error PCRE_ERROR_BADUTF8 or PCRE_ERROR_BADUTF16. However, in the
136: special case of a truncated character at the end of the subject,
137: PCRE_ERROR_SHORTUTF8 or PCRE_ERROR_SHORTUTF16 is returned when
1.1 misho 138: PCRE_PARTIAL_HARD is set.
139: .
140: .
141: .SS "Comparing hard and soft partial matching"
142: .rs
143: .sp
144: The difference between the two partial matching options can be illustrated by a
145: pattern such as:
146: .sp
147: /dog(sbody)?/
148: .sp
149: This matches either "dog" or "dogsbody", greedily (that is, it prefers the
150: longer string if possible). If it is matched against the string "dog" with
151: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if
152: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand,
153: if the pattern is made ungreedy the result is different:
154: .sp
155: /dog(sbody)??/
156: .sp
1.1.1.2 misho 157: In this case the result is always a complete match because that is found first,
158: and matching never continues after finding a complete match. It might be easier
159: to follow this explanation by thinking of the two patterns like this:
1.1 misho 160: .sp
161: /dog(sbody)?/ is the same as /dogsbody|dog/
162: /dog(sbody)??/ is the same as /dog|dogsbody/
163: .sp
1.1.1.2 misho 164: The second pattern will never match "dogsbody", because it will always find the
165: shorter match first.
1.1 misho 166: .
167: .
1.1.1.4 ! misho 168: .SH "PARTIAL MATCHING USING pcre_dfa_exec() OR pcre[16|32]_dfa_exec()"
1.1 misho 169: .rs
170: .sp
1.1.1.2 misho 171: The DFA functions move along the subject string character by character, without
172: backtracking, searching for all possible matches simultaneously. If the end of
173: the subject is reached before the end of the pattern, there is the possibility
174: of a partial match, again provided that at least one character has been
175: inspected.
1.1 misho 176: .P
177: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there
178: have been no complete matches. Otherwise, the complete matches are returned.
179: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any
180: complete matches. The portion of the string that was inspected when the longest
181: partial match was found is set as the first matching string, provided there are
182: at least two slots in the offsets vector.
183: .P
1.1.1.2 misho 184: Because the DFA functions always search for all possible matches, and there is
185: no difference between greedy and ungreedy repetition, their behaviour is
186: different from the standard functions when PCRE_PARTIAL_HARD is set. Consider
187: the string "dog" matched against the ungreedy pattern shown above:
1.1 misho 188: .sp
189: /dog(sbody)??/
190: .sp
1.1.1.2 misho 191: Whereas the standard functions stop as soon as they find the complete match for
192: "dog", the DFA functions also find the partial match for "dogsbody", and so
193: return that when PCRE_PARTIAL_HARD is set.
1.1 misho 194: .
195: .
196: .SH "PARTIAL MATCHING AND WORD BOUNDARIES"
197: .rs
198: .sp
199: If a pattern ends with one of sequences \eb or \eB, which test for word
200: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive
201: results. Consider this pattern:
202: .sp
203: /\ebcat\eb/
204: .sp
205: This matches "cat", provided there is a word boundary at either end. If the
206: subject string is "the cat", the comparison of the final "t" with a following
1.1.1.2 misho 207: character cannot take place, so a partial match is found. However, normal
208: matching carries on, and \eb matches at the end of the subject when the last
209: character is a letter, so a complete match is found. The result, therefore, is
210: \fInot\fP PCRE_ERROR_PARTIAL. Using PCRE_PARTIAL_HARD in this case does yield
211: PCRE_ERROR_PARTIAL, because then the partial match takes precedence.
1.1 misho 212: .
213: .
214: .SH "FORMERLY RESTRICTED PATTERNS"
215: .rs
216: .sp
217: For releases of PCRE prior to 8.00, because of the way certain internal
218: optimizations were implemented in the \fBpcre_exec()\fP function, the
219: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with
220: all patterns. From release 8.00 onwards, the restrictions no longer apply, and
1.1.1.2 misho 221: partial matching with can be requested for any pattern.
1.1 misho 222: .P
223: Items that were formerly restricted were repeated single characters and
224: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not
225: conform to the restrictions, \fBpcre_exec()\fP returned the error code
226: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The
227: PCRE_INFO_OKPARTIAL call to \fBpcre_fullinfo()\fP to find out if a compiled
228: pattern can be used for partial matching now always returns 1.
229: .
230: .
231: .SH "EXAMPLE OF PARTIAL MATCHING USING PCRETEST"
232: .rs
233: .sp
234: If the escape sequence \eP is present in a \fBpcretest\fP data line, the
235: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of \fBpcretest\fP
236: that uses the date example quoted above:
237: .sp
238: re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
239: data> 25jun04\eP
240: 0: 25jun04
241: 1: jun
242: data> 25dec3\eP
243: Partial match: 23dec3
244: data> 3ju\eP
245: Partial match: 3ju
246: data> 3juj\eP
247: No match
248: data> j\eP
249: No match
250: .sp
251: The first data string is matched completely, so \fBpcretest\fP shows the
252: matched substrings. The remaining four strings do not match the complete
253: pattern, but the first two are partial matches. Similar output is obtained
1.1.1.2 misho 254: if DFA matching is used.
1.1 misho 255: .P
256: If the escape sequence \eP is present more than once in a \fBpcretest\fP data
257: line, the PCRE_PARTIAL_HARD option is set for the match.
258: .
259: .
1.1.1.4 ! misho 260: .SH "MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre[16|32]_dfa_exec()"
1.1 misho 261: .rs
262: .sp
1.1.1.2 misho 263: When a partial match has been found using a DFA matching function, it is
264: possible to continue the match by providing additional subject data and calling
265: the function again with the same compiled regular expression, this time setting
266: the PCRE_DFA_RESTART option. You must pass the same working space as before,
267: because this is where details of the previous partial match are stored. Here is
268: an example using \fBpcretest\fP, using the \eR escape sequence to set the
269: PCRE_DFA_RESTART option (\eD specifies the use of the DFA matching function):
1.1 misho 270: .sp
271: re> /^\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed$/
272: data> 23ja\eP\eD
273: Partial match: 23ja
274: data> n05\eR\eD
275: 0: n05
276: .sp
277: The first call has "23ja" as the subject, and requests partial matching; the
278: second call has "n05" as the subject for the continued (restarted) match.
279: Notice that when the match is complete, only the last part is shown; PCRE does
280: not retain the previously partially-matched string. It is up to the calling
281: program to do that if it needs to.
282: .P
283: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with
284: PCRE_DFA_RESTART to continue partial matching over multiple segments. This
1.1.1.2 misho 285: facility can be used to pass very long subject strings to the DFA matching
286: functions.
1.1 misho 287: .
288: .
1.1.1.4 ! misho 289: .SH "MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre[16|32]_exec()"
1.1 misho 290: .rs
291: .sp
1.1.1.2 misho 292: From release 8.00, the standard matching functions can also be used to do
293: multi-segment matching. Unlike the DFA functions, it is not possible to
294: restart the previous match with a new segment of data. Instead, new data must
295: be added to the previous subject string, and the entire match re-run, starting
296: from the point where the partial match occurred. Earlier data can be discarded.
297: .P
298: It is best to use PCRE_PARTIAL_HARD in this situation, because it does not
299: treat the end of a segment as the end of the subject when matching \ez, \eZ,
300: \eb, \eB, and $. Consider an unanchored pattern that matches dates:
1.1 misho 301: .sp
302: re> /\ed?\ed(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\ed\ed/
303: data> The date is 23ja\eP\eP
304: Partial match: 23ja
305: .sp
306: At this stage, an application could discard the text preceding "23ja", add on
1.1.1.2 misho 307: text from the next segment, and call the matching function again. Unlike the
1.1.1.3 misho 308: DFA matching functions, the entire matching string must always be available,
309: and the complete matching process occurs for each call, so more memory and more
1.1 misho 310: processing time is needed.
311: .P
312: \fBNote:\fP If the pattern contains lookbehind assertions, or \eK, or starts
1.1.1.2 misho 313: with \eb or \eB, the string that is returned for a partial match includes
1.1.1.4 ! misho 314: characters that precede the start of what would be returned for a complete
! 315: match, because it contains all the characters that were inspected during the
! 316: partial match.
1.1 misho 317: .
318: .
319: .SH "ISSUES WITH MULTI-SEGMENT MATCHING"
320: .rs
321: .sp
322: Certain types of pattern may give problems with multi-segment matching,
323: whichever matching function is used.
324: .P
325: 1. If the pattern contains a test for the beginning of a line, you need to pass
326: the PCRE_NOTBOL option when the subject string for any call does start at the
327: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when
328: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which
329: includes the effect of PCRE_NOTEOL.
330: .P
1.1.1.3 misho 331: 2. Lookbehind assertions that have already been obeyed are catered for in the
332: offsets that are returned for a partial match. However a lookbehind assertion
333: later in the pattern could require even earlier characters to be inspected. You
334: can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the
1.1.1.4 ! misho 335: \fBpcre_fullinfo()\fP or \fBpcre[16|32]_fullinfo()\fP functions to obtain the
! 336: length of the longest lookbehind in the pattern. This length is given in
! 337: characters, not bytes. If you always retain at least that many characters
! 338: before the partially matched string, all should be well. (Of course, near the
! 339: start of the subject, fewer characters may be present; in that case all
! 340: characters should be retained.)
! 341: .P
! 342: From release 8.33, there is a more accurate way of deciding which characters to
! 343: retain. Instead of subtracting the length of the longest lookbehind from the
! 344: earliest inspected character (\fIoffsets[0]\fP), the match start position
! 345: (\fIoffsets[2]\fP) should be used, and the next match attempt started at the
! 346: \fIoffsets[2]\fP character by setting the \fIstartoffset\fP argument of
! 347: \fBpcre_exec()\fP or \fBpcre_dfa_exec()\fP.
! 348: .P
! 349: For example, if the pattern "(?<=123)abc" is partially
! 350: matched against the string "xx123a", the three offset values returned are 2, 6,
! 351: and 5. This indicates that the matching process that gave a partial match
! 352: started at offset 5, but the characters "123a" were all inspected. The maximum
! 353: lookbehind for that pattern is 3, so taking that away from 5 shows that we need
! 354: only keep "123a", and the next match attempt can be started at offset 3 (that
! 355: is, at "a") when further characters have been added. When the match start is
! 356: not the earliest inspected character, \fBpcretest\fP shows it explicitly:
! 357: .sp
! 358: re> "(?<=123)abc"
! 359: data> xx123a\eP\eP
! 360: Partial match at offset 5: 123a
1.1.1.3 misho 361: .P
362: 3. Because a partial match must always contain at least one character, what
363: might be considered a partial match of an empty string actually gives a "no
364: match" result. For example:
365: .sp
366: re> /c(?<=abc)x/
367: data> ab\eP
368: No match
369: .sp
370: If the next segment begins "cx", a match should be found, but this will only
371: happen if characters from the previous segment are retained. For this reason, a
372: "no match" result should be interpreted as "partial match of an empty string"
373: when the pattern contains lookbehinds.
1.1 misho 374: .P
1.1.1.3 misho 375: 4. Matching a subject string that is split into multiple segments may not
1.1 misho 376: always produce exactly the same result as matching over one single long string,
377: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and
378: Word Boundaries" above describes an issue that arises if the pattern ends with
379: \eb or \eB. Another kind of difference may occur when there are multiple
380: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result
381: is given only when there are no completed matches. This means that as soon as
382: the shortest match has been found, continuation to a new subject segment is no
383: longer possible. Consider again this \fBpcretest\fP example:
384: .sp
385: re> /dog(sbody)?/
386: data> dogsb\eP
387: 0: dog
388: data> do\eP\eD
389: Partial match: do
390: data> gsb\eR\eP\eD
391: 0: g
392: data> dogsbody\eD
393: 0: dogsbody
394: 1: dog
395: .sp
1.1.1.2 misho 396: The first data line passes the string "dogsb" to a standard matching function,
397: setting the PCRE_PARTIAL_SOFT option. Although the string is a partial match
398: for "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter
399: string "dog" is a complete match. Similarly, when the subject is presented to
400: a DFA matching function in several parts ("do" and "gsb" being the first two)
401: the match stops when "dog" has been found, and it is not possible to continue.
402: On the other hand, if "dogsbody" is presented as a single string, a DFA
403: matching function finds both matches.
1.1 misho 404: .P
405: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching
406: multi-segment data. The example above then behaves differently:
407: .sp
408: re> /dog(sbody)?/
409: data> dogsb\eP\eP
410: Partial match: dogsb
411: data> do\eP\eD
412: Partial match: do
413: data> gsb\eR\eP\eP\eD
414: Partial match: gsb
415: .sp
1.1.1.3 misho 416: 5. Patterns that contain alternatives at the top level which do not all start
1.1.1.2 misho 417: with the same pattern item may not work as expected when PCRE_DFA_RESTART is
418: used. For example, consider this pattern:
1.1 misho 419: .sp
420: 1234|3789
421: .sp
422: If the first part of the subject is "ABC123", a partial match of the first
423: alternative is found at offset 3. There is no partial match for the second
424: alternative, because such a match does not start at the same point in the
425: subject string. Attempting to continue with the string "7890" does not yield a
426: match because only those alternatives that match at one point in the subject
427: are remembered. The problem arises because the start of the second alternative
428: matches within the first alternative. There is no problem with anchored
429: patterns or patterns such as:
430: .sp
431: 1234|ABCD
432: .sp
433: where no string can be a partial match for both alternatives. This is not a
1.1.1.2 misho 434: problem if a standard matching function is used, because the entire match has
435: to be rerun each time:
1.1 misho 436: .sp
437: re> /1234|3789/
438: data> ABC123\eP\eP
439: Partial match: 123
440: data> 1237890
441: 0: 3789
442: .sp
443: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running
1.1.1.2 misho 444: the entire match can also be used with the DFA matching functions. Another
1.1 misho 445: possibility is to work with two buffers. If a partial match at offset \fIn\fP
446: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on
447: the second buffer, you can then try a new match starting at offset \fIn+1\fP in
448: the first buffer.
449: .
450: .
451: .SH AUTHOR
452: .rs
453: .sp
454: .nf
455: Philip Hazel
456: University Computing Service
457: Cambridge CB2 3QH, England.
458: .fi
459: .
460: .
461: .SH REVISION
462: .rs
463: .sp
464: .nf
1.1.1.4 ! misho 465: Last updated: 20 February 2013
! 466: Copyright (c) 1997-2013 University of Cambridge.
1.1 misho 467: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrepartial.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc