embedaddon/pcre/doc/html/pcrepartial.html - view

File: [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html / pcrepartial.html
Revision 1.1.1.5 (vendor branch): download - view: text, annotated - select for diffs - revision graph
Sun Jun 15 19:46:05 2014 UTC (10 years, 9 months ago) by misho
Branches: pcre, MAIN
CVS tags: v8_34, HEAD

pcre 8.34

1: <html> 2: <head> 3: <title>pcrepartial specification</title> 4: </head> 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6: <h1>pcrepartial man page</h1> 7:  8: Return to the <a href="index.html">PCRE index page</a>. 9:  10:  11: This page is part of the PCRE HTML documentation. It was generated automatically 12: from the original man page. If there is any nonsense in it, please consult the 13: man page, in case the conversion went wrong. 14:   15: <ul> 16: <li><a name="TOC1" href="#SEC1">PARTIAL MATCHING IN PCRE</a> 17: <li><a name="TOC2" href="#SEC2">PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec()</a> 18: <li><a name="TOC3" href="#SEC3">PARTIAL MATCHING USING pcre_dfa_exec() OR pcre[16|32]_dfa_exec()</a> 19: <li><a name="TOC4" href="#SEC4">PARTIAL MATCHING AND WORD BOUNDARIES</a> 20: <li><a name="TOC5" href="#SEC5">FORMERLY RESTRICTED PATTERNS</a> 21: <li><a name="TOC6" href="#SEC6">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a> 22: <li><a name="TOC7" href="#SEC7">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre[16|32]_dfa_exec()</a> 23: <li><a name="TOC8" href="#SEC8">MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre[16|32]_exec()</a> 24: <li><a name="TOC9" href="#SEC9">ISSUES WITH MULTI-SEGMENT MATCHING</a> 25: <li><a name="TOC10" href="#SEC10">AUTHOR</a> 26: <li><a name="TOC11" href="#SEC11">REVISION</a> 27: </ul> 28:  <a name="SEC1" href="#TOC1">PARTIAL MATCHING IN PCRE</a>  29:  30: In normal use of PCRE, if the subject string that is passed to a matching 31: function matches as far as it goes, but is too short to match the entire 32: pattern, PCRE_ERROR_NOMATCH is returned. There are circumstances where it might 33: be helpful to distinguish this case from other cases in which there is no 34: match. 35:  36:  37: Consider, for example, an application where a human is required to type in data 38: for a field with specific formatting requirements. An example might be a date 39: in the form ddmmmyy, defined by this pattern: 40: <pre> 41: ^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$ 42: </pre> 43: If the application sees the user's keystrokes one by one, and can check that 44: what has been typed so far is potentially valid, it is able to raise an error 45: as soon as a mistake is made, by beeping and not reflecting the character that 46: has been typed, for example. This immediate feedback is likely to be a better 47: user interface than a check that is delayed until the entire string has been 48: entered. Partial matching can also be useful when the subject string is very 49: long and is not all available at once. 50:  51:  52: PCRE supports partial matching by means of the PCRE_PARTIAL_SOFT and 53: PCRE_PARTIAL_HARD options, which can be set when calling any of the matching 54: functions. For backwards compatibility, PCRE_PARTIAL is a synonym for 55: PCRE_PARTIAL_SOFT. The essential difference between the two options is whether 56: or not a partial match is preferred to an alternative complete match, though 57: the details differ between the two types of matching function. If both options 58: are set, PCRE_PARTIAL_HARD takes precedence. 59:  60:  61: If you want to use partial matching with just-in-time optimized code, you must 62: call pcre_study(), pcre16_study() or pcre32_study() with one 63: or both of these options: 64: <pre> 65: PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE 66: PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE 67: </pre> 68: PCRE_STUDY_JIT_COMPILE should also be set if you are going to run non-partial 69: matches on the same pattern. If the appropriate JIT study mode has not been set 70: for a match, the interpretive matching code is used. 71:  72:  73: Setting a partial matching option disables two of PCRE's standard 74: optimizations. PCRE remembers the last literal data unit in a pattern, and 75: abandons matching immediately if it is not present in the subject string. This 76: optimization cannot be used for a subject string that might match only 77: partially. If the pattern was studied, PCRE knows the minimum length of a 78: matching string, and does not bother to run the matching function on shorter 79: strings. This optimization is also disabled for partial matching. 80:  81:  <a name="SEC2" href="#TOC1">PARTIAL MATCHING USING pcre_exec() OR pcre[16|32]_exec()</a>  82:  83: A partial match occurs during a call to pcre_exec() or 84: pcre[16|32]_exec() when the end of the subject string is reached 85: successfully, but matching cannot continue because more characters are needed. 86: However, at least one character in the subject must have been inspected. This 87: character need not form part of the final matched string; lookbehind assertions 88: and the \K escape sequence provide ways of inspecting characters before the 89: start of a matched substring. The requirement for inspecting at least one 90: character exists because an empty string can always be matched; without such a 91: restriction there would always be a partial match of an empty string at the end 92: of the subject. 93:  94:  95: If there are at least two slots in the offsets vector when a partial match is 96: returned, the first slot is set to the offset of the earliest character that 97: was inspected. For convenience, the second offset points to the end of the 98: subject so that a substring can easily be identified. If there are at least 99: three slots in the offsets vector, the third slot is set to the offset of the 100: character where matching started. 101:  102:  103: For the majority of patterns, the contents of the first and third slots will be 104: the same. However, for patterns that contain lookbehind assertions, or begin 105: with \b or \B, characters before the one where matching started may have been 106: inspected while carrying out the match. For example, consider this pattern: 107: <pre> 108: /(?<=abc)123/ 109: </pre> 110: This pattern matches "123", but only if it is preceded by "abc". If the subject 111: string is "xyzabc12", the first two offsets after a partial match are for the 112: substring "abc12", because all these characters were inspected. However, the 113: third offset is set to 6, because that is the offset where matching began. 114:  115:  116: What happens when a partial match is identified depends on which of the two 117: partial matching options are set. 118:  119:   120: PCRE_PARTIAL_SOFT WITH pcre_exec() OR pcre[16|32]_exec() 121:   122:  123: If PCRE_PARTIAL_SOFT is set when pcre_exec() or pcre[16|32]_exec() 124: identifies a partial match, the partial match is remembered, but matching 125: continues as normal, and other alternatives in the pattern are tried. If no 126: complete match can be found, PCRE_ERROR_PARTIAL is returned instead of 127: PCRE_ERROR_NOMATCH. 128:  129:  130: This option is "soft" because it prefers a complete match over a partial match. 131: All the various matching items in a pattern behave as if the subject string is 132: potentially complete. For example, \z, \Z, and $ match at the end of the 133: subject, as normal, and for \b and \B the end of the subject is treated as a 134: non-alphanumeric. 135:  136:  137: If there is more than one partial match, the first one that was found provides 138: the data that is returned. Consider this pattern: 139: <pre> 140: /123\w+X|dogY/ 141: </pre> 142: If this is matched against the subject string "abc123dog", both 143: alternatives fail to match, but the end of the subject is reached during 144: matching, so PCRE_ERROR_PARTIAL is returned. The offsets are set to 3 and 9, 145: identifying "123dog" as the first partial match that was found. (In this 146: example, there are two partial matches, because "dog" on its own partially 147: matches the second alternative.) 148:  149:   150: PCRE_PARTIAL_HARD WITH pcre_exec() OR pcre[16|32]_exec() 151:   152:  153: If PCRE_PARTIAL_HARD is set for pcre_exec() or pcre[16|32]_exec(), 154: PCRE_ERROR_PARTIAL is returned as soon as a partial match is found, without 155: continuing to search for possible complete matches. This option is "hard" 156: because it prefers an earlier partial match over a later complete match. For 157: this reason, the assumption is made that the end of the supplied subject string 158: may not be the true end of the available data, and so, if \z, \Z, \b, \B, 159: or $ are encountered at the end of the subject, the result is 160: PCRE_ERROR_PARTIAL, provided that at least one character in the subject has 161: been inspected. 162:  163:  164: Setting PCRE_PARTIAL_HARD also affects the way UTF-8 and UTF-16 165: subject strings are checked for validity. Normally, an invalid sequence 166: causes the error PCRE_ERROR_BADUTF8 or PCRE_ERROR_BADUTF16. However, in the 167: special case of a truncated character at the end of the subject, 168: PCRE_ERROR_SHORTUTF8 or PCRE_ERROR_SHORTUTF16 is returned when 169: PCRE_PARTIAL_HARD is set. 170:  171:   172: Comparing hard and soft partial matching 173:   174:  175: The difference between the two partial matching options can be illustrated by a 176: pattern such as: 177: <pre> 178: /dog(sbody)?/ 179: </pre> 180: This matches either "dog" or "dogsbody", greedily (that is, it prefers the 181: longer string if possible). If it is matched against the string "dog" with 182: PCRE_PARTIAL_SOFT, it yields a complete match for "dog". However, if 183: PCRE_PARTIAL_HARD is set, the result is PCRE_ERROR_PARTIAL. On the other hand, 184: if the pattern is made ungreedy the result is different: 185: <pre> 186: /dog(sbody)??/ 187: </pre> 188: In this case the result is always a complete match because that is found first, 189: and matching never continues after finding a complete match. It might be easier 190: to follow this explanation by thinking of the two patterns like this: 191: <pre> 192: /dog(sbody)?/ is the same as /dogsbody|dog/ 193: /dog(sbody)??/ is the same as /dog|dogsbody/ 194: </pre> 195: The second pattern will never match "dogsbody", because it will always find the 196: shorter match first. 197:  198:  <a name="SEC3" href="#TOC1">PARTIAL MATCHING USING pcre_dfa_exec() OR pcre[16|32]_dfa_exec()</a>  199:  200: The DFA functions move along the subject string character by character, without 201: backtracking, searching for all possible matches simultaneously. If the end of 202: the subject is reached before the end of the pattern, there is the possibility 203: of a partial match, again provided that at least one character has been 204: inspected. 205:  206:  207: When PCRE_PARTIAL_SOFT is set, PCRE_ERROR_PARTIAL is returned only if there 208: have been no complete matches. Otherwise, the complete matches are returned. 209: However, if PCRE_PARTIAL_HARD is set, a partial match takes precedence over any 210: complete matches. The portion of the string that was inspected when the longest 211: partial match was found is set as the first matching string, provided there are 212: at least two slots in the offsets vector. 213:  214:  215: Because the DFA functions always search for all possible matches, and there is 216: no difference between greedy and ungreedy repetition, their behaviour is 217: different from the standard functions when PCRE_PARTIAL_HARD is set. Consider 218: the string "dog" matched against the ungreedy pattern shown above: 219: <pre> 220: /dog(sbody)??/ 221: </pre> 222: Whereas the standard functions stop as soon as they find the complete match for 223: "dog", the DFA functions also find the partial match for "dogsbody", and so 224: return that when PCRE_PARTIAL_HARD is set. 225:  226:  <a name="SEC4" href="#TOC1">PARTIAL MATCHING AND WORD BOUNDARIES</a>  227:  228: If a pattern ends with one of sequences \b or \B, which test for word 229: boundaries, partial matching with PCRE_PARTIAL_SOFT can give counter-intuitive 230: results. Consider this pattern: 231: <pre> 232: /\bcat\b/ 233: </pre> 234: This matches "cat", provided there is a word boundary at either end. If the 235: subject string is "the cat", the comparison of the final "t" with a following 236: character cannot take place, so a partial match is found. However, normal 237: matching carries on, and \b matches at the end of the subject when the last 238: character is a letter, so a complete match is found. The result, therefore, is 239: not PCRE_ERROR_PARTIAL. Using PCRE_PARTIAL_HARD in this case does yield 240: PCRE_ERROR_PARTIAL, because then the partial match takes precedence. 241:  242:  <a name="SEC5" href="#TOC1">FORMERLY RESTRICTED PATTERNS</a>  243:  244: For releases of PCRE prior to 8.00, because of the way certain internal 245: optimizations were implemented in the pcre_exec() function, the 246: PCRE_PARTIAL option (predecessor of PCRE_PARTIAL_SOFT) could not be used with 247: all patterns. From release 8.00 onwards, the restrictions no longer apply, and 248: partial matching with can be requested for any pattern. 249:  250:  251: Items that were formerly restricted were repeated single characters and 252: repeated metasequences. If PCRE_PARTIAL was set for a pattern that did not 253: conform to the restrictions, pcre_exec() returned the error code 254: PCRE_ERROR_BADPARTIAL (-13). This error code is no longer in use. The 255: PCRE_INFO_OKPARTIAL call to pcre_fullinfo() to find out if a compiled 256: pattern can be used for partial matching now always returns 1. 257:  258:  <a name="SEC6" href="#TOC1">EXAMPLE OF PARTIAL MATCHING USING PCRETEST</a>  259:  260: If the escape sequence \P is present in a pcretest data line, the 261: PCRE_PARTIAL_SOFT option is used for the match. Here is a run of pcretest 262: that uses the date example quoted above: 263: <pre> 264: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 265: data> 25jun04\P 266: 0: 25jun04 267: 1: jun 268: data> 25dec3\P 269: Partial match: 23dec3 270: data> 3ju\P 271: Partial match: 3ju 272: data> 3juj\P 273: No match 274: data> j\P 275: No match 276: </pre> 277: The first data string is matched completely, so pcretest shows the 278: matched substrings. The remaining four strings do not match the complete 279: pattern, but the first two are partial matches. Similar output is obtained 280: if DFA matching is used. 281:  282:  283: If the escape sequence \P is present more than once in a pcretest data 284: line, the PCRE_PARTIAL_HARD option is set for the match. 285:  286:  <a name="SEC7" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_dfa_exec() OR pcre[16|32]_dfa_exec()</a>  287:  288: When a partial match has been found using a DFA matching function, it is 289: possible to continue the match by providing additional subject data and calling 290: the function again with the same compiled regular expression, this time setting 291: the PCRE_DFA_RESTART option. You must pass the same working space as before, 292: because this is where details of the previous partial match are stored. Here is 293: an example using pcretest, using the \R escape sequence to set the 294: PCRE_DFA_RESTART option (\D specifies the use of the DFA matching function): 295: <pre> 296: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 297: data> 23ja\P\D 298: Partial match: 23ja 299: data> n05\R\D 300: 0: n05 301: </pre> 302: The first call has "23ja" as the subject, and requests partial matching; the 303: second call has "n05" as the subject for the continued (restarted) match. 304: Notice that when the match is complete, only the last part is shown; PCRE does 305: not retain the previously partially-matched string. It is up to the calling 306: program to do that if it needs to. 307:  308:  309: That means that, for an unanchored pattern, if a continued match fails, it is 310: not possible to try again at a new starting point. All this facility is capable 311: of doing is continuing with the previous match attempt. In the previous 312: example, if the second set of data is "ug23" the result is no match, even 313: though there would be a match for "aug23" if the entire string were given at 314: once. Depending on the application, this may or may not be what you want. 315: The only way to allow for starting again at the next character is to retain the 316: matched part of the subject and try a new complete match. 317:  318:  319: You can set the PCRE_PARTIAL_SOFT or PCRE_PARTIAL_HARD options with 320: PCRE_DFA_RESTART to continue partial matching over multiple segments. This 321: facility can be used to pass very long subject strings to the DFA matching 322: functions. 323:  324:  <a name="SEC8" href="#TOC1">MULTI-SEGMENT MATCHING WITH pcre_exec() OR pcre[16|32]_exec()</a>  325:  326: From release 8.00, the standard matching functions can also be used to do 327: multi-segment matching. Unlike the DFA functions, it is not possible to 328: restart the previous match with a new segment of data. Instead, new data must 329: be added to the previous subject string, and the entire match re-run, starting 330: from the point where the partial match occurred. Earlier data can be discarded. 331:  332:  333: It is best to use PCRE_PARTIAL_HARD in this situation, because it does not 334: treat the end of a segment as the end of the subject when matching \z, \Z, 335: \b, \B, and $. Consider an unanchored pattern that matches dates: 336: <pre> 337: re> /\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d/ 338: data> The date is 23ja\P\P 339: Partial match: 23ja 340: </pre> 341: At this stage, an application could discard the text preceding "23ja", add on 342: text from the next segment, and call the matching function again. Unlike the 343: DFA matching functions, the entire matching string must always be available, 344: and the complete matching process occurs for each call, so more memory and more 345: processing time is needed. 346:  347:  348: Note: If the pattern contains lookbehind assertions, or \K, or starts 349: with \b or \B, the string that is returned for a partial match includes 350: characters that precede the start of what would be returned for a complete 351: match, because it contains all the characters that were inspected during the 352: partial match. 353:  354:  <a name="SEC9" href="#TOC1">ISSUES WITH MULTI-SEGMENT MATCHING</a>  355:  356: Certain types of pattern may give problems with multi-segment matching, 357: whichever matching function is used. 358:  359:  360: 1. If the pattern contains a test for the beginning of a line, you need to pass 361: the PCRE_NOTBOL option when the subject string for any call does start at the 362: beginning of a line. There is also a PCRE_NOTEOL option, but in practice when 363: doing multi-segment matching you should be using PCRE_PARTIAL_HARD, which 364: includes the effect of PCRE_NOTEOL. 365:  366:  367: 2. Lookbehind assertions that have already been obeyed are catered for in the 368: offsets that are returned for a partial match. However a lookbehind assertion 369: later in the pattern could require even earlier characters to be inspected. You 370: can handle this case by using the PCRE_INFO_MAXLOOKBEHIND option of the 371: pcre_fullinfo() or pcre[16|32]_fullinfo() functions to obtain the 372: length of the longest lookbehind in the pattern. This length is given in 373: characters, not bytes. If you always retain at least that many characters 374: before the partially matched string, all should be well. (Of course, near the 375: start of the subject, fewer characters may be present; in that case all 376: characters should be retained.) 377:  378:  379: From release 8.33, there is a more accurate way of deciding which characters to 380: retain. Instead of subtracting the length of the longest lookbehind from the 381: earliest inspected character (offsets[0]), the match start position 382: (offsets[2]) should be used, and the next match attempt started at the 383: offsets[2] character by setting the startoffset argument of 384: pcre_exec() or pcre_dfa_exec(). 385:  386:  387: For example, if the pattern "(?<=123)abc" is partially 388: matched against the string "xx123a", the three offset values returned are 2, 6, 389: and 5. This indicates that the matching process that gave a partial match 390: started at offset 5, but the characters "123a" were all inspected. The maximum 391: lookbehind for that pattern is 3, so taking that away from 5 shows that we need 392: only keep "123a", and the next match attempt can be started at offset 3 (that 393: is, at "a") when further characters have been added. When the match start is 394: not the earliest inspected character, pcretest shows it explicitly: 395: <pre> 396: re> "(?<=123)abc" 397: data> xx123a\P\P 398: Partial match at offset 5: 123a 399: </PRE> 400:  401:  402: 3. Because a partial match must always contain at least one character, what 403: might be considered a partial match of an empty string actually gives a "no 404: match" result. For example: 405: <pre> 406: re> /c(?<=abc)x/ 407: data> ab\P 408: No match 409: </pre> 410: If the next segment begins "cx", a match should be found, but this will only 411: happen if characters from the previous segment are retained. For this reason, a 412: "no match" result should be interpreted as "partial match of an empty string" 413: when the pattern contains lookbehinds. 414:  415:  416: 4. Matching a subject string that is split into multiple segments may not 417: always produce exactly the same result as matching over one single long string, 418: especially when PCRE_PARTIAL_SOFT is used. The section "Partial Matching and 419: Word Boundaries" above describes an issue that arises if the pattern ends with 420: \b or \B. Another kind of difference may occur when there are multiple 421: matching possibilities, because (for PCRE_PARTIAL_SOFT) a partial match result 422: is given only when there are no completed matches. This means that as soon as 423: the shortest match has been found, continuation to a new subject segment is no 424: longer possible. Consider again this pcretest example: 425: <pre> 426: re> /dog(sbody)?/ 427: data> dogsb\P 428: 0: dog 429: data> do\P\D 430: Partial match: do 431: data> gsb\R\P\D 432: 0: g 433: data> dogsbody\D 434: 0: dogsbody 435: 1: dog 436: </pre> 437: The first data line passes the string "dogsb" to a standard matching function, 438: setting the PCRE_PARTIAL_SOFT option. Although the string is a partial match 439: for "dogsbody", the result is not PCRE_ERROR_PARTIAL, because the shorter 440: string "dog" is a complete match. Similarly, when the subject is presented to 441: a DFA matching function in several parts ("do" and "gsb" being the first two) 442: the match stops when "dog" has been found, and it is not possible to continue. 443: On the other hand, if "dogsbody" is presented as a single string, a DFA 444: matching function finds both matches. 445:  446:  447: Because of these problems, it is best to use PCRE_PARTIAL_HARD when matching 448: multi-segment data. The example above then behaves differently: 449: <pre> 450: re> /dog(sbody)?/ 451: data> dogsb\P\P 452: Partial match: dogsb 453: data> do\P\D 454: Partial match: do 455: data> gsb\R\P\P\D 456: Partial match: gsb 457: </pre> 458: 5. Patterns that contain alternatives at the top level which do not all start 459: with the same pattern item may not work as expected when PCRE_DFA_RESTART is 460: used. For example, consider this pattern: 461: <pre> 462: 1234|3789 463: </pre> 464: If the first part of the subject is "ABC123", a partial match of the first 465: alternative is found at offset 3. There is no partial match for the second 466: alternative, because such a match does not start at the same point in the 467: subject string. Attempting to continue with the string "7890" does not yield a 468: match because only those alternatives that match at one point in the subject 469: are remembered. The problem arises because the start of the second alternative 470: matches within the first alternative. There is no problem with anchored 471: patterns or patterns such as: 472: <pre> 473: 1234|ABCD 474: </pre> 475: where no string can be a partial match for both alternatives. This is not a 476: problem if a standard matching function is used, because the entire match has 477: to be rerun each time: 478: <pre> 479: re> /1234|3789/ 480: data> ABC123\P\P 481: Partial match: 123 482: data> 1237890 483: 0: 3789 484: </pre> 485: Of course, instead of using PCRE_DFA_RESTART, the same technique of re-running 486: the entire match can also be used with the DFA matching functions. Another 487: possibility is to work with two buffers. If a partial match at offset n 488: in the first buffer is followed by "no match" when PCRE_DFA_RESTART is used on 489: the second buffer, you can then try a new match starting at offset n+1 in 490: the first buffer. 491:  492:  <a name="SEC10" href="#TOC1">AUTHOR</a>  493:  494: Philip Hazel 495:   496: University Computing Service 497:   498: Cambridge CB2 3QH, England. 499:   500:  501:  <a name="SEC11" href="#TOC1">REVISION</a>  502:  503: Last updated: 02 July 2013 504:   505: Copyright © 1997-2013 University of Cambridge. 506:   507:  508: Return to the <a href="index.html">PCRE index page</a>. 509: 