embedaddon/pcre/doc/html/pcretest.html - view

File: [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html / pcretest.html
Revision 1.1.1.5 (vendor branch): download - view: text, annotated - select for diffs - revision graph
Sun Jun 15 19:46:05 2014 UTC (10 years, 9 months ago) by misho
Branches: pcre, MAIN
CVS tags: v8_34, HEAD

pcre 8.34

1: <html> 2: <head> 3: <title>pcretest specification</title> 4: </head> 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB"> 6: <h1>pcretest man page</h1> 7:  8: Return to the <a href="index.html">PCRE index page</a>. 9:  10:  11: This page is part of the PCRE HTML documentation. It was generated automatically 12: from the original man page. If there is any nonsense in it, please consult the 13: man page, in case the conversion went wrong. 14:   15: <ul> 16: <li><a name="TOC1" href="#SEC1">SYNOPSIS</a> 17: <li><a name="TOC2" href="#SEC2">INPUT DATA FORMAT</a> 18: <li><a name="TOC3" href="#SEC3">PCRE's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a> 19: <li><a name="TOC4" href="#SEC4">COMMAND LINE OPTIONS</a> 20: <li><a name="TOC5" href="#SEC5">DESCRIPTION</a> 21: <li><a name="TOC6" href="#SEC6">PATTERN MODIFIERS</a> 22: <li><a name="TOC7" href="#SEC7">DATA LINES</a> 23: <li><a name="TOC8" href="#SEC8">THE ALTERNATIVE MATCHING FUNCTION</a> 24: <li><a name="TOC9" href="#SEC9">DEFAULT OUTPUT FROM PCRETEST</a> 25: <li><a name="TOC10" href="#SEC10">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a> 26: <li><a name="TOC11" href="#SEC11">RESTARTING AFTER A PARTIAL MATCH</a> 27: <li><a name="TOC12" href="#SEC12">CALLOUTS</a> 28: <li><a name="TOC13" href="#SEC13">NON-PRINTING CHARACTERS</a> 29: <li><a name="TOC14" href="#SEC14">SAVING AND RELOADING COMPILED PATTERNS</a> 30: <li><a name="TOC15" href="#SEC15">SEE ALSO</a> 31: <li><a name="TOC16" href="#SEC16">AUTHOR</a> 32: <li><a name="TOC17" href="#SEC17">REVISION</a> 33: </ul> 34:  <a name="SEC1" href="#TOC1">SYNOPSIS</a>  35:  36: pcretest [options] [input file [output file]] 37:   38:   39: pcretest was written as a test program for the PCRE regular expression 40: library itself, but it can also be used for experimenting with regular 41: expressions. This document describes the features of the test program; for 42: details of the regular expressions themselves, see the 43: <a href="pcrepattern.html">pcrepattern</a> 44: documentation. For details of the PCRE library function calls and their 45: options, see the 46: <a href="pcreapi.html">pcreapi</a> 47: , 48: <a href="pcre16.html">pcre16</a> 49: and 50: <a href="pcre32.html">pcre32</a> 51: documentation. 52:  53:  54: The input for pcretest is a sequence of regular expression patterns and 55: strings to be matched, as described below. The output shows the result of each 56: match. Options on the command line and the patterns control PCRE options and 57: exactly what is output. 58:  59:  60: As PCRE has evolved, it has acquired many different features, and as a result, 61: pcretest now has rather a lot of obscure options for testing every 62: possible feature. Some of these options are specifically designed for use in 63: conjunction with the test script and data files that are distributed as part of 64: PCRE, and are unlikely to be of use otherwise. They are all documented here, 65: but without much justification. 66:  67:  <a name="SEC2" href="#TOC1">INPUT DATA FORMAT</a>  68:  69: Input to pcretest is processed line by line, either by calling the C 70: library's fgets() function, or via the libreadline library (see 71: below). In Unix-like environments, fgets() treats any bytes other than 72: newline as data characters. However, in some Windows environments character 26 73: (hex 1A) causes an immediate end of file, and no further data is read. For 74: maximum portability, therefore, it is safest to use only ASCII characters in 75: pcretest input files. 76:  77:  <a name="SEC3" href="#TOC1">PCRE's 8-BIT, 16-BIT AND 32-BIT LIBRARIES</a>  78:  79: From release 8.30, two separate PCRE libraries can be built. The original one 80: supports 8-bit character strings, whereas the newer 16-bit library supports 81: character strings encoded in 16-bit units. From release 8.32, a third library 82: can be built, supporting character strings encoded in 32-bit units. The 83: pcretest program can be used to test all three libraries. However, it is 84: itself still an 8-bit program, reading 8-bit input and writing 8-bit output. 85: When testing the 16-bit or 32-bit library, the patterns and data strings are 86: converted to 16- or 32-bit format before being passed to the PCRE library 87: functions. Results are converted to 8-bit for output. 88:  89:  90: References to functions and structures of the form pcre[16|32]_xx below 91: mean "pcre_xx when using the 8-bit library, pcre16_xx when using 92: the 16-bit library, or pcre32_xx when using the 32-bit library". 93:  94:  <a name="SEC4" href="#TOC1">COMMAND LINE OPTIONS</a>  95:  96: -8 97: If both the 8-bit library has been built, this option causes the 8-bit library 98: to be used (which is the default); if the 8-bit library has not been built, 99: this option causes an error. 100:  101:  102: -16 103: If both the 8-bit or the 32-bit, and the 16-bit libraries have been built, this 104: option causes the 16-bit library to be used. If only the 16-bit library has been 105: built, this is the default (so has no effect). If only the 8-bit or the 32-bit 106: library has been built, this option causes an error. 107:  108:  109: -32 110: If both the 8-bit or the 16-bit, and the 32-bit libraries have been built, this 111: option causes the 32-bit library to be used. If only the 32-bit library has been 112: built, this is the default (so has no effect). If only the 8-bit or the 16-bit 113: library has been built, this option causes an error. 114:  115:  116: -b 117: Behave as if each pattern has the /B (show byte code) modifier; the 118: internal form is output after compilation. 119:  120:  121: -C 122: Output the version number of the PCRE library, and all available information 123: about the optional features that are included, and then exit with zero exit 124: code. All other options are ignored. 125:  126:  127: -C option 128: Output information about a specific build-time option, then exit. This 129: functionality is intended for use in scripts such as RunTest. The 130: following options output the value and set the exit code as indicated: 131: <pre> 132: ebcdic-nl the code for LF (= NL) in an EBCDIC environment: 133: 0x15 or 0x25 134: 0 if used in an ASCII environment 135: exit code is always 0 136: linksize the configured internal link size (2, 3, or 4) 137: exit code is set to the link size 138: newline the default newline setting: 139: CR, LF, CRLF, ANYCRLF, or ANY 140: exit code is always 0 141: </pre> 142: The following options output 1 for true or 0 for false, and set the exit code 143: to the same value: 144: <pre> 145: ebcdic compiled for an EBCDIC environment 146: jit just-in-time support is available 147: pcre16 the 16-bit library was built 148: pcre32 the 32-bit library was built 149: pcre8 the 8-bit library was built 150: ucp Unicode property support is available 151: utf UTF-8 and/or UTF-16 and/or UTF-32 support 152: is available 153: </pre> 154: If an unknown option is given, an error message is output; the exit code is 0. 155:  156:  157: -d 158: Behave as if each pattern has the /D (debug) modifier; the internal 159: form and information about the compiled pattern is output after compilation; 160: -d is equivalent to -b -i. 161:  162:  163: -dfa 164: Behave as if each data line contains the \D escape sequence; this causes the 165: alternative matching function, pcre[16|32]_dfa_exec(), to be used instead 166: of the standard pcre[16|32]_exec() function (more detail is given below). 167:  168:  169: -help 170: Output a brief summary these options and then exit. 171:  172:  173: -i 174: Behave as if each pattern has the /I modifier; information about the 175: compiled pattern is given after compilation. 176:  177:  178: -M 179: Behave as if each data line contains the \M escape sequence; this causes 180: PCRE to discover the minimum MATCH_LIMIT and MATCH_LIMIT_RECURSION settings by 181: calling pcre[16|32]_exec() repeatedly with different limits. 182:  183:  184: -m 185: Output the size of each compiled pattern after it has been compiled. This is 186: equivalent to adding /M to each regular expression. The size is given in 187: bytes for both libraries. 188:  189:  190: -O 191: Behave as if each pattern has the /O modifier, that is disable 192: auto-possessification for all patterns. 193:  194:  195: -o osize 196: Set the number of elements in the output vector that is used when calling 197: pcre[16|32]_exec() or pcre[16|32]_dfa_exec() to be osize. The 198: default value is 45, which is enough for 14 capturing subexpressions for 199: pcre[16|32]_exec() or 22 different matches for 200: pcre[16|32]_dfa_exec(). 201: The vector size can be changed for individual matching calls by including \O 202: in the data line (see below). 203:  204:  205: -p 206: Behave as if each pattern has the /P modifier; the POSIX wrapper API is 207: used to call PCRE. None of the other options has any effect when -p is 208: set. This option can be used only with the 8-bit library. 209:  210:  211: -q 212: Do not output the version number of pcretest at the start of execution. 213:  214:  215: -S size 216: On Unix-like systems, set the size of the run-time stack to size 217: megabytes. 218:  219:  220: -s or -s+ 221: Behave as if each pattern has the /S modifier; in other words, force each 222: pattern to be studied. If -s+ is used, all the JIT compile options are 223: passed to pcre[16|32]_study(), causing just-in-time optimization to be set 224: up if it is available, for both full and partial matching. Specific JIT compile 225: options can be selected by following -s+ with a digit in the range 1 to 226: 7, which selects the JIT compile modes as follows: 227: <pre> 228: 1 normal match only 229: 2 soft partial match only 230: 3 normal match and soft partial match 231: 4 hard partial match only 232: 6 soft and hard partial match 233: 7 all three modes (default) 234: </pre> 235: If -s++ is used instead of -s+ (with or without a following digit), 236: the text "(JIT)" is added to the first output line after a match or no match 237: when JIT-compiled code was actually used. 238:   239:   240: Note that there are pattern options that can override -s, either 241: specifying no studying at all, or suppressing JIT compilation. 242:   243:   244: If the /I or /D option is present on a pattern (requesting output 245: about the compiled pattern), information about the result of studying is not 246: included when studying is caused only by -s and neither -i nor 247: -d is present on the command line. This behaviour means that the output 248: from tests that are run with and without -s should be identical, except 249: when options that output information about the actual running of a match are 250: set. 251:   252:   253: The -M, -t, and -tm options, which give information about 254: resources used, are likely to produce different output with and without 255: -s. Output may also differ if the /C option is present on an 256: individual pattern. This uses callouts to trace the the matching process, and 257: this may be different between studied and non-studied patterns. If the pattern 258: contains (*MARK) items there may also be differences, for the same reason. The 259: -s command line option can be overridden for specific patterns that 260: should never be studied (see the /S pattern modifier below). 261:  262:  263: -t 264: Run each compile, study, and match many times with a timer, and output the 265: resulting times per compile, study, or match (in milliseconds). Do not set 266: -m with -t, because you will then get the size output a zillion 267: times, and the timing will be distorted. You can control the number of 268: iterations that are used for timing by following -t with a number (as a 269: separate item on the command line). For example, "-t 1000" iterates 1000 times. 270: The default is to iterate 500000 times. 271:  272:  273: -tm 274: This is like -t except that it times only the matching phase, not the 275: compile or study phases. 276:  277:  278: -T -TM 279: These behave like -t and -tm, but in addition, at the end of a run, 280: the total times for all compiles, studies, and matches are output. 281:  282:  <a name="SEC5" href="#TOC1">DESCRIPTION</a>  283:  284: If pcretest is given two filename arguments, it reads from the first and 285: writes to the second. If it is given only one filename argument, it reads from 286: that file and writes to stdout. Otherwise, it reads from stdin and writes to 287: stdout, and prompts for each line of input, using "re>" to prompt for regular 288: expressions, and "data>" to prompt for data lines. 289:  290:  291: When pcretest is built, a configuration option can specify that it should 292: be linked with the libreadline library. When this is done, if the input 293: is from a terminal, it is read using the readline() function. This 294: provides line-editing and history facilities. The output from the -help 295: option states whether or not readline() will be used. 296:  297:  298: The program handles any number of sets of input on a single input file. Each 299: set starts with a regular expression, and continues with any number of data 300: lines to be matched against that pattern. 301:  302:  303: Each data line is matched separately and independently. If you want to do 304: multi-line matches, you have to use the \n escape sequence (or \r or \r\n, 305: etc., depending on the newline setting) in a single line of input to encode the 306: newline sequences. There is no limit on the length of data lines; the input 307: buffer is automatically extended if it is too small. 308:  309:  310: An empty line signals the end of the data lines, at which point a new regular 311: expression is read. The regular expressions are given enclosed in any 312: non-alphanumeric delimiters other than backslash, for example: 313: <pre> 314: /(a|bc)x+yz/ 315: </pre> 316: White space before the initial delimiter is ignored. A regular expression may 317: be continued over several input lines, in which case the newline characters are 318: included within it. It is possible to include the delimiter within the pattern 319: by escaping it, for example 320: <pre> 321: /abc\/def/ 322: </pre> 323: If you do so, the escape and the delimiter form part of the pattern, but since 324: delimiters are always non-alphanumeric, this does not affect its interpretation. 325: If the terminating delimiter is immediately followed by a backslash, for 326: example, 327: <pre> 328: /abc/\ 329: </pre> 330: then a backslash is added to the end of the pattern. This is done to provide a 331: way of testing the error condition that arises if a pattern finishes with a 332: backslash, because 333: <pre> 334: /abc\/ 335: </pre> 336: is interpreted as the first line of a pattern that starts with "abc/", causing 337: pcretest to read the next line as a continuation of the regular expression. 338:  339:  <a name="SEC6" href="#TOC1">PATTERN MODIFIERS</a>  340:  341: A pattern may be followed by any number of modifiers, which are mostly single 342: characters, though some of these can be qualified by further characters. 343: Following Perl usage, these are referred to below as, for example, "the 344: /i modifier", even though the delimiter of the pattern need not always be 345: a slash, and no slash is used when writing modifiers. White space may appear 346: between the final pattern delimiter and the first modifier, and between the 347: modifiers themselves. For reference, here is a complete list of modifiers. They 348: fall into several groups that are described in detail in the following 349: sections. 350: <pre> 351: /8 set UTF mode 352: /9 set PCRE_NEVER_UTF (locks out UTF mode) 353: /? disable UTF validity check 354: /+ show remainder of subject after match 355: /= show all captures (not just those that are set) 356: 357: /A set PCRE_ANCHORED 358: /B show compiled code 359: /C set PCRE_AUTO_CALLOUT 360: /D same as /B plus /I 361: /E set PCRE_DOLLAR_ENDONLY 362: /F flip byte order in compiled pattern 363: /f set PCRE_FIRSTLINE 364: /G find all matches (shorten string) 365: /g find all matches (use startoffset) 366: /I show information about pattern 367: /i set PCRE_CASELESS 368: /J set PCRE_DUPNAMES 369: /K show backtracking control names 370: /L set locale 371: /M show compiled memory size 372: /m set PCRE_MULTILINE 373: /N set PCRE_NO_AUTO_CAPTURE 374: /O set PCRE_NO_AUTO_POSSESS 375: /P use the POSIX wrapper 376: /S study the pattern after compilation 377: /s set PCRE_DOTALL 378: /T select character tables 379: /U set PCRE_UNGREEDY 380: /W set PCRE_UCP 381: /X set PCRE_EXTRA 382: /x set PCRE_EXTENDED 383: /Y set PCRE_NO_START_OPTIMIZE 384: /Z don't show lengths in /B output 385: 386: /<any> set PCRE_NEWLINE_ANY 387: /<anycrlf> set PCRE_NEWLINE_ANYCRLF 388: /<cr> set PCRE_NEWLINE_CR 389: /<crlf> set PCRE_NEWLINE_CRLF 390: /<lf> set PCRE_NEWLINE_LF 391: /<bsr_anycrlf> set PCRE_BSR_ANYCRLF 392: /<bsr_unicode> set PCRE_BSR_UNICODE 393: /<JS> set PCRE_JAVASCRIPT_COMPAT 394: 395: </PRE> 396:  397:   398: Perl-compatible modifiers 399:   400:  401: The /i, /m, /s, and /x modifiers set the PCRE_CASELESS, 402: PCRE_MULTILINE, PCRE_DOTALL, or PCRE_EXTENDED options, respectively, when 403: pcre[16|32]_compile() is called. These four modifier letters have the same 404: effect as they do in Perl. For example: 405: <pre> 406: /caseless/i 407: 408: </PRE> 409:  410:   411: Modifiers for other PCRE options 412:   413:  414: The following table shows additional modifiers for setting PCRE compile-time 415: options that do not correspond to anything in Perl: 416: <pre> 417: /8 PCRE_UTF8 ) when using the 8-bit 418: /? PCRE_NO_UTF8_CHECK ) library 419: 420: /8 PCRE_UTF16 ) when using the 16-bit 421: /? PCRE_NO_UTF16_CHECK ) library 422: 423: /8 PCRE_UTF32 ) when using the 32-bit 424: /? PCRE_NO_UTF32_CHECK ) library 425: 426: /9 PCRE_NEVER_UTF 427: /A PCRE_ANCHORED 428: /C PCRE_AUTO_CALLOUT 429: /E PCRE_DOLLAR_ENDONLY 430: /f PCRE_FIRSTLINE 431: /J PCRE_DUPNAMES 432: /N PCRE_NO_AUTO_CAPTURE 433: /O PCRE_NO_AUTO_POSSESS 434: /U PCRE_UNGREEDY 435: /W PCRE_UCP 436: /X PCRE_EXTRA 437: /Y PCRE_NO_START_OPTIMIZE 438: /<any> PCRE_NEWLINE_ANY 439: /<anycrlf> PCRE_NEWLINE_ANYCRLF 440: /<cr> PCRE_NEWLINE_CR 441: /<crlf> PCRE_NEWLINE_CRLF 442: /<lf> PCRE_NEWLINE_LF 443: /<bsr_anycrlf> PCRE_BSR_ANYCRLF 444: /<bsr_unicode> PCRE_BSR_UNICODE 445: /<JS> PCRE_JAVASCRIPT_COMPAT 446: </pre> 447: The modifiers that are enclosed in angle brackets are literal strings as shown, 448: including the angle brackets, but the letters within can be in either case. 449: This example sets multiline matching with CRLF as the line ending sequence: 450: <pre> 451: /^abc/m<CRLF> 452: </pre> 453: As well as turning on the PCRE_UTF8/16/32 option, the /8 modifier causes 454: all non-printing characters in output strings to be printed using the 455: \x{hh...} notation. Otherwise, those less than 0x100 are output in hex without 456: the curly brackets. 457:  458:  459: Full details of the PCRE options are given in the 460: <a href="pcreapi.html">pcreapi</a> 461: documentation. 462:  463:   464: Finding all matches in a string 465:   466:  467: Searching for all possible matches within each subject string can be requested 468: by the /g or /G modifier. After finding a match, PCRE is called 469: again to search the remainder of the subject string. The difference between 470: /g and /G is that the former uses the startoffset argument to 471: pcre[16|32]_exec() to start searching at a new point within the entire 472: string (which is in effect what Perl does), whereas the latter passes over a 473: shortened substring. This makes a difference to the matching process if the 474: pattern begins with a lookbehind assertion (including \b or \B). 475:  476:  477: If any call to pcre[16|32]_exec() in a /g or /G sequence matches 478: an empty string, the next call is done with the PCRE_NOTEMPTY_ATSTART and 479: PCRE_ANCHORED flags set in order to search for another, non-empty, match at the 480: same point. If this second match fails, the start offset is advanced, and the 481: normal match is retried. This imitates the way Perl handles such cases when 482: using the /g modifier or the split() function. Normally, the start 483: offset is advanced by one character, but if the newline convention recognizes 484: CRLF as a newline, and the current character is CR followed by LF, an advance 485: of two is used. 486:  487:   488: Other modifiers 489:   490:  491: There are yet more modifiers for controlling the way pcretest 492: operates. 493:  494:  495: The /+ modifier requests that as well as outputting the substring that 496: matched the entire pattern, pcretest should in addition output the 497: remainder of the subject string. This is useful for tests where the subject 498: contains multiple copies of the same substring. If the + modifier appears 499: twice, the same action is taken for captured substrings. In each case the 500: remainder is output on the following line with a plus character following the 501: capture number. Note that this modifier must not immediately follow the /S 502: modifier because /S+ and /S++ have other meanings. 503:  504:  505: The /= modifier requests that the values of all potential captured 506: parentheses be output after a match. By default, only those up to the highest 507: one actually used in the match are output (corresponding to the return code 508: from pcre[16|32]_exec()). Values in the offsets vector corresponding to 509: higher numbers should be set to -1, and these are output as "<unset>". This 510: modifier gives a way of checking that this is happening. 511:  512:  513: The /B modifier is a debugging feature. It requests that pcretest 514: output a representation of the compiled code after compilation. Normally this 515: information contains length and offset values; however, if /Z is also 516: present, this data is replaced by spaces. This is a special feature for use in 517: the automatic test scripts; it ensures that the same output is generated for 518: different internal link sizes. 519:  520:  521: The /D modifier is a PCRE debugging feature, and is equivalent to 522: /BI, that is, both the /B and the /I modifiers. 523:  524:  525: The /F modifier causes pcretest to flip the byte order of the 526: 2-byte and 4-byte fields in the compiled pattern. This facility is for testing 527: the feature in PCRE that allows it to execute patterns that were compiled on a 528: host with a different endianness. This feature is not available when the POSIX 529: interface to PCRE is being used, that is, when the /P pattern modifier is 530: specified. See also the section about saving and reloading compiled patterns 531: below. 532:  533:  534: The /I modifier requests that pcretest output information about the 535: compiled pattern (whether it is anchored, has a fixed first character, and 536: so on). It does this by calling pcre[16|32]_fullinfo() after compiling a 537: pattern. If the pattern is studied, the results of that are also output. 538:  539:  540: The /K modifier requests pcretest to show names from backtracking 541: control verbs that are returned from calls to pcre[16|32]_exec(). It causes 542: pcretest to create a pcre[16|32]_extra block if one has not already 543: been created by a call to pcre[16|32]_study(), and to set the 544: PCRE_EXTRA_MARK flag and the mark field within it, every time that 545: pcre[16|32]_exec() is called. If the variable that the mark field 546: points to is non-NULL for a match, non-match, or partial match, pcretest 547: prints the string to which it points. For a match, this is shown on a line by 548: itself, tagged with "MK:". For a non-match it is added to the message. 549:  550:  551: The /L modifier must be followed directly by the name of a locale, for 552: example, 553: <pre> 554: /pattern/Lfr_FR 555: </pre> 556: For this reason, it must be the last modifier. The given locale is set, 557: pcre[16|32]_maketables() is called to build a set of character tables for 558: the locale, and this is then passed to pcre[16|32]_compile() when compiling 559: the regular expression. Without an /L (or /T) modifier, NULL is 560: passed as the tables pointer; that is, /L applies only to the expression 561: on which it appears. 562:  563:  564: The /M modifier causes the size in bytes of the memory block used to hold 565: the compiled pattern to be output. This does not include the size of the 566: pcre[16|32] block; it is just the actual compiled data. If the pattern is 567: successfully studied with the PCRE_STUDY_JIT_COMPILE option, the size of the 568: JIT compiled code is also output. 569:  570:  571: The /S modifier causes pcre[16|32]_study() to be called after the 572: expression has been compiled, and the results used when the expression is 573: matched. There are a number of qualifying characters that may follow /S. 574: They may appear in any order. 575:  576:  577: If /S is followed by an exclamation mark, pcre[16|32]_study() is 578: called with the PCRE_STUDY_EXTRA_NEEDED option, causing it always to return a 579: pcre_extra block, even when studying discovers no useful information. 580:  581:  582: If /S is followed by a second S character, it suppresses studying, even 583: if it was requested externally by the -s command line option. This makes 584: it possible to specify that certain patterns are always studied, and others are 585: never studied, independently of -s. This feature is used in the test 586: files in a few cases where the output is different when the pattern is studied. 587:  588:  589: If the /S modifier is followed by a + character, the call to 590: pcre[16|32]_study() is made with all the JIT study options, requesting 591: just-in-time optimization support if it is available, for both normal and 592: partial matching. If you want to restrict the JIT compiling modes, you can 593: follow /S+ with a digit in the range 1 to 7: 594: <pre> 595: 1 normal match only 596: 2 soft partial match only 597: 3 normal match and soft partial match 598: 4 hard partial match only 599: 6 soft and hard partial match 600: 7 all three modes (default) 601: </pre> 602: If /S++ is used instead of /S+ (with or without a following digit), 603: the text "(JIT)" is added to the first output line after a match or no match 604: when JIT-compiled code was actually used. 605:  606:  607: Note that there is also an independent /+ modifier; it must not be given 608: immediately after /S or /S+ because this will be misinterpreted. 609:  610:  611: If JIT studying is successful, the compiled JIT code will automatically be used 612: when pcre[16|32]_exec() is run, except when incompatible run-time options 613: are specified. For more details, see the 614: <a href="pcrejit.html">pcrejit</a> 615: documentation. See also the \J escape sequence below for a way of 616: setting the size of the JIT stack. 617:  618:  619: Finally, if /S is followed by a minus character, JIT compilation is 620: suppressed, even if it was requested externally by the -s command line 621: option. This makes it possible to specify that JIT is never to be used for 622: certain patterns. 623:  624:  625: The /T modifier must be followed by a single digit. It causes a specific 626: set of built-in character tables to be passed to pcre[16|32]_compile(). It 627: is used in the standard PCRE tests to check behaviour with different character 628: tables. The digit specifies the tables as follows: 629: <pre> 630: 0 the default ASCII tables, as distributed in 631: pcre_chartables.c.dist 632: 1 a set of tables defining ISO 8859 characters 633: </pre> 634: In table 1, some characters whose codes are greater than 128 are identified as 635: letters, digits, spaces, etc. 636:  637:   638: Using the POSIX wrapper API 639:   640:  641: The /P modifier causes pcretest to call PCRE via the POSIX wrapper 642: API rather than its native API. This supports only the 8-bit library. When 643: /P is set, the following modifiers set options for the regcomp() 644: function: 645: <pre> 646: /i REG_ICASE 647: /m REG_NEWLINE 648: /N REG_NOSUB 649: /s REG_DOTALL ) 650: /U REG_UNGREEDY ) These options are not part of 651: /W REG_UCP ) the POSIX standard 652: /8 REG_UTF8 ) 653: </pre> 654: The /+ modifier works as described above. All other modifiers are 655: ignored. 656:  657:   658: Locking out certain modifiers 659:   660:  661: PCRE can be compiled with or without support for certain features such as 662: UTF-8/16/32 or Unicode properties. Accordingly, the standard tests are split up 663: into a number of different files that are selected for running depending on 664: which features are available. When updating the tests, it is all too easy to 665: put a new test into the wrong file by mistake; for example, to put a test that 666: requires UTF support into a file that is used when it is not available. To help 667: detect such mistakes as early as possible, there is a facility for locking out 668: specific modifiers. If an input line for pcretest starts with the string 669: "< forbid " the following sequence of characters is taken as a list of 670: forbidden modifiers. For example, in the test files that must not use UTF or 671: Unicode property support, this line appears: 672: <pre> 673: < forbid 8W 674: </pre> 675: This locks out the /8 and /W modifiers. An immediate error is given if they are 676: subsequently encountered. If the character string contains < but not >, all the 677: multi-character modifiers that begin with < are locked out. Otherwise, such 678: modifiers must be explicitly listed, for example: 679: <pre> 680: < forbid <JS><cr> 681: </pre> 682: There must be a single space between < and "forbid" for this feature to be 683: recognised. If there is not, the line is interpreted either as a request to 684: re-load a pre-compiled pattern (see "SAVING AND RELOADING COMPILED PATTERNS" 685: below) or, if there is a another < character, as a pattern that uses < as its 686: delimiter. 687:  688:  <a name="SEC7" href="#TOC1">DATA LINES</a>  689:  690: Before each data line is passed to pcre[16|32]_exec(), leading and trailing 691: white space is removed, and it is then scanned for \ escapes. Some of these 692: are pretty esoteric features, intended for checking out some of the more 693: complicated features of PCRE. If you are just testing "ordinary" regular 694: expressions, you probably don't need any of these. The following escapes are 695: recognized: 696: <pre> 697: \a alarm (BEL, \x07) 698: \b backspace (\x08) 699: \e escape (\x27) 700: \f form feed (\x0c) 701: \n newline (\x0a) 702: \qdd set the PCRE_MATCH_LIMIT limit to dd (any number of digits) 703: \r carriage return (\x0d) 704: \t tab (\x09) 705: \v vertical tab (\x0b) 706: \nnn octal character (up to 3 octal digits); always 707: a byte unless > 255 in UTF-8 or 16-bit or 32-bit mode 708: \o{dd...} octal character (any number of octal digits} 709: \xhh hexadecimal byte (up to 2 hex digits) 710: \x{hh...} hexadecimal character (any number of hex digits) 711: \A pass the PCRE_ANCHORED option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 712: \B pass the PCRE_NOTBOL option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 713: \Cdd call pcre[16|32]_copy_substring() for substring dd after a successful match (number less than 32) 714: \Cname call pcre[16|32]_copy_named_substring() for substring "name" after a successful match (name termin- 715: ated by next non alphanumeric character) 716: \C+ show the current captured substrings at callout time 717: \C- do not supply a callout function 718: \C!n return 1 instead of 0 when callout number n is reached 719: \C!n!m return 1 instead of 0 when callout number n is reached for the nth time 720: \C*n pass the number n (may be negative) as callout data; this is used as the callout return value 721: \D use the pcre[16|32]_dfa_exec() match function 722: \F only shortest match for pcre[16|32]_dfa_exec() 723: \Gdd call pcre[16|32]_get_substring() for substring dd after a successful match (number less than 32) 724: \Gname call pcre[16|32]_get_named_substring() for substring "name" after a successful match (name termin- 725: ated by next non-alphanumeric character) 726: \Jdd set up a JIT stack of dd kilobytes maximum (any number of digits) 727: \L call pcre[16|32]_get_substringlist() after a successful match 728: \M discover the minimum MATCH_LIMIT and MATCH_LIMIT_RECURSION settings 729: \N pass the PCRE_NOTEMPTY option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec(); if used twice, pass the 730: PCRE_NOTEMPTY_ATSTART option 731: \Odd set the size of the output vector passed to pcre[16|32]_exec() to dd (any number of digits) 732: \P pass the PCRE_PARTIAL_SOFT option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec(); if used twice, pass the 733: PCRE_PARTIAL_HARD option 734: \Qdd set the PCRE_MATCH_LIMIT_RECURSION limit to dd (any number of digits) 735: \R pass the PCRE_DFA_RESTART option to pcre[16|32]_dfa_exec() 736: \S output details of memory get/free calls during matching 737: \Y pass the PCRE_NO_START_OPTIMIZE option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 738: \Z pass the PCRE_NOTEOL option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 739: \? pass the PCRE_NO_UTF[8|16|32]_CHECK option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 740: \>dd start the match at offset dd (optional "-"; then any number of digits); this sets the startoffset 741: argument for pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 742: \<cr> pass the PCRE_NEWLINE_CR option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 743: \<lf> pass the PCRE_NEWLINE_LF option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 744: \<crlf> pass the PCRE_NEWLINE_CRLF option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 745: \<anycrlf> pass the PCRE_NEWLINE_ANYCRLF option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 746: \<any> pass the PCRE_NEWLINE_ANY option to pcre[16|32]_exec() or pcre[16|32]_dfa_exec() 747: </pre> 748: The use of \x{hh...} is not dependent on the use of the /8 modifier on 749: the pattern. It is recognized always. There may be any number of hexadecimal 750: digits inside the braces; invalid values provoke error messages. 751:  752:  753: Note that \xhh specifies one byte rather than one character in UTF-8 mode; 754: this makes it possible to construct invalid UTF-8 sequences for testing 755: purposes. On the other hand, \x{hh} is interpreted as a UTF-8 character in 756: UTF-8 mode, generating more than one byte if the value is greater than 127. 757: When testing the 8-bit library not in UTF-8 mode, \x{hh} generates one byte 758: for values less than 256, and causes an error for greater values. 759:  760:  761: In UTF-16 mode, all 4-digit \x{hhhh} values are accepted. This makes it 762: possible to construct invalid UTF-16 sequences for testing purposes. 763:  764:  765: In UTF-32 mode, all 4- to 8-digit \x{...} values are accepted. This makes it 766: possible to construct invalid UTF-32 sequences for testing purposes. 767:  768:  769: The escapes that specify line ending sequences are literal strings, exactly as 770: shown. No more than one newline setting should be present in any data line. 771:  772:  773: A backslash followed by anything else just escapes the anything else. If 774: the very last character is a backslash, it is ignored. This gives a way of 775: passing an empty line as data, since a real empty line terminates the data 776: input. 777:  778:  779: The \J escape provides a way of setting the maximum stack size that is 780: used by the just-in-time optimization code. It is ignored if JIT optimization 781: is not being used. Providing a stack that is larger than the default 32K is 782: necessary only for very complicated patterns. 783:  784:  785: If \M is present, pcretest calls pcre[16|32]_exec() several times, 786: with different values in the match_limit and match_limit_recursion 787: fields of the pcre[16|32]_extra data structure, until it finds the minimum 788: numbers for each parameter that allow pcre[16|32]_exec() to complete without 789: error. Because this is testing a specific feature of the normal interpretive 790: pcre[16|32]_exec() execution, the use of any JIT optimization that might 791: have been set up by the /S+ qualifier of -s+ option is disabled. 792:  793:  794: The match_limit number is a measure of the amount of backtracking 795: that takes place, and checking it out can be instructive. For most simple 796: matches, the number is quite small, but for patterns with very large numbers of 797: matching possibilities, it can become large very quickly with increasing length 798: of subject string. The match_limit_recursion number is a measure of how 799: much stack (or, if PCRE is compiled with NO_RECURSE, how much heap) memory is 800: needed to complete the match attempt. 801:  802:  803: When \O is used, the value specified may be higher or lower than the size set 804: by the -O command line option (or defaulted to 45); \O applies only to 805: the call of pcre[16|32]_exec() for the line in which it appears. 806:  807:  808: If the /P modifier was present on the pattern, causing the POSIX wrapper 809: API to be used, the only option-setting sequences that have any effect are \B, 810: \N, and \Z, causing REG_NOTBOL, REG_NOTEMPTY, and REG_NOTEOL, respectively, 811: to be passed to regexec(). 812:  813:  <a name="SEC8" href="#TOC1">THE ALTERNATIVE MATCHING FUNCTION</a>  814:  815: By default, pcretest uses the standard PCRE matching function, 816: pcre[16|32]_exec() to match each data line. PCRE also supports an 817: alternative matching function, pcre[16|32]_dfa_test(), which operates in a 818: different way, and has some restrictions. The differences between the two 819: functions are described in the 820: <a href="pcrematching.html">pcrematching</a> 821: documentation. 822:  823:  824: If a data line contains the \D escape sequence, or if the command line 825: contains the -dfa option, the alternative matching function is used. 826: This function finds all possible matches at a given point. If, however, the \F 827: escape sequence is present in the data line, it stops after the first match is 828: found. This is always the shortest possible match. 829:  830:  <a name="SEC9" href="#TOC1">DEFAULT OUTPUT FROM PCRETEST</a>  831:  832: This section describes the output when the normal matching function, 833: pcre[16|32]_exec(), is being used. 834:  835:  836: When a match succeeds, pcretest outputs the list of captured substrings 837: that pcre[16|32]_exec() returns, starting with number 0 for the string that 838: matched the whole pattern. Otherwise, it outputs "No match" when the return is 839: PCRE_ERROR_NOMATCH, and "Partial match:" followed by the partially matching 840: substring when pcre[16|32]_exec() returns PCRE_ERROR_PARTIAL. (Note that 841: this is the entire substring that was inspected during the partial match; it 842: may include characters before the actual match start if a lookbehind assertion, 843: \K, \b, or \B was involved.) For any other return, pcretest outputs 844: the PCRE negative error number and a short descriptive phrase. If the error is 845: a failed UTF string check, the offset of the start of the failing character and 846: the reason code are also output, provided that the size of the output vector is 847: at least two. Here is an example of an interactive pcretest run. 848: <pre> 849: $ pcretest 850: PCRE version 8.13 2011-04-30 851: 852: re> /^abc(\d+)/ 853: data> abc123 854: 0: abc123 855: 1: 123 856: data> xyz 857: No match 858: </pre> 859: Unset capturing substrings that are not followed by one that is set are not 860: returned by pcre[16|32]_exec(), and are not shown by pcretest. In the 861: following example, there are two capturing substrings, but when the first data 862: line is matched, the second, unset substring is not shown. An "internal" unset 863: substring is shown as "<unset>", as for the second data line. 864: <pre> 865: re> /(a)|(b)/ 866: data> a 867: 0: a 868: 1: a 869: data> b 870: 0: b 871: 1: <unset> 872: 2: b 873: </pre> 874: If the strings contain any non-printing characters, they are output as \xhh 875: escapes if the value is less than 256 and UTF mode is not set. Otherwise they 876: are output as \x{hh...} escapes. See below for the definition of non-printing 877: characters. If the pattern has the /+ modifier, the output for substring 878: 0 is followed by the the rest of the subject string, identified by "0+" like 879: this: 880: <pre> 881: re> /cat/+ 882: data> cataract 883: 0: cat 884: 0+ aract 885: </pre> 886: If the pattern has the /g or /G modifier, the results of successive 887: matching attempts are output in sequence, like this: 888: <pre> 889: re> /\Bi(\w\w)/g 890: data> Mississippi 891: 0: iss 892: 1: ss 893: 0: iss 894: 1: ss 895: 0: ipp 896: 1: pp 897: </pre> 898: "No match" is output only if the first match attempt fails. Here is an example 899: of a failure message (the offset 4 that is specified by \>4 is past the end of 900: the subject string): 901: <pre> 902: re> /xyz/ 903: data> xyz\>4 904: Error -24 (bad offset value) 905: </PRE> 906:  907:  908: If any of the sequences \C, \G, or \L are present in a 909: data line that is successfully matched, the substrings extracted by the 910: convenience functions are output with C, G, or L after the string number 911: instead of a colon. This is in addition to the normal full list. The string 912: length (that is, the return from the extraction function) is given in 913: parentheses after each string for \C and \G. 914:  915:  916: Note that whereas patterns can be continued over several lines (a plain ">" 917: prompt is used for continuations), data lines may not. However newlines can be 918: included in data by means of the \n escape (or \r, \r\n, etc., depending on 919: the newline sequence setting). 920:  921:  <a name="SEC10" href="#TOC1">OUTPUT FROM THE ALTERNATIVE MATCHING FUNCTION</a>  922:  923: When the alternative matching function, pcre[16|32]_dfa_exec(), is used (by 924: means of the \D escape sequence or the -dfa command line option), the 925: output consists of a list of all the matches that start at the first point in 926: the subject where there is at least one match. For example: 927: <pre> 928: re> /(tang|tangerine|tan)/ 929: data> yellow tangerine\D 930: 0: tangerine 931: 1: tang 932: 2: tan 933: </pre> 934: (Using the normal matching function on this data finds only "tang".) The 935: longest matching string is always given first (and numbered zero). After a 936: PCRE_ERROR_PARTIAL return, the output is "Partial match:", followed by the 937: partially matching substring. (Note that this is the entire substring that was 938: inspected during the partial match; it may include characters before the actual 939: match start if a lookbehind assertion, \K, \b, or \B was involved.) 940:  941:  942: If /g is present on the pattern, the search for further matches resumes 943: at the end of the longest match. For example: 944: <pre> 945: re> /(tang|tangerine|tan)/g 946: data> yellow tangerine and tangy sultana\D 947: 0: tangerine 948: 1: tang 949: 2: tan 950: 0: tang 951: 1: tan 952: 0: tan 953: </pre> 954: Since the matching function does not support substring capture, the escape 955: sequences that are concerned with captured substrings are not relevant. 956:  957:  <a name="SEC11" href="#TOC1">RESTARTING AFTER A PARTIAL MATCH</a>  958:  959: When the alternative matching function has given the PCRE_ERROR_PARTIAL return, 960: indicating that the subject partially matched the pattern, you can restart the 961: match with additional subject data by means of the \R escape sequence. For 962: example: 963: <pre> 964: re> /^\d?\d(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\d\d$/ 965: data> 23ja\P\D 966: Partial match: 23ja 967: data> n05\R\D 968: 0: n05 969: </pre> 970: For further information about partial matching, see the 971: <a href="pcrepartial.html">pcrepartial</a> 972: documentation. 973:  974:  <a name="SEC12" href="#TOC1">CALLOUTS</a>  975:  976: If the pattern contains any callout requests, pcretest's callout function 977: is called during matching. This works with both matching functions. By default, 978: the called function displays the callout number, the start and current 979: positions in the text at the callout time, and the next pattern item to be 980: tested. For example: 981: <pre> 982: --->pqrabcdef 983: 0 ^ ^ \d 984: </pre> 985: This output indicates that callout number 0 occurred for a match attempt 986: starting at the fourth character of the subject string, when the pointer was at 987: the seventh character of the data, and when the next pattern item was \d. Just 988: one circumflex is output if the start and current positions are the same. 989:  990:  991: Callouts numbered 255 are assumed to be automatic callouts, inserted as a 992: result of the /C pattern modifier. In this case, instead of showing the 993: callout number, the offset in the pattern, preceded by a plus, is output. For 994: example: 995: <pre> 996: re> /\d?[A-E]\*/C 997: data> E* 998: --->E* 999: +0 ^ \d? 1000: +3 ^ [A-E] 1001: +8 ^^ \* 1002: +10 ^ ^ 1003: 0: E* 1004: </pre> 1005: If a pattern contains (*MARK) items, an additional line is output whenever 1006: a change of latest mark is passed to the callout function. For example: 1007: <pre> 1008: re> /a(*MARK:X)bc/C 1009: data> abc 1010: --->abc 1011: +0 ^ a 1012: +1 ^^ (*MARK:X) 1013: +10 ^^ b 1014: Latest Mark: X 1015: +11 ^ ^ c 1016: +12 ^ ^ 1017: 0: abc 1018: </pre> 1019: The mark changes between matching "a" and "b", but stays the same for the rest 1020: of the match, so nothing more is output. If, as a result of backtracking, the 1021: mark reverts to being unset, the text "<unset>" is output. 1022:  1023:  1024: The callout function in pcretest returns zero (carry on matching) by 1025: default, but you can use a \C item in a data line (as described above) to 1026: change this and other parameters of the callout. 1027:  1028:  1029: Inserting callouts can be helpful when using pcretest to check 1030: complicated regular expressions. For further information about callouts, see 1031: the 1032: <a href="pcrecallout.html">pcrecallout</a> 1033: documentation. 1034:  1035:  <a name="SEC13" href="#TOC1">NON-PRINTING CHARACTERS</a>  1036:  1037: When pcretest is outputting text in the compiled version of a pattern, 1038: bytes other than 32-126 are always treated as non-printing characters are are 1039: therefore shown as hex escapes. 1040:  1041:  1042: When pcretest is outputting text that is a matched part of a subject 1043: string, it behaves in the same way, unless a different locale has been set for 1044: the pattern (using the /L modifier). In this case, the isprint() 1045: function to distinguish printing and non-printing characters. 1046:  1047:  <a name="SEC14" href="#TOC1">SAVING AND RELOADING COMPILED PATTERNS</a>  1048:  1049: The facilities described in this section are not available when the POSIX 1050: interface to PCRE is being used, that is, when the /P pattern modifier is 1051: specified. 1052:  1053:  1054: When the POSIX interface is not in use, you can cause pcretest to write a 1055: compiled pattern to a file, by following the modifiers with > and a file name. 1056: For example: 1057: <pre> 1058: /pattern/im >/some/file 1059: </pre> 1060: See the 1061: <a href="pcreprecompile.html">pcreprecompile</a> 1062: documentation for a discussion about saving and re-using compiled patterns. 1063: Note that if the pattern was successfully studied with JIT optimization, the 1064: JIT data cannot be saved. 1065:  1066:  1067: The data that is written is binary. The first eight bytes are the length of the 1068: compiled pattern data followed by the length of the optional study data, each 1069: written as four bytes in big-endian order (most significant byte first). If 1070: there is no study data (either the pattern was not studied, or studying did not 1071: return any data), the second length is zero. The lengths are followed by an 1072: exact copy of the compiled pattern. If there is additional study data, this 1073: (excluding any JIT data) follows immediately after the compiled pattern. After 1074: writing the file, pcretest expects to read a new pattern. 1075:  1076:  1077: A saved pattern can be reloaded into pcretest by specifying < and a file 1078: name instead of a pattern. There must be no space between < and the file name, 1079: which must not contain a < character, as otherwise pcretest will 1080: interpret the line as a pattern delimited by < characters. For example: 1081: <pre> 1082: re> </some/file 1083: Compiled pattern loaded from /some/file 1084: No study data 1085: </pre> 1086: If the pattern was previously studied with the JIT optimization, the JIT 1087: information cannot be saved and restored, and so is lost. When the pattern has 1088: been loaded, pcretest proceeds to read data lines in the usual way. 1089:  1090:  1091: You can copy a file written by pcretest to a different host and reload it 1092: there, even if the new host has opposite endianness to the one on which the 1093: pattern was compiled. For example, you can compile on an i86 machine and run on 1094: a SPARC machine. When a pattern is reloaded on a host with different 1095: endianness, the confirmation message is changed to: 1096: <pre> 1097: Compiled pattern (byte-inverted) loaded from /some/file 1098: </pre> 1099: The test suite contains some saved pre-compiled patterns with different 1100: endianness. These are reloaded using "<!" instead of just "<". This suppresses 1101: the "(byte-inverted)" text so that the output is the same on all hosts. It also 1102: forces debugging output once the pattern has been reloaded. 1103:  1104:  1105: File names for saving and reloading can be absolute or relative, but note that 1106: the shell facility of expanding a file name that starts with a tilde (~) is not 1107: available. 1108:  1109:  1110: The ability to save and reload files in pcretest is intended for testing 1111: and experimentation. It is not intended for production use because only a 1112: single pattern can be written to a file. Furthermore, there is no facility for 1113: supplying custom character tables for use with a reloaded pattern. If the 1114: original pattern was compiled with custom tables, an attempt to match a subject 1115: string using a reloaded pattern is likely to cause pcretest to crash. 1116: Finally, if you attempt to load a file that is not in the correct format, the 1117: result is undefined. 1118:  1119:  <a name="SEC15" href="#TOC1">SEE ALSO</a>  1120:  1121: pcre(3), pcre16(3), pcre32(3), pcreapi(3), 1122: pcrecallout(3), 1123: pcrejit, pcrematching(3), pcrepartial(d), 1124: pcrepattern(3), pcreprecompile(3). 1125:  1126:  <a name="SEC16" href="#TOC1">AUTHOR</a>  1127:  1128: Philip Hazel 1129:   1130: University Computing Service 1131:   1132: Cambridge CB2 3QH, England. 1133:   1134:  1135:  <a name="SEC17" href="#TOC1">REVISION</a>  1136:  1137: Last updated: 12 November 2013 1138:   1139: Copyright © 1997-2013 University of Cambridge. 1140:   1141:  1142: Return to the <a href="index.html">PCRE index page</a>. 1143: 