Annotation of embedaddon/pcre/doc/html/pcrejit.html, revision 1.1
1.1 ! misho 1: <html>
! 2: <head>
! 3: <title>pcrejit specification</title>
! 4: </head>
! 5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
! 6: <h1>pcrejit man page</h1>
! 7: <p>
! 8: Return to the <a href="index.html">PCRE index page</a>.
! 9: </p>
! 10: <p>
! 11: This page is part of the PCRE HTML documentation. It was generated automatically
! 12: from the original man page. If there is any nonsense in it, please consult the
! 13: man page, in case the conversion went wrong.
! 14: <br>
! 15: <ul>
! 16: <li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a>
! 17: <li><a name="TOC2" href="#SEC2">AVAILABILITY OF JIT SUPPORT</a>
! 18: <li><a name="TOC3" href="#SEC3">SIMPLE USE OF JIT</a>
! 19: <li><a name="TOC4" href="#SEC4">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a>
! 20: <li><a name="TOC5" href="#SEC5">RETURN VALUES FROM JIT EXECUTION</a>
! 21: <li><a name="TOC6" href="#SEC6">SAVING AND RESTORING COMPILED PATTERNS</a>
! 22: <li><a name="TOC7" href="#SEC7">CONTROLLING THE JIT STACK</a>
! 23: <li><a name="TOC8" href="#SEC8">JIT STACK FAQ</a>
! 24: <li><a name="TOC9" href="#SEC9">EXAMPLE CODE</a>
! 25: <li><a name="TOC10" href="#SEC10">SEE ALSO</a>
! 26: <li><a name="TOC11" href="#SEC11">AUTHOR</a>
! 27: <li><a name="TOC12" href="#SEC12">REVISION</a>
! 28: </ul>
! 29: <br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br>
! 30: <P>
! 31: Just-in-time compiling is a heavyweight optimization that can greatly speed up
! 32: pattern matching. However, it comes at the cost of extra processing before the
! 33: match is performed. Therefore, it is of most benefit when the same pattern is
! 34: going to be matched many times. This does not necessarily mean many calls of
! 35: \fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
! 36: place many times at various positions in the subject, even for a single call to
! 37: <b>pcre_exec()</b>. If the subject string is very long, it may still pay to use
! 38: JIT for one-off matches.
! 39: </P>
! 40: <P>
! 41: JIT support applies only to the traditional matching function,
! 42: <b>pcre_exec()</b>. It does not apply when <b>pcre_dfa_exec()</b> is being used.
! 43: The code for this support was written by Zoltan Herczeg.
! 44: </P>
! 45: <br><a name="SEC2" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br>
! 46: <P>
! 47: JIT support is an optional feature of PCRE. The "configure" option --enable-jit
! 48: (or equivalent CMake option) must be set when PCRE is built if you want to use
! 49: JIT. The support is limited to the following hardware platforms:
! 50: <pre>
! 51: ARM v5, v7, and Thumb2
! 52: Intel x86 32-bit and 64-bit
! 53: MIPS 32-bit
! 54: Power PC 32-bit and 64-bit (experimental)
! 55: </pre>
! 56: The Power PC support is designated as experimental because it has not been
! 57: fully tested. If --enable-jit is set on an unsupported platform, compilation
! 58: fails.
! 59: </P>
! 60: <P>
! 61: A program that is linked with PCRE 8.20 or later can tell if JIT support is
! 62: available by calling <b>pcre_config()</b> with the PCRE_CONFIG_JIT option. The
! 63: result is 1 when JIT is available, and 0 otherwise. However, a simple program
! 64: does not need to check this in order to use JIT. The API is implemented in a
! 65: way that falls back to the ordinary PCRE code if JIT is not available.
! 66: </P>
! 67: <P>
! 68: If your program may sometimes be linked with versions of PCRE that are older
! 69: than 8.20, but you want to use JIT when it is available, you can test
! 70: the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
! 71: as PCRE_CONFIG_JIT, for compile-time control of your code.
! 72: </P>
! 73: <br><a name="SEC3" href="#TOC1">SIMPLE USE OF JIT</a><br>
! 74: <P>
! 75: You have to do two things to make use of the JIT support in the simplest way:
! 76: <pre>
! 77: (1) Call <b>pcre_study()</b> with the PCRE_STUDY_JIT_COMPILE option for
! 78: each compiled pattern, and pass the resulting <b>pcre_extra</b> block to
! 79: <b>pcre_exec()</b>.
! 80:
! 81: (2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is
! 82: no longer needed instead of just freeing it yourself. This
! 83: ensures that any JIT data is also freed.
! 84: </pre>
! 85: For a program that may be linked with pre-8.20 versions of PCRE, you can insert
! 86: <pre>
! 87: #ifndef PCRE_STUDY_JIT_COMPILE
! 88: #define PCRE_STUDY_JIT_COMPILE 0
! 89: #endif
! 90: </pre>
! 91: so that no option is passed to <b>pcre_study()</b>, and then use something like
! 92: this to free the study data:
! 93: <pre>
! 94: #ifdef PCRE_CONFIG_JIT
! 95: pcre_free_study(study_ptr);
! 96: #else
! 97: pcre_free(study_ptr);
! 98: #endif
! 99: </pre>
! 100: In some circumstances you may need to call additional functions. These are
! 101: described in the section entitled
! 102: <a href="#stackcontrol">"Controlling the JIT stack"</a>
! 103: below.
! 104: </P>
! 105: <P>
! 106: If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
! 107: data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
! 108: which turns it into machine code that executes much faster than the normal
! 109: interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b> block
! 110: containing a pointer to JIT code, it obeys that instead of the normal code. The
! 111: result is identical, but the code runs much faster.
! 112: </P>
! 113: <P>
! 114: There are some <b>pcre_exec()</b> options that are not supported for JIT
! 115: execution. There are also some pattern items that JIT cannot handle. Details
! 116: are given below. In both cases, execution automatically falls back to the
! 117: interpretive code.
! 118: </P>
! 119: <P>
! 120: If the JIT compiler finds an unsupported item, no JIT data is generated. You
! 121: can find out if JIT execution is available after studying a pattern by calling
! 122: <b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that
! 123: JIT compilation was successful. A result of 0 means that JIT support is not
! 124: available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
! 125: JIT compiler was not able to handle the pattern.
! 126: </P>
! 127: <P>
! 128: Once a pattern has been studied, with or without JIT, it can be used as many
! 129: times as you like for matching different subject strings.
! 130: </P>
! 131: <br><a name="SEC4" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br>
! 132: <P>
! 133: The only <b>pcre_exec()</b> options that are supported for JIT execution are
! 134: PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
! 135: PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
! 136: supported.
! 137: </P>
! 138: <P>
! 139: The unsupported pattern items are:
! 140: <pre>
! 141: \C match a single byte; not supported in UTF-8 mode
! 142: (?Cn) callouts
! 143: (*COMMIT) )
! 144: (*MARK) )
! 145: (*PRUNE) ) the backtracking control verbs
! 146: (*SKIP) )
! 147: (*THEN) )
! 148: </pre>
! 149: Support for some of these may be added in future.
! 150: </P>
! 151: <br><a name="SEC5" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br>
! 152: <P>
! 153: When a pattern is matched using JIT execution, the return values are the same
! 154: as those given by the interpretive <b>pcre_exec()</b> code, with the addition of
! 155: one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
! 156: for the JIT stack was insufficient. See
! 157: <a href="#stackcontrol">"Controlling the JIT stack"</a>
! 158: below for a discussion of JIT stack usage. For compatibility with the
! 159: interpretive <b>pcre_exec()</b> code, no more than two-thirds of the
! 160: <i>ovector</i> argument is used for passing back captured substrings.
! 161: </P>
! 162: <P>
! 163: The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
! 164: very large pattern tree goes on for too long, as it is in the same circumstance
! 165: when JIT is not used, but the details of exactly what is counted are not the
! 166: same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
! 167: execution.
! 168: </P>
! 169: <br><a name="SEC6" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br>
! 170: <P>
! 171: The code that is generated by the JIT compiler is architecture-specific, and is
! 172: also position dependent. For those reasons it cannot be saved (in a file or
! 173: database) and restored later like the bytecode and other data of a compiled
! 174: pattern. Saving and restoring compiled patterns is not something many people
! 175: do. More detail about this facility is given in the
! 176: <a href="pcreprecompile.html"><b>pcreprecompile</b></a>
! 177: documentation. It should be possible to run <b>pcre_study()</b> on a saved and
! 178: restored pattern, and thereby recreate the JIT data, but because JIT
! 179: compilation uses significant resources, it is probably not worth doing this;
! 180: you might as well recompile the original pattern.
! 181: <a name="stackcontrol"></a></P>
! 182: <br><a name="SEC7" href="#TOC1">CONTROLLING THE JIT STACK</a><br>
! 183: <P>
! 184: When the compiled JIT code runs, it needs a block of memory to use as a stack.
! 185: By default, it uses 32K on the machine stack. However, some large or
! 186: complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
! 187: is given when there is not enough stack. Three functions are provided for
! 188: managing blocks of memory for use as JIT stacks. There is further discussion
! 189: about the use of JIT stacks in the section entitled
! 190: <a href="#stackcontrol">"JIT stack FAQ"</a>
! 191: below.
! 192: </P>
! 193: <P>
! 194: The <b>pcre_jit_stack_alloc()</b> function creates a JIT stack. Its arguments
! 195: are a starting size and a maximum size, and it returns a pointer to an opaque
! 196: structure of type <b>pcre_jit_stack</b>, or NULL if there is an error. The
! 197: <b>pcre_jit_stack_free()</b> function can be used to free a stack that is no
! 198: longer needed. (For the technically minded: the address space is allocated by
! 199: mmap or VirtualAlloc.)
! 200: </P>
! 201: <P>
! 202: JIT uses far less memory for recursion than the interpretive code,
! 203: and a maximum stack size of 512K to 1M should be more than enough for any
! 204: pattern.
! 205: </P>
! 206: <P>
! 207: The <b>pcre_assign_jit_stack()</b> function specifies which stack JIT code
! 208: should use. Its arguments are as follows:
! 209: <pre>
! 210: pcre_extra *extra
! 211: pcre_jit_callback callback
! 212: void *data
! 213: </pre>
! 214: The <i>extra</i> argument must be the result of studying a pattern with
! 215: PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
! 216: options:
! 217: <pre>
! 218: (1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block
! 219: on the machine stack is used.
! 220:
! 221: (2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be
! 222: a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>.
! 223:
! 224: (3) If <i>callback</i> not NULL, it must point to a function that is called
! 225: with <i>data</i> as an argument at the start of matching, in order to
! 226: set up a JIT stack. If the result is NULL, the internal 32K stack
! 227: is used; otherwise the return value must be a valid JIT stack,
! 228: the result of calling <b>pcre_jit_stack_alloc()</b>.
! 229: </pre>
! 230: You may safely assign the same JIT stack to more than one pattern, as long as
! 231: they are all matched sequentially in the same thread. In a multithread
! 232: application, each thread must use its own JIT stack.
! 233: </P>
! 234: <P>
! 235: Strictly speaking, even more is allowed. You can assign the same stack to any
! 236: number of patterns as long as they are not used for matching by multiple
! 237: threads at the same time. For example, you can assign the same stack to all
! 238: compiled patterns, and use a global mutex in the callback to wait until the
! 239: stack is available for use. However, this is an inefficient solution, and
! 240: not recommended.
! 241: </P>
! 242: <P>
! 243: This is a suggestion for how a typical multithreaded program might operate:
! 244: <pre>
! 245: During thread initalization
! 246: thread_local_var = pcre_jit_stack_alloc(...)
! 247:
! 248: During thread exit
! 249: pcre_jit_stack_free(thread_local_var)
! 250:
! 251: Use a one-line callback function
! 252: return thread_local_var
! 253: </pre>
! 254: All the functions described in this section do nothing if JIT is not available,
! 255: and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument
! 256: is non-NULL and points to a <b>pcre_extra</b> block that is the result of a
! 257: successful study with PCRE_STUDY_JIT_COMPILE.
! 258: <a name="stackfaq"></a></P>
! 259: <br><a name="SEC8" href="#TOC1">JIT STACK FAQ</a><br>
! 260: <P>
! 261: (1) Why do we need JIT stacks?
! 262: <br>
! 263: <br>
! 264: PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
! 265: the local data of the current node is pushed before checking its child nodes.
! 266: Allocating real machine stack on some platforms is difficult. For example, the
! 267: stack chain needs to be updated every time if we extend the stack on PowerPC.
! 268: Although it is possible, its updating time overhead decreases performance. So
! 269: we do the recursion in memory.
! 270: </P>
! 271: <P>
! 272: (2) Why don't we simply allocate blocks of memory with <b>malloc()</b>?
! 273: <br>
! 274: <br>
! 275: Modern operating systems have a nice feature: they can reserve an address space
! 276: instead of allocating memory. We can safely allocate memory pages inside this
! 277: address space, so the stack could grow without moving memory data (this is
! 278: important because of pointers). Thus we can allocate 1M address space, and use
! 279: only a single memory page (usually 4K) if that is enough. However, we can still
! 280: grow up to 1M anytime if needed.
! 281: </P>
! 282: <P>
! 283: (3) Who "owns" a JIT stack?
! 284: <br>
! 285: <br>
! 286: The owner of the stack is the user program, not the JIT studied pattern or
! 287: anything else. The user program must ensure that if a stack is used by
! 288: <b>pcre_exec()</b>, (that is, it is assigned to the pattern currently running),
! 289: that stack must not be used by any other threads (to avoid overwriting the same
! 290: memory area). The best practice for multithreaded programs is to allocate a
! 291: stack for each thread, and return this stack through the JIT callback function.
! 292: </P>
! 293: <P>
! 294: (4) When should a JIT stack be freed?
! 295: <br>
! 296: <br>
! 297: You can free a JIT stack at any time, as long as it will not be used by
! 298: <b>pcre_exec()</b> again. When you assign the stack to a pattern, only a pointer
! 299: is set. There is no reference counting or any other magic. You can free the
! 300: patterns and stacks in any order, anytime. Just <i>do not</i> call
! 301: <b>pcre_exec()</b> with a pattern pointing to an already freed stack, as that
! 302: will cause SEGFAULT. (Also, do not free a stack currently used by
! 303: <b>pcre_exec()</b> in another thread). You can also replace the stack for a
! 304: pattern at any time. You can even free the previous stack before assigning a
! 305: replacement.
! 306: </P>
! 307: <P>
! 308: (5) Should I allocate/free a stack every time before/after calling
! 309: <b>pcre_exec()</b>?
! 310: <br>
! 311: <br>
! 312: No, because this is too costly in terms of resources. However, you could
! 313: implement some clever idea which release the stack if it is not used in let's
! 314: say two minutes. The JIT callback can help to achive this without keeping a
! 315: list of the currently JIT studied patterns.
! 316: </P>
! 317: <P>
! 318: (6) OK, the stack is for long term memory allocation. But what happens if a
! 319: pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
! 320: stack is freed?
! 321: <br>
! 322: <br>
! 323: Especially on embedded sytems, it might be a good idea to release
! 324: memory sometimes without freeing the stack. There is no API for this at the
! 325: moment. Probably a function call which returns with the currently allocated
! 326: memory for any stack and another which allows releasing memory (shrinking the
! 327: stack) would be a good idea if someone needs this.
! 328: </P>
! 329: <P>
! 330: (7) This is too much of a headache. Isn't there any better solution for JIT
! 331: stack handling?
! 332: <br>
! 333: <br>
! 334: No, thanks to Windows. If POSIX threads were used everywhere, we could throw
! 335: out this complicated API.
! 336: </P>
! 337: <br><a name="SEC9" href="#TOC1">EXAMPLE CODE</a><br>
! 338: <P>
! 339: This is a single-threaded example that specifies a JIT stack without using a
! 340: callback.
! 341: <pre>
! 342: int rc;
! 343: int ovector[30];
! 344: pcre *re;
! 345: pcre_extra *extra;
! 346: pcre_jit_stack *jit_stack;
! 347:
! 348: re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
! 349: /* Check for errors */
! 350: extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
! 351: jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
! 352: /* Check for error (NULL) */
! 353: pcre_assign_jit_stack(extra, NULL, jit_stack);
! 354: rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
! 355: /* Check results */
! 356: pcre_free(re);
! 357: pcre_free_study(extra);
! 358: pcre_jit_stack_free(jit_stack);
! 359:
! 360: </PRE>
! 361: </P>
! 362: <br><a name="SEC10" href="#TOC1">SEE ALSO</a><br>
! 363: <P>
! 364: <b>pcreapi</b>(3)
! 365: </P>
! 366: <br><a name="SEC11" href="#TOC1">AUTHOR</a><br>
! 367: <P>
! 368: Philip Hazel (FAQ by Zoltan Herczeg)
! 369: <br>
! 370: University Computing Service
! 371: <br>
! 372: Cambridge CB2 3QH, England.
! 373: <br>
! 374: </P>
! 375: <br><a name="SEC12" href="#TOC1">REVISION</a><br>
! 376: <P>
! 377: Last updated: 26 November 2011
! 378: <br>
! 379: Copyright © 1997-2011 University of Cambridge.
! 380: <br>
! 381: <p>
! 382: Return to the <a href="index.html">PCRE index page</a>.
! 383: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrejit.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html