Annotation of embedaddon/pcre/doc/pcrejit.3, revision 1.1
1.1 ! misho 1: .TH PCREJIT 3
! 2: .SH NAME
! 3: PCRE - Perl-compatible regular expressions
! 4: .SH "PCRE JUST-IN-TIME COMPILER SUPPORT"
! 5: .rs
! 6: .sp
! 7: Just-in-time compiling is a heavyweight optimization that can greatly speed up
! 8: pattern matching. However, it comes at the cost of extra processing before the
! 9: match is performed. Therefore, it is of most benefit when the same pattern is
! 10: going to be matched many times. This does not necessarily mean many calls of
! 11: \fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
! 12: place many times at various positions in the subject, even for a single call to
! 13: \fBpcre_exec()\fP. If the subject string is very long, it may still pay to use
! 14: JIT for one-off matches.
! 15: .P
! 16: JIT support applies only to the traditional matching function,
! 17: \fBpcre_exec()\fP. It does not apply when \fBpcre_dfa_exec()\fP is being used.
! 18: The code for this support was written by Zoltan Herczeg.
! 19: .
! 20: .
! 21: .SH "AVAILABILITY OF JIT SUPPORT"
! 22: .rs
! 23: .sp
! 24: JIT support is an optional feature of PCRE. The "configure" option --enable-jit
! 25: (or equivalent CMake option) must be set when PCRE is built if you want to use
! 26: JIT. The support is limited to the following hardware platforms:
! 27: .sp
! 28: ARM v5, v7, and Thumb2
! 29: Intel x86 32-bit and 64-bit
! 30: MIPS 32-bit
! 31: Power PC 32-bit and 64-bit (experimental)
! 32: .sp
! 33: The Power PC support is designated as experimental because it has not been
! 34: fully tested. If --enable-jit is set on an unsupported platform, compilation
! 35: fails.
! 36: .P
! 37: A program that is linked with PCRE 8.20 or later can tell if JIT support is
! 38: available by calling \fBpcre_config()\fP with the PCRE_CONFIG_JIT option. The
! 39: result is 1 when JIT is available, and 0 otherwise. However, a simple program
! 40: does not need to check this in order to use JIT. The API is implemented in a
! 41: way that falls back to the ordinary PCRE code if JIT is not available.
! 42: .P
! 43: If your program may sometimes be linked with versions of PCRE that are older
! 44: than 8.20, but you want to use JIT when it is available, you can test
! 45: the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
! 46: as PCRE_CONFIG_JIT, for compile-time control of your code.
! 47: .
! 48: .
! 49: .SH "SIMPLE USE OF JIT"
! 50: .rs
! 51: .sp
! 52: You have to do two things to make use of the JIT support in the simplest way:
! 53: .sp
! 54: (1) Call \fBpcre_study()\fP with the PCRE_STUDY_JIT_COMPILE option for
! 55: each compiled pattern, and pass the resulting \fBpcre_extra\fP block to
! 56: \fBpcre_exec()\fP.
! 57: .sp
! 58: (2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is
! 59: no longer needed instead of just freeing it yourself. This
! 60: ensures that any JIT data is also freed.
! 61: .sp
! 62: For a program that may be linked with pre-8.20 versions of PCRE, you can insert
! 63: .sp
! 64: #ifndef PCRE_STUDY_JIT_COMPILE
! 65: #define PCRE_STUDY_JIT_COMPILE 0
! 66: #endif
! 67: .sp
! 68: so that no option is passed to \fBpcre_study()\fP, and then use something like
! 69: this to free the study data:
! 70: .sp
! 71: #ifdef PCRE_CONFIG_JIT
! 72: pcre_free_study(study_ptr);
! 73: #else
! 74: pcre_free(study_ptr);
! 75: #endif
! 76: .sp
! 77: In some circumstances you may need to call additional functions. These are
! 78: described in the section entitled
! 79: .\" HTML <a href="#stackcontrol">
! 80: .\" </a>
! 81: "Controlling the JIT stack"
! 82: .\"
! 83: below.
! 84: .P
! 85: If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
! 86: data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
! 87: which turns it into machine code that executes much faster than the normal
! 88: interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block
! 89: containing a pointer to JIT code, it obeys that instead of the normal code. The
! 90: result is identical, but the code runs much faster.
! 91: .P
! 92: There are some \fBpcre_exec()\fP options that are not supported for JIT
! 93: execution. There are also some pattern items that JIT cannot handle. Details
! 94: are given below. In both cases, execution automatically falls back to the
! 95: interpretive code.
! 96: .P
! 97: If the JIT compiler finds an unsupported item, no JIT data is generated. You
! 98: can find out if JIT execution is available after studying a pattern by calling
! 99: \fBpcre_fullinfo()\fP with the PCRE_INFO_JIT option. A result of 1 means that
! 100: JIT compilation was successful. A result of 0 means that JIT support is not
! 101: available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
! 102: JIT compiler was not able to handle the pattern.
! 103: .P
! 104: Once a pattern has been studied, with or without JIT, it can be used as many
! 105: times as you like for matching different subject strings.
! 106: .
! 107: .
! 108: .SH "UNSUPPORTED OPTIONS AND PATTERN ITEMS"
! 109: .rs
! 110: .sp
! 111: The only \fBpcre_exec()\fP options that are supported for JIT execution are
! 112: PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
! 113: PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
! 114: supported.
! 115: .P
! 116: The unsupported pattern items are:
! 117: .sp
! 118: \eC match a single byte; not supported in UTF-8 mode
! 119: (?Cn) callouts
! 120: (*COMMIT) )
! 121: (*MARK) )
! 122: (*PRUNE) ) the backtracking control verbs
! 123: (*SKIP) )
! 124: (*THEN) )
! 125: .sp
! 126: Support for some of these may be added in future.
! 127: .
! 128: .
! 129: .SH "RETURN VALUES FROM JIT EXECUTION"
! 130: .rs
! 131: .sp
! 132: When a pattern is matched using JIT execution, the return values are the same
! 133: as those given by the interpretive \fBpcre_exec()\fP code, with the addition of
! 134: one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
! 135: for the JIT stack was insufficient. See
! 136: .\" HTML <a href="#stackcontrol">
! 137: .\" </a>
! 138: "Controlling the JIT stack"
! 139: .\"
! 140: below for a discussion of JIT stack usage. For compatibility with the
! 141: interpretive \fBpcre_exec()\fP code, no more than two-thirds of the
! 142: \fIovector\fP argument is used for passing back captured substrings.
! 143: .P
! 144: The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
! 145: very large pattern tree goes on for too long, as it is in the same circumstance
! 146: when JIT is not used, but the details of exactly what is counted are not the
! 147: same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
! 148: execution.
! 149: .
! 150: .
! 151: .SH "SAVING AND RESTORING COMPILED PATTERNS"
! 152: .rs
! 153: .sp
! 154: The code that is generated by the JIT compiler is architecture-specific, and is
! 155: also position dependent. For those reasons it cannot be saved (in a file or
! 156: database) and restored later like the bytecode and other data of a compiled
! 157: pattern. Saving and restoring compiled patterns is not something many people
! 158: do. More detail about this facility is given in the
! 159: .\" HREF
! 160: \fBpcreprecompile\fP
! 161: .\"
! 162: documentation. It should be possible to run \fBpcre_study()\fP on a saved and
! 163: restored pattern, and thereby recreate the JIT data, but because JIT
! 164: compilation uses significant resources, it is probably not worth doing this;
! 165: you might as well recompile the original pattern.
! 166: .
! 167: .
! 168: .\" HTML <a name="stackcontrol"></a>
! 169: .SH "CONTROLLING THE JIT STACK"
! 170: .rs
! 171: .sp
! 172: When the compiled JIT code runs, it needs a block of memory to use as a stack.
! 173: By default, it uses 32K on the machine stack. However, some large or
! 174: complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
! 175: is given when there is not enough stack. Three functions are provided for
! 176: managing blocks of memory for use as JIT stacks. There is further discussion
! 177: about the use of JIT stacks in the section entitled
! 178: .\" HTML <a href="#stackcontrol">
! 179: .\" </a>
! 180: "JIT stack FAQ"
! 181: .\"
! 182: below.
! 183: .P
! 184: The \fBpcre_jit_stack_alloc()\fP function creates a JIT stack. Its arguments
! 185: are a starting size and a maximum size, and it returns a pointer to an opaque
! 186: structure of type \fBpcre_jit_stack\fP, or NULL if there is an error. The
! 187: \fBpcre_jit_stack_free()\fP function can be used to free a stack that is no
! 188: longer needed. (For the technically minded: the address space is allocated by
! 189: mmap or VirtualAlloc.)
! 190: .P
! 191: JIT uses far less memory for recursion than the interpretive code,
! 192: and a maximum stack size of 512K to 1M should be more than enough for any
! 193: pattern.
! 194: .P
! 195: The \fBpcre_assign_jit_stack()\fP function specifies which stack JIT code
! 196: should use. Its arguments are as follows:
! 197: .sp
! 198: pcre_extra *extra
! 199: pcre_jit_callback callback
! 200: void *data
! 201: .sp
! 202: The \fIextra\fP argument must be the result of studying a pattern with
! 203: PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
! 204: options:
! 205: .sp
! 206: (1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block
! 207: on the machine stack is used.
! 208: .sp
! 209: (2) If \fIcallback\fP is NULL and \fIdata\fP is not NULL, \fIdata\fP must be
! 210: a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP.
! 211: .sp
! 212: (3) If \fIcallback\fP not NULL, it must point to a function that is called
! 213: with \fIdata\fP as an argument at the start of matching, in order to
! 214: set up a JIT stack. If the result is NULL, the internal 32K stack
! 215: is used; otherwise the return value must be a valid JIT stack,
! 216: the result of calling \fBpcre_jit_stack_alloc()\fP.
! 217: .sp
! 218: You may safely assign the same JIT stack to more than one pattern, as long as
! 219: they are all matched sequentially in the same thread. In a multithread
! 220: application, each thread must use its own JIT stack.
! 221: .P
! 222: Strictly speaking, even more is allowed. You can assign the same stack to any
! 223: number of patterns as long as they are not used for matching by multiple
! 224: threads at the same time. For example, you can assign the same stack to all
! 225: compiled patterns, and use a global mutex in the callback to wait until the
! 226: stack is available for use. However, this is an inefficient solution, and
! 227: not recommended.
! 228: .P
! 229: This is a suggestion for how a typical multithreaded program might operate:
! 230: .sp
! 231: During thread initalization
! 232: thread_local_var = pcre_jit_stack_alloc(...)
! 233: .sp
! 234: During thread exit
! 235: pcre_jit_stack_free(thread_local_var)
! 236: .sp
! 237: Use a one-line callback function
! 238: return thread_local_var
! 239: .sp
! 240: All the functions described in this section do nothing if JIT is not available,
! 241: and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument
! 242: is non-NULL and points to a \fBpcre_extra\fP block that is the result of a
! 243: successful study with PCRE_STUDY_JIT_COMPILE.
! 244: .
! 245: .
! 246: .\" HTML <a name="stackfaq"></a>
! 247: .SH "JIT STACK FAQ"
! 248: .rs
! 249: .sp
! 250: (1) Why do we need JIT stacks?
! 251: .sp
! 252: PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
! 253: the local data of the current node is pushed before checking its child nodes.
! 254: Allocating real machine stack on some platforms is difficult. For example, the
! 255: stack chain needs to be updated every time if we extend the stack on PowerPC.
! 256: Although it is possible, its updating time overhead decreases performance. So
! 257: we do the recursion in memory.
! 258: .P
! 259: (2) Why don't we simply allocate blocks of memory with \fBmalloc()\fP?
! 260: .sp
! 261: Modern operating systems have a nice feature: they can reserve an address space
! 262: instead of allocating memory. We can safely allocate memory pages inside this
! 263: address space, so the stack could grow without moving memory data (this is
! 264: important because of pointers). Thus we can allocate 1M address space, and use
! 265: only a single memory page (usually 4K) if that is enough. However, we can still
! 266: grow up to 1M anytime if needed.
! 267: .P
! 268: (3) Who "owns" a JIT stack?
! 269: .sp
! 270: The owner of the stack is the user program, not the JIT studied pattern or
! 271: anything else. The user program must ensure that if a stack is used by
! 272: \fBpcre_exec()\fP, (that is, it is assigned to the pattern currently running),
! 273: that stack must not be used by any other threads (to avoid overwriting the same
! 274: memory area). The best practice for multithreaded programs is to allocate a
! 275: stack for each thread, and return this stack through the JIT callback function.
! 276: .P
! 277: (4) When should a JIT stack be freed?
! 278: .sp
! 279: You can free a JIT stack at any time, as long as it will not be used by
! 280: \fBpcre_exec()\fP again. When you assign the stack to a pattern, only a pointer
! 281: is set. There is no reference counting or any other magic. You can free the
! 282: patterns and stacks in any order, anytime. Just \fIdo not\fP call
! 283: \fBpcre_exec()\fP with a pattern pointing to an already freed stack, as that
! 284: will cause SEGFAULT. (Also, do not free a stack currently used by
! 285: \fBpcre_exec()\fP in another thread). You can also replace the stack for a
! 286: pattern at any time. You can even free the previous stack before assigning a
! 287: replacement.
! 288: .P
! 289: (5) Should I allocate/free a stack every time before/after calling
! 290: \fBpcre_exec()\fP?
! 291: .sp
! 292: No, because this is too costly in terms of resources. However, you could
! 293: implement some clever idea which release the stack if it is not used in let's
! 294: say two minutes. The JIT callback can help to achive this without keeping a
! 295: list of the currently JIT studied patterns.
! 296: .P
! 297: (6) OK, the stack is for long term memory allocation. But what happens if a
! 298: pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
! 299: stack is freed?
! 300: .sp
! 301: Especially on embedded sytems, it might be a good idea to release
! 302: memory sometimes without freeing the stack. There is no API for this at the
! 303: moment. Probably a function call which returns with the currently allocated
! 304: memory for any stack and another which allows releasing memory (shrinking the
! 305: stack) would be a good idea if someone needs this.
! 306: .P
! 307: (7) This is too much of a headache. Isn't there any better solution for JIT
! 308: stack handling?
! 309: .sp
! 310: No, thanks to Windows. If POSIX threads were used everywhere, we could throw
! 311: out this complicated API.
! 312: .
! 313: .
! 314: .SH "EXAMPLE CODE"
! 315: .rs
! 316: .sp
! 317: This is a single-threaded example that specifies a JIT stack without using a
! 318: callback.
! 319: .sp
! 320: int rc;
! 321: int ovector[30];
! 322: pcre *re;
! 323: pcre_extra *extra;
! 324: pcre_jit_stack *jit_stack;
! 325: .sp
! 326: re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
! 327: /* Check for errors */
! 328: extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
! 329: jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
! 330: /* Check for error (NULL) */
! 331: pcre_assign_jit_stack(extra, NULL, jit_stack);
! 332: rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
! 333: /* Check results */
! 334: pcre_free(re);
! 335: pcre_free_study(extra);
! 336: pcre_jit_stack_free(jit_stack);
! 337: .sp
! 338: .
! 339: .
! 340: .SH "SEE ALSO"
! 341: .rs
! 342: .sp
! 343: \fBpcreapi\fP(3)
! 344: .
! 345: .
! 346: .SH AUTHOR
! 347: .rs
! 348: .sp
! 349: .nf
! 350: Philip Hazel (FAQ by Zoltan Herczeg)
! 351: University Computing Service
! 352: Cambridge CB2 3QH, England.
! 353: .fi
! 354: .
! 355: .
! 356: .SH REVISION
! 357: .rs
! 358: .sp
! 359: .nf
! 360: Last updated: 26 November 2011
! 361: Copyright (c) 1997-2011 University of Cambridge.
! 362: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrejit.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc