Annotation of embedaddon/pcre/doc/html/pcrejit.html, revision 1.1.1.3
1.1 misho 1: <html>
2: <head>
3: <title>pcrejit specification</title>
4: </head>
5: <body bgcolor="#FFFFFF" text="#00005A" link="#0066FF" alink="#3399FF" vlink="#2222BB">
6: <h1>pcrejit man page</h1>
7: <p>
8: Return to the <a href="index.html">PCRE index page</a>.
9: </p>
10: <p>
11: This page is part of the PCRE HTML documentation. It was generated automatically
12: from the original man page. If there is any nonsense in it, please consult the
13: man page, in case the conversion went wrong.
14: <br>
15: <ul>
16: <li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a>
1.1.1.2 misho 17: <li><a name="TOC2" href="#SEC2">8-BIT and 16-BIT SUPPORT</a>
18: <li><a name="TOC3" href="#SEC3">AVAILABILITY OF JIT SUPPORT</a>
19: <li><a name="TOC4" href="#SEC4">SIMPLE USE OF JIT</a>
20: <li><a name="TOC5" href="#SEC5">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a>
21: <li><a name="TOC6" href="#SEC6">RETURN VALUES FROM JIT EXECUTION</a>
22: <li><a name="TOC7" href="#SEC7">SAVING AND RESTORING COMPILED PATTERNS</a>
23: <li><a name="TOC8" href="#SEC8">CONTROLLING THE JIT STACK</a>
24: <li><a name="TOC9" href="#SEC9">JIT STACK FAQ</a>
25: <li><a name="TOC10" href="#SEC10">EXAMPLE CODE</a>
26: <li><a name="TOC11" href="#SEC11">SEE ALSO</a>
27: <li><a name="TOC12" href="#SEC12">AUTHOR</a>
28: <li><a name="TOC13" href="#SEC13">REVISION</a>
1.1 misho 29: </ul>
30: <br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br>
31: <P>
32: Just-in-time compiling is a heavyweight optimization that can greatly speed up
33: pattern matching. However, it comes at the cost of extra processing before the
34: match is performed. Therefore, it is of most benefit when the same pattern is
1.1.1.2 misho 35: going to be matched many times. This does not necessarily mean many calls of a
36: matching function; if the pattern is not anchored, matching attempts may take
37: place many times at various positions in the subject, even for a single call.
38: Therefore, if the subject string is very long, it may still pay to use JIT for
39: one-off matches.
40: </P>
41: <P>
42: JIT support applies only to the traditional Perl-compatible matching function.
43: It does not apply when the DFA matching function is being used. The code for
44: this support was written by Zoltan Herczeg.
45: </P>
46: <br><a name="SEC2" href="#TOC1">8-BIT and 16-BIT SUPPORT</a><br>
47: <P>
48: JIT support is available for both the 8-bit and 16-bit PCRE libraries. To keep
49: this documentation simple, only the 8-bit interface is described in what
50: follows. If you are using the 16-bit library, substitute the 16-bit functions
51: and 16-bit structures (for example, <i>pcre16_jit_stack</i> instead of
52: <i>pcre_jit_stack</i>).
1.1 misho 53: </P>
1.1.1.2 misho 54: <br><a name="SEC3" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br>
1.1 misho 55: <P>
56: JIT support is an optional feature of PCRE. The "configure" option --enable-jit
57: (or equivalent CMake option) must be set when PCRE is built if you want to use
58: JIT. The support is limited to the following hardware platforms:
59: <pre>
60: ARM v5, v7, and Thumb2
61: Intel x86 32-bit and 64-bit
62: MIPS 32-bit
1.1.1.2 misho 63: Power PC 32-bit and 64-bit
1.1 misho 64: </pre>
1.1.1.3 ! misho 65: If --enable-jit is set on an unsupported platform, compilation fails.
1.1 misho 66: </P>
67: <P>
68: A program that is linked with PCRE 8.20 or later can tell if JIT support is
69: available by calling <b>pcre_config()</b> with the PCRE_CONFIG_JIT option. The
70: result is 1 when JIT is available, and 0 otherwise. However, a simple program
71: does not need to check this in order to use JIT. The API is implemented in a
1.1.1.3 ! misho 72: way that falls back to the interpretive code if JIT is not available.
1.1 misho 73: </P>
74: <P>
75: If your program may sometimes be linked with versions of PCRE that are older
76: than 8.20, but you want to use JIT when it is available, you can test
77: the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
78: as PCRE_CONFIG_JIT, for compile-time control of your code.
79: </P>
1.1.1.2 misho 80: <br><a name="SEC4" href="#TOC1">SIMPLE USE OF JIT</a><br>
1.1 misho 81: <P>
82: You have to do two things to make use of the JIT support in the simplest way:
83: <pre>
84: (1) Call <b>pcre_study()</b> with the PCRE_STUDY_JIT_COMPILE option for
85: each compiled pattern, and pass the resulting <b>pcre_extra</b> block to
86: <b>pcre_exec()</b>.
87:
88: (2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is
1.1.1.3 ! misho 89: no longer needed, instead of just freeing it yourself. This
1.1 misho 90: ensures that any JIT data is also freed.
91: </pre>
92: For a program that may be linked with pre-8.20 versions of PCRE, you can insert
93: <pre>
94: #ifndef PCRE_STUDY_JIT_COMPILE
95: #define PCRE_STUDY_JIT_COMPILE 0
96: #endif
97: </pre>
98: so that no option is passed to <b>pcre_study()</b>, and then use something like
99: this to free the study data:
100: <pre>
101: #ifdef PCRE_CONFIG_JIT
102: pcre_free_study(study_ptr);
103: #else
104: pcre_free(study_ptr);
105: #endif
106: </pre>
1.1.1.3 ! misho 107: PCRE_STUDY_JIT_COMPILE requests the JIT compiler to generate code for complete
! 108: matches. If you want to run partial matches using the PCRE_PARTIAL_HARD or
! 109: PCRE_PARTIAL_SOFT options of <b>pcre_exec()</b>, you should set one or both of
! 110: the following options in addition to, or instead of, PCRE_STUDY_JIT_COMPILE
! 111: when you call <b>pcre_study()</b>:
! 112: <pre>
! 113: PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE
! 114: PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE
! 115: </pre>
! 116: The JIT compiler generates different optimized code for each of the three
! 117: modes (normal, soft partial, hard partial). When <b>pcre_exec()</b> is called,
! 118: the appropriate code is run if it is available. Otherwise, the pattern is
! 119: matched using interpretive code.
! 120: </P>
! 121: <P>
1.1 misho 122: In some circumstances you may need to call additional functions. These are
123: described in the section entitled
124: <a href="#stackcontrol">"Controlling the JIT stack"</a>
125: below.
126: </P>
127: <P>
1.1.1.3 ! misho 128: If JIT support is not available, PCRE_STUDY_JIT_COMPILE etc. are ignored, and
! 129: no JIT data is created. Otherwise, the compiled pattern is passed to the JIT
! 130: compiler, which turns it into machine code that executes much faster than the
! 131: normal interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b>
! 132: block containing a pointer to JIT code of the appropriate mode (normal or
! 133: hard/soft partial), it obeys that code instead of running the interpreter. The
! 134: result is identical, but the compiled JIT code runs much faster.
1.1 misho 135: </P>
136: <P>
137: There are some <b>pcre_exec()</b> options that are not supported for JIT
138: execution. There are also some pattern items that JIT cannot handle. Details
139: are given below. In both cases, execution automatically falls back to the
1.1.1.3 ! misho 140: interpretive code. If you want to know whether JIT was actually used for a
! 141: particular match, you should arrange for a JIT callback function to be set up
! 142: as described in the section entitled
! 143: <a href="#stackcontrol">"Controlling the JIT stack"</a>
! 144: below, even if you do not need to supply a non-default JIT stack. Such a
! 145: callback function is called whenever JIT code is about to be obeyed. If the
! 146: execution options are not right for JIT execution, the callback function is not
! 147: obeyed.
1.1 misho 148: </P>
149: <P>
150: If the JIT compiler finds an unsupported item, no JIT data is generated. You
151: can find out if JIT execution is available after studying a pattern by calling
152: <b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that
153: JIT compilation was successful. A result of 0 means that JIT support is not
1.1.1.3 ! misho 154: available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE etc., or
! 155: the JIT compiler was not able to handle the pattern.
1.1 misho 156: </P>
157: <P>
158: Once a pattern has been studied, with or without JIT, it can be used as many
159: times as you like for matching different subject strings.
160: </P>
1.1.1.2 misho 161: <br><a name="SEC5" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br>
1.1 misho 162: <P>
163: The only <b>pcre_exec()</b> options that are supported for JIT execution are
1.1.1.3 ! misho 164: PCRE_NO_UTF8_CHECK, PCRE_NO_UTF16_CHECK, PCRE_NOTBOL, PCRE_NOTEOL,
! 165: PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT.
1.1 misho 166: </P>
167: <P>
168: The unsupported pattern items are:
169: <pre>
170: \C match a single byte; not supported in UTF-8 mode
171: (?Cn) callouts
1.1.1.3 ! misho 172: (*PRUNE) )
! 173: (*SKIP) ) backtracking control verbs
1.1 misho 174: (*THEN) )
175: </pre>
176: Support for some of these may be added in future.
177: </P>
1.1.1.2 misho 178: <br><a name="SEC6" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br>
1.1 misho 179: <P>
180: When a pattern is matched using JIT execution, the return values are the same
181: as those given by the interpretive <b>pcre_exec()</b> code, with the addition of
182: one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
183: for the JIT stack was insufficient. See
184: <a href="#stackcontrol">"Controlling the JIT stack"</a>
185: below for a discussion of JIT stack usage. For compatibility with the
186: interpretive <b>pcre_exec()</b> code, no more than two-thirds of the
187: <i>ovector</i> argument is used for passing back captured substrings.
188: </P>
189: <P>
190: The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
191: very large pattern tree goes on for too long, as it is in the same circumstance
192: when JIT is not used, but the details of exactly what is counted are not the
193: same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
194: execution.
195: </P>
1.1.1.2 misho 196: <br><a name="SEC7" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br>
1.1 misho 197: <P>
198: The code that is generated by the JIT compiler is architecture-specific, and is
199: also position dependent. For those reasons it cannot be saved (in a file or
200: database) and restored later like the bytecode and other data of a compiled
201: pattern. Saving and restoring compiled patterns is not something many people
202: do. More detail about this facility is given in the
203: <a href="pcreprecompile.html"><b>pcreprecompile</b></a>
204: documentation. It should be possible to run <b>pcre_study()</b> on a saved and
205: restored pattern, and thereby recreate the JIT data, but because JIT
206: compilation uses significant resources, it is probably not worth doing this;
207: you might as well recompile the original pattern.
208: <a name="stackcontrol"></a></P>
1.1.1.2 misho 209: <br><a name="SEC8" href="#TOC1">CONTROLLING THE JIT STACK</a><br>
1.1 misho 210: <P>
211: When the compiled JIT code runs, it needs a block of memory to use as a stack.
212: By default, it uses 32K on the machine stack. However, some large or
213: complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
214: is given when there is not enough stack. Three functions are provided for
215: managing blocks of memory for use as JIT stacks. There is further discussion
216: about the use of JIT stacks in the section entitled
217: <a href="#stackcontrol">"JIT stack FAQ"</a>
218: below.
219: </P>
220: <P>
221: The <b>pcre_jit_stack_alloc()</b> function creates a JIT stack. Its arguments
222: are a starting size and a maximum size, and it returns a pointer to an opaque
223: structure of type <b>pcre_jit_stack</b>, or NULL if there is an error. The
224: <b>pcre_jit_stack_free()</b> function can be used to free a stack that is no
225: longer needed. (For the technically minded: the address space is allocated by
226: mmap or VirtualAlloc.)
227: </P>
228: <P>
229: JIT uses far less memory for recursion than the interpretive code,
230: and a maximum stack size of 512K to 1M should be more than enough for any
231: pattern.
232: </P>
233: <P>
234: The <b>pcre_assign_jit_stack()</b> function specifies which stack JIT code
235: should use. Its arguments are as follows:
236: <pre>
237: pcre_extra *extra
238: pcre_jit_callback callback
239: void *data
240: </pre>
241: The <i>extra</i> argument must be the result of studying a pattern with
1.1.1.3 ! misho 242: PCRE_STUDY_JIT_COMPILE etc. There are three cases for the values of the other
! 243: two options:
1.1 misho 244: <pre>
245: (1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block
246: on the machine stack is used.
247:
248: (2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be
249: a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>.
250:
1.1.1.3 ! misho 251: (3) If <i>callback</i> is not NULL, it must point to a function that is
! 252: called with <i>data</i> as an argument at the start of matching, in
! 253: order to set up a JIT stack. If the return from the callback
! 254: function is NULL, the internal 32K stack is used; otherwise the
! 255: return value must be a valid JIT stack, the result of calling
! 256: <b>pcre_jit_stack_alloc()</b>.
1.1 misho 257: </pre>
1.1.1.3 ! misho 258: A callback function is obeyed whenever JIT code is about to be run; it is not
! 259: obeyed when <b>pcre_exec()</b> is called with options that are incompatible for
! 260: JIT execution. A callback function can therefore be used to determine whether a
! 261: match operation was executed by JIT or by the interpreter.
! 262: </P>
! 263: <P>
! 264: You may safely use the same JIT stack for more than one pattern (either by
! 265: assigning directly or by callback), as long as the patterns are all matched
! 266: sequentially in the same thread. In a multithread application, if you do not
! 267: specify a JIT stack, or if you assign or pass back NULL from a callback, that
! 268: is thread-safe, because each thread has its own machine stack. However, if you
! 269: assign or pass back a non-NULL JIT stack, this must be a different stack for
! 270: each thread so that the application is thread-safe.
1.1 misho 271: </P>
272: <P>
1.1.1.3 ! misho 273: Strictly speaking, even more is allowed. You can assign the same non-NULL stack
! 274: to any number of patterns as long as they are not used for matching by multiple
1.1 misho 275: threads at the same time. For example, you can assign the same stack to all
276: compiled patterns, and use a global mutex in the callback to wait until the
1.1.1.3 ! misho 277: stack is available for use. However, this is an inefficient solution, and not
! 278: recommended.
1.1 misho 279: </P>
280: <P>
1.1.1.3 ! misho 281: This is a suggestion for how a multithreaded program that needs to set up
! 282: non-default JIT stacks might operate:
1.1 misho 283: <pre>
284: During thread initalization
285: thread_local_var = pcre_jit_stack_alloc(...)
286:
287: During thread exit
288: pcre_jit_stack_free(thread_local_var)
289:
290: Use a one-line callback function
291: return thread_local_var
292: </pre>
293: All the functions described in this section do nothing if JIT is not available,
294: and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument
295: is non-NULL and points to a <b>pcre_extra</b> block that is the result of a
1.1.1.3 ! misho 296: successful study with PCRE_STUDY_JIT_COMPILE etc.
1.1 misho 297: <a name="stackfaq"></a></P>
1.1.1.2 misho 298: <br><a name="SEC9" href="#TOC1">JIT STACK FAQ</a><br>
1.1 misho 299: <P>
300: (1) Why do we need JIT stacks?
301: <br>
302: <br>
303: PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
304: the local data of the current node is pushed before checking its child nodes.
305: Allocating real machine stack on some platforms is difficult. For example, the
306: stack chain needs to be updated every time if we extend the stack on PowerPC.
307: Although it is possible, its updating time overhead decreases performance. So
308: we do the recursion in memory.
309: </P>
310: <P>
311: (2) Why don't we simply allocate blocks of memory with <b>malloc()</b>?
312: <br>
313: <br>
314: Modern operating systems have a nice feature: they can reserve an address space
315: instead of allocating memory. We can safely allocate memory pages inside this
316: address space, so the stack could grow without moving memory data (this is
317: important because of pointers). Thus we can allocate 1M address space, and use
318: only a single memory page (usually 4K) if that is enough. However, we can still
319: grow up to 1M anytime if needed.
320: </P>
321: <P>
322: (3) Who "owns" a JIT stack?
323: <br>
324: <br>
325: The owner of the stack is the user program, not the JIT studied pattern or
326: anything else. The user program must ensure that if a stack is used by
327: <b>pcre_exec()</b>, (that is, it is assigned to the pattern currently running),
328: that stack must not be used by any other threads (to avoid overwriting the same
329: memory area). The best practice for multithreaded programs is to allocate a
330: stack for each thread, and return this stack through the JIT callback function.
331: </P>
332: <P>
333: (4) When should a JIT stack be freed?
334: <br>
335: <br>
336: You can free a JIT stack at any time, as long as it will not be used by
337: <b>pcre_exec()</b> again. When you assign the stack to a pattern, only a pointer
338: is set. There is no reference counting or any other magic. You can free the
339: patterns and stacks in any order, anytime. Just <i>do not</i> call
340: <b>pcre_exec()</b> with a pattern pointing to an already freed stack, as that
341: will cause SEGFAULT. (Also, do not free a stack currently used by
342: <b>pcre_exec()</b> in another thread). You can also replace the stack for a
343: pattern at any time. You can even free the previous stack before assigning a
344: replacement.
345: </P>
346: <P>
347: (5) Should I allocate/free a stack every time before/after calling
348: <b>pcre_exec()</b>?
349: <br>
350: <br>
351: No, because this is too costly in terms of resources. However, you could
352: implement some clever idea which release the stack if it is not used in let's
353: say two minutes. The JIT callback can help to achive this without keeping a
354: list of the currently JIT studied patterns.
355: </P>
356: <P>
357: (6) OK, the stack is for long term memory allocation. But what happens if a
358: pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
359: stack is freed?
360: <br>
361: <br>
1.1.1.3 ! misho 362: Especially on embedded sytems, it might be a good idea to release memory
! 363: sometimes without freeing the stack. There is no API for this at the moment.
! 364: Probably a function call which returns with the currently allocated memory for
! 365: any stack and another which allows releasing memory (shrinking the stack) would
! 366: be a good idea if someone needs this.
1.1 misho 367: </P>
368: <P>
369: (7) This is too much of a headache. Isn't there any better solution for JIT
370: stack handling?
371: <br>
372: <br>
373: No, thanks to Windows. If POSIX threads were used everywhere, we could throw
374: out this complicated API.
375: </P>
1.1.1.2 misho 376: <br><a name="SEC10" href="#TOC1">EXAMPLE CODE</a><br>
1.1 misho 377: <P>
378: This is a single-threaded example that specifies a JIT stack without using a
379: callback.
380: <pre>
381: int rc;
382: int ovector[30];
383: pcre *re;
384: pcre_extra *extra;
385: pcre_jit_stack *jit_stack;
386:
387: re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
388: /* Check for errors */
389: extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
390: jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
391: /* Check for error (NULL) */
392: pcre_assign_jit_stack(extra, NULL, jit_stack);
393: rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
394: /* Check results */
395: pcre_free(re);
396: pcre_free_study(extra);
397: pcre_jit_stack_free(jit_stack);
398:
399: </PRE>
400: </P>
1.1.1.2 misho 401: <br><a name="SEC11" href="#TOC1">SEE ALSO</a><br>
1.1 misho 402: <P>
403: <b>pcreapi</b>(3)
404: </P>
1.1.1.2 misho 405: <br><a name="SEC12" href="#TOC1">AUTHOR</a><br>
1.1 misho 406: <P>
407: Philip Hazel (FAQ by Zoltan Herczeg)
408: <br>
409: University Computing Service
410: <br>
411: Cambridge CB2 3QH, England.
412: <br>
413: </P>
1.1.1.2 misho 414: <br><a name="SEC13" href="#TOC1">REVISION</a><br>
1.1 misho 415: <P>
1.1.1.3 ! misho 416: Last updated: 04 May 2012
1.1 misho 417: <br>
1.1.1.2 misho 418: Copyright © 1997-2012 University of Cambridge.
1.1 misho 419: <br>
420: <p>
421: Return to the <a href="index.html">PCRE index page</a>.
422: </p>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrejit.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc / html