Annotation of embedaddon/pcre/doc/pcrejit.3, revision 1.1.1.1
1.1 misho 1: .TH PCREJIT 3
2: .SH NAME
3: PCRE - Perl-compatible regular expressions
4: .SH "PCRE JUST-IN-TIME COMPILER SUPPORT"
5: .rs
6: .sp
7: Just-in-time compiling is a heavyweight optimization that can greatly speed up
8: pattern matching. However, it comes at the cost of extra processing before the
9: match is performed. Therefore, it is of most benefit when the same pattern is
10: going to be matched many times. This does not necessarily mean many calls of
11: \fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take
12: place many times at various positions in the subject, even for a single call to
13: \fBpcre_exec()\fP. If the subject string is very long, it may still pay to use
14: JIT for one-off matches.
15: .P
16: JIT support applies only to the traditional matching function,
17: \fBpcre_exec()\fP. It does not apply when \fBpcre_dfa_exec()\fP is being used.
18: The code for this support was written by Zoltan Herczeg.
19: .
20: .
21: .SH "AVAILABILITY OF JIT SUPPORT"
22: .rs
23: .sp
24: JIT support is an optional feature of PCRE. The "configure" option --enable-jit
25: (or equivalent CMake option) must be set when PCRE is built if you want to use
26: JIT. The support is limited to the following hardware platforms:
27: .sp
28: ARM v5, v7, and Thumb2
29: Intel x86 32-bit and 64-bit
30: MIPS 32-bit
31: Power PC 32-bit and 64-bit (experimental)
32: .sp
33: The Power PC support is designated as experimental because it has not been
34: fully tested. If --enable-jit is set on an unsupported platform, compilation
35: fails.
36: .P
37: A program that is linked with PCRE 8.20 or later can tell if JIT support is
38: available by calling \fBpcre_config()\fP with the PCRE_CONFIG_JIT option. The
39: result is 1 when JIT is available, and 0 otherwise. However, a simple program
40: does not need to check this in order to use JIT. The API is implemented in a
41: way that falls back to the ordinary PCRE code if JIT is not available.
42: .P
43: If your program may sometimes be linked with versions of PCRE that are older
44: than 8.20, but you want to use JIT when it is available, you can test
45: the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
46: as PCRE_CONFIG_JIT, for compile-time control of your code.
47: .
48: .
49: .SH "SIMPLE USE OF JIT"
50: .rs
51: .sp
52: You have to do two things to make use of the JIT support in the simplest way:
53: .sp
54: (1) Call \fBpcre_study()\fP with the PCRE_STUDY_JIT_COMPILE option for
55: each compiled pattern, and pass the resulting \fBpcre_extra\fP block to
56: \fBpcre_exec()\fP.
57: .sp
58: (2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is
59: no longer needed instead of just freeing it yourself. This
60: ensures that any JIT data is also freed.
61: .sp
62: For a program that may be linked with pre-8.20 versions of PCRE, you can insert
63: .sp
64: #ifndef PCRE_STUDY_JIT_COMPILE
65: #define PCRE_STUDY_JIT_COMPILE 0
66: #endif
67: .sp
68: so that no option is passed to \fBpcre_study()\fP, and then use something like
69: this to free the study data:
70: .sp
71: #ifdef PCRE_CONFIG_JIT
72: pcre_free_study(study_ptr);
73: #else
74: pcre_free(study_ptr);
75: #endif
76: .sp
77: In some circumstances you may need to call additional functions. These are
78: described in the section entitled
79: .\" HTML <a href="#stackcontrol">
80: .\" </a>
81: "Controlling the JIT stack"
82: .\"
83: below.
84: .P
85: If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
86: data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
87: which turns it into machine code that executes much faster than the normal
88: interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block
89: containing a pointer to JIT code, it obeys that instead of the normal code. The
90: result is identical, but the code runs much faster.
91: .P
92: There are some \fBpcre_exec()\fP options that are not supported for JIT
93: execution. There are also some pattern items that JIT cannot handle. Details
94: are given below. In both cases, execution automatically falls back to the
95: interpretive code.
96: .P
97: If the JIT compiler finds an unsupported item, no JIT data is generated. You
98: can find out if JIT execution is available after studying a pattern by calling
99: \fBpcre_fullinfo()\fP with the PCRE_INFO_JIT option. A result of 1 means that
100: JIT compilation was successful. A result of 0 means that JIT support is not
101: available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
102: JIT compiler was not able to handle the pattern.
103: .P
104: Once a pattern has been studied, with or without JIT, it can be used as many
105: times as you like for matching different subject strings.
106: .
107: .
108: .SH "UNSUPPORTED OPTIONS AND PATTERN ITEMS"
109: .rs
110: .sp
111: The only \fBpcre_exec()\fP options that are supported for JIT execution are
112: PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
113: PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
114: supported.
115: .P
116: The unsupported pattern items are:
117: .sp
118: \eC match a single byte; not supported in UTF-8 mode
119: (?Cn) callouts
120: (*COMMIT) )
121: (*MARK) )
122: (*PRUNE) ) the backtracking control verbs
123: (*SKIP) )
124: (*THEN) )
125: .sp
126: Support for some of these may be added in future.
127: .
128: .
129: .SH "RETURN VALUES FROM JIT EXECUTION"
130: .rs
131: .sp
132: When a pattern is matched using JIT execution, the return values are the same
133: as those given by the interpretive \fBpcre_exec()\fP code, with the addition of
134: one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
135: for the JIT stack was insufficient. See
136: .\" HTML <a href="#stackcontrol">
137: .\" </a>
138: "Controlling the JIT stack"
139: .\"
140: below for a discussion of JIT stack usage. For compatibility with the
141: interpretive \fBpcre_exec()\fP code, no more than two-thirds of the
142: \fIovector\fP argument is used for passing back captured substrings.
143: .P
144: The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
145: very large pattern tree goes on for too long, as it is in the same circumstance
146: when JIT is not used, but the details of exactly what is counted are not the
147: same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
148: execution.
149: .
150: .
151: .SH "SAVING AND RESTORING COMPILED PATTERNS"
152: .rs
153: .sp
154: The code that is generated by the JIT compiler is architecture-specific, and is
155: also position dependent. For those reasons it cannot be saved (in a file or
156: database) and restored later like the bytecode and other data of a compiled
157: pattern. Saving and restoring compiled patterns is not something many people
158: do. More detail about this facility is given in the
159: .\" HREF
160: \fBpcreprecompile\fP
161: .\"
162: documentation. It should be possible to run \fBpcre_study()\fP on a saved and
163: restored pattern, and thereby recreate the JIT data, but because JIT
164: compilation uses significant resources, it is probably not worth doing this;
165: you might as well recompile the original pattern.
166: .
167: .
168: .\" HTML <a name="stackcontrol"></a>
169: .SH "CONTROLLING THE JIT STACK"
170: .rs
171: .sp
172: When the compiled JIT code runs, it needs a block of memory to use as a stack.
173: By default, it uses 32K on the machine stack. However, some large or
174: complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
175: is given when there is not enough stack. Three functions are provided for
176: managing blocks of memory for use as JIT stacks. There is further discussion
177: about the use of JIT stacks in the section entitled
178: .\" HTML <a href="#stackcontrol">
179: .\" </a>
180: "JIT stack FAQ"
181: .\"
182: below.
183: .P
184: The \fBpcre_jit_stack_alloc()\fP function creates a JIT stack. Its arguments
185: are a starting size and a maximum size, and it returns a pointer to an opaque
186: structure of type \fBpcre_jit_stack\fP, or NULL if there is an error. The
187: \fBpcre_jit_stack_free()\fP function can be used to free a stack that is no
188: longer needed. (For the technically minded: the address space is allocated by
189: mmap or VirtualAlloc.)
190: .P
191: JIT uses far less memory for recursion than the interpretive code,
192: and a maximum stack size of 512K to 1M should be more than enough for any
193: pattern.
194: .P
195: The \fBpcre_assign_jit_stack()\fP function specifies which stack JIT code
196: should use. Its arguments are as follows:
197: .sp
198: pcre_extra *extra
199: pcre_jit_callback callback
200: void *data
201: .sp
202: The \fIextra\fP argument must be the result of studying a pattern with
203: PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
204: options:
205: .sp
206: (1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block
207: on the machine stack is used.
208: .sp
209: (2) If \fIcallback\fP is NULL and \fIdata\fP is not NULL, \fIdata\fP must be
210: a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP.
211: .sp
212: (3) If \fIcallback\fP not NULL, it must point to a function that is called
213: with \fIdata\fP as an argument at the start of matching, in order to
214: set up a JIT stack. If the result is NULL, the internal 32K stack
215: is used; otherwise the return value must be a valid JIT stack,
216: the result of calling \fBpcre_jit_stack_alloc()\fP.
217: .sp
218: You may safely assign the same JIT stack to more than one pattern, as long as
219: they are all matched sequentially in the same thread. In a multithread
220: application, each thread must use its own JIT stack.
221: .P
222: Strictly speaking, even more is allowed. You can assign the same stack to any
223: number of patterns as long as they are not used for matching by multiple
224: threads at the same time. For example, you can assign the same stack to all
225: compiled patterns, and use a global mutex in the callback to wait until the
226: stack is available for use. However, this is an inefficient solution, and
227: not recommended.
228: .P
229: This is a suggestion for how a typical multithreaded program might operate:
230: .sp
231: During thread initalization
232: thread_local_var = pcre_jit_stack_alloc(...)
233: .sp
234: During thread exit
235: pcre_jit_stack_free(thread_local_var)
236: .sp
237: Use a one-line callback function
238: return thread_local_var
239: .sp
240: All the functions described in this section do nothing if JIT is not available,
241: and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument
242: is non-NULL and points to a \fBpcre_extra\fP block that is the result of a
243: successful study with PCRE_STUDY_JIT_COMPILE.
244: .
245: .
246: .\" HTML <a name="stackfaq"></a>
247: .SH "JIT STACK FAQ"
248: .rs
249: .sp
250: (1) Why do we need JIT stacks?
251: .sp
252: PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
253: the local data of the current node is pushed before checking its child nodes.
254: Allocating real machine stack on some platforms is difficult. For example, the
255: stack chain needs to be updated every time if we extend the stack on PowerPC.
256: Although it is possible, its updating time overhead decreases performance. So
257: we do the recursion in memory.
258: .P
259: (2) Why don't we simply allocate blocks of memory with \fBmalloc()\fP?
260: .sp
261: Modern operating systems have a nice feature: they can reserve an address space
262: instead of allocating memory. We can safely allocate memory pages inside this
263: address space, so the stack could grow without moving memory data (this is
264: important because of pointers). Thus we can allocate 1M address space, and use
265: only a single memory page (usually 4K) if that is enough. However, we can still
266: grow up to 1M anytime if needed.
267: .P
268: (3) Who "owns" a JIT stack?
269: .sp
270: The owner of the stack is the user program, not the JIT studied pattern or
271: anything else. The user program must ensure that if a stack is used by
272: \fBpcre_exec()\fP, (that is, it is assigned to the pattern currently running),
273: that stack must not be used by any other threads (to avoid overwriting the same
274: memory area). The best practice for multithreaded programs is to allocate a
275: stack for each thread, and return this stack through the JIT callback function.
276: .P
277: (4) When should a JIT stack be freed?
278: .sp
279: You can free a JIT stack at any time, as long as it will not be used by
280: \fBpcre_exec()\fP again. When you assign the stack to a pattern, only a pointer
281: is set. There is no reference counting or any other magic. You can free the
282: patterns and stacks in any order, anytime. Just \fIdo not\fP call
283: \fBpcre_exec()\fP with a pattern pointing to an already freed stack, as that
284: will cause SEGFAULT. (Also, do not free a stack currently used by
285: \fBpcre_exec()\fP in another thread). You can also replace the stack for a
286: pattern at any time. You can even free the previous stack before assigning a
287: replacement.
288: .P
289: (5) Should I allocate/free a stack every time before/after calling
290: \fBpcre_exec()\fP?
291: .sp
292: No, because this is too costly in terms of resources. However, you could
293: implement some clever idea which release the stack if it is not used in let's
294: say two minutes. The JIT callback can help to achive this without keeping a
295: list of the currently JIT studied patterns.
296: .P
297: (6) OK, the stack is for long term memory allocation. But what happens if a
298: pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
299: stack is freed?
300: .sp
301: Especially on embedded sytems, it might be a good idea to release
302: memory sometimes without freeing the stack. There is no API for this at the
303: moment. Probably a function call which returns with the currently allocated
304: memory for any stack and another which allows releasing memory (shrinking the
305: stack) would be a good idea if someone needs this.
306: .P
307: (7) This is too much of a headache. Isn't there any better solution for JIT
308: stack handling?
309: .sp
310: No, thanks to Windows. If POSIX threads were used everywhere, we could throw
311: out this complicated API.
312: .
313: .
314: .SH "EXAMPLE CODE"
315: .rs
316: .sp
317: This is a single-threaded example that specifies a JIT stack without using a
318: callback.
319: .sp
320: int rc;
321: int ovector[30];
322: pcre *re;
323: pcre_extra *extra;
324: pcre_jit_stack *jit_stack;
325: .sp
326: re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
327: /* Check for errors */
328: extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
329: jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
330: /* Check for error (NULL) */
331: pcre_assign_jit_stack(extra, NULL, jit_stack);
332: rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
333: /* Check results */
334: pcre_free(re);
335: pcre_free_study(extra);
336: pcre_jit_stack_free(jit_stack);
337: .sp
338: .
339: .
340: .SH "SEE ALSO"
341: .rs
342: .sp
343: \fBpcreapi\fP(3)
344: .
345: .
346: .SH AUTHOR
347: .rs
348: .sp
349: .nf
350: Philip Hazel (FAQ by Zoltan Herczeg)
351: University Computing Service
352: Cambridge CB2 3QH, England.
353: .fi
354: .
355: .
356: .SH REVISION
357: .rs
358: .sp
359: .nf
360: Last updated: 26 November 2011
361: Copyright (c) 1997-2011 University of Cambridge.
362: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrejit.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc