Annotation of embedaddon/pcre/doc/pcrejit.3, revision 1.1.1.2
1.1 misho 1: .TH PCREJIT 3
2: .SH NAME
3: PCRE - Perl-compatible regular expressions
4: .SH "PCRE JUST-IN-TIME COMPILER SUPPORT"
5: .rs
6: .sp
7: Just-in-time compiling is a heavyweight optimization that can greatly speed up
8: pattern matching. However, it comes at the cost of extra processing before the
9: match is performed. Therefore, it is of most benefit when the same pattern is
1.1.1.2 ! misho 10: going to be matched many times. This does not necessarily mean many calls of a
! 11: matching function; if the pattern is not anchored, matching attempts may take
! 12: place many times at various positions in the subject, even for a single call.
! 13: Therefore, if the subject string is very long, it may still pay to use JIT for
! 14: one-off matches.
! 15: .P
! 16: JIT support applies only to the traditional Perl-compatible matching function.
! 17: It does not apply when the DFA matching function is being used. The code for
! 18: this support was written by Zoltan Herczeg.
! 19: .
! 20: .
! 21: .SH "8-BIT and 16-BIT SUPPORT"
! 22: .rs
! 23: .sp
! 24: JIT support is available for both the 8-bit and 16-bit PCRE libraries. To keep
! 25: this documentation simple, only the 8-bit interface is described in what
! 26: follows. If you are using the 16-bit library, substitute the 16-bit functions
! 27: and 16-bit structures (for example, \fIpcre16_jit_stack\fP instead of
! 28: \fIpcre_jit_stack\fP).
1.1 misho 29: .
30: .
31: .SH "AVAILABILITY OF JIT SUPPORT"
32: .rs
33: .sp
34: JIT support is an optional feature of PCRE. The "configure" option --enable-jit
35: (or equivalent CMake option) must be set when PCRE is built if you want to use
36: JIT. The support is limited to the following hardware platforms:
37: .sp
38: ARM v5, v7, and Thumb2
39: Intel x86 32-bit and 64-bit
40: MIPS 32-bit
1.1.1.2 ! misho 41: Power PC 32-bit and 64-bit
1.1 misho 42: .sp
43: The Power PC support is designated as experimental because it has not been
44: fully tested. If --enable-jit is set on an unsupported platform, compilation
45: fails.
46: .P
47: A program that is linked with PCRE 8.20 or later can tell if JIT support is
48: available by calling \fBpcre_config()\fP with the PCRE_CONFIG_JIT option. The
49: result is 1 when JIT is available, and 0 otherwise. However, a simple program
50: does not need to check this in order to use JIT. The API is implemented in a
51: way that falls back to the ordinary PCRE code if JIT is not available.
52: .P
53: If your program may sometimes be linked with versions of PCRE that are older
54: than 8.20, but you want to use JIT when it is available, you can test
55: the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such
56: as PCRE_CONFIG_JIT, for compile-time control of your code.
57: .
58: .
59: .SH "SIMPLE USE OF JIT"
60: .rs
61: .sp
62: You have to do two things to make use of the JIT support in the simplest way:
63: .sp
64: (1) Call \fBpcre_study()\fP with the PCRE_STUDY_JIT_COMPILE option for
65: each compiled pattern, and pass the resulting \fBpcre_extra\fP block to
66: \fBpcre_exec()\fP.
67: .sp
68: (2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is
69: no longer needed instead of just freeing it yourself. This
70: ensures that any JIT data is also freed.
71: .sp
72: For a program that may be linked with pre-8.20 versions of PCRE, you can insert
73: .sp
74: #ifndef PCRE_STUDY_JIT_COMPILE
75: #define PCRE_STUDY_JIT_COMPILE 0
76: #endif
77: .sp
78: so that no option is passed to \fBpcre_study()\fP, and then use something like
79: this to free the study data:
80: .sp
81: #ifdef PCRE_CONFIG_JIT
82: pcre_free_study(study_ptr);
83: #else
84: pcre_free(study_ptr);
85: #endif
86: .sp
87: In some circumstances you may need to call additional functions. These are
88: described in the section entitled
89: .\" HTML <a href="#stackcontrol">
90: .\" </a>
91: "Controlling the JIT stack"
92: .\"
93: below.
94: .P
95: If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT
96: data is set up. Otherwise, the compiled pattern is passed to the JIT compiler,
97: which turns it into machine code that executes much faster than the normal
98: interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block
99: containing a pointer to JIT code, it obeys that instead of the normal code. The
100: result is identical, but the code runs much faster.
101: .P
102: There are some \fBpcre_exec()\fP options that are not supported for JIT
103: execution. There are also some pattern items that JIT cannot handle. Details
104: are given below. In both cases, execution automatically falls back to the
105: interpretive code.
106: .P
107: If the JIT compiler finds an unsupported item, no JIT data is generated. You
108: can find out if JIT execution is available after studying a pattern by calling
109: \fBpcre_fullinfo()\fP with the PCRE_INFO_JIT option. A result of 1 means that
110: JIT compilation was successful. A result of 0 means that JIT support is not
111: available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the
112: JIT compiler was not able to handle the pattern.
113: .P
114: Once a pattern has been studied, with or without JIT, it can be used as many
115: times as you like for matching different subject strings.
116: .
117: .
118: .SH "UNSUPPORTED OPTIONS AND PATTERN ITEMS"
119: .rs
120: .sp
121: The only \fBpcre_exec()\fP options that are supported for JIT execution are
122: PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and
123: PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not
124: supported.
125: .P
126: The unsupported pattern items are:
127: .sp
128: \eC match a single byte; not supported in UTF-8 mode
129: (?Cn) callouts
130: (*COMMIT) )
131: (*MARK) )
132: (*PRUNE) ) the backtracking control verbs
133: (*SKIP) )
134: (*THEN) )
135: .sp
136: Support for some of these may be added in future.
137: .
138: .
139: .SH "RETURN VALUES FROM JIT EXECUTION"
140: .rs
141: .sp
142: When a pattern is matched using JIT execution, the return values are the same
143: as those given by the interpretive \fBpcre_exec()\fP code, with the addition of
144: one new error code: PCRE_ERROR_JIT_STACKLIMIT. This means that the memory used
145: for the JIT stack was insufficient. See
146: .\" HTML <a href="#stackcontrol">
147: .\" </a>
148: "Controlling the JIT stack"
149: .\"
150: below for a discussion of JIT stack usage. For compatibility with the
151: interpretive \fBpcre_exec()\fP code, no more than two-thirds of the
152: \fIovector\fP argument is used for passing back captured substrings.
153: .P
154: The error code PCRE_ERROR_MATCHLIMIT is returned by the JIT code if searching a
155: very large pattern tree goes on for too long, as it is in the same circumstance
156: when JIT is not used, but the details of exactly what is counted are not the
157: same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT
158: execution.
159: .
160: .
161: .SH "SAVING AND RESTORING COMPILED PATTERNS"
162: .rs
163: .sp
164: The code that is generated by the JIT compiler is architecture-specific, and is
165: also position dependent. For those reasons it cannot be saved (in a file or
166: database) and restored later like the bytecode and other data of a compiled
167: pattern. Saving and restoring compiled patterns is not something many people
168: do. More detail about this facility is given in the
169: .\" HREF
170: \fBpcreprecompile\fP
171: .\"
172: documentation. It should be possible to run \fBpcre_study()\fP on a saved and
173: restored pattern, and thereby recreate the JIT data, but because JIT
174: compilation uses significant resources, it is probably not worth doing this;
175: you might as well recompile the original pattern.
176: .
177: .
178: .\" HTML <a name="stackcontrol"></a>
179: .SH "CONTROLLING THE JIT STACK"
180: .rs
181: .sp
182: When the compiled JIT code runs, it needs a block of memory to use as a stack.
183: By default, it uses 32K on the machine stack. However, some large or
184: complicated patterns need more than this. The error PCRE_ERROR_JIT_STACKLIMIT
185: is given when there is not enough stack. Three functions are provided for
186: managing blocks of memory for use as JIT stacks. There is further discussion
187: about the use of JIT stacks in the section entitled
188: .\" HTML <a href="#stackcontrol">
189: .\" </a>
190: "JIT stack FAQ"
191: .\"
192: below.
193: .P
194: The \fBpcre_jit_stack_alloc()\fP function creates a JIT stack. Its arguments
195: are a starting size and a maximum size, and it returns a pointer to an opaque
196: structure of type \fBpcre_jit_stack\fP, or NULL if there is an error. The
197: \fBpcre_jit_stack_free()\fP function can be used to free a stack that is no
198: longer needed. (For the technically minded: the address space is allocated by
199: mmap or VirtualAlloc.)
200: .P
201: JIT uses far less memory for recursion than the interpretive code,
202: and a maximum stack size of 512K to 1M should be more than enough for any
203: pattern.
204: .P
205: The \fBpcre_assign_jit_stack()\fP function specifies which stack JIT code
206: should use. Its arguments are as follows:
207: .sp
208: pcre_extra *extra
209: pcre_jit_callback callback
210: void *data
211: .sp
212: The \fIextra\fP argument must be the result of studying a pattern with
213: PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two
214: options:
215: .sp
216: (1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block
217: on the machine stack is used.
218: .sp
219: (2) If \fIcallback\fP is NULL and \fIdata\fP is not NULL, \fIdata\fP must be
220: a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP.
221: .sp
222: (3) If \fIcallback\fP not NULL, it must point to a function that is called
223: with \fIdata\fP as an argument at the start of matching, in order to
224: set up a JIT stack. If the result is NULL, the internal 32K stack
225: is used; otherwise the return value must be a valid JIT stack,
226: the result of calling \fBpcre_jit_stack_alloc()\fP.
227: .sp
228: You may safely assign the same JIT stack to more than one pattern, as long as
229: they are all matched sequentially in the same thread. In a multithread
230: application, each thread must use its own JIT stack.
231: .P
232: Strictly speaking, even more is allowed. You can assign the same stack to any
233: number of patterns as long as they are not used for matching by multiple
234: threads at the same time. For example, you can assign the same stack to all
235: compiled patterns, and use a global mutex in the callback to wait until the
236: stack is available for use. However, this is an inefficient solution, and
237: not recommended.
238: .P
239: This is a suggestion for how a typical multithreaded program might operate:
240: .sp
241: During thread initalization
242: thread_local_var = pcre_jit_stack_alloc(...)
243: .sp
244: During thread exit
245: pcre_jit_stack_free(thread_local_var)
246: .sp
247: Use a one-line callback function
248: return thread_local_var
249: .sp
250: All the functions described in this section do nothing if JIT is not available,
251: and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument
252: is non-NULL and points to a \fBpcre_extra\fP block that is the result of a
253: successful study with PCRE_STUDY_JIT_COMPILE.
254: .
255: .
256: .\" HTML <a name="stackfaq"></a>
257: .SH "JIT STACK FAQ"
258: .rs
259: .sp
260: (1) Why do we need JIT stacks?
261: .sp
262: PCRE (and JIT) is a recursive, depth-first engine, so it needs a stack where
263: the local data of the current node is pushed before checking its child nodes.
264: Allocating real machine stack on some platforms is difficult. For example, the
265: stack chain needs to be updated every time if we extend the stack on PowerPC.
266: Although it is possible, its updating time overhead decreases performance. So
267: we do the recursion in memory.
268: .P
269: (2) Why don't we simply allocate blocks of memory with \fBmalloc()\fP?
270: .sp
271: Modern operating systems have a nice feature: they can reserve an address space
272: instead of allocating memory. We can safely allocate memory pages inside this
273: address space, so the stack could grow without moving memory data (this is
274: important because of pointers). Thus we can allocate 1M address space, and use
275: only a single memory page (usually 4K) if that is enough. However, we can still
276: grow up to 1M anytime if needed.
277: .P
278: (3) Who "owns" a JIT stack?
279: .sp
280: The owner of the stack is the user program, not the JIT studied pattern or
281: anything else. The user program must ensure that if a stack is used by
282: \fBpcre_exec()\fP, (that is, it is assigned to the pattern currently running),
283: that stack must not be used by any other threads (to avoid overwriting the same
284: memory area). The best practice for multithreaded programs is to allocate a
285: stack for each thread, and return this stack through the JIT callback function.
286: .P
287: (4) When should a JIT stack be freed?
288: .sp
289: You can free a JIT stack at any time, as long as it will not be used by
290: \fBpcre_exec()\fP again. When you assign the stack to a pattern, only a pointer
291: is set. There is no reference counting or any other magic. You can free the
292: patterns and stacks in any order, anytime. Just \fIdo not\fP call
293: \fBpcre_exec()\fP with a pattern pointing to an already freed stack, as that
294: will cause SEGFAULT. (Also, do not free a stack currently used by
295: \fBpcre_exec()\fP in another thread). You can also replace the stack for a
296: pattern at any time. You can even free the previous stack before assigning a
297: replacement.
298: .P
299: (5) Should I allocate/free a stack every time before/after calling
300: \fBpcre_exec()\fP?
301: .sp
302: No, because this is too costly in terms of resources. However, you could
303: implement some clever idea which release the stack if it is not used in let's
304: say two minutes. The JIT callback can help to achive this without keeping a
305: list of the currently JIT studied patterns.
306: .P
307: (6) OK, the stack is for long term memory allocation. But what happens if a
308: pattern causes stack overflow with a stack of 1M? Is that 1M kept until the
309: stack is freed?
310: .sp
311: Especially on embedded sytems, it might be a good idea to release
312: memory sometimes without freeing the stack. There is no API for this at the
313: moment. Probably a function call which returns with the currently allocated
314: memory for any stack and another which allows releasing memory (shrinking the
315: stack) would be a good idea if someone needs this.
316: .P
317: (7) This is too much of a headache. Isn't there any better solution for JIT
318: stack handling?
319: .sp
320: No, thanks to Windows. If POSIX threads were used everywhere, we could throw
321: out this complicated API.
322: .
323: .
324: .SH "EXAMPLE CODE"
325: .rs
326: .sp
327: This is a single-threaded example that specifies a JIT stack without using a
328: callback.
329: .sp
330: int rc;
331: int ovector[30];
332: pcre *re;
333: pcre_extra *extra;
334: pcre_jit_stack *jit_stack;
335: .sp
336: re = pcre_compile(pattern, 0, &error, &erroffset, NULL);
337: /* Check for errors */
338: extra = pcre_study(re, PCRE_STUDY_JIT_COMPILE, &error);
339: jit_stack = pcre_jit_stack_alloc(32*1024, 512*1024);
340: /* Check for error (NULL) */
341: pcre_assign_jit_stack(extra, NULL, jit_stack);
342: rc = pcre_exec(re, extra, subject, length, 0, 0, ovector, 30);
343: /* Check results */
344: pcre_free(re);
345: pcre_free_study(extra);
346: pcre_jit_stack_free(jit_stack);
347: .sp
348: .
349: .
350: .SH "SEE ALSO"
351: .rs
352: .sp
353: \fBpcreapi\fP(3)
354: .
355: .
356: .SH AUTHOR
357: .rs
358: .sp
359: .nf
360: Philip Hazel (FAQ by Zoltan Herczeg)
361: University Computing Service
362: Cambridge CB2 3QH, England.
363: .fi
364: .
365: .
366: .SH REVISION
367: .rs
368: .sp
369: .nf
1.1.1.2 ! misho 370: Last updated: 08 January 2012
! 371: Copyright (c) 1997-2012 University of Cambridge.
1.1 misho 372: .fi
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to pcrejit.3 CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre / doc