version 1.1, 2012/02/21 23:05:52
|
version 1.1.1.3, 2012/10/09 09:19:18
|
Line 14 man page, in case the conversion went wrong.
|
Line 14 man page, in case the conversion went wrong.
|
<br> |
<br> |
<ul> |
<ul> |
<li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a> |
<li><a name="TOC1" href="#SEC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a> |
<li><a name="TOC2" href="#SEC2">AVAILABILITY OF JIT SUPPORT</a> | <li><a name="TOC2" href="#SEC2">8-BIT and 16-BIT SUPPORT</a> |
<li><a name="TOC3" href="#SEC3">SIMPLE USE OF JIT</a> | <li><a name="TOC3" href="#SEC3">AVAILABILITY OF JIT SUPPORT</a> |
<li><a name="TOC4" href="#SEC4">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a> | <li><a name="TOC4" href="#SEC4">SIMPLE USE OF JIT</a> |
<li><a name="TOC5" href="#SEC5">RETURN VALUES FROM JIT EXECUTION</a> | <li><a name="TOC5" href="#SEC5">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a> |
<li><a name="TOC6" href="#SEC6">SAVING AND RESTORING COMPILED PATTERNS</a> | <li><a name="TOC6" href="#SEC6">RETURN VALUES FROM JIT EXECUTION</a> |
<li><a name="TOC7" href="#SEC7">CONTROLLING THE JIT STACK</a> | <li><a name="TOC7" href="#SEC7">SAVING AND RESTORING COMPILED PATTERNS</a> |
<li><a name="TOC8" href="#SEC8">JIT STACK FAQ</a> | <li><a name="TOC8" href="#SEC8">CONTROLLING THE JIT STACK</a> |
<li><a name="TOC9" href="#SEC9">EXAMPLE CODE</a> | <li><a name="TOC9" href="#SEC9">JIT STACK FAQ</a> |
<li><a name="TOC10" href="#SEC10">SEE ALSO</a> | <li><a name="TOC10" href="#SEC10">EXAMPLE CODE</a> |
<li><a name="TOC11" href="#SEC11">AUTHOR</a> | <li><a name="TOC11" href="#SEC11">SEE ALSO</a> |
<li><a name="TOC12" href="#SEC12">REVISION</a> | <li><a name="TOC12" href="#SEC12">AUTHOR</a> |
| <li><a name="TOC13" href="#SEC13">REVISION</a> |
</ul> |
</ul> |
<br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br> |
<br><a name="SEC1" href="#TOC1">PCRE JUST-IN-TIME COMPILER SUPPORT</a><br> |
<P> |
<P> |
Just-in-time compiling is a heavyweight optimization that can greatly speed up |
Just-in-time compiling is a heavyweight optimization that can greatly speed up |
pattern matching. However, it comes at the cost of extra processing before the |
pattern matching. However, it comes at the cost of extra processing before the |
match is performed. Therefore, it is of most benefit when the same pattern is |
match is performed. Therefore, it is of most benefit when the same pattern is |
going to be matched many times. This does not necessarily mean many calls of | going to be matched many times. This does not necessarily mean many calls of a |
\fPpcre_exec()\fP; if the pattern is not anchored, matching attempts may take | matching function; if the pattern is not anchored, matching attempts may take |
place many times at various positions in the subject, even for a single call to | place many times at various positions in the subject, even for a single call. |
<b>pcre_exec()</b>. If the subject string is very long, it may still pay to use | Therefore, if the subject string is very long, it may still pay to use JIT for |
JIT for one-off matches. | one-off matches. |
</P> |
</P> |
<P> |
<P> |
JIT support applies only to the traditional matching function, | JIT support applies only to the traditional Perl-compatible matching function. |
<b>pcre_exec()</b>. It does not apply when <b>pcre_dfa_exec()</b> is being used. | It does not apply when the DFA matching function is being used. The code for |
The code for this support was written by Zoltan Herczeg. | this support was written by Zoltan Herczeg. |
</P> |
</P> |
<br><a name="SEC2" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br> | <br><a name="SEC2" href="#TOC1">8-BIT and 16-BIT SUPPORT</a><br> |
<P> |
<P> |
|
JIT support is available for both the 8-bit and 16-bit PCRE libraries. To keep |
|
this documentation simple, only the 8-bit interface is described in what |
|
follows. If you are using the 16-bit library, substitute the 16-bit functions |
|
and 16-bit structures (for example, <i>pcre16_jit_stack</i> instead of |
|
<i>pcre_jit_stack</i>). |
|
</P> |
|
<br><a name="SEC3" href="#TOC1">AVAILABILITY OF JIT SUPPORT</a><br> |
|
<P> |
JIT support is an optional feature of PCRE. The "configure" option --enable-jit |
JIT support is an optional feature of PCRE. The "configure" option --enable-jit |
(or equivalent CMake option) must be set when PCRE is built if you want to use |
(or equivalent CMake option) must be set when PCRE is built if you want to use |
JIT. The support is limited to the following hardware platforms: |
JIT. The support is limited to the following hardware platforms: |
Line 51 JIT. The support is limited to the following hardware
|
Line 60 JIT. The support is limited to the following hardware
|
ARM v5, v7, and Thumb2 |
ARM v5, v7, and Thumb2 |
Intel x86 32-bit and 64-bit |
Intel x86 32-bit and 64-bit |
MIPS 32-bit |
MIPS 32-bit |
Power PC 32-bit and 64-bit (experimental) | Power PC 32-bit and 64-bit |
</pre> |
</pre> |
The Power PC support is designated as experimental because it has not been | If --enable-jit is set on an unsupported platform, compilation fails. |
fully tested. If --enable-jit is set on an unsupported platform, compilation | |
fails. | |
</P> |
</P> |
<P> |
<P> |
A program that is linked with PCRE 8.20 or later can tell if JIT support is |
A program that is linked with PCRE 8.20 or later can tell if JIT support is |
available by calling <b>pcre_config()</b> with the PCRE_CONFIG_JIT option. The |
available by calling <b>pcre_config()</b> with the PCRE_CONFIG_JIT option. The |
result is 1 when JIT is available, and 0 otherwise. However, a simple program |
result is 1 when JIT is available, and 0 otherwise. However, a simple program |
does not need to check this in order to use JIT. The API is implemented in a |
does not need to check this in order to use JIT. The API is implemented in a |
way that falls back to the ordinary PCRE code if JIT is not available. | way that falls back to the interpretive code if JIT is not available. |
</P> |
</P> |
<P> |
<P> |
If your program may sometimes be linked with versions of PCRE that are older |
If your program may sometimes be linked with versions of PCRE that are older |
Line 70 than 8.20, but you want to use JIT when it is availabl
|
Line 77 than 8.20, but you want to use JIT when it is availabl
|
the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such |
the values of PCRE_MAJOR and PCRE_MINOR, or the existence of a JIT macro such |
as PCRE_CONFIG_JIT, for compile-time control of your code. |
as PCRE_CONFIG_JIT, for compile-time control of your code. |
</P> |
</P> |
<br><a name="SEC3" href="#TOC1">SIMPLE USE OF JIT</a><br> | <br><a name="SEC4" href="#TOC1">SIMPLE USE OF JIT</a><br> |
<P> |
<P> |
You have to do two things to make use of the JIT support in the simplest way: |
You have to do two things to make use of the JIT support in the simplest way: |
<pre> |
<pre> |
Line 79 You have to do two things to make use of the JIT suppo
|
Line 86 You have to do two things to make use of the JIT suppo
|
<b>pcre_exec()</b>. |
<b>pcre_exec()</b>. |
|
|
(2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is |
(2) Use <b>pcre_free_study()</b> to free the <b>pcre_extra</b> block when it is |
no longer needed instead of just freeing it yourself. This | no longer needed, instead of just freeing it yourself. This |
ensures that any JIT data is also freed. |
ensures that any JIT data is also freed. |
</pre> |
</pre> |
For a program that may be linked with pre-8.20 versions of PCRE, you can insert |
For a program that may be linked with pre-8.20 versions of PCRE, you can insert |
Line 97 this to free the study data:
|
Line 104 this to free the study data:
|
pcre_free(study_ptr); |
pcre_free(study_ptr); |
#endif |
#endif |
</pre> |
</pre> |
|
PCRE_STUDY_JIT_COMPILE requests the JIT compiler to generate code for complete |
|
matches. If you want to run partial matches using the PCRE_PARTIAL_HARD or |
|
PCRE_PARTIAL_SOFT options of <b>pcre_exec()</b>, you should set one or both of |
|
the following options in addition to, or instead of, PCRE_STUDY_JIT_COMPILE |
|
when you call <b>pcre_study()</b>: |
|
<pre> |
|
PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE |
|
PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE |
|
</pre> |
|
The JIT compiler generates different optimized code for each of the three |
|
modes (normal, soft partial, hard partial). When <b>pcre_exec()</b> is called, |
|
the appropriate code is run if it is available. Otherwise, the pattern is |
|
matched using interpretive code. |
|
</P> |
|
<P> |
In some circumstances you may need to call additional functions. These are |
In some circumstances you may need to call additional functions. These are |
described in the section entitled |
described in the section entitled |
<a href="#stackcontrol">"Controlling the JIT stack"</a> |
<a href="#stackcontrol">"Controlling the JIT stack"</a> |
below. |
below. |
</P> |
</P> |
<P> |
<P> |
If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT | If JIT support is not available, PCRE_STUDY_JIT_COMPILE etc. are ignored, and |
data is set up. Otherwise, the compiled pattern is passed to the JIT compiler, | no JIT data is created. Otherwise, the compiled pattern is passed to the JIT |
which turns it into machine code that executes much faster than the normal | compiler, which turns it into machine code that executes much faster than the |
interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b> block | normal interpretive code. When <b>pcre_exec()</b> is passed a <b>pcre_extra</b> |
containing a pointer to JIT code, it obeys that instead of the normal code. The | block containing a pointer to JIT code of the appropriate mode (normal or |
result is identical, but the code runs much faster. | hard/soft partial), it obeys that code instead of running the interpreter. The |
| result is identical, but the compiled JIT code runs much faster. |
</P> |
</P> |
<P> |
<P> |
There are some <b>pcre_exec()</b> options that are not supported for JIT |
There are some <b>pcre_exec()</b> options that are not supported for JIT |
execution. There are also some pattern items that JIT cannot handle. Details |
execution. There are also some pattern items that JIT cannot handle. Details |
are given below. In both cases, execution automatically falls back to the |
are given below. In both cases, execution automatically falls back to the |
interpretive code. | interpretive code. If you want to know whether JIT was actually used for a |
| particular match, you should arrange for a JIT callback function to be set up |
| as described in the section entitled |
| <a href="#stackcontrol">"Controlling the JIT stack"</a> |
| below, even if you do not need to supply a non-default JIT stack. Such a |
| callback function is called whenever JIT code is about to be obeyed. If the |
| execution options are not right for JIT execution, the callback function is not |
| obeyed. |
</P> |
</P> |
<P> |
<P> |
If the JIT compiler finds an unsupported item, no JIT data is generated. You |
If the JIT compiler finds an unsupported item, no JIT data is generated. You |
can find out if JIT execution is available after studying a pattern by calling |
can find out if JIT execution is available after studying a pattern by calling |
<b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that |
<b>pcre_fullinfo()</b> with the PCRE_INFO_JIT option. A result of 1 means that |
JIT compilation was successful. A result of 0 means that JIT support is not |
JIT compilation was successful. A result of 0 means that JIT support is not |
available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the | available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE etc., or |
JIT compiler was not able to handle the pattern. | the JIT compiler was not able to handle the pattern. |
</P> |
</P> |
<P> |
<P> |
Once a pattern has been studied, with or without JIT, it can be used as many |
Once a pattern has been studied, with or without JIT, it can be used as many |
times as you like for matching different subject strings. |
times as you like for matching different subject strings. |
</P> |
</P> |
<br><a name="SEC4" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br> | <br><a name="SEC5" href="#TOC1">UNSUPPORTED OPTIONS AND PATTERN ITEMS</a><br> |
<P> |
<P> |
The only <b>pcre_exec()</b> options that are supported for JIT execution are |
The only <b>pcre_exec()</b> options that are supported for JIT execution are |
PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and | PCRE_NO_UTF8_CHECK, PCRE_NO_UTF16_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, |
PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not | PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and PCRE_PARTIAL_SOFT. |
supported. | |
</P> |
</P> |
<P> |
<P> |
The unsupported pattern items are: |
The unsupported pattern items are: |
<pre> |
<pre> |
\C match a single byte; not supported in UTF-8 mode |
\C match a single byte; not supported in UTF-8 mode |
(?Cn) callouts |
(?Cn) callouts |
(*COMMIT) ) | (*PRUNE) ) |
(*MARK) ) | (*SKIP) ) backtracking control verbs |
(*PRUNE) ) the backtracking control verbs | |
(*SKIP) ) | |
(*THEN) ) |
(*THEN) ) |
</pre> |
</pre> |
Support for some of these may be added in future. |
Support for some of these may be added in future. |
</P> |
</P> |
<br><a name="SEC5" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br> | <br><a name="SEC6" href="#TOC1">RETURN VALUES FROM JIT EXECUTION</a><br> |
<P> |
<P> |
When a pattern is matched using JIT execution, the return values are the same |
When a pattern is matched using JIT execution, the return values are the same |
as those given by the interpretive <b>pcre_exec()</b> code, with the addition of |
as those given by the interpretive <b>pcre_exec()</b> code, with the addition of |
Line 166 when JIT is not used, but the details of exactly what
|
Line 193 when JIT is not used, but the details of exactly what
|
same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT |
same. The PCRE_ERROR_RECURSIONLIMIT error code is never returned by JIT |
execution. |
execution. |
</P> |
</P> |
<br><a name="SEC6" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br> | <br><a name="SEC7" href="#TOC1">SAVING AND RESTORING COMPILED PATTERNS</a><br> |
<P> |
<P> |
The code that is generated by the JIT compiler is architecture-specific, and is |
The code that is generated by the JIT compiler is architecture-specific, and is |
also position dependent. For those reasons it cannot be saved (in a file or |
also position dependent. For those reasons it cannot be saved (in a file or |
Line 179 restored pattern, and thereby recreate the JIT data, b
|
Line 206 restored pattern, and thereby recreate the JIT data, b
|
compilation uses significant resources, it is probably not worth doing this; |
compilation uses significant resources, it is probably not worth doing this; |
you might as well recompile the original pattern. |
you might as well recompile the original pattern. |
<a name="stackcontrol"></a></P> |
<a name="stackcontrol"></a></P> |
<br><a name="SEC7" href="#TOC1">CONTROLLING THE JIT STACK</a><br> | <br><a name="SEC8" href="#TOC1">CONTROLLING THE JIT STACK</a><br> |
<P> |
<P> |
When the compiled JIT code runs, it needs a block of memory to use as a stack. |
When the compiled JIT code runs, it needs a block of memory to use as a stack. |
By default, it uses 32K on the machine stack. However, some large or |
By default, it uses 32K on the machine stack. However, some large or |
Line 212 should use. Its arguments are as follows:
|
Line 239 should use. Its arguments are as follows:
|
void *data |
void *data |
</pre> |
</pre> |
The <i>extra</i> argument must be the result of studying a pattern with |
The <i>extra</i> argument must be the result of studying a pattern with |
PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two | PCRE_STUDY_JIT_COMPILE etc. There are three cases for the values of the other |
options: | two options: |
<pre> |
<pre> |
(1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block |
(1) If <i>callback</i> is NULL and <i>data</i> is NULL, an internal 32K block |
on the machine stack is used. |
on the machine stack is used. |
Line 221 options:
|
Line 248 options:
|
(2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be |
(2) If <i>callback</i> is NULL and <i>data</i> is not NULL, <i>data</i> must be |
a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>. |
a valid JIT stack, the result of calling <b>pcre_jit_stack_alloc()</b>. |
|
|
(3) If <i>callback</i> not NULL, it must point to a function that is called | (3) If <i>callback</i> is not NULL, it must point to a function that is |
with <i>data</i> as an argument at the start of matching, in order to | called with <i>data</i> as an argument at the start of matching, in |
set up a JIT stack. If the result is NULL, the internal 32K stack | order to set up a JIT stack. If the return from the callback |
is used; otherwise the return value must be a valid JIT stack, | function is NULL, the internal 32K stack is used; otherwise the |
the result of calling <b>pcre_jit_stack_alloc()</b>. | return value must be a valid JIT stack, the result of calling |
| <b>pcre_jit_stack_alloc()</b>. |
</pre> |
</pre> |
You may safely assign the same JIT stack to more than one pattern, as long as | A callback function is obeyed whenever JIT code is about to be run; it is not |
they are all matched sequentially in the same thread. In a multithread | obeyed when <b>pcre_exec()</b> is called with options that are incompatible for |
application, each thread must use its own JIT stack. | JIT execution. A callback function can therefore be used to determine whether a |
| match operation was executed by JIT or by the interpreter. |
</P> |
</P> |
<P> |
<P> |
Strictly speaking, even more is allowed. You can assign the same stack to any | You may safely use the same JIT stack for more than one pattern (either by |
number of patterns as long as they are not used for matching by multiple | assigning directly or by callback), as long as the patterns are all matched |
| sequentially in the same thread. In a multithread application, if you do not |
| specify a JIT stack, or if you assign or pass back NULL from a callback, that |
| is thread-safe, because each thread has its own machine stack. However, if you |
| assign or pass back a non-NULL JIT stack, this must be a different stack for |
| each thread so that the application is thread-safe. |
| </P> |
| <P> |
| Strictly speaking, even more is allowed. You can assign the same non-NULL stack |
| to any number of patterns as long as they are not used for matching by multiple |
threads at the same time. For example, you can assign the same stack to all |
threads at the same time. For example, you can assign the same stack to all |
compiled patterns, and use a global mutex in the callback to wait until the |
compiled patterns, and use a global mutex in the callback to wait until the |
stack is available for use. However, this is an inefficient solution, and | stack is available for use. However, this is an inefficient solution, and not |
not recommended. | recommended. |
</P> |
</P> |
<P> |
<P> |
This is a suggestion for how a typical multithreaded program might operate: | This is a suggestion for how a multithreaded program that needs to set up |
| non-default JIT stacks might operate: |
<pre> |
<pre> |
During thread initalization |
During thread initalization |
thread_local_var = pcre_jit_stack_alloc(...) |
thread_local_var = pcre_jit_stack_alloc(...) |
Line 254 This is a suggestion for how a typical multithreaded p
|
Line 293 This is a suggestion for how a typical multithreaded p
|
All the functions described in this section do nothing if JIT is not available, |
All the functions described in this section do nothing if JIT is not available, |
and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument |
and <b>pcre_assign_jit_stack()</b> does nothing unless the <b>extra</b> argument |
is non-NULL and points to a <b>pcre_extra</b> block that is the result of a |
is non-NULL and points to a <b>pcre_extra</b> block that is the result of a |
successful study with PCRE_STUDY_JIT_COMPILE. | successful study with PCRE_STUDY_JIT_COMPILE etc. |
<a name="stackfaq"></a></P> |
<a name="stackfaq"></a></P> |
<br><a name="SEC8" href="#TOC1">JIT STACK FAQ</a><br> | <br><a name="SEC9" href="#TOC1">JIT STACK FAQ</a><br> |
<P> |
<P> |
(1) Why do we need JIT stacks? |
(1) Why do we need JIT stacks? |
<br> |
<br> |
Line 320 pattern causes stack overflow with a stack of 1M? Is t
|
Line 359 pattern causes stack overflow with a stack of 1M? Is t
|
stack is freed? |
stack is freed? |
<br> |
<br> |
<br> |
<br> |
Especially on embedded sytems, it might be a good idea to release | Especially on embedded sytems, it might be a good idea to release memory |
memory sometimes without freeing the stack. There is no API for this at the | sometimes without freeing the stack. There is no API for this at the moment. |
moment. Probably a function call which returns with the currently allocated | Probably a function call which returns with the currently allocated memory for |
memory for any stack and another which allows releasing memory (shrinking the | any stack and another which allows releasing memory (shrinking the stack) would |
stack) would be a good idea if someone needs this. | be a good idea if someone needs this. |
</P> |
</P> |
<P> |
<P> |
(7) This is too much of a headache. Isn't there any better solution for JIT |
(7) This is too much of a headache. Isn't there any better solution for JIT |
Line 334 stack handling?
|
Line 373 stack handling?
|
No, thanks to Windows. If POSIX threads were used everywhere, we could throw |
No, thanks to Windows. If POSIX threads were used everywhere, we could throw |
out this complicated API. |
out this complicated API. |
</P> |
</P> |
<br><a name="SEC9" href="#TOC1">EXAMPLE CODE</a><br> | <br><a name="SEC10" href="#TOC1">EXAMPLE CODE</a><br> |
<P> |
<P> |
This is a single-threaded example that specifies a JIT stack without using a |
This is a single-threaded example that specifies a JIT stack without using a |
callback. |
callback. |
Line 359 callback.
|
Line 398 callback.
|
|
|
</PRE> |
</PRE> |
</P> |
</P> |
<br><a name="SEC10" href="#TOC1">SEE ALSO</a><br> | <br><a name="SEC11" href="#TOC1">SEE ALSO</a><br> |
<P> |
<P> |
<b>pcreapi</b>(3) |
<b>pcreapi</b>(3) |
</P> |
</P> |
<br><a name="SEC11" href="#TOC1">AUTHOR</a><br> | <br><a name="SEC12" href="#TOC1">AUTHOR</a><br> |
<P> |
<P> |
Philip Hazel (FAQ by Zoltan Herczeg) |
Philip Hazel (FAQ by Zoltan Herczeg) |
<br> |
<br> |
Line 372 University Computing Service
|
Line 411 University Computing Service
|
Cambridge CB2 3QH, England. |
Cambridge CB2 3QH, England. |
<br> |
<br> |
</P> |
</P> |
<br><a name="SEC12" href="#TOC1">REVISION</a><br> | <br><a name="SEC13" href="#TOC1">REVISION</a><br> |
<P> |
<P> |
Last updated: 26 November 2011 | Last updated: 04 May 2012 |
<br> |
<br> |
Copyright © 1997-2011 University of Cambridge. | Copyright © 1997-2012 University of Cambridge. |
<br> |
<br> |
<p> |
<p> |
Return to the <a href="index.html">PCRE index page</a>. |
Return to the <a href="index.html">PCRE index page</a>. |