version 1.1.1.2, 2012/02/21 23:50:25
|
version 1.1.1.4, 2013/07/22 08:25:57
|
Line 1
|
Line 1
|
.TH PCREJIT 3 | .TH PCREJIT 3 "17 March 2013" "PCRE 8.33" |
.SH NAME |
.SH NAME |
PCRE - Perl-compatible regular expressions |
PCRE - Perl-compatible regular expressions |
.SH "PCRE JUST-IN-TIME COMPILER SUPPORT" |
.SH "PCRE JUST-IN-TIME COMPILER SUPPORT" |
Line 18 It does not apply when the DFA matching function is be
|
Line 18 It does not apply when the DFA matching function is be
|
this support was written by Zoltan Herczeg. |
this support was written by Zoltan Herczeg. |
. |
. |
. |
. |
.SH "8-BIT and 16-BIT SUPPORT" | .SH "8-BIT, 16-BIT AND 32-BIT SUPPORT" |
.rs |
.rs |
.sp |
.sp |
JIT support is available for both the 8-bit and 16-bit PCRE libraries. To keep | JIT support is available for all of the 8-bit, 16-bit and 32-bit PCRE |
this documentation simple, only the 8-bit interface is described in what | libraries. To keep this documentation simple, only the 8-bit interface is |
follows. If you are using the 16-bit library, substitute the 16-bit functions | described in what follows. If you are using the 16-bit library, substitute the |
and 16-bit structures (for example, \fIpcre16_jit_stack\fP instead of | 16-bit functions and 16-bit structures (for example, \fIpcre16_jit_stack\fP |
\fIpcre_jit_stack\fP). | instead of \fIpcre_jit_stack\fP). If you are using the 32-bit library, |
| substitute the 32-bit functions and 32-bit structures (for example, |
| \fIpcre32_jit_stack\fP instead of \fIpcre_jit_stack\fP). |
. |
. |
. |
. |
.SH "AVAILABILITY OF JIT SUPPORT" |
.SH "AVAILABILITY OF JIT SUPPORT" |
Line 39 JIT. The support is limited to the following hardware
|
Line 41 JIT. The support is limited to the following hardware
|
Intel x86 32-bit and 64-bit |
Intel x86 32-bit and 64-bit |
MIPS 32-bit |
MIPS 32-bit |
Power PC 32-bit and 64-bit |
Power PC 32-bit and 64-bit |
|
SPARC 32-bit (experimental) |
.sp |
.sp |
The Power PC support is designated as experimental because it has not been | If --enable-jit is set on an unsupported platform, compilation fails. |
fully tested. If --enable-jit is set on an unsupported platform, compilation | |
fails. | |
.P |
.P |
A program that is linked with PCRE 8.20 or later can tell if JIT support is |
A program that is linked with PCRE 8.20 or later can tell if JIT support is |
available by calling \fBpcre_config()\fP with the PCRE_CONFIG_JIT option. The |
available by calling \fBpcre_config()\fP with the PCRE_CONFIG_JIT option. The |
result is 1 when JIT is available, and 0 otherwise. However, a simple program |
result is 1 when JIT is available, and 0 otherwise. However, a simple program |
does not need to check this in order to use JIT. The API is implemented in a | does not need to check this in order to use JIT. The normal API is implemented |
way that falls back to the ordinary PCRE code if JIT is not available. | in a way that falls back to the interpretive code if JIT is not available. For |
| programs that need the best possible performance, there is also a "fast path" |
| API that is JIT-specific. |
.P |
.P |
If your program may sometimes be linked with versions of PCRE that are older |
If your program may sometimes be linked with versions of PCRE that are older |
than 8.20, but you want to use JIT when it is available, you can test |
than 8.20, but you want to use JIT when it is available, you can test |
Line 66 You have to do two things to make use of the JIT suppo
|
Line 69 You have to do two things to make use of the JIT suppo
|
\fBpcre_exec()\fP. |
\fBpcre_exec()\fP. |
.sp |
.sp |
(2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is |
(2) Use \fBpcre_free_study()\fP to free the \fBpcre_extra\fP block when it is |
no longer needed instead of just freeing it yourself. This | no longer needed, instead of just freeing it yourself. This ensures that |
ensures that any JIT data is also freed. | any JIT data is also freed. |
.sp |
.sp |
For a program that may be linked with pre-8.20 versions of PCRE, you can insert |
For a program that may be linked with pre-8.20 versions of PCRE, you can insert |
.sp |
.sp |
Line 84 this to free the study data:
|
Line 87 this to free the study data:
|
pcre_free(study_ptr); |
pcre_free(study_ptr); |
#endif |
#endif |
.sp |
.sp |
|
PCRE_STUDY_JIT_COMPILE requests the JIT compiler to generate code for complete |
|
matches. If you want to run partial matches using the PCRE_PARTIAL_HARD or |
|
PCRE_PARTIAL_SOFT options of \fBpcre_exec()\fP, you should set one or both of |
|
the following options in addition to, or instead of, PCRE_STUDY_JIT_COMPILE |
|
when you call \fBpcre_study()\fP: |
|
.sp |
|
PCRE_STUDY_JIT_PARTIAL_HARD_COMPILE |
|
PCRE_STUDY_JIT_PARTIAL_SOFT_COMPILE |
|
.sp |
|
The JIT compiler generates different optimized code for each of the three |
|
modes (normal, soft partial, hard partial). When \fBpcre_exec()\fP is called, |
|
the appropriate code is run if it is available. Otherwise, the pattern is |
|
matched using interpretive code. |
|
.P |
In some circumstances you may need to call additional functions. These are |
In some circumstances you may need to call additional functions. These are |
described in the section entitled |
described in the section entitled |
.\" HTML <a href="#stackcontrol"> |
.\" HTML <a href="#stackcontrol"> |
Line 92 described in the section entitled
|
Line 109 described in the section entitled
|
.\" |
.\" |
below. |
below. |
.P |
.P |
If JIT support is not available, PCRE_STUDY_JIT_COMPILE is ignored, and no JIT | If JIT support is not available, PCRE_STUDY_JIT_COMPILE etc. are ignored, and |
data is set up. Otherwise, the compiled pattern is passed to the JIT compiler, | no JIT data is created. Otherwise, the compiled pattern is passed to the JIT |
which turns it into machine code that executes much faster than the normal | compiler, which turns it into machine code that executes much faster than the |
interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP block | normal interpretive code. When \fBpcre_exec()\fP is passed a \fBpcre_extra\fP |
containing a pointer to JIT code, it obeys that instead of the normal code. The | block containing a pointer to JIT code of the appropriate mode (normal or |
result is identical, but the code runs much faster. | hard/soft partial), it obeys that code instead of running the interpreter. The |
| result is identical, but the compiled JIT code runs much faster. |
.P |
.P |
There are some \fBpcre_exec()\fP options that are not supported for JIT |
There are some \fBpcre_exec()\fP options that are not supported for JIT |
execution. There are also some pattern items that JIT cannot handle. Details |
execution. There are also some pattern items that JIT cannot handle. Details |
are given below. In both cases, execution automatically falls back to the |
are given below. In both cases, execution automatically falls back to the |
interpretive code. | interpretive code. If you want to know whether JIT was actually used for a |
| particular match, you should arrange for a JIT callback function to be set up |
| as described in the section entitled |
| .\" HTML <a href="#stackcontrol"> |
| .\" </a> |
| "Controlling the JIT stack" |
| .\" |
| below, even if you do not need to supply a non-default JIT stack. Such a |
| callback function is called whenever JIT code is about to be obeyed. If the |
| execution options are not right for JIT execution, the callback function is not |
| obeyed. |
.P |
.P |
If the JIT compiler finds an unsupported item, no JIT data is generated. You |
If the JIT compiler finds an unsupported item, no JIT data is generated. You |
can find out if JIT execution is available after studying a pattern by calling |
can find out if JIT execution is available after studying a pattern by calling |
\fBpcre_fullinfo()\fP with the PCRE_INFO_JIT option. A result of 1 means that |
\fBpcre_fullinfo()\fP with the PCRE_INFO_JIT option. A result of 1 means that |
JIT compilation was successful. A result of 0 means that JIT support is not |
JIT compilation was successful. A result of 0 means that JIT support is not |
available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE, or the | available, or the pattern was not studied with PCRE_STUDY_JIT_COMPILE etc., or |
JIT compiler was not able to handle the pattern. | the JIT compiler was not able to handle the pattern. |
.P |
.P |
Once a pattern has been studied, with or without JIT, it can be used as many |
Once a pattern has been studied, with or without JIT, it can be used as many |
times as you like for matching different subject strings. |
times as you like for matching different subject strings. |
Line 119 times as you like for matching different subject strin
|
Line 147 times as you like for matching different subject strin
|
.rs |
.rs |
.sp |
.sp |
The only \fBpcre_exec()\fP options that are supported for JIT execution are |
The only \fBpcre_exec()\fP options that are supported for JIT execution are |
PCRE_NO_UTF8_CHECK, PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and | PCRE_NO_UTF8_CHECK, PCRE_NO_UTF16_CHECK, PCRE_NO_UTF32_CHECK, PCRE_NOTBOL, |
PCRE_NOTEMPTY_ATSTART. Note in particular that partial matching is not | PCRE_NOTEOL, PCRE_NOTEMPTY, PCRE_NOTEMPTY_ATSTART, PCRE_PARTIAL_HARD, and |
supported. | PCRE_PARTIAL_SOFT. |
.P |
.P |
The unsupported pattern items are: | The only unsupported pattern items are \eC (match a single data unit) when |
.sp | running in a UTF mode, and a callout immediately before an assertion condition |
\eC match a single byte; not supported in UTF-8 mode | in a conditional group. |
(?Cn) callouts | |
(*COMMIT) ) | |
(*MARK) ) | |
(*PRUNE) ) the backtracking control verbs | |
(*SKIP) ) | |
(*THEN) ) | |
.sp | |
Support for some of these may be added in future. | |
. |
. |
. |
. |
.SH "RETURN VALUES FROM JIT EXECUTION" |
.SH "RETURN VALUES FROM JIT EXECUTION" |
Line 210 should use. Its arguments are as follows:
|
Line 230 should use. Its arguments are as follows:
|
void *data |
void *data |
.sp |
.sp |
The \fIextra\fP argument must be the result of studying a pattern with |
The \fIextra\fP argument must be the result of studying a pattern with |
PCRE_STUDY_JIT_COMPILE. There are three cases for the values of the other two | PCRE_STUDY_JIT_COMPILE etc. There are three cases for the values of the other |
options: | two options: |
.sp |
.sp |
(1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block |
(1) If \fIcallback\fP is NULL and \fIdata\fP is NULL, an internal 32K block |
on the machine stack is used. |
on the machine stack is used. |
Line 219 options:
|
Line 239 options:
|
(2) If \fIcallback\fP is NULL and \fIdata\fP is not NULL, \fIdata\fP must be |
(2) If \fIcallback\fP is NULL and \fIdata\fP is not NULL, \fIdata\fP must be |
a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP. |
a valid JIT stack, the result of calling \fBpcre_jit_stack_alloc()\fP. |
.sp |
.sp |
(3) If \fIcallback\fP not NULL, it must point to a function that is called | (3) If \fIcallback\fP is not NULL, it must point to a function that is |
with \fIdata\fP as an argument at the start of matching, in order to | called with \fIdata\fP as an argument at the start of matching, in |
set up a JIT stack. If the result is NULL, the internal 32K stack | order to set up a JIT stack. If the return from the callback |
is used; otherwise the return value must be a valid JIT stack, | function is NULL, the internal 32K stack is used; otherwise the |
the result of calling \fBpcre_jit_stack_alloc()\fP. | return value must be a valid JIT stack, the result of calling |
| \fBpcre_jit_stack_alloc()\fP. |
.sp |
.sp |
You may safely assign the same JIT stack to more than one pattern, as long as | A callback function is obeyed whenever JIT code is about to be run; it is not |
they are all matched sequentially in the same thread. In a multithread | obeyed when \fBpcre_exec()\fP is called with options that are incompatible for |
application, each thread must use its own JIT stack. | JIT execution. A callback function can therefore be used to determine whether a |
| match operation was executed by JIT or by the interpreter. |
.P |
.P |
Strictly speaking, even more is allowed. You can assign the same stack to any | You may safely use the same JIT stack for more than one pattern (either by |
number of patterns as long as they are not used for matching by multiple | assigning directly or by callback), as long as the patterns are all matched |
| sequentially in the same thread. In a multithread application, if you do not |
| specify a JIT stack, or if you assign or pass back NULL from a callback, that |
| is thread-safe, because each thread has its own machine stack. However, if you |
| assign or pass back a non-NULL JIT stack, this must be a different stack for |
| each thread so that the application is thread-safe. |
| .P |
| Strictly speaking, even more is allowed. You can assign the same non-NULL stack |
| to any number of patterns as long as they are not used for matching by multiple |
threads at the same time. For example, you can assign the same stack to all |
threads at the same time. For example, you can assign the same stack to all |
compiled patterns, and use a global mutex in the callback to wait until the |
compiled patterns, and use a global mutex in the callback to wait until the |
stack is available for use. However, this is an inefficient solution, and | stack is available for use. However, this is an inefficient solution, and not |
not recommended. | recommended. |
.P |
.P |
This is a suggestion for how a typical multithreaded program might operate: | This is a suggestion for how a multithreaded program that needs to set up |
| non-default JIT stacks might operate: |
.sp |
.sp |
During thread initalization |
During thread initalization |
thread_local_var = pcre_jit_stack_alloc(...) |
thread_local_var = pcre_jit_stack_alloc(...) |
Line 250 This is a suggestion for how a typical multithreaded p
|
Line 281 This is a suggestion for how a typical multithreaded p
|
All the functions described in this section do nothing if JIT is not available, |
All the functions described in this section do nothing if JIT is not available, |
and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument |
and \fBpcre_assign_jit_stack()\fP does nothing unless the \fBextra\fP argument |
is non-NULL and points to a \fBpcre_extra\fP block that is the result of a |
is non-NULL and points to a \fBpcre_extra\fP block that is the result of a |
successful study with PCRE_STUDY_JIT_COMPILE. | successful study with PCRE_STUDY_JIT_COMPILE etc. |
. |
. |
. |
. |
.\" HTML <a name="stackfaq"></a> |
.\" HTML <a name="stackfaq"></a> |
Line 301 replacement.
|
Line 332 replacement.
|
.sp |
.sp |
No, because this is too costly in terms of resources. However, you could |
No, because this is too costly in terms of resources. However, you could |
implement some clever idea which release the stack if it is not used in let's |
implement some clever idea which release the stack if it is not used in let's |
say two minutes. The JIT callback can help to achive this without keeping a | say two minutes. The JIT callback can help to achieve this without keeping a |
list of the currently JIT studied patterns. |
list of the currently JIT studied patterns. |
.P |
.P |
(6) OK, the stack is for long term memory allocation. But what happens if a |
(6) OK, the stack is for long term memory allocation. But what happens if a |
pattern causes stack overflow with a stack of 1M? Is that 1M kept until the |
pattern causes stack overflow with a stack of 1M? Is that 1M kept until the |
stack is freed? |
stack is freed? |
.sp |
.sp |
Especially on embedded sytems, it might be a good idea to release | Especially on embedded sytems, it might be a good idea to release memory |
memory sometimes without freeing the stack. There is no API for this at the | sometimes without freeing the stack. There is no API for this at the moment. |
moment. Probably a function call which returns with the currently allocated | Probably a function call which returns with the currently allocated memory for |
memory for any stack and another which allows releasing memory (shrinking the | any stack and another which allows releasing memory (shrinking the stack) would |
stack) would be a good idea if someone needs this. | be a good idea if someone needs this. |
.P |
.P |
(7) This is too much of a headache. Isn't there any better solution for JIT |
(7) This is too much of a headache. Isn't there any better solution for JIT |
stack handling? |
stack handling? |
Line 347 callback.
|
Line 378 callback.
|
.sp |
.sp |
. |
. |
. |
. |
|
.SH "JIT FAST PATH API" |
|
.rs |
|
.sp |
|
Because the API described above falls back to interpreted execution when JIT is |
|
not available, it is convenient for programs that are written for general use |
|
in many environments. However, calling JIT via \fBpcre_exec()\fP does have a |
|
performance impact. Programs that are written for use where JIT is known to be |
|
available, and which need the best possible performance, can instead use a |
|
"fast path" API to call JIT execution directly instead of calling |
|
\fBpcre_exec()\fP (obviously only for patterns that have been successfully |
|
studied by JIT). |
|
.P |
|
The fast path function is called \fBpcre_jit_exec()\fP, and it takes exactly |
|
the same arguments as \fBpcre_exec()\fP, plus one additional argument that |
|
must point to a JIT stack. The JIT stack arrangements described above do not |
|
apply. The return values are the same as for \fBpcre_exec()\fP. |
|
.P |
|
When you call \fBpcre_exec()\fP, as well as testing for invalid options, a |
|
number of other sanity checks are performed on the arguments. For example, if |
|
the subject pointer is NULL, or its length is negative, an immediate error is |
|
given. Also, unless PCRE_NO_UTF[8|16|32] is set, a UTF subject string is tested |
|
for validity. In the interests of speed, these checks do not happen on the JIT |
|
fast path, and if invalid data is passed, the result is undefined. |
|
.P |
|
Bypassing the sanity checks and the \fBpcre_exec()\fP wrapping can give |
|
speedups of more than 10%. |
|
. |
|
. |
.SH "SEE ALSO" |
.SH "SEE ALSO" |
.rs |
.rs |
.sp |
.sp |
Line 367 Cambridge CB2 3QH, England.
|
Line 426 Cambridge CB2 3QH, England.
|
.rs |
.rs |
.sp |
.sp |
.nf |
.nf |
Last updated: 08 January 2012 | Last updated: 17 March 2013 |
Copyright (c) 1997-2012 University of Cambridge. | Copyright (c) 1997-2013 University of Cambridge. |
.fi |
.fi |