version 1.1, 2012/02/21 23:05:52
|
version 1.1.1.3, 2013/07/22 08:25:57
|
Line 16 man page, in case the conversion went wrong.
|
Line 16 man page, in case the conversion went wrong.
|
PCRE DISCUSSION OF STACK USAGE |
PCRE DISCUSSION OF STACK USAGE |
</b><br> |
</b><br> |
<P> |
<P> |
When you call <b>pcre_exec()</b>, it makes use of an internal function called | When you call <b>pcre[16|32]_exec()</b>, it makes use of an internal function |
<b>match()</b>. This calls itself recursively at branch points in the pattern, | called <b>match()</b>. This calls itself recursively at branch points in the |
in order to remember the state of the match so that it can back up and try a | pattern, in order to remember the state of the match so that it can back up and |
different alternative if the first one fails. As matching proceeds deeper and | try a different alternative if the first one fails. As matching proceeds deeper |
deeper into the tree of possibilities, the recursion depth increases. The | and deeper into the tree of possibilities, the recursion depth increases. The |
<b>match()</b> function is also called in other circumstances, for example, |
<b>match()</b> function is also called in other circumstances, for example, |
whenever a parenthesized sub-pattern is entered, and in certain cases of |
whenever a parenthesized sub-pattern is entered, and in certain cases of |
repetition. |
repetition. |
Line 33 the recursive call would immediately be passed back as
|
Line 33 the recursive call would immediately be passed back as
|
current call (a "tail recursion"), the function is just restarted instead. |
current call (a "tail recursion"), the function is just restarted instead. |
</P> |
</P> |
<P> |
<P> |
The above comments apply when <b>pcre_exec()</b> is run in its normal | The above comments apply when <b>pcre[16|32]_exec()</b> is run in its normal |
interpretive manner. If the pattern was studied with the |
interpretive manner. If the pattern was studied with the |
PCRE_STUDY_JIT_COMPILE option, and just-in-time compiling was successful, and |
PCRE_STUDY_JIT_COMPILE option, and just-in-time compiling was successful, and |
the options passed to <b>pcre_exec()</b> were not incompatible, the matching | the options passed to <b>pcre[16|32]_exec()</b> were not incompatible, the matching |
process uses the JIT-compiled code instead of the <b>match()</b> function. In |
process uses the JIT-compiled code instead of the <b>match()</b> function. In |
this case, the memory requirements are handled entirely differently. See the |
this case, the memory requirements are handled entirely differently. See the |
<a href="pcrejit.html"><b>pcrejit</b></a> |
<a href="pcrejit.html"><b>pcrejit</b></a> |
documentation for details. |
documentation for details. |
</P> |
</P> |
<P> |
<P> |
The <b>pcre_dfa_exec()</b> function operates in an entirely different way, and | The <b>pcre[16|32]_dfa_exec()</b> function operates in an entirely different way, |
uses recursion only when there is a regular expression recursion or subroutine | and uses recursion only when there is a regular expression recursion or |
call in the pattern. This includes the processing of assertion and "once-only" | subroutine call in the pattern. This includes the processing of assertion and |
subpatterns, which are handled like subroutine calls. Normally, these are never | "once-only" subpatterns, which are handled like subroutine calls. Normally, |
very deep, and the limit on the complexity of <b>pcre_dfa_exec()</b> is | these are never very deep, and the limit on the complexity of |
controlled by the amount of workspace it is given. However, it is possible to | <b>pcre[16|32]_dfa_exec()</b> is controlled by the amount of workspace it is given. |
write patterns with runaway infinite recursions; such patterns will cause | However, it is possible to write patterns with runaway infinite recursions; |
<b>pcre_dfa_exec()</b> to run out of stack. At present, there is no protection | such patterns will cause <b>pcre[16|32]_dfa_exec()</b> to run out of stack. At |
against this. | present, there is no protection against this. |
</P> |
</P> |
<P> |
<P> |
The comments that follow do NOT apply to <b>pcre_dfa_exec()</b>; they are | The comments that follow do NOT apply to <b>pcre[16|32]_dfa_exec()</b>; they are |
relevant only for <b>pcre_exec()</b> without the JIT optimization. | relevant only for <b>pcre[16|32]_exec()</b> without the JIT optimization. |
</P> |
</P> |
<br><b> |
<br><b> |
Reducing <b>pcre_exec()</b>'s stack usage | Reducing <b>pcre[16|32]_exec()</b>'s stack usage |
</b><br> |
</b><br> |
<P> |
<P> |
Each time that <b>match()</b> is actually called recursively, it uses memory |
Each time that <b>match()</b> is actually called recursively, it uses memory |
Line 94 subject strings is to write repeated parenthesized sub
|
Line 94 subject strings is to write repeated parenthesized sub
|
than one character whenever possible. |
than one character whenever possible. |
</P> |
</P> |
<br><b> |
<br><b> |
Compiling PCRE to use heap instead of stack for <b>pcre_exec()</b> | Compiling PCRE to use heap instead of stack for <b>pcre[16|32]_exec()</b> |
</b><br> |
</b><br> |
<P> |
<P> |
In environments where stack memory is constrained, you might want to compile |
In environments where stack memory is constrained, you might want to compile |
PCRE to use heap memory instead of stack for remembering back-up points when |
PCRE to use heap memory instead of stack for remembering back-up points when |
<b>pcre_exec()</b> is running. This makes it run a lot more slowly, however. | <b>pcre[16|32]_exec()</b> is running. This makes it run a lot more slowly, however. |
Details of how to do this are given in the |
Details of how to do this are given in the |
<a href="pcrebuild.html"><b>pcrebuild</b></a> |
<a href="pcrebuild.html"><b>pcrebuild</b></a> |
documentation. When built in this way, instead of using the stack, PCRE obtains |
documentation. When built in this way, instead of using the stack, PCRE obtains |
and frees memory by calling the functions that are pointed to by the |
and frees memory by calling the functions that are pointed to by the |
<b>pcre_stack_malloc</b> and <b>pcre_stack_free</b> variables. By default, these | <b>pcre[16|32]_stack_malloc</b> and <b>pcre[16|32]_stack_free</b> variables. By |
point to <b>malloc()</b> and <b>free()</b>, but you can replace the pointers to | default, these point to <b>malloc()</b> and <b>free()</b>, but you can replace |
cause PCRE to use your own functions. Since the block sizes are always the | the pointers to cause PCRE to use your own functions. Since the block sizes are |
same, and are always freed in reverse order, it may be possible to implement | always the same, and are always freed in reverse order, it may be possible to |
customized memory handlers that are more efficient than the standard functions. | implement customized memory handlers that are more efficient than the standard |
| functions. |
</P> |
</P> |
<br><b> |
<br><b> |
Limiting <b>pcre_exec()</b>'s stack usage | Limiting <b>pcre[16|32]_exec()</b>'s stack usage |
</b><br> |
</b><br> |
<P> |
<P> |
You can set limits on the number of times that <b>match()</b> is called, both in |
You can set limits on the number of times that <b>match()</b> is called, both in |
total and recursively. If a limit is exceeded, <b>pcre_exec()</b> returns an | total and recursively. If a limit is exceeded, <b>pcre[16|32]_exec()</b> returns an |
error code. Setting suitable limits should prevent it from running out of |
error code. Setting suitable limits should prevent it from running out of |
stack. The default values of the limits are very large, and unlikely ever to |
stack. The default values of the limits are very large, and unlikely ever to |
operate. They can be changed when PCRE is built, and they can also be set when |
operate. They can be changed when PCRE is built, and they can also be set when |
<b>pcre_exec()</b> is called. For details of these interfaces, see the | <b>pcre[16|32]_exec()</b> is called. For details of these interfaces, see the |
<a href="pcrebuild.html"><b>pcrebuild</b></a> |
<a href="pcrebuild.html"><b>pcrebuild</b></a> |
documentation and the |
documentation and the |
<a href="pcreapi.html#extradata">section on extra data for <b>pcre_exec()</b></a> | <a href="pcreapi.html#extradata">section on extra data for <b>pcre[16|32]_exec()</b></a> |
in the |
in the |
<a href="pcreapi.html"><b>pcreapi</b></a> |
<a href="pcreapi.html"><b>pcreapi</b></a> |
documentation. |
documentation. |
</P> |
</P> |
<P> |
<P> |
As a very rough rule of thumb, you should reckon on about 500 bytes per |
As a very rough rule of thumb, you should reckon on about 500 bytes per |
recursion. Thus, if you want to limit your stack usage to 8Mb, you | recursion. Thus, if you want to limit your stack usage to 8Mb, you should set |
should set the limit at 16000 recursions. A 64Mb stack, on the other hand, can | the limit at 16000 recursions. A 64Mb stack, on the other hand, can support |
support around 128000 recursions. | around 128000 recursions. |
</P> |
</P> |
<P> |
<P> |
In Unix-like environments, the <b>pcretest</b> test program has a command line |
In Unix-like environments, the <b>pcretest</b> test program has a command line |
option (<b>-S</b>) that can be used to increase the size of its stack. As long |
option (<b>-S</b>) that can be used to increase the size of its stack. As long |
as the stack is large enough, another option (<b>-M</b>) can be used to find the |
as the stack is large enough, another option (<b>-M</b>) can be used to find the |
smallest limits that allow a particular pattern to match a given subject |
smallest limits that allow a particular pattern to match a given subject |
string. This is done by calling <b>pcre_exec()</b> repeatedly with different | string. This is done by calling <b>pcre[16|32]_exec()</b> repeatedly with different |
limits. |
limits. |
</P> |
</P> |
<br><b> |
<br><b> |
|
Obtaining an estimate of stack usage |
|
</b><br> |
|
<P> |
|
The actual amount of stack used per recursion can vary quite a lot, depending |
|
on the compiler that was used to build PCRE and the optimization or debugging |
|
options that were set for it. The rule of thumb value of 500 bytes mentioned |
|
above may be larger or smaller than what is actually needed. A better |
|
approximation can be obtained by running this command: |
|
<pre> |
|
pcretest -m -C |
|
</pre> |
|
The <b>-C</b> option causes <b>pcretest</b> to output information about the |
|
options with which PCRE was compiled. When <b>-m</b> is also given (before |
|
<b>-C</b>), information about stack use is given in a line like this: |
|
<pre> |
|
Match recursion uses stack: approximate frame size = 640 bytes |
|
</pre> |
|
The value is approximate because some recursions need a bit more (up to perhaps |
|
16 more bytes). |
|
</P> |
|
<P> |
|
If the above command is given when PCRE is compiled to use the heap instead of |
|
the stack for recursion, the value that is output is the size of each block |
|
that is obtained from the heap. |
|
</P> |
|
<br><b> |
Changing stack size in Unix-like systems |
Changing stack size in Unix-like systems |
</b><br> |
</b><br> |
<P> |
<P> |
Line 163 limit on stack size by code such as this:
|
Line 190 limit on stack size by code such as this:
|
</pre> |
</pre> |
This reads the current limits (soft and hard) using <b>getrlimit()</b>, then |
This reads the current limits (soft and hard) using <b>getrlimit()</b>, then |
attempts to increase the soft limit to 100Mb using <b>setrlimit()</b>. You must |
attempts to increase the soft limit to 100Mb using <b>setrlimit()</b>. You must |
do this before calling <b>pcre_exec()</b>. | do this before calling <b>pcre[16|32]_exec()</b>. |
</P> |
</P> |
<br><b> |
<br><b> |
Changing stack size in Mac OS X |
Changing stack size in Mac OS X |
Line 189 Cambridge CB2 3QH, England.
|
Line 216 Cambridge CB2 3QH, England.
|
REVISION |
REVISION |
</b><br> |
</b><br> |
<P> |
<P> |
Last updated: 26 August 2011 | Last updated: 24 June 2012 |
<br> |
<br> |
Copyright © 1997-2011 University of Cambridge. | Copyright © 1997-2012 University of Cambridge. |
<br> |
<br> |
<p> |
<p> |
Return to the <a href="index.html">PCRE index page</a>. |
Return to the <a href="index.html">PCRE index page</a>. |