1: README file for PCRE (Perl-compatible regular expression library)
2: -----------------------------------------------------------------
3:
4: The latest release of PCRE is always available in three alternative formats
5: from:
6:
7: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.gz
8: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.bz2
9: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.zip
10:
11: There is a mailing list for discussion about the development of PCRE at
12:
13: pcre-dev@exim.org
14:
15: Please read the NEWS file if you are upgrading from a previous release.
16: The contents of this README file are:
17:
18: The PCRE APIs
19: Documentation for PCRE
20: Contributions by users of PCRE
21: Building PCRE on non-Unix-like systems
22: Building PCRE without using autotools
23: Building PCRE using autotools
24: Retrieving configuration information
25: Shared libraries
26: Cross-compiling using autotools
27: Using HP's ANSI C++ compiler (aCC)
28: Using PCRE from MySQL
29: Making new tarballs
30: Testing PCRE
31: Character tables
32: File manifest
33:
34:
35: The PCRE APIs
36: -------------
37:
38: PCRE is written in C, and it has its own API. There are two sets of functions,
39: one for the 8-bit library, which processes strings of bytes, and one for the
40: 16-bit library, which processes strings of 16-bit values. The distribution also
41: includes a set of C++ wrapper functions (see the pcrecpp man page for details),
42: courtesy of Google Inc., which can be used to call the 8-bit PCRE library from
43: C++.
44:
45: In addition, there is a set of C wrapper functions (again, just for the 8-bit
46: library) that are based on the POSIX regular expression API (see the pcreposix
47: man page). These end up in the library called libpcreposix. Note that this just
48: provides a POSIX calling interface to PCRE; the regular expressions themselves
49: still follow Perl syntax and semantics. The POSIX API is restricted, and does
50: not give full access to all of PCRE's facilities.
51:
52: The header file for the POSIX-style functions is called pcreposix.h. The
53: official POSIX name is regex.h, but I did not want to risk possible problems
54: with existing files of that name by distributing it that way. To use PCRE with
55: an existing program that uses the POSIX API, pcreposix.h will have to be
56: renamed or pointed at by a link.
57:
58: If you are using the POSIX interface to PCRE and there is already a POSIX regex
59: library installed on your system, as well as worrying about the regex.h header
60: file (as mentioned above), you must also take care when linking programs to
61: ensure that they link with PCRE's libpcreposix library. Otherwise they may pick
62: up the POSIX functions of the same name from the other library.
63:
64: One way of avoiding this confusion is to compile PCRE with the addition of
65: -Dregcomp=PCREregcomp (and similarly for the other POSIX functions) to the
66: compiler flags (CFLAGS if you are using "configure" -- see below). This has the
67: effect of renaming the functions so that the names no longer clash. Of course,
68: you have to do the same thing for your applications, or write them using the
69: new names.
70:
71:
72: Documentation for PCRE
73: ----------------------
74:
75: If you install PCRE in the normal way on a Unix-like system, you will end up
76: with a set of man pages whose names all start with "pcre". The one that is just
77: called "pcre" lists all the others. In addition to these man pages, the PCRE
78: documentation is supplied in two other forms:
79:
80: 1. There are files called doc/pcre.txt, doc/pcregrep.txt, and
81: doc/pcretest.txt in the source distribution. The first of these is a
82: concatenation of the text forms of all the section 3 man pages except
83: those that summarize individual functions. The other two are the text
84: forms of the section 1 man pages for the pcregrep and pcretest commands.
85: These text forms are provided for ease of scanning with text editors or
86: similar tools. They are installed in <prefix>/share/doc/pcre, where
87: <prefix> is the installation prefix (defaulting to /usr/local).
88:
89: 2. A set of files containing all the documentation in HTML form, hyperlinked
90: in various ways, and rooted in a file called index.html, is distributed in
91: doc/html and installed in <prefix>/share/doc/pcre/html.
92:
93: Users of PCRE have contributed files containing the documentation for various
94: releases in CHM format. These can be found in the Contrib directory of the FTP
95: site (see next section).
96:
97:
98: Contributions by users of PCRE
99: ------------------------------
100:
101: You can find contributions from PCRE users in the directory
102:
103: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/Contrib
104:
105: There is a README file giving brief descriptions of what they are. Some are
106: complete in themselves; others are pointers to URLs containing relevant files.
107: Some of this material is likely to be well out-of-date. Several of the earlier
108: contributions provided support for compiling PCRE on various flavours of
109: Windows (I myself do not use Windows). Nowadays there is more Windows support
110: in the standard distribution, so these contibutions have been archived.
111:
112:
113: Building PCRE on non-Unix-like systems
114: --------------------------------------
115:
116: For a non-Unix-like system, please read the comments in the file
117: NON-AUTOTOOLS-BUILD, though if your system supports the use of "configure" and
118: "make" you may be able to build PCRE using autotools in the same way as for
119: many Unix-like systems.
120:
121: PCRE can also be configured using the GUI facility provided by CMake's
122: cmake-gui command. This creates Makefiles, solution files, etc. The file
123: NON-AUTOTOOLS-BUILD has information about CMake.
124:
125: PCRE has been compiled on many different operating systems. It should be
126: straightforward to build PCRE on any system that has a Standard C compiler and
127: library, because it uses only Standard C functions.
128:
129:
130: Building PCRE without using autotools
131: -------------------------------------
132:
133: The use of autotools (in particular, libtool) is problematic in some
134: environments, even some that are Unix or Unix-like. See the NON-AUTOTOOLS-BUILD
135: file for ways of building PCRE without using autotools.
136:
137:
138: Building PCRE using autotools
139: -----------------------------
140:
141: If you are using HP's ANSI C++ compiler (aCC), please see the special note
142: in the section entitled "Using HP's ANSI C++ compiler (aCC)" below.
143:
144: The following instructions assume the use of the widely used "configure; make;
145: make install" (autotools) process.
146:
147: To build PCRE on system that supports autotools, first run the "configure"
148: command from the PCRE distribution directory, with your current directory set
149: to the directory where you want the files to be created. This command is a
150: standard GNU "autoconf" configuration script, for which generic instructions
151: are supplied in the file INSTALL.
152:
153: Most commonly, people build PCRE within its own distribution directory, and in
154: this case, on many systems, just running "./configure" is sufficient. However,
155: the usual methods of changing standard defaults are available. For example:
156:
157: CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local
158:
159: This command specifies that the C compiler should be run with the flags '-O2
160: -Wall' instead of the default, and that "make install" should install PCRE
161: under /opt/local instead of the default /usr/local.
162:
163: If you want to build in a different directory, just run "configure" with that
164: directory as current. For example, suppose you have unpacked the PCRE source
165: into /source/pcre/pcre-xxx, but you want to build it in /build/pcre/pcre-xxx:
166:
167: cd /build/pcre/pcre-xxx
168: /source/pcre/pcre-xxx/configure
169:
170: PCRE is written in C and is normally compiled as a C library. However, it is
171: possible to build it as a C++ library, though the provided building apparatus
172: does not have any features to support this.
173:
174: There are some optional features that can be included or omitted from the PCRE
175: library. They are also documented in the pcrebuild man page.
176:
177: . By default, both shared and static libraries are built. You can change this
178: by adding one of these options to the "configure" command:
179:
180: --disable-shared
181: --disable-static
182:
183: (See also "Shared libraries on Unix-like systems" below.)
184:
185: . By default, only the 8-bit library is built. If you add --enable-pcre16 to
186: the "configure" command, the 16-bit library is also built. If you want only
187: the 16-bit library, use "./configure --enable-pcre16 --disable-pcre8".
188:
189: . If you are building the 8-bit library and want to suppress the building of
190: the C++ wrapper library, you can add --disable-cpp to the "configure"
191: command. Otherwise, when "configure" is run without --disable-pcre8, it will
192: try to find a C++ compiler and C++ header files, and if it succeeds, it will
193: try to build the C++ wrapper.
194:
195: . If you want to include support for just-in-time compiling, which can give
196: large performance improvements on certain platforms, add --enable-jit to the
197: "configure" command. This support is available only for certain hardware
198: architectures. If you try to enable it on an unsupported architecture, there
199: will be a compile time error.
200:
201: . When JIT support is enabled, pcregrep automatically makes use of it, unless
202: you add --disable-pcregrep-jit to the "configure" command.
203:
204: . If you want to make use of the support for UTF-8 Unicode character strings in
205: the 8-bit library, or UTF-16 Unicode character strings in the 16-bit library,
206: you must add --enable-utf to the "configure" command. Without it, the code
207: for handling UTF-8 and UTF-16 is not included in the relevant library. Even
208: when --enable-utf is included, the use of a UTF encoding still has to be
209: enabled by an option at run time. When PCRE is compiled with this option, its
210: input can only either be ASCII or UTF-8/16, even when running on EBCDIC
211: platforms. It is not possible to use both --enable-utf and --enable-ebcdic at
212: the same time.
213:
214: . There are no separate options for enabling UTF-8 and UTF-16 independently
215: because that would allow ridiculous settings such as requesting UTF-16
216: support while building only the 8-bit library. However, the option
217: --enable-utf8 is retained for backwards compatibility with earlier releases
218: that did not support 16-bit character strings. It is synonymous with
219: --enable-utf. It is not possible to configure one library with UTF support
220: and the other without in the same configuration.
221:
222: . If, in addition to support for UTF-8/16 character strings, you want to
223: include support for the \P, \p, and \X sequences that recognize Unicode
224: character properties, you must add --enable-unicode-properties to the
225: "configure" command. This adds about 30K to the size of the library (in the
226: form of a property table); only the basic two-letter properties such as Lu
227: are supported.
228:
229: . You can build PCRE to recognize either CR or LF or the sequence CRLF or any
230: of the preceding, or any of the Unicode newline sequences as indicating the
231: end of a line. Whatever you specify at build time is the default; the caller
232: of PCRE can change the selection at run time. The default newline indicator
233: is a single LF character (the Unix standard). You can specify the default
234: newline indicator by adding --enable-newline-is-cr or --enable-newline-is-lf
235: or --enable-newline-is-crlf or --enable-newline-is-anycrlf or
236: --enable-newline-is-any to the "configure" command, respectively.
237:
238: If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of
239: the standard tests will fail, because the lines in the test files end with
240: LF. Even if the files are edited to change the line endings, there are likely
241: to be some failures. With --enable-newline-is-anycrlf or
242: --enable-newline-is-any, many tests should succeed, but there may be some
243: failures.
244:
245: . By default, the sequence \R in a pattern matches any Unicode line ending
246: sequence. This is independent of the option specifying what PCRE considers to
247: be the end of a line (see above). However, the caller of PCRE can restrict \R
248: to match only CR, LF, or CRLF. You can make this the default by adding
249: --enable-bsr-anycrlf to the "configure" command (bsr = "backslash R").
250:
251: . When called via the POSIX interface, PCRE uses malloc() to get additional
252: storage for processing capturing parentheses if there are more than 10 of
253: them in a pattern. You can increase this threshold by setting, for example,
254:
255: --with-posix-malloc-threshold=20
256:
257: on the "configure" command.
258:
259: . PCRE has a counter that can be set to limit the amount of resources it uses.
260: If the limit is exceeded during a match, the match fails. The default is ten
261: million. You can change the default by setting, for example,
262:
263: --with-match-limit=500000
264:
265: on the "configure" command. This is just the default; individual calls to
266: pcre_exec() can supply their own value. There is more discussion on the
267: pcreapi man page.
268:
269: . There is a separate counter that limits the depth of recursive function calls
270: during a matching process. This also has a default of ten million, which is
271: essentially "unlimited". You can change the default by setting, for example,
272:
273: --with-match-limit-recursion=500000
274:
275: Recursive function calls use up the runtime stack; running out of stack can
276: cause programs to crash in strange ways. There is a discussion about stack
277: sizes in the pcrestack man page.
278:
279: . The default maximum compiled pattern size is around 64K. You can increase
280: this by adding --with-link-size=3 to the "configure" command. In the 8-bit
281: library, PCRE then uses three bytes instead of two for offsets to different
282: parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is
283: the same as --with-link-size=4, which (in both libraries) uses four-byte
284: offsets. Increasing the internal link size reduces performance.
285:
286: . You can build PCRE so that its internal match() function that is called from
287: pcre_exec() does not call itself recursively. Instead, it uses memory blocks
288: obtained from the heap via the special functions pcre_stack_malloc() and
289: pcre_stack_free() to save data that would otherwise be saved on the stack. To
290: build PCRE like this, use
291:
292: --disable-stack-for-recursion
293:
294: on the "configure" command. PCRE runs more slowly in this mode, but it may be
295: necessary in environments with limited stack sizes. This applies only to the
296: normal execution of the pcre_exec() function; if JIT support is being
297: successfully used, it is not relevant. Equally, it does not apply to
298: pcre_dfa_exec(), which does not use deeply nested recursion. There is a
299: discussion about stack sizes in the pcrestack man page.
300:
301: . For speed, PCRE uses four tables for manipulating and identifying characters
302: whose code point values are less than 256. By default, it uses a set of
303: tables for ASCII encoding that is part of the distribution. If you specify
304:
305: --enable-rebuild-chartables
306:
307: a program called dftables is compiled and run in the default C locale when
308: you obey "make". It builds a source file called pcre_chartables.c. If you do
309: not specify this option, pcre_chartables.c is created as a copy of
310: pcre_chartables.c.dist. See "Character tables" below for further information.
311:
312: . It is possible to compile PCRE for use on systems that use EBCDIC as their
313: character code (as opposed to ASCII) by specifying
314:
315: --enable-ebcdic
316:
317: This automatically implies --enable-rebuild-chartables (see above). However,
318: when PCRE is built this way, it always operates in EBCDIC. It cannot support
319: both EBCDIC and UTF-8/16.
320:
321: . The pcregrep program currently supports only 8-bit data files, and so
322: requires the 8-bit PCRE library. It is possible to compile pcregrep to use
323: libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by
324: specifying one or both of
325:
326: --enable-pcregrep-libz
327: --enable-pcregrep-libbz2
328:
329: Of course, the relevant libraries must be installed on your system.
330:
331: . The default size of internal buffer used by pcregrep can be set by, for
332: example:
333:
334: --with-pcregrep-bufsize=50K
335:
336: The default value is 20K.
337:
338: . It is possible to compile pcretest so that it links with the libreadline
339: or libedit libraries, by specifying, respectively,
340:
341: --enable-pcretest-libreadline or --enable-pcretest-libedit
342:
343: If this is done, when pcretest's input is from a terminal, it reads it using
344: the readline() function. This provides line-editing and history facilities.
345: Note that libreadline is GPL-licenced, so if you distribute a binary of
346: pcretest linked in this way, there may be licensing issues. These can be
347: avoided by linking with libedit (which has a BSD licence) instead.
348:
349: Enabling libreadline causes the -lreadline option to be added to the pcretest
350: build. In many operating environments with a sytem-installed readline
351: library this is sufficient. However, in some environments (e.g. if an
352: unmodified distribution version of readline is in use), it may be necessary
353: to specify something like LIBS="-lncurses" as well. This is because, to quote
354: the readline INSTALL, "Readline uses the termcap functions, but does not link
355: with the termcap or curses library itself, allowing applications which link
356: with readline the to choose an appropriate library." If you get error
357: messages about missing functions tgetstr, tgetent, tputs, tgetflag, or tgoto,
358: this is the problem, and linking with the ncurses library should fix it.
359:
360: The "configure" script builds the following files for the basic C library:
361:
362: . Makefile the makefile that builds the library
363: . config.h build-time configuration options for the library
364: . pcre.h the public PCRE header file
365: . pcre-config script that shows the building settings such as CFLAGS
366: that were set for "configure"
367: . libpcre.pc ) data for the pkg-config command
368: . libpcre16.pc )
369: . libpcreposix.pc )
370: . libtool script that builds shared and/or static libraries
371:
372: Versions of config.h and pcre.h are distributed in the PCRE tarballs under the
373: names config.h.generic and pcre.h.generic. These are provided for those who
374: have to built PCRE without using "configure" or CMake. If you use "configure"
375: or CMake, the .generic versions are not used.
376:
377: When building the 8-bit library, if a C++ compiler is found, the following
378: files are also built:
379:
380: . libpcrecpp.pc data for the pkg-config command
381: . pcrecpparg.h header file for calling PCRE via the C++ wrapper
382: . pcre_stringpiece.h header for the C++ "stringpiece" functions
383:
384: The "configure" script also creates config.status, which is an executable
385: script that can be run to recreate the configuration, and config.log, which
386: contains compiler output from tests that "configure" runs.
387:
388: Once "configure" has run, you can run "make". This builds either or both of the
389: libraries libpcre and libpcre16, and a test program called pcretest. If you
390: enabled JIT support with --enable-jit, a test program called pcre_jit_test is
391: built as well.
392:
393: If the 8-bit library is built, libpcreposix and the pcregrep command are also
394: built, and if a C++ compiler was found on your system, and you did not disable
395: it with --disable-cpp, "make" builds the C++ wrapper library, which is called
396: libpcrecpp, as well as some test programs called pcrecpp_unittest,
397: pcre_scanner_unittest, and pcre_stringpiece_unittest.
398:
399: The command "make check" runs all the appropriate tests. Details of the PCRE
400: tests are given below in a separate section of this document.
401:
402: You can use "make install" to install PCRE into live directories on your
403: system. The following are installed (file names are all relative to the
404: <prefix> that is set when "configure" is run):
405:
406: Commands (bin):
407: pcretest
408: pcregrep (if 8-bit support is enabled)
409: pcre-config
410:
411: Libraries (lib):
412: libpcre16 (if 16-bit support is enabled)
413: libpcre (if 8-bit support is enabled)
414: libpcreposix (if 8-bit support is enabled)
415: libpcrecpp (if 8-bit and C++ support is enabled)
416:
417: Configuration information (lib/pkgconfig):
418: libpcre16.pc
419: libpcre.pc
420: libpcreposix.pc
421: libpcrecpp.pc (if C++ support is enabled)
422:
423: Header files (include):
424: pcre.h
425: pcreposix.h
426: pcre_scanner.h )
427: pcre_stringpiece.h ) if C++ support is enabled
428: pcrecpp.h )
429: pcrecpparg.h )
430:
431: Man pages (share/man/man{1,3}):
432: pcregrep.1
433: pcretest.1
434: pcre-config.1
435: pcre.3
436: pcre*.3 (lots more pages, all starting "pcre")
437:
438: HTML documentation (share/doc/pcre/html):
439: index.html
440: *.html (lots more pages, hyperlinked from index.html)
441:
442: Text file documentation (share/doc/pcre):
443: AUTHORS
444: COPYING
445: ChangeLog
446: LICENCE
447: NEWS
448: README
449: pcre.txt (a concatenation of the man(3) pages)
450: pcretest.txt the pcretest man page
451: pcregrep.txt the pcregrep man page
452: pcre-config.txt the pcre-config man page
453:
454: If you want to remove PCRE from your system, you can run "make uninstall".
455: This removes all the files that "make install" installed. However, it does not
456: remove any directories, because these are often shared with other programs.
457:
458:
459: Retrieving configuration information
460: ------------------------------------
461:
462: Running "make install" installs the command pcre-config, which can be used to
463: recall information about the PCRE configuration and installation. For example:
464:
465: pcre-config --version
466:
467: prints the version number, and
468:
469: pcre-config --libs
470:
471: outputs information about where the library is installed. This command can be
472: included in makefiles for programs that use PCRE, saving the programmer from
473: having to remember too many details.
474:
475: The pkg-config command is another system for saving and retrieving information
476: about installed libraries. Instead of separate commands for each library, a
477: single command is used. For example:
478:
479: pkg-config --cflags pcre
480:
481: The data is held in *.pc files that are installed in a directory called
482: <prefix>/lib/pkgconfig.
483:
484:
485: Shared libraries
486: ----------------
487:
488: The default distribution builds PCRE as shared libraries and static libraries,
489: as long as the operating system supports shared libraries. Shared library
490: support relies on the "libtool" script which is built as part of the
491: "configure" process.
492:
493: The libtool script is used to compile and link both shared and static
494: libraries. They are placed in a subdirectory called .libs when they are newly
495: built. The programs pcretest and pcregrep are built to use these uninstalled
496: libraries (by means of wrapper scripts in the case of shared libraries). When
497: you use "make install" to install shared libraries, pcregrep and pcretest are
498: automatically re-built to use the newly installed shared libraries before being
499: installed themselves. However, the versions left in the build directory still
500: use the uninstalled libraries.
501:
502: To build PCRE using static libraries only you must use --disable-shared when
503: configuring it. For example:
504:
505: ./configure --prefix=/usr/gnu --disable-shared
506:
507: Then run "make" in the usual way. Similarly, you can use --disable-static to
508: build only shared libraries.
509:
510:
511: Cross-compiling using autotools
512: -------------------------------
513:
514: You can specify CC and CFLAGS in the normal way to the "configure" command, in
515: order to cross-compile PCRE for some other host. However, you should NOT
516: specify --enable-rebuild-chartables, because if you do, the dftables.c source
517: file is compiled and run on the local host, in order to generate the inbuilt
518: character tables (the pcre_chartables.c file). This will probably not work,
519: because dftables.c needs to be compiled with the local compiler, not the cross
520: compiler.
521:
522: When --enable-rebuild-chartables is not specified, pcre_chartables.c is created
523: by making a copy of pcre_chartables.c.dist, which is a default set of tables
524: that assumes ASCII code. Cross-compiling with the default tables should not be
525: a problem.
526:
527: If you need to modify the character tables when cross-compiling, you should
528: move pcre_chartables.c.dist out of the way, then compile dftables.c by hand and
529: run it on the local host to make a new version of pcre_chartables.c.dist.
530: Then when you cross-compile PCRE this new version of the tables will be used.
531:
532:
533: Using HP's ANSI C++ compiler (aCC)
534: ----------------------------------
535:
536: Unless C++ support is disabled by specifying the "--disable-cpp" option of the
537: "configure" script, you must include the "-AA" option in the CXXFLAGS
538: environment variable in order for the C++ components to compile correctly.
539:
540: Also, note that the aCC compiler on PA-RISC platforms may have a defect whereby
541: needed libraries fail to get included when specifying the "-AA" compiler
542: option. If you experience unresolved symbols when linking the C++ programs,
543: use the workaround of specifying the following environment variable prior to
544: running the "configure" script:
545:
546: CXXLDFLAGS="-lstd_v2 -lCsup_v2"
547:
548:
549: Using Sun's compilers for Solaris
550: ---------------------------------
551:
552: A user reports that the following configurations work on Solaris 9 sparcv9 and
553: Solaris 9 x86 (32-bit):
554:
555: Solaris 9 sparcv9: ./configure --disable-cpp CC=/bin/cc CFLAGS="-m64 -g"
556: Solaris 9 x86: ./configure --disable-cpp CC=/bin/cc CFLAGS="-g"
557:
558:
559: Using PCRE from MySQL
560: ---------------------
561:
562: On systems where both PCRE and MySQL are installed, it is possible to make use
563: of PCRE from within MySQL, as an alternative to the built-in pattern matching.
564: There is a web page that tells you how to do this:
565:
566: http://www.mysqludf.org/lib_mysqludf_preg/index.php
567:
568:
569: Making new tarballs
570: -------------------
571:
572: The command "make dist" creates three PCRE tarballs, in tar.gz, tar.bz2, and
573: zip formats. The command "make distcheck" does the same, but then does a trial
574: build of the new distribution to ensure that it works.
575:
576: If you have modified any of the man page sources in the doc directory, you
577: should first run the PrepareRelease script before making a distribution. This
578: script creates the .txt and HTML forms of the documentation from the man pages.
579:
580:
581: Testing PCRE
582: ------------
583:
584: To test the basic PCRE library on a Unix-like system, run the RunTest script.
585: There is another script called RunGrepTest that tests the options of the
586: pcregrep command. If the C++ wrapper library is built, three test programs
587: called pcrecpp_unittest, pcre_scanner_unittest, and pcre_stringpiece_unittest
588: are also built. When JIT support is enabled, another test program called
589: pcre_jit_test is built.
590:
591: Both the scripts and all the program tests are run if you obey "make check" or
592: "make test". For other environments, see the instructions in
593: NON-AUTOTOOLS-BUILD.
594:
595: The RunTest script runs the pcretest test program (which is documented in its
596: own man page) on each of the relevant testinput files in the testdata
597: directory, and compares the output with the contents of the corresponding
598: testoutput files. Some tests are relevant only when certain build-time options
599: were selected. For example, the tests for UTF-8/16 support are run only if
600: --enable-utf was used. RunTest outputs a comment when it skips a test.
601:
602: Many of the tests that are not skipped are run up to three times. The second
603: run forces pcre_study() to be called for all patterns except for a few in some
604: tests that are marked "never study" (see the pcretest program for how this is
605: done). If JIT support is available, the non-DFA tests are run a third time,
606: this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.
607:
608: When both 8-bit and 16-bit support is enabled, the entire set of tests is run
609: twice, once for each library. If you want to run just one set of tests, call
610: RunTest with either the -8 or -16 option.
611:
612: RunTest uses a file called testtry to hold the main output from pcretest.
613: Other files whose names begin with "test" are used as working files in some
614: tests. To run pcretest on just one or more specific test files, give their
615: numbers as arguments to RunTest, for example:
616:
617: RunTest 2 7 11
618:
619: You can also call RunTest with the single argument "list" to cause it to output
620: a list of tests.
621:
622: The first test file can be fed directly into the perltest.pl script to check
623: that Perl gives the same results. The only difference you should see is in the
624: first few lines, where the Perl version is given instead of the PCRE version.
625:
626: The second set of tests check pcre_fullinfo(), pcre_study(),
627: pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error
628: detection, and run-time flags that are specific to PCRE, as well as the POSIX
629: wrapper API. It also uses the debugging flags to check some of the internals of
630: pcre_compile().
631:
632: If you build PCRE with a locale setting that is not the standard C locale, the
633: character tables may be different (see next paragraph). In some cases, this may
634: cause failures in the second set of tests. For example, in a locale where the
635: isprint() function yields TRUE for characters in the range 128-255, the use of
636: [:isascii:] inside a character class defines a different set of characters, and
637: this shows up in this test as a difference in the compiled code, which is being
638: listed for checking. Where the comparison test output contains [\x00-\x7f] the
639: test will contain [\x00-\xff], and similarly in some other cases. This is not a
640: bug in PCRE.
641:
642: The third set of tests checks pcre_maketables(), the facility for building a
643: set of character tables for a specific locale and using them instead of the
644: default tables. The tests make use of the "fr_FR" (French) locale. Before
645: running the test, the script checks for the presence of this locale by running
646: the "locale" command. If that command fails, or if it doesn't include "fr_FR"
647: in the list of available locales, the third test cannot be run, and a comment
648: is output to say why. If running this test produces instances of the error
649:
650: ** Failed to set locale "fr_FR"
651:
652: in the comparison output, it means that locale is not available on your system,
653: despite being listed by "locale". This does not mean that PCRE is broken.
654:
655: [If you are trying to run this test on Windows, you may be able to get it to
656: work by changing "fr_FR" to "french" everywhere it occurs. Alternatively, use
657: RunTest.bat. The version of RunTest.bat included with PCRE 7.4 and above uses
658: Windows versions of test 2. More info on using RunTest.bat is included in the
659: document entitled NON-UNIX-USE.]
660:
661: The fourth and fifth tests check the UTF-8/16 support and error handling and
662: internal UTF features of PCRE that are not relevant to Perl, respectively. The
663: sixth and seventh tests do the same for Unicode character properties support.
664:
665: The eighth, ninth, and tenth tests check the pcre_dfa_exec() alternative
666: matching function, in non-UTF-8/16 mode, UTF-8/16 mode, and UTF-8/16 mode with
667: Unicode property support, respectively.
668:
669: The eleventh test checks some internal offsets and code size features; it is
670: run only when the default "link size" of 2 is set (in other cases the sizes
671: change) and when Unicode property support is enabled.
672:
673: The twelfth test is run only when JIT support is available, and the thirteenth
674: test is run only when JIT support is not available. They test some JIT-specific
675: features such as information output from pcretest about JIT compilation.
676:
677: The fourteenth, fifteenth, and sixteenth tests are run only in 8-bit mode, and
678: the seventeenth, eighteenth, and nineteenth tests are run only in 16-bit mode.
679: These are tests that generate different output in the two modes. They are for
680: general cases, UTF-8/16 support, and Unicode property support, respectively.
681:
682: The twentieth test is run only in 16-bit mode. It tests some specific 16-bit
683: features of the DFA matching engine.
684:
685: The twenty-first and twenty-second tests are run only in 16-bit mode, when the
686: link size is set to 2. They test reloading pre-compiled patterns.
687:
688:
689: Character tables
690: ----------------
691:
692: For speed, PCRE uses four tables for manipulating and identifying characters
693: whose code point values are less than 256. The final argument of the
694: pcre_compile() function is a pointer to a block of memory containing the
695: concatenated tables. A call to pcre_maketables() can be used to generate a set
696: of tables in the current locale. If the final argument for pcre_compile() is
697: passed as NULL, a set of default tables that is built into the binary is used.
698:
699: The source file called pcre_chartables.c contains the default set of tables. By
700: default, this is created as a copy of pcre_chartables.c.dist, which contains
701: tables for ASCII coding. However, if --enable-rebuild-chartables is specified
702: for ./configure, a different version of pcre_chartables.c is built by the
703: program dftables (compiled from dftables.c), which uses the ANSI C character
704: handling functions such as isalnum(), isalpha(), isupper(), islower(), etc. to
705: build the table sources. This means that the default C locale which is set for
706: your system will control the contents of these default tables. You can change
707: the default tables by editing pcre_chartables.c and then re-building PCRE. If
708: you do this, you should take care to ensure that the file does not get
709: automatically re-generated. The best way to do this is to move
710: pcre_chartables.c.dist out of the way and replace it with your customized
711: tables.
712:
713: When the dftables program is run as a result of --enable-rebuild-chartables,
714: it uses the default C locale that is set on your system. It does not pay
715: attention to the LC_xxx environment variables. In other words, it uses the
716: system's default locale rather than whatever the compiling user happens to have
717: set. If you really do want to build a source set of character tables in a
718: locale that is specified by the LC_xxx variables, you can run the dftables
719: program by hand with the -L option. For example:
720:
721: ./dftables -L pcre_chartables.c.special
722:
723: The first two 256-byte tables provide lower casing and case flipping functions,
724: respectively. The next table consists of three 32-byte bit maps which identify
725: digits, "word" characters, and white space, respectively. These are used when
726: building 32-byte bit maps that represent character classes for code points less
727: than 256.
728:
729: The final 256-byte table has bits indicating various character types, as
730: follows:
731:
732: 1 white space character
733: 2 letter
734: 4 decimal digit
735: 8 hexadecimal digit
736: 16 alphanumeric or '_'
737: 128 regular expression metacharacter or binary zero
738:
739: You should not alter the set of characters that contain the 128 bit, as that
740: will cause PCRE to malfunction.
741:
742:
743: File manifest
744: -------------
745:
746: The distribution should contain the files listed below. Where a file name is
747: given as pcre[16]_xxx it means that there are two files, one with the name
748: pcre_xxx and the other with the name pcre16_xxx.
749:
750: (A) Source files of the PCRE library functions and their headers:
751:
752: dftables.c auxiliary program for building pcre_chartables.c
753: when --enable-rebuild-chartables is specified
754:
755: pcre_chartables.c.dist a default set of character tables that assume ASCII
756: coding; used, unless --enable-rebuild-chartables is
757: specified, by copying to pcre[16]_chartables.c
758:
759: pcreposix.c )
760: pcre[16]_byte_order.c )
761: pcre[16]_compile.c )
762: pcre[16]_config.c )
763: pcre[16]_dfa_exec.c )
764: pcre[16]_exec.c )
765: pcre[16]_fullinfo.c )
766: pcre[16]_get.c ) sources for the functions in the library,
767: pcre[16]_globals.c ) and some internal functions that they use
768: pcre[16]_jit_compile.c )
769: pcre[16]_maketables.c )
770: pcre[16]_newline.c )
771: pcre[16]_refcount.c )
772: pcre[16]_string_utils.c )
773: pcre[16]_study.c )
774: pcre[16]_tables.c )
775: pcre[16]_ucd.c )
776: pcre[16]_version.c )
777: pcre[16]_xclass.c )
778: pcre_ord2utf8.c )
779: pcre_valid_utf8.c )
780: pcre16_ord2utf16.c )
781: pcre16_utf16_utils.c )
782: pcre16_valid_utf16.c )
783:
784: pcre[16]_printint.c ) debugging function that is used by pcretest,
785: ) and can also be #included in pcre_compile()
786:
787: pcre.h.in template for pcre.h when built by "configure"
788: pcreposix.h header for the external POSIX wrapper API
789: pcre_internal.h header for internal use
790: sljit/* 16 files that make up the JIT compiler
791: ucp.h header for Unicode property handling
792:
793: config.h.in template for config.h, which is built by "configure"
794:
795: pcrecpp.h public header file for the C++ wrapper
796: pcrecpparg.h.in template for another C++ header file
797: pcre_scanner.h public header file for C++ scanner functions
798: pcrecpp.cc )
799: pcre_scanner.cc ) source for the C++ wrapper library
800:
801: pcre_stringpiece.h.in template for pcre_stringpiece.h, the header for the
802: C++ stringpiece functions
803: pcre_stringpiece.cc source for the C++ stringpiece functions
804:
805: (B) Source files for programs that use PCRE:
806:
807: pcredemo.c simple demonstration of coding calls to PCRE
808: pcregrep.c source of a grep utility that uses PCRE
809: pcretest.c comprehensive test program
810:
811: (C) Auxiliary files:
812:
813: 132html script to turn "man" pages into HTML
814: AUTHORS information about the author of PCRE
815: ChangeLog log of changes to the code
816: CleanTxt script to clean nroff output for txt man pages
817: Detrail script to remove trailing spaces
818: HACKING some notes about the internals of PCRE
819: INSTALL generic installation instructions
820: LICENCE conditions for the use of PCRE
821: COPYING the same, using GNU's standard name
822: Makefile.in ) template for Unix Makefile, which is built by
823: ) "configure"
824: Makefile.am ) the automake input that was used to create
825: ) Makefile.in
826: NEWS important changes in this release
827: NON-UNIX-USE the previous name for NON-AUTOTOOLS-BUILD
828: NON-AUTOTOOLS-BUILD notes on building PCRE without using autotools
829: PrepareRelease script to make preparations for "make dist"
830: README this file
831: RunTest a Unix shell script for running tests
832: RunGrepTest a Unix shell script for pcregrep tests
833: aclocal.m4 m4 macros (generated by "aclocal")
834: config.guess ) files used by libtool,
835: config.sub ) used only when building a shared library
836: configure a configuring shell script (built by autoconf)
837: configure.ac ) the autoconf input that was used to build
838: ) "configure" and config.h
839: depcomp ) script to find program dependencies, generated by
840: ) automake
841: doc/*.3 man page sources for PCRE
842: doc/*.1 man page sources for pcregrep and pcretest
843: doc/index.html.src the base HTML page
844: doc/html/* HTML documentation
845: doc/pcre.txt plain text version of the man pages
846: doc/pcretest.txt plain text documentation of test program
847: doc/perltest.txt plain text documentation of Perl test program
848: install-sh a shell script for installing files
849: libpcre16.pc.in template for libpcre16.pc for pkg-config
850: libpcre.pc.in template for libpcre.pc for pkg-config
851: libpcreposix.pc.in template for libpcreposix.pc for pkg-config
852: libpcrecpp.pc.in template for libpcrecpp.pc for pkg-config
853: ltmain.sh file used to build a libtool script
854: missing ) common stub for a few missing GNU programs while
855: ) installing, generated by automake
856: mkinstalldirs script for making install directories
857: perltest.pl Perl test program
858: pcre-config.in source of script which retains PCRE information
859: pcre_jit_test.c test program for the JIT compiler
860: pcrecpp_unittest.cc )
861: pcre_scanner_unittest.cc ) test programs for the C++ wrapper
862: pcre_stringpiece_unittest.cc )
863: testdata/testinput* test data for main library tests
864: testdata/testoutput* expected test results
865: testdata/grep* input and output for pcregrep tests
866: testdata/* other supporting test files
867:
868: (D) Auxiliary files for cmake support
869:
870: cmake/COPYING-CMAKE-SCRIPTS
871: cmake/FindPackageHandleStandardArgs.cmake
872: cmake/FindEditline.cmake
873: cmake/FindReadline.cmake
874: CMakeLists.txt
875: config-cmake.h.in
876:
877: (E) Auxiliary files for VPASCAL
878:
879: makevp.bat
880: makevp_c.txt
881: makevp_l.txt
882: pcregexp.pas
883:
884: (F) Auxiliary files for building PCRE "by hand"
885:
886: pcre.h.generic ) a version of the public PCRE header file
887: ) for use in non-"configure" environments
888: config.h.generic ) a version of config.h for use in non-"configure"
889: ) environments
890:
891: (F) Miscellaneous
892:
893: RunTest.bat a script for running tests under Windows
894:
895: Philip Hazel
896: Email local part: ph10
897: Email domain: cam.ac.uk
898: Last updated: 18 June 2012
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre