Annotation of embedaddon/pcre/README, revision 1.1.1.2
1.1 misho 1: README file for PCRE (Perl-compatible regular expression library)
2: -----------------------------------------------------------------
3:
4: The latest release of PCRE is always available in three alternative formats
5: from:
6:
7: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.gz
8: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.tar.bz2
9: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-xxx.zip
10:
11: There is a mailing list for discussion about the development of PCRE at
12:
13: pcre-dev@exim.org
14:
15: Please read the NEWS file if you are upgrading from a previous release.
16: The contents of this README file are:
17:
18: The PCRE APIs
19: Documentation for PCRE
20: Contributions by users of PCRE
21: Building PCRE on non-Unix systems
22: Building PCRE on Unix-like systems
23: Retrieving configuration information on Unix-like systems
24: Shared libraries on Unix-like systems
25: Cross-compiling on Unix-like systems
26: Using HP's ANSI C++ compiler (aCC)
27: Using PCRE from MySQL
28: Making new tarballs
29: Testing PCRE
30: Character tables
31: File manifest
32:
33:
34: The PCRE APIs
35: -------------
36:
1.1.1.2 ! misho 37: PCRE is written in C, and it has its own API. There are two sets of functions,
! 38: one for the 8-bit library, which processes strings of bytes, and one for the
! 39: 16-bit library, which processes strings of 16-bit values. The distribution also
! 40: includes a set of C++ wrapper functions (see the pcrecpp man page for details),
! 41: courtesy of Google Inc., which can be used to call the 8-bit PCRE library from
! 42: C++.
! 43:
! 44: In addition, there is a set of C wrapper functions (again, just for the 8-bit
! 45: library) that are based on the POSIX regular expression API (see the pcreposix
! 46: man page). These end up in the library called libpcreposix. Note that this just
! 47: provides a POSIX calling interface to PCRE; the regular expressions themselves
! 48: still follow Perl syntax and semantics. The POSIX API is restricted, and does
! 49: not give full access to all of PCRE's facilities.
1.1 misho 50:
51: The header file for the POSIX-style functions is called pcreposix.h. The
52: official POSIX name is regex.h, but I did not want to risk possible problems
53: with existing files of that name by distributing it that way. To use PCRE with
54: an existing program that uses the POSIX API, pcreposix.h will have to be
55: renamed or pointed at by a link.
56:
57: If you are using the POSIX interface to PCRE and there is already a POSIX regex
58: library installed on your system, as well as worrying about the regex.h header
59: file (as mentioned above), you must also take care when linking programs to
60: ensure that they link with PCRE's libpcreposix library. Otherwise they may pick
61: up the POSIX functions of the same name from the other library.
62:
63: One way of avoiding this confusion is to compile PCRE with the addition of
64: -Dregcomp=PCREregcomp (and similarly for the other POSIX functions) to the
65: compiler flags (CFLAGS if you are using "configure" -- see below). This has the
66: effect of renaming the functions so that the names no longer clash. Of course,
67: you have to do the same thing for your applications, or write them using the
68: new names.
69:
70:
71: Documentation for PCRE
72: ----------------------
73:
74: If you install PCRE in the normal way on a Unix-like system, you will end up
75: with a set of man pages whose names all start with "pcre". The one that is just
76: called "pcre" lists all the others. In addition to these man pages, the PCRE
77: documentation is supplied in two other forms:
78:
79: 1. There are files called doc/pcre.txt, doc/pcregrep.txt, and
80: doc/pcretest.txt in the source distribution. The first of these is a
81: concatenation of the text forms of all the section 3 man pages except
82: those that summarize individual functions. The other two are the text
83: forms of the section 1 man pages for the pcregrep and pcretest commands.
84: These text forms are provided for ease of scanning with text editors or
85: similar tools. They are installed in <prefix>/share/doc/pcre, where
86: <prefix> is the installation prefix (defaulting to /usr/local).
87:
88: 2. A set of files containing all the documentation in HTML form, hyperlinked
89: in various ways, and rooted in a file called index.html, is distributed in
90: doc/html and installed in <prefix>/share/doc/pcre/html.
91:
92: Users of PCRE have contributed files containing the documentation for various
93: releases in CHM format. These can be found in the Contrib directory of the FTP
94: site (see next section).
95:
96:
97: Contributions by users of PCRE
98: ------------------------------
99:
100: You can find contributions from PCRE users in the directory
101:
102: ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/Contrib
103:
104: There is a README file giving brief descriptions of what they are. Some are
105: complete in themselves; others are pointers to URLs containing relevant files.
106: Some of this material is likely to be well out-of-date. Several of the earlier
107: contributions provided support for compiling PCRE on various flavours of
108: Windows (I myself do not use Windows). Nowadays there is more Windows support
109: in the standard distribution, so these contibutions have been archived.
110:
111:
112: Building PCRE on non-Unix systems
113: ---------------------------------
114:
115: For a non-Unix system, please read the comments in the file NON-UNIX-USE,
116: though if your system supports the use of "configure" and "make" you may be
117: able to build PCRE in the same way as for Unix-like systems. PCRE can also be
118: configured in many platform environments using the GUI facility provided by
119: CMake's cmake-gui command. This creates Makefiles, solution files, etc.
120:
121: PCRE has been compiled on many different operating systems. It should be
122: straightforward to build PCRE on any system that has a Standard C compiler and
123: library, because it uses only Standard C functions.
124:
125:
126: Building PCRE on Unix-like systems
127: ----------------------------------
128:
129: If you are using HP's ANSI C++ compiler (aCC), please see the special note
130: in the section entitled "Using HP's ANSI C++ compiler (aCC)" below.
131:
132: The following instructions assume the use of the widely used "configure, make,
133: make install" process. There is also support for CMake in the PCRE
134: distribution; there are some comments about using CMake in the NON-UNIX-USE
135: file, though it can also be used in Unix-like systems.
136:
137: To build PCRE on a Unix-like system, first run the "configure" command from the
138: PCRE distribution directory, with your current directory set to the directory
139: where you want the files to be created. This command is a standard GNU
140: "autoconf" configuration script, for which generic instructions are supplied in
141: the file INSTALL.
142:
143: Most commonly, people build PCRE within its own distribution directory, and in
144: this case, on many systems, just running "./configure" is sufficient. However,
145: the usual methods of changing standard defaults are available. For example:
146:
147: CFLAGS='-O2 -Wall' ./configure --prefix=/opt/local
148:
1.1.1.2 ! misho 149: This command specifies that the C compiler should be run with the flags '-O2
! 150: -Wall' instead of the default, and that "make install" should install PCRE
! 151: under /opt/local instead of the default /usr/local.
1.1 misho 152:
153: If you want to build in a different directory, just run "configure" with that
154: directory as current. For example, suppose you have unpacked the PCRE source
155: into /source/pcre/pcre-xxx, but you want to build it in /build/pcre/pcre-xxx:
156:
157: cd /build/pcre/pcre-xxx
158: /source/pcre/pcre-xxx/configure
159:
160: PCRE is written in C and is normally compiled as a C library. However, it is
161: possible to build it as a C++ library, though the provided building apparatus
162: does not have any features to support this.
163:
164: There are some optional features that can be included or omitted from the PCRE
165: library. They are also documented in the pcrebuild man page.
166:
167: . By default, both shared and static libraries are built. You can change this
168: by adding one of these options to the "configure" command:
169:
170: --disable-shared
171: --disable-static
172:
173: (See also "Shared libraries on Unix-like systems" below.)
174:
1.1.1.2 ! misho 175: . By default, only the 8-bit library is built. If you add --enable-pcre16 to
! 176: the "configure" command, the 16-bit library is also built. If you want only
! 177: the 16-bit library, use "./configure --enable-pcre16 --disable-pcre8".
! 178:
! 179: . If you are building the 8-bit library and want to suppress the building of
! 180: the C++ wrapper library, you can add --disable-cpp to the "configure"
! 181: command. Otherwise, when "configure" is run without --disable-pcre8, it will
! 182: try to find a C++ compiler and C++ header files, and if it succeeds, it will
! 183: try to build the C++ wrapper.
1.1 misho 184:
185: . If you want to include support for just-in-time compiling, which can give
186: large performance improvements on certain platforms, add --enable-jit to the
187: "configure" command. This support is available only for certain hardware
188: architectures. If you try to enable it on an unsupported architecture, there
189: will be a compile time error.
190:
191: . When JIT support is enabled, pcregrep automatically makes use of it, unless
192: you add --disable-pcregrep-jit to the "configure" command.
193:
194: . If you want to make use of the support for UTF-8 Unicode character strings in
1.1.1.2 ! misho 195: the 8-bit library, or UTF-16 Unicode character strings in the 16-bit library,
! 196: you must add --enable-utf to the "configure" command. Without it, the code
! 197: for handling UTF-8 and UTF-16 is not included in the relevant library. Even
! 198: when --enable-utf is included, the use of a UTF encoding still has to be
! 199: enabled by an option at run time. When PCRE is compiled with this option, its
! 200: input can only either be ASCII or UTF-8/16, even when running on EBCDIC
! 201: platforms. It is not possible to use both --enable-utf and --enable-ebcdic at
! 202: the same time.
! 203:
! 204: . There are no separate options for enabling UTF-8 and UTF-16 independently
! 205: because that would allow ridiculous settings such as requesting UTF-16
! 206: support while building only the 8-bit library. However, the option
! 207: --enable-utf8 is retained for backwards compatibility with earlier releases
! 208: that did not support 16-bit character strings. It is synonymous with
! 209: --enable-utf. It is not possible to configure one library with UTF support
! 210: and the other without in the same configuration.
! 211:
! 212: . If, in addition to support for UTF-8/16 character strings, you want to
! 213: include support for the \P, \p, and \X sequences that recognize Unicode
! 214: character properties, you must add --enable-unicode-properties to the
! 215: "configure" command. This adds about 30K to the size of the library (in the
! 216: form of a property table); only the basic two-letter properties such as Lu
! 217: are supported.
1.1 misho 218:
219: . You can build PCRE to recognize either CR or LF or the sequence CRLF or any
220: of the preceding, or any of the Unicode newline sequences as indicating the
221: end of a line. Whatever you specify at build time is the default; the caller
222: of PCRE can change the selection at run time. The default newline indicator
223: is a single LF character (the Unix standard). You can specify the default
224: newline indicator by adding --enable-newline-is-cr or --enable-newline-is-lf
225: or --enable-newline-is-crlf or --enable-newline-is-anycrlf or
226: --enable-newline-is-any to the "configure" command, respectively.
227:
228: If you specify --enable-newline-is-cr or --enable-newline-is-crlf, some of
229: the standard tests will fail, because the lines in the test files end with
230: LF. Even if the files are edited to change the line endings, there are likely
231: to be some failures. With --enable-newline-is-anycrlf or
232: --enable-newline-is-any, many tests should succeed, but there may be some
233: failures.
234:
235: . By default, the sequence \R in a pattern matches any Unicode line ending
236: sequence. This is independent of the option specifying what PCRE considers to
237: be the end of a line (see above). However, the caller of PCRE can restrict \R
238: to match only CR, LF, or CRLF. You can make this the default by adding
239: --enable-bsr-anycrlf to the "configure" command (bsr = "backslash R").
240:
241: . When called via the POSIX interface, PCRE uses malloc() to get additional
242: storage for processing capturing parentheses if there are more than 10 of
243: them in a pattern. You can increase this threshold by setting, for example,
244:
245: --with-posix-malloc-threshold=20
246:
247: on the "configure" command.
248:
249: . PCRE has a counter that can be set to limit the amount of resources it uses.
250: If the limit is exceeded during a match, the match fails. The default is ten
251: million. You can change the default by setting, for example,
252:
253: --with-match-limit=500000
254:
255: on the "configure" command. This is just the default; individual calls to
256: pcre_exec() can supply their own value. There is more discussion on the
257: pcreapi man page.
258:
259: . There is a separate counter that limits the depth of recursive function calls
260: during a matching process. This also has a default of ten million, which is
261: essentially "unlimited". You can change the default by setting, for example,
262:
263: --with-match-limit-recursion=500000
264:
265: Recursive function calls use up the runtime stack; running out of stack can
266: cause programs to crash in strange ways. There is a discussion about stack
267: sizes in the pcrestack man page.
268:
269: . The default maximum compiled pattern size is around 64K. You can increase
1.1.1.2 ! misho 270: this by adding --with-link-size=3 to the "configure" command. In the 8-bit
! 271: library, PCRE then uses three bytes instead of two for offsets to different
! 272: parts of the compiled pattern. In the 16-bit library, --with-link-size=3 is
! 273: the same as --with-link-size=4, which (in both libraries) uses four-byte
! 274: offsets. Increasing the internal link size reduces performance.
1.1 misho 275:
276: . You can build PCRE so that its internal match() function that is called from
277: pcre_exec() does not call itself recursively. Instead, it uses memory blocks
278: obtained from the heap via the special functions pcre_stack_malloc() and
279: pcre_stack_free() to save data that would otherwise be saved on the stack. To
280: build PCRE like this, use
281:
282: --disable-stack-for-recursion
283:
284: on the "configure" command. PCRE runs more slowly in this mode, but it may be
285: necessary in environments with limited stack sizes. This applies only to the
286: normal execution of the pcre_exec() function; if JIT support is being
287: successfully used, it is not relevant. Equally, it does not apply to
288: pcre_dfa_exec(), which does not use deeply nested recursion. There is a
289: discussion about stack sizes in the pcrestack man page.
290:
291: . For speed, PCRE uses four tables for manipulating and identifying characters
292: whose code point values are less than 256. By default, it uses a set of
293: tables for ASCII encoding that is part of the distribution. If you specify
294:
295: --enable-rebuild-chartables
296:
297: a program called dftables is compiled and run in the default C locale when
298: you obey "make". It builds a source file called pcre_chartables.c. If you do
299: not specify this option, pcre_chartables.c is created as a copy of
300: pcre_chartables.c.dist. See "Character tables" below for further information.
301:
302: . It is possible to compile PCRE for use on systems that use EBCDIC as their
303: character code (as opposed to ASCII) by specifying
304:
305: --enable-ebcdic
306:
307: This automatically implies --enable-rebuild-chartables (see above). However,
308: when PCRE is built this way, it always operates in EBCDIC. It cannot support
1.1.1.2 ! misho 309: both EBCDIC and UTF-8/16.
1.1 misho 310:
1.1.1.2 ! misho 311: . The pcregrep program currently supports only 8-bit data files, and so
! 312: requires the 8-bit PCRE library. It is possible to compile pcregrep to use
! 313: libz and/or libbz2, in order to read .gz and .bz2 files (respectively), by
! 314: specifying one or both of
1.1 misho 315:
316: --enable-pcregrep-libz
317: --enable-pcregrep-libbz2
318:
319: Of course, the relevant libraries must be installed on your system.
320:
321: . The default size of internal buffer used by pcregrep can be set by, for
322: example:
323:
324: --with-pcregrep-bufsize=50K
325:
326: The default value is 20K.
327:
328: . It is possible to compile pcretest so that it links with the libreadline
329: library, by specifying
330:
331: --enable-pcretest-libreadline
332:
333: If this is done, when pcretest's input is from a terminal, it reads it using
334: the readline() function. This provides line-editing and history facilities.
335: Note that libreadline is GPL-licenced, so if you distribute a binary of
336: pcretest linked in this way, there may be licensing issues.
337:
338: Setting this option causes the -lreadline option to be added to the pcretest
339: build. In many operating environments with a sytem-installed readline
340: library this is sufficient. However, in some environments (e.g. if an
341: unmodified distribution version of readline is in use), it may be necessary
342: to specify something like LIBS="-lncurses" as well. This is because, to quote
343: the readline INSTALL, "Readline uses the termcap functions, but does not link
344: with the termcap or curses library itself, allowing applications which link
345: with readline the to choose an appropriate library." If you get error
346: messages about missing functions tgetstr, tgetent, tputs, tgetflag, or tgoto,
347: this is the problem, and linking with the ncurses library should fix it.
348:
349: The "configure" script builds the following files for the basic C library:
350:
351: . Makefile the makefile that builds the library
352: . config.h build-time configuration options for the library
353: . pcre.h the public PCRE header file
354: . pcre-config script that shows the building settings such as CFLAGS
355: that were set for "configure"
356: . libpcre.pc ) data for the pkg-config command
1.1.1.2 ! misho 357: . libpcre16.pc )
1.1 misho 358: . libpcreposix.pc )
359: . libtool script that builds shared and/or static libraries
360: . RunTest script for running tests on the basic C library
361: . RunGrepTest script for running tests on the pcregrep command
362:
363: Versions of config.h and pcre.h are distributed in the PCRE tarballs under the
364: names config.h.generic and pcre.h.generic. These are provided for those who
365: have to built PCRE without using "configure" or CMake. If you use "configure"
366: or CMake, the .generic versions are not used.
367:
1.1.1.2 ! misho 368: When building the 8-bit library, if a C++ compiler is found, the following
! 369: files are also built:
1.1 misho 370:
371: . libpcrecpp.pc data for the pkg-config command
372: . pcrecpparg.h header file for calling PCRE via the C++ wrapper
373: . pcre_stringpiece.h header for the C++ "stringpiece" functions
374:
375: The "configure" script also creates config.status, which is an executable
376: script that can be run to recreate the configuration, and config.log, which
377: contains compiler output from tests that "configure" runs.
378:
1.1.1.2 ! misho 379: Once "configure" has run, you can run "make". This builds either or both of the
! 380: libraries libpcre and libpcre16, and a test program called pcretest. If you
! 381: enabled JIT support with --enable-jit, a test program called pcre_jit_test is
! 382: built as well.
! 383:
! 384: If the 8-bit library is built, libpcreposix and the pcregrep command are also
! 385: built, and if a C++ compiler was found on your system, and you did not disable
! 386: it with --disable-cpp, "make" builds the C++ wrapper library, which is called
! 387: libpcrecpp, as well as some test programs called pcrecpp_unittest,
! 388: pcre_scanner_unittest, and pcre_stringpiece_unittest.
1.1 misho 389:
390: The command "make check" runs all the appropriate tests. Details of the PCRE
391: tests are given below in a separate section of this document.
392:
393: You can use "make install" to install PCRE into live directories on your
394: system. The following are installed (file names are all relative to the
395: <prefix> that is set when "configure" is run):
396:
397: Commands (bin):
398: pcretest
1.1.1.2 ! misho 399: pcregrep (if 8-bit support is enabled)
1.1 misho 400: pcre-config
401:
402: Libraries (lib):
1.1.1.2 ! misho 403: libpcre16 (if 16-bit support is enabled)
! 404: libpcre (if 8-bit support is enabled)
! 405: libpcreposix (if 8-bit support is enabled)
! 406: libpcrecpp (if 8-bit and C++ support is enabled)
1.1 misho 407:
408: Configuration information (lib/pkgconfig):
1.1.1.2 ! misho 409: libpcre16.pc
1.1 misho 410: libpcre.pc
411: libpcreposix.pc
412: libpcrecpp.pc (if C++ support is enabled)
413:
414: Header files (include):
415: pcre.h
416: pcreposix.h
417: pcre_scanner.h )
418: pcre_stringpiece.h ) if C++ support is enabled
419: pcrecpp.h )
420: pcrecpparg.h )
421:
422: Man pages (share/man/man{1,3}):
423: pcregrep.1
424: pcretest.1
425: pcre-config.1
426: pcre.3
427: pcre*.3 (lots more pages, all starting "pcre")
428:
429: HTML documentation (share/doc/pcre/html):
430: index.html
431: *.html (lots more pages, hyperlinked from index.html)
432:
433: Text file documentation (share/doc/pcre):
434: AUTHORS
435: COPYING
436: ChangeLog
437: LICENCE
438: NEWS
439: README
440: pcre.txt (a concatenation of the man(3) pages)
441: pcretest.txt the pcretest man page
442: pcregrep.txt the pcregrep man page
443: pcre-config.txt the pcre-config man page
444:
445: If you want to remove PCRE from your system, you can run "make uninstall".
446: This removes all the files that "make install" installed. However, it does not
447: remove any directories, because these are often shared with other programs.
448:
449:
450: Retrieving configuration information on Unix-like systems
451: ---------------------------------------------------------
452:
453: Running "make install" installs the command pcre-config, which can be used to
454: recall information about the PCRE configuration and installation. For example:
455:
456: pcre-config --version
457:
458: prints the version number, and
459:
460: pcre-config --libs
461:
462: outputs information about where the library is installed. This command can be
463: included in makefiles for programs that use PCRE, saving the programmer from
464: having to remember too many details.
465:
466: The pkg-config command is another system for saving and retrieving information
467: about installed libraries. Instead of separate commands for each library, a
468: single command is used. For example:
469:
470: pkg-config --cflags pcre
471:
472: The data is held in *.pc files that are installed in a directory called
473: <prefix>/lib/pkgconfig.
474:
475:
476: Shared libraries on Unix-like systems
477: -------------------------------------
478:
479: The default distribution builds PCRE as shared libraries and static libraries,
480: as long as the operating system supports shared libraries. Shared library
481: support relies on the "libtool" script which is built as part of the
482: "configure" process.
483:
484: The libtool script is used to compile and link both shared and static
485: libraries. They are placed in a subdirectory called .libs when they are newly
486: built. The programs pcretest and pcregrep are built to use these uninstalled
487: libraries (by means of wrapper scripts in the case of shared libraries). When
488: you use "make install" to install shared libraries, pcregrep and pcretest are
489: automatically re-built to use the newly installed shared libraries before being
490: installed themselves. However, the versions left in the build directory still
491: use the uninstalled libraries.
492:
493: To build PCRE using static libraries only you must use --disable-shared when
494: configuring it. For example:
495:
496: ./configure --prefix=/usr/gnu --disable-shared
497:
498: Then run "make" in the usual way. Similarly, you can use --disable-static to
499: build only shared libraries.
500:
501:
502: Cross-compiling on Unix-like systems
503: ------------------------------------
504:
505: You can specify CC and CFLAGS in the normal way to the "configure" command, in
506: order to cross-compile PCRE for some other host. However, you should NOT
507: specify --enable-rebuild-chartables, because if you do, the dftables.c source
508: file is compiled and run on the local host, in order to generate the inbuilt
509: character tables (the pcre_chartables.c file). This will probably not work,
510: because dftables.c needs to be compiled with the local compiler, not the cross
511: compiler.
512:
513: When --enable-rebuild-chartables is not specified, pcre_chartables.c is created
514: by making a copy of pcre_chartables.c.dist, which is a default set of tables
515: that assumes ASCII code. Cross-compiling with the default tables should not be
516: a problem.
517:
518: If you need to modify the character tables when cross-compiling, you should
519: move pcre_chartables.c.dist out of the way, then compile dftables.c by hand and
520: run it on the local host to make a new version of pcre_chartables.c.dist.
521: Then when you cross-compile PCRE this new version of the tables will be used.
522:
523:
524: Using HP's ANSI C++ compiler (aCC)
525: ----------------------------------
526:
527: Unless C++ support is disabled by specifying the "--disable-cpp" option of the
528: "configure" script, you must include the "-AA" option in the CXXFLAGS
529: environment variable in order for the C++ components to compile correctly.
530:
531: Also, note that the aCC compiler on PA-RISC platforms may have a defect whereby
532: needed libraries fail to get included when specifying the "-AA" compiler
533: option. If you experience unresolved symbols when linking the C++ programs,
534: use the workaround of specifying the following environment variable prior to
535: running the "configure" script:
536:
537: CXXLDFLAGS="-lstd_v2 -lCsup_v2"
538:
539:
540: Using Sun's compilers for Solaris
541: ---------------------------------
542:
543: A user reports that the following configurations work on Solaris 9 sparcv9 and
544: Solaris 9 x86 (32-bit):
545:
546: Solaris 9 sparcv9: ./configure --disable-cpp CC=/bin/cc CFLAGS="-m64 -g"
547: Solaris 9 x86: ./configure --disable-cpp CC=/bin/cc CFLAGS="-g"
548:
549:
550: Using PCRE from MySQL
551: ---------------------
552:
553: On systems where both PCRE and MySQL are installed, it is possible to make use
554: of PCRE from within MySQL, as an alternative to the built-in pattern matching.
555: There is a web page that tells you how to do this:
556:
557: http://www.mysqludf.org/lib_mysqludf_preg/index.php
558:
559:
560: Making new tarballs
561: -------------------
562:
563: The command "make dist" creates three PCRE tarballs, in tar.gz, tar.bz2, and
564: zip formats. The command "make distcheck" does the same, but then does a trial
565: build of the new distribution to ensure that it works.
566:
567: If you have modified any of the man page sources in the doc directory, you
568: should first run the PrepareRelease script before making a distribution. This
569: script creates the .txt and HTML forms of the documentation from the man pages.
570:
571:
572: Testing PCRE
573: ------------
574:
575: To test the basic PCRE library on a Unix system, run the RunTest script that is
576: created by the configuring process. There is also a script called RunGrepTest
577: that tests the options of the pcregrep command. If the C++ wrapper library is
578: built, three test programs called pcrecpp_unittest, pcre_scanner_unittest, and
579: pcre_stringpiece_unittest are also built. When JIT support is enabled, another
580: test program called pcre_jit_test is built.
581:
582: Both the scripts and all the program tests are run if you obey "make check" or
583: "make test". For other systems, see the instructions in NON-UNIX-USE.
584:
585: The RunTest script runs the pcretest test program (which is documented in its
586: own man page) on each of the relevant testinput files in the testdata
587: directory, and compares the output with the contents of the corresponding
588: testoutput files. Some tests are relevant only when certain build-time options
1.1.1.2 ! misho 589: were selected. For example, the tests for UTF-8/16 support are run only if
! 590: --enable-utf was used. RunTest outputs a comment when it skips a test.
1.1 misho 591:
592: Many of the tests that are not skipped are run up to three times. The second
593: run forces pcre_study() to be called for all patterns except for a few in some
594: tests that are marked "never study" (see the pcretest program for how this is
595: done). If JIT support is available, the non-DFA tests are run a third time,
596: this time with a forced pcre_study() with the PCRE_STUDY_JIT_COMPILE option.
597:
1.1.1.2 ! misho 598: When both 8-bit and 16-bit support is enabled, the entire set of tests is run
! 599: twice, once for each library. If you want to run just one set of tests, call
! 600: RunTest with either the -8 or -16 option.
! 601:
! 602: RunTest uses a file called testtry to hold the main output from pcretest.
! 603: Other files whose names begin with "test" are used as working files in some
! 604: tests. To run pcretest on just one or more specific test files, give their
! 605: numbers as arguments to RunTest, for example:
1.1 misho 606:
1.1.1.2 ! misho 607: RunTest 2 7 11
1.1 misho 608:
609: The first test file can be fed directly into the perltest.pl script to check
610: that Perl gives the same results. The only difference you should see is in the
611: first few lines, where the Perl version is given instead of the PCRE version.
612:
1.1.1.2 ! misho 613: The second set of tests check pcre_fullinfo(), pcre_study(),
1.1 misho 614: pcre_copy_substring(), pcre_get_substring(), pcre_get_substring_list(), error
615: detection, and run-time flags that are specific to PCRE, as well as the POSIX
616: wrapper API. It also uses the debugging flags to check some of the internals of
617: pcre_compile().
618:
619: If you build PCRE with a locale setting that is not the standard C locale, the
620: character tables may be different (see next paragraph). In some cases, this may
621: cause failures in the second set of tests. For example, in a locale where the
622: isprint() function yields TRUE for characters in the range 128-255, the use of
623: [:isascii:] inside a character class defines a different set of characters, and
624: this shows up in this test as a difference in the compiled code, which is being
625: listed for checking. Where the comparison test output contains [\x00-\x7f] the
626: test will contain [\x00-\xff], and similarly in some other cases. This is not a
627: bug in PCRE.
628:
629: The third set of tests checks pcre_maketables(), the facility for building a
630: set of character tables for a specific locale and using them instead of the
631: default tables. The tests make use of the "fr_FR" (French) locale. Before
632: running the test, the script checks for the presence of this locale by running
633: the "locale" command. If that command fails, or if it doesn't include "fr_FR"
634: in the list of available locales, the third test cannot be run, and a comment
635: is output to say why. If running this test produces instances of the error
636:
637: ** Failed to set locale "fr_FR"
638:
639: in the comparison output, it means that locale is not available on your system,
640: despite being listed by "locale". This does not mean that PCRE is broken.
641:
642: [If you are trying to run this test on Windows, you may be able to get it to
643: work by changing "fr_FR" to "french" everywhere it occurs. Alternatively, use
644: RunTest.bat. The version of RunTest.bat included with PCRE 7.4 and above uses
645: Windows versions of test 2. More info on using RunTest.bat is included in the
646: document entitled NON-UNIX-USE.]
647:
1.1.1.2 ! misho 648: The fourth and fifth tests check the UTF-8/16 support and error handling and
! 649: internal UTF features of PCRE that are not relevant to Perl, respectively. The
! 650: sixth and seventh tests do the same for Unicode character properties support.
! 651:
! 652: The eighth, ninth, and tenth tests check the pcre_dfa_exec() alternative
! 653: matching function, in non-UTF-8/16 mode, UTF-8/16 mode, and UTF-8/16 mode with
! 654: Unicode property support, respectively.
1.1 misho 655:
1.1.1.2 ! misho 656: The eleventh test checks some internal offsets and code size features; it is
! 657: run only when the default "link size" of 2 is set (in other cases the sizes
! 658: change) and when Unicode property support is enabled.
1.1 misho 659:
1.1.1.2 ! misho 660: The twelfth test is run only when JIT support is available, and the thirteenth
! 661: test is run only when JIT support is not available. They test some JIT-specific
! 662: features such as information output from pcretest about JIT compilation.
! 663:
! 664: The fourteenth, fifteenth, and sixteenth tests are run only in 8-bit mode, and
! 665: the seventeenth, eighteenth, and nineteenth tests are run only in 16-bit mode.
! 666: These are tests that generate different output in the two modes. They are for
! 667: general cases, UTF-8/16 support, and Unicode property support, respectively.
1.1 misho 668:
1.1.1.2 ! misho 669: The twentieth test is run only in 16-bit mode. It tests some specific 16-bit
! 670: features of the DFA matching engine.
1.1 misho 671:
1.1.1.2 ! misho 672: The twenty-first and twenty-second tests are run only in 16-bit mode, when the
! 673: link size is set to 2. They test reloading pre-compiled patterns.
1.1 misho 674:
675:
676: Character tables
677: ----------------
678:
679: For speed, PCRE uses four tables for manipulating and identifying characters
680: whose code point values are less than 256. The final argument of the
681: pcre_compile() function is a pointer to a block of memory containing the
682: concatenated tables. A call to pcre_maketables() can be used to generate a set
683: of tables in the current locale. If the final argument for pcre_compile() is
684: passed as NULL, a set of default tables that is built into the binary is used.
685:
686: The source file called pcre_chartables.c contains the default set of tables. By
687: default, this is created as a copy of pcre_chartables.c.dist, which contains
688: tables for ASCII coding. However, if --enable-rebuild-chartables is specified
689: for ./configure, a different version of pcre_chartables.c is built by the
690: program dftables (compiled from dftables.c), which uses the ANSI C character
691: handling functions such as isalnum(), isalpha(), isupper(), islower(), etc. to
692: build the table sources. This means that the default C locale which is set for
693: your system will control the contents of these default tables. You can change
694: the default tables by editing pcre_chartables.c and then re-building PCRE. If
695: you do this, you should take care to ensure that the file does not get
696: automatically re-generated. The best way to do this is to move
697: pcre_chartables.c.dist out of the way and replace it with your customized
698: tables.
699:
700: When the dftables program is run as a result of --enable-rebuild-chartables,
701: it uses the default C locale that is set on your system. It does not pay
702: attention to the LC_xxx environment variables. In other words, it uses the
703: system's default locale rather than whatever the compiling user happens to have
704: set. If you really do want to build a source set of character tables in a
705: locale that is specified by the LC_xxx variables, you can run the dftables
706: program by hand with the -L option. For example:
707:
708: ./dftables -L pcre_chartables.c.special
709:
710: The first two 256-byte tables provide lower casing and case flipping functions,
711: respectively. The next table consists of three 32-byte bit maps which identify
712: digits, "word" characters, and white space, respectively. These are used when
713: building 32-byte bit maps that represent character classes for code points less
714: than 256.
715:
716: The final 256-byte table has bits indicating various character types, as
717: follows:
718:
719: 1 white space character
720: 2 letter
721: 4 decimal digit
722: 8 hexadecimal digit
723: 16 alphanumeric or '_'
724: 128 regular expression metacharacter or binary zero
725:
726: You should not alter the set of characters that contain the 128 bit, as that
727: will cause PCRE to malfunction.
728:
729:
730: File manifest
731: -------------
732:
1.1.1.2 ! misho 733: The distribution should contain the files listed below. Where a file name is
! 734: given as pcre[16]_xxx it means that there are two files, one with the name
! 735: pcre_xxx and the other with the name pcre16_xxx.
1.1 misho 736:
737: (A) Source files of the PCRE library functions and their headers:
738:
739: dftables.c auxiliary program for building pcre_chartables.c
740: when --enable-rebuild-chartables is specified
741:
742: pcre_chartables.c.dist a default set of character tables that assume ASCII
743: coding; used, unless --enable-rebuild-chartables is
1.1.1.2 ! misho 744: specified, by copying to pcre[16]_chartables.c
1.1 misho 745:
746: pcreposix.c )
1.1.1.2 ! misho 747: pcre[16]_byte_order.c )
! 748: pcre[16]_compile.c )
! 749: pcre[16]_config.c )
! 750: pcre[16]_dfa_exec.c )
! 751: pcre[16]_exec.c )
! 752: pcre[16]_fullinfo.c )
! 753: pcre[16]_get.c ) sources for the functions in the library,
! 754: pcre[16]_globals.c ) and some internal functions that they use
! 755: pcre[16]_jit_compile.c )
! 756: pcre[16]_maketables.c )
! 757: pcre[16]_newline.c )
! 758: pcre[16]_refcount.c )
! 759: pcre[16]_string_utils.c )
! 760: pcre[16]_study.c )
! 761: pcre[16]_tables.c )
! 762: pcre[16]_ucd.c )
! 763: pcre[16]_version.c )
! 764: pcre[16]_xclass.c )
1.1 misho 765: pcre_ord2utf8.c )
766: pcre_valid_utf8.c )
1.1.1.2 ! misho 767: pcre16_ord2utf16.c )
! 768: pcre16_utf16_utils.c )
! 769: pcre16_valid_utf16.c )
! 770:
! 771: pcre[16]_printint.c ) debugging function that is used by pcretest,
1.1 misho 772: ) and can also be #included in pcre_compile()
1.1.1.2 ! misho 773:
1.1 misho 774: pcre.h.in template for pcre.h when built by "configure"
775: pcreposix.h header for the external POSIX wrapper API
776: pcre_internal.h header for internal use
777: sljit/* 16 files that make up the JIT compiler
778: ucp.h header for Unicode property handling
779:
780: config.h.in template for config.h, which is built by "configure"
781:
782: pcrecpp.h public header file for the C++ wrapper
783: pcrecpparg.h.in template for another C++ header file
784: pcre_scanner.h public header file for C++ scanner functions
785: pcrecpp.cc )
786: pcre_scanner.cc ) source for the C++ wrapper library
787:
788: pcre_stringpiece.h.in template for pcre_stringpiece.h, the header for the
789: C++ stringpiece functions
790: pcre_stringpiece.cc source for the C++ stringpiece functions
791:
792: (B) Source files for programs that use PCRE:
793:
794: pcredemo.c simple demonstration of coding calls to PCRE
795: pcregrep.c source of a grep utility that uses PCRE
796: pcretest.c comprehensive test program
797:
798: (C) Auxiliary files:
799:
800: 132html script to turn "man" pages into HTML
801: AUTHORS information about the author of PCRE
802: ChangeLog log of changes to the code
803: CleanTxt script to clean nroff output for txt man pages
804: Detrail script to remove trailing spaces
805: HACKING some notes about the internals of PCRE
806: INSTALL generic installation instructions
807: LICENCE conditions for the use of PCRE
808: COPYING the same, using GNU's standard name
809: Makefile.in ) template for Unix Makefile, which is built by
810: ) "configure"
811: Makefile.am ) the automake input that was used to create
812: ) Makefile.in
813: NEWS important changes in this release
814: NON-UNIX-USE notes on building PCRE on non-Unix systems
815: PrepareRelease script to make preparations for "make dist"
816: README this file
817: RunTest a Unix shell script for running tests
818: RunGrepTest a Unix shell script for pcregrep tests
819: aclocal.m4 m4 macros (generated by "aclocal")
820: config.guess ) files used by libtool,
821: config.sub ) used only when building a shared library
822: configure a configuring shell script (built by autoconf)
823: configure.ac ) the autoconf input that was used to build
824: ) "configure" and config.h
825: depcomp ) script to find program dependencies, generated by
826: ) automake
827: doc/*.3 man page sources for PCRE
828: doc/*.1 man page sources for pcregrep and pcretest
829: doc/index.html.src the base HTML page
830: doc/html/* HTML documentation
831: doc/pcre.txt plain text version of the man pages
832: doc/pcretest.txt plain text documentation of test program
833: doc/perltest.txt plain text documentation of Perl test program
834: install-sh a shell script for installing files
1.1.1.2 ! misho 835: libpcre16.pc.in template for libpcre16.pc for pkg-config
1.1 misho 836: libpcre.pc.in template for libpcre.pc for pkg-config
837: libpcreposix.pc.in template for libpcreposix.pc for pkg-config
838: libpcrecpp.pc.in template for libpcrecpp.pc for pkg-config
839: ltmain.sh file used to build a libtool script
840: missing ) common stub for a few missing GNU programs while
841: ) installing, generated by automake
842: mkinstalldirs script for making install directories
843: perltest.pl Perl test program
844: pcre-config.in source of script which retains PCRE information
845: pcre_jit_test.c test program for the JIT compiler
846: pcrecpp_unittest.cc )
847: pcre_scanner_unittest.cc ) test programs for the C++ wrapper
848: pcre_stringpiece_unittest.cc )
849: testdata/testinput* test data for main library tests
850: testdata/testoutput* expected test results
851: testdata/grep* input and output for pcregrep tests
1.1.1.2 ! misho 852: testdata/* other supporting test files
1.1 misho 853:
854: (D) Auxiliary files for cmake support
855:
856: cmake/COPYING-CMAKE-SCRIPTS
857: cmake/FindPackageHandleStandardArgs.cmake
858: cmake/FindReadline.cmake
859: CMakeLists.txt
860: config-cmake.h.in
861:
862: (E) Auxiliary files for VPASCAL
863:
864: makevp.bat
865: makevp_c.txt
866: makevp_l.txt
867: pcregexp.pas
868:
869: (F) Auxiliary files for building PCRE "by hand"
870:
871: pcre.h.generic ) a version of the public PCRE header file
872: ) for use in non-"configure" environments
873: config.h.generic ) a version of config.h for use in non-"configure"
874: ) environments
875:
876: (F) Miscellaneous
877:
878: RunTest.bat a script for running tests under Windows
879:
880: Philip Hazel
881: Email local part: ph10
882: Email domain: cam.ac.uk
1.1.1.2 ! misho 883: Last updated: 20 January 2012
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / pcre