Annotation of embedaddon/php/README.TESTING, revision 1.1.1.2
1.1 misho 1: [IMPORTANT NOTICE]
2: ------------------
1.1.1.2 ! misho 3: Failed tests usually indicate a problem with your local system setup
1.1 misho 4: and not within PHP itself (at least for official PHP release versions).
5: You may decide to automaticaly submit a test summary to our QA workflow
6: at the end of a test run.
7: Please do *not* submit a failed test as a bug or ask for help on why
8: it failed on your system without providing substantial backup information
9: on *why* the test failed on your special setup. Thank you :-)
10:
11:
12: [Testing Basics]
13: ----------------
14: The easiest way to test your PHP build is to run "make test" from the
15: command line after successfully compiling. This will run the complete
16: tests for all enabled functionalities and extensions using the PHP
17: CLI binary.
18: To execute test scripts, you must build PHP with some SAPI, then you
19: type "make test" to execute all or some test scripts saved under
20: "tests" directory under source root directory.
21:
22: Usage:
23: make test
24:
25: "make test" basically executes "run-tests.php" script
26: under the source root (parallel builds will not work). Therefore you
27: can execute the script as follows:
28:
29: TEST_PHP_EXECUTABLE=sapi/cli/php \
30: sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
31:
32:
33: [Which "php" executable "make test" look for]
34: ---------------------------------------------
35: If you are running the run-tests.php script from the command line (as above)
36: you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
37: select the PHP executable that is to be tested, that is, used to run the test scripts.
38:
39: If you run the tests using make test, the PHP CLI and CGI executables are
40: automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some
41: test scripts such as session must be executed by CGI SAPI. Therefore,
42: you must build PHP with CGI SAPI to perform all tests.
43:
44: NOTE: PHP binary executing "run-tests.php" and php binary used for
45: executing test scripts may differ. If you use different PHP binary for
46: executing "run-tests.php" script, you may get errors.
47:
48:
49: [Which php.ini is used]
50: -----------------------
51: "make test" uses the same php.ini file as it would once installed.
52: The tests have been written to be independent of that php.ini file,
53: so if you find a test that is affected by a setting, please report
54: this, so we can address the issue.
55:
56:
57: [Which test scripts are executed]
58: ---------------------------------
59: "run-tests.php" ("make test"), without any arguments executes all
60: test scripts by extracting all directories named "tests"
61: from the source root and any subdirectories below. If there are files,
62: which have a "phpt" extension, "run-tests.php" looks at the sections
63: in these files, determines whether it should run it, by evaluating
64: the 'SKIP' section. If the test is eligible for execution, the 'FILE'
65: section is extracted into a ".php" file (with the same name besides
66: the extension) and gets executed.
67: When an argument is given or TESTS environment variable is set, the
68: GLOB is expanded by the shell and any file with extension "*.phpt" is
69: regarded as a test file.
70:
71: Tester can easily execute tests selectively with as follows.
72:
73: Examples:
74: ./sapi/cli/php run-tests.php ext/mbstring/*
75: ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
76:
77:
78: [Test results]
79: --------------
80: Test results are printed to standard output. If there is a failed test,
81: the "run-tests.php" script saves the result, the expected result and the
82: code executed to the test script directory. For example, if
83: ext/myext/tests/myext.phpt fails to pass, the following files are created:
84:
85: ext/myext/tests/myext.php - actual test file executed
86: ext/myext/tests/myext.log - log of test execution (L)
87: ext/myext/tests/myext.exp - expected output (E)
88: ext/myext/tests/myext.out - output from test script (O)
89: ext/myext/tests/myext.diff - diff of .out and .exp (D)
90:
91: Failed tests are always bugs. Either the test is bugged or not considering
92: factors applying to the tester's environment, or there is a bug in PHP.
93: If this is a known bug, we strive to provide bug numbers, in either the
94: test name or the file name. You can check the status of such a bug, by
95: going to: http://bugs.php.net/12345 where 12345 is the bug number.
96: For clarity and automated processing, bug numbers are prefixed by a hash
97: sign '#' in test names and/or test cases are named bug12345.phpt.
98:
99: NOTE: The files generated by tests can be selected by setting the
100: environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
101: generated use the character in brackets as shown above (default is LEOD).
102: The php file will be generated always.
103:
104: NOTE: You can set environment variable TEST_PHP_DETAILED to enable
105: detailed test information.
106:
107: [Automated testing]
108: If you like to keep up to speed, with latest developments and quality
109: assurance, setting the environment variable NO_INTERACTION to 1, will not
110: prompt the tester for any user input.
111:
112: Normally, the exit status of "make test" is zero, regardless of the results
113: of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
114: and "make test" will set the exit status ("$?") to non-zero, when an
115: individual test has failed.
116:
117: Example script to be run by cron(1):
118: ========== qa-test.sh =============
119: #!/bin/sh
120:
121: CO_DIR=$HOME/cvs/php5
122: MYMAIL=qa-test@domain.com
123: TMPDIR=/var/tmp
124: TODAY=`date +"%Y%m%d"`
125:
126: # Make sure compilation enviroment is correct
127: CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
128: export MAKE=gmake
129: export CC=gcc
130:
131: # Set test environment
132: export NO_INTERACTION=1
133: export REPORT_EXIT_STATUS=1
134:
135: cd $CO_DIR
136: cvs update . >>$TMPDIR/phpqatest.$TODAY
137: ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
138: $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
139: if test $? -gt 0
140: then
141: cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
142: fi
143: ========== end of qa-test.sh =============
144:
145: NOTE: the exit status of run-tests.php will be 1 when
146: REPORT_EXIT_STATUS is set. The result of "make test" may be higher
147: than that. At present, gmake 3.79.1 returns 2, so it is
148: advised to test for non-zero, rather then a specific value.
149:
150:
151: [Creating new test files]
152: -------------------------
153: Writing test file is very easy if you are used to PHP.
154: See the HOWTO at http://qa.php.net/write-test.php
155:
156:
157: [How to help us]
158: ----------------
159: If you find bug in PHP, you can submit bug report AND test script
160: for us. You don't have to write complete script, just give us test
161: script with following format. Please test the script and make sure
162: you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
163: submit.
164:
165: <?php
166: /*
167: Bug #12345
168: substr() bug. Do not return expected string.
169:
170: ACTUAL OUTPUT
171: XYXA
172:
173: EXPECTED OUTPUT
174: ABCD
175: */
176:
177: $str = "XYZABCD";
178: echo substr($str,3,7);
179:
180: ?>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README.TESTING CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / php