Annotation of embedaddon/php/README.TESTING, revision 1.1
1.1 ! misho 1: [IMPORTANT NOTICE]
! 2: ------------------
! 3: Failed tests usualy indicate a problem with your local system setup
! 4: and not within PHP itself (at least for official PHP release versions).
! 5: You may decide to automaticaly submit a test summary to our QA workflow
! 6: at the end of a test run.
! 7: Please do *not* submit a failed test as a bug or ask for help on why
! 8: it failed on your system without providing substantial backup information
! 9: on *why* the test failed on your special setup. Thank you :-)
! 10:
! 11:
! 12: [Testing Basics]
! 13: ----------------
! 14: The easiest way to test your PHP build is to run "make test" from the
! 15: command line after successfully compiling. This will run the complete
! 16: tests for all enabled functionalities and extensions using the PHP
! 17: CLI binary.
! 18: To execute test scripts, you must build PHP with some SAPI, then you
! 19: type "make test" to execute all or some test scripts saved under
! 20: "tests" directory under source root directory.
! 21:
! 22: Usage:
! 23: make test
! 24:
! 25: "make test" basically executes "run-tests.php" script
! 26: under the source root (parallel builds will not work). Therefore you
! 27: can execute the script as follows:
! 28:
! 29: TEST_PHP_EXECUTABLE=sapi/cli/php \
! 30: sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
! 31:
! 32:
! 33: [Which "php" executable "make test" look for]
! 34: ---------------------------------------------
! 35: If you are running the run-tests.php script from the command line (as above)
! 36: you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
! 37: select the PHP executable that is to be tested, that is, used to run the test scripts.
! 38:
! 39: If you run the tests using make test, the PHP CLI and CGI executables are
! 40: automatically set for you. "make test" executes "run-tests.php" script with the CLI binary. Some
! 41: test scripts such as session must be executed by CGI SAPI. Therefore,
! 42: you must build PHP with CGI SAPI to perform all tests.
! 43:
! 44: NOTE: PHP binary executing "run-tests.php" and php binary used for
! 45: executing test scripts may differ. If you use different PHP binary for
! 46: executing "run-tests.php" script, you may get errors.
! 47:
! 48:
! 49: [Which php.ini is used]
! 50: -----------------------
! 51: "make test" uses the same php.ini file as it would once installed.
! 52: The tests have been written to be independent of that php.ini file,
! 53: so if you find a test that is affected by a setting, please report
! 54: this, so we can address the issue.
! 55:
! 56:
! 57: [Which test scripts are executed]
! 58: ---------------------------------
! 59: "run-tests.php" ("make test"), without any arguments executes all
! 60: test scripts by extracting all directories named "tests"
! 61: from the source root and any subdirectories below. If there are files,
! 62: which have a "phpt" extension, "run-tests.php" looks at the sections
! 63: in these files, determines whether it should run it, by evaluating
! 64: the 'SKIP' section. If the test is eligible for execution, the 'FILE'
! 65: section is extracted into a ".php" file (with the same name besides
! 66: the extension) and gets executed.
! 67: When an argument is given or TESTS environment variable is set, the
! 68: GLOB is expanded by the shell and any file with extension "*.phpt" is
! 69: regarded as a test file.
! 70:
! 71: Tester can easily execute tests selectively with as follows.
! 72:
! 73: Examples:
! 74: ./sapi/cli/php run-tests.php ext/mbstring/*
! 75: ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
! 76:
! 77:
! 78: [Test results]
! 79: --------------
! 80: Test results are printed to standard output. If there is a failed test,
! 81: the "run-tests.php" script saves the result, the expected result and the
! 82: code executed to the test script directory. For example, if
! 83: ext/myext/tests/myext.phpt fails to pass, the following files are created:
! 84:
! 85: ext/myext/tests/myext.php - actual test file executed
! 86: ext/myext/tests/myext.log - log of test execution (L)
! 87: ext/myext/tests/myext.exp - expected output (E)
! 88: ext/myext/tests/myext.out - output from test script (O)
! 89: ext/myext/tests/myext.diff - diff of .out and .exp (D)
! 90:
! 91: Failed tests are always bugs. Either the test is bugged or not considering
! 92: factors applying to the tester's environment, or there is a bug in PHP.
! 93: If this is a known bug, we strive to provide bug numbers, in either the
! 94: test name or the file name. You can check the status of such a bug, by
! 95: going to: http://bugs.php.net/12345 where 12345 is the bug number.
! 96: For clarity and automated processing, bug numbers are prefixed by a hash
! 97: sign '#' in test names and/or test cases are named bug12345.phpt.
! 98:
! 99: NOTE: The files generated by tests can be selected by setting the
! 100: environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
! 101: generated use the character in brackets as shown above (default is LEOD).
! 102: The php file will be generated always.
! 103:
! 104: NOTE: You can set environment variable TEST_PHP_DETAILED to enable
! 105: detailed test information.
! 106:
! 107: [Automated testing]
! 108: If you like to keep up to speed, with latest developments and quality
! 109: assurance, setting the environment variable NO_INTERACTION to 1, will not
! 110: prompt the tester for any user input.
! 111:
! 112: Normally, the exit status of "make test" is zero, regardless of the results
! 113: of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
! 114: and "make test" will set the exit status ("$?") to non-zero, when an
! 115: individual test has failed.
! 116:
! 117: Example script to be run by cron(1):
! 118: ========== qa-test.sh =============
! 119: #!/bin/sh
! 120:
! 121: CO_DIR=$HOME/cvs/php5
! 122: MYMAIL=qa-test@domain.com
! 123: TMPDIR=/var/tmp
! 124: TODAY=`date +"%Y%m%d"`
! 125:
! 126: # Make sure compilation enviroment is correct
! 127: CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
! 128: export MAKE=gmake
! 129: export CC=gcc
! 130:
! 131: # Set test environment
! 132: export NO_INTERACTION=1
! 133: export REPORT_EXIT_STATUS=1
! 134:
! 135: cd $CO_DIR
! 136: cvs update . >>$TMPDIR/phpqatest.$TODAY
! 137: ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
! 138: $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
! 139: if test $? -gt 0
! 140: then
! 141: cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
! 142: fi
! 143: ========== end of qa-test.sh =============
! 144:
! 145: NOTE: the exit status of run-tests.php will be 1 when
! 146: REPORT_EXIT_STATUS is set. The result of "make test" may be higher
! 147: than that. At present, gmake 3.79.1 returns 2, so it is
! 148: advised to test for non-zero, rather then a specific value.
! 149:
! 150:
! 151: [Creating new test files]
! 152: -------------------------
! 153: Writing test file is very easy if you are used to PHP.
! 154: See the HOWTO at http://qa.php.net/write-test.php
! 155:
! 156:
! 157: [How to help us]
! 158: ----------------
! 159: If you find bug in PHP, you can submit bug report AND test script
! 160: for us. You don't have to write complete script, just give us test
! 161: script with following format. Please test the script and make sure
! 162: you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
! 163: submit.
! 164:
! 165: <?php
! 166: /*
! 167: Bug #12345
! 168: substr() bug. Do not return expected string.
! 169:
! 170: ACTUAL OUTPUT
! 171: XYXA
! 172:
! 173: EXPECTED OUTPUT
! 174: ABCD
! 175: */
! 176:
! 177: $str = "XYZABCD";
! 178: echo substr($str,3,7);
! 179:
! 180: ?>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README.TESTING CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / php