Annotation of embedaddon/php/README.TESTING, revision 1.1.1.1

1.1       misho       1: [IMPORTANT NOTICE]
                      2: ------------------
                      3:  Failed tests usualy indicate a problem with your local system setup
                      4: and not within PHP itself (at least for official PHP release versions).
                      5: You may decide to automaticaly submit a test summary to our QA workflow
                      6: at the end of a test run.
                      7:  Please do *not* submit a failed test as a bug or ask for help on why
                      8: it failed on your system without providing substantial backup information
                      9: on *why* the test failed on your special setup. Thank you :-)
                     10: 
                     11: 
                     12: [Testing Basics] 
                     13: ----------------
                     14:  The easiest way to test your PHP build is to run "make test" from the
                     15: command line after successfully compiling. This will run the complete
                     16: tests for all enabled functionalities and extensions using the PHP
                     17: CLI binary.
                     18:  To execute test scripts, you must build PHP with some SAPI, then you
                     19: type "make test" to execute all or some test scripts saved under
                     20: "tests" directory under source root directory.
                     21: 
                     22: Usage:
                     23: make test
                     24: 
                     25:  "make test" basically executes "run-tests.php" script
                     26: under the source root (parallel builds will not work). Therefore you
                     27: can execute the script as follows:
                     28: 
                     29: TEST_PHP_EXECUTABLE=sapi/cli/php \
                     30: sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
                     31: 
                     32: 
                     33: [Which "php" executable "make test" look for]
                     34: ---------------------------------------------
                     35: If you are running the run-tests.php script from the command line (as above)
                     36: you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
                     37: select the PHP executable that is to be tested, that is, used to run the test scripts.
                     38: 
                     39: If you run the tests using make test, the PHP CLI and CGI executables are 
                     40: automatically set for you. "make test" executes "run-tests.php" script with the CLI binary.  Some
                     41: test scripts such as session must be executed by CGI SAPI. Therefore,
                     42: you must build PHP with CGI SAPI to perform all tests.
                     43: 
                     44: NOTE: PHP binary executing "run-tests.php" and php binary used for
                     45: executing test scripts may differ. If you use different PHP binary for
                     46: executing "run-tests.php" script, you may get errors.
                     47: 
                     48: 
                     49: [Which php.ini is used]
                     50: -----------------------
                     51:  "make test" uses the same php.ini file as it would once installed.
                     52: The tests have been written to be independent of that php.ini file,
                     53: so if you find a test that is affected by a setting, please report
                     54: this, so we can address the issue.
                     55: 
                     56: 
                     57: [Which test scripts are executed]
                     58: ---------------------------------
                     59:  "run-tests.php" ("make test"), without any arguments executes all
                     60: test scripts by extracting all directories named "tests"
                     61: from the source root and any subdirectories below. If there are files,
                     62: which have a "phpt" extension, "run-tests.php" looks at the sections
                     63: in these files, determines whether it should run it, by evaluating
                     64: the 'SKIP' section. If the test is eligible for execution, the 'FILE'
                     65: section is extracted into a ".php" file (with the same name besides 
                     66: the extension) and gets executed.
                     67: When an argument is given or TESTS environment variable is set, the
                     68: GLOB is expanded by the shell and any file with extension "*.phpt" is
                     69: regarded as a test file.
                     70: 
                     71:  Tester can easily execute tests selectively with as follows.
                     72: 
                     73: Examples:
                     74: ./sapi/cli/php run-tests.php ext/mbstring/*
                     75: ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
                     76: 
                     77: 
                     78: [Test results]
                     79: --------------
                     80:  Test results are printed to standard output. If there is a failed test, 
                     81: the "run-tests.php" script saves the result, the expected result and the
                     82: code executed to the test script directory. For example, if 
                     83: ext/myext/tests/myext.phpt fails to pass, the following files are created:
                     84: 
                     85: ext/myext/tests/myext.php   - actual test file executed
                     86: ext/myext/tests/myext.log   - log of test execution (L)
                     87: ext/myext/tests/myext.exp   - expected output (E)
                     88: ext/myext/tests/myext.out   - output from test script (O)
                     89: ext/myext/tests/myext.diff  - diff of .out and .exp (D)
                     90: 
                     91:  Failed tests are always bugs. Either the test is bugged or not considering
                     92: factors applying to the tester's environment, or there is a bug in PHP.
                     93: If this is a known bug, we strive to provide bug numbers, in either the
                     94: test name or the file name. You can check the status of such a bug, by
                     95: going to: http://bugs.php.net/12345 where 12345 is the bug number.
                     96: For clarity and automated processing, bug numbers are prefixed by a hash
                     97: sign '#' in test names and/or test cases are named bug12345.phpt.
                     98: 
                     99: NOTE: The files generated by tests can be selected by setting the
                    100: environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
                    101: generated use the character in brackets as shown above (default is LEOD).
                    102: The php file will be generated always.
                    103: 
                    104: NOTE: You can set environment variable TEST_PHP_DETAILED to enable
                    105: detailed test information.
                    106: 
                    107: [Automated testing]
                    108:  If you like to keep up to speed, with latest developments and quality
                    109: assurance, setting the environment variable NO_INTERACTION to 1, will not
                    110: prompt the tester for any user input.
                    111: 
                    112: Normally, the exit status of "make test" is zero, regardless of the results
                    113: of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
                    114: and "make test" will set the exit status ("$?") to non-zero, when an
                    115: individual test has failed.
                    116: 
                    117: Example script to be run by cron(1):
                    118: ========== qa-test.sh =============
                    119: #!/bin/sh
                    120: 
                    121: CO_DIR=$HOME/cvs/php5
                    122: MYMAIL=qa-test@domain.com
                    123: TMPDIR=/var/tmp
                    124: TODAY=`date +"%Y%m%d"`
                    125: 
                    126: # Make sure compilation enviroment is correct
                    127: CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
                    128: export MAKE=gmake
                    129: export CC=gcc
                    130: 
                    131: # Set test environment
                    132: export NO_INTERACTION=1
                    133: export REPORT_EXIT_STATUS=1
                    134: 
                    135: cd $CO_DIR
                    136: cvs update . >>$TMPDIR/phpqatest.$TODAY
                    137: ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
                    138: $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
                    139: if test $? -gt 0
                    140: then
                    141:         cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
                    142: fi
                    143: ========== end of qa-test.sh =============
                    144: 
                    145: NOTE: the exit status of run-tests.php will be 1 when
                    146: REPORT_EXIT_STATUS is set. The result of "make test" may be higher
                    147: than that. At present, gmake 3.79.1 returns 2, so it is
                    148: advised to test for non-zero, rather then a specific value.
                    149: 
                    150: 
                    151: [Creating new test files]
                    152: -------------------------
                    153:  Writing test file is very easy if you are used to PHP. 
                    154: See the HOWTO at http://qa.php.net/write-test.php
                    155: 
                    156: 
                    157: [How to help us]
                    158: ----------------
                    159:  If you find bug in PHP, you can submit bug report AND test script 
                    160: for us. You don't have to write complete script, just give us test
                    161: script with following format. Please test the script and make sure
                    162: you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
                    163: submit.
                    164: 
                    165: <?php
                    166: /* 
                    167: Bug #12345
                    168: substr() bug. Do not return expected string.
                    169: 
                    170: ACTUAL OUTPUT
                    171: XYXA
                    172: 
                    173: EXPECTED OUTPUT
                    174: ABCD
                    175: */
                    176: 
                    177: $str = "XYZABCD";
                    178: echo substr($str,3,7);
                    179: 
                    180: ?>

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>