File:  [ELWIX - Embedded LightWeight unIX -] / embedaddon / php / README.TESTING
Revision 1.1.1.2 (vendor branch): download - view: text, annotated - select for diffs - revision graph
Mon Jul 22 01:31:37 2013 UTC (11 years, 5 months ago) by misho
Branches: php, MAIN
CVS tags: v5_4_29p0, v5_4_29, v5_4_20p0, v5_4_20, v5_4_17, HEAD
5.4.17

    1: [IMPORTANT NOTICE]
    2: ------------------
    3:  Failed tests usually indicate a problem with your local system setup
    4: and not within PHP itself (at least for official PHP release versions).
    5: You may decide to automaticaly submit a test summary to our QA workflow
    6: at the end of a test run.
    7:  Please do *not* submit a failed test as a bug or ask for help on why
    8: it failed on your system without providing substantial backup information
    9: on *why* the test failed on your special setup. Thank you :-)
   10: 
   11: 
   12: [Testing Basics] 
   13: ----------------
   14:  The easiest way to test your PHP build is to run "make test" from the
   15: command line after successfully compiling. This will run the complete
   16: tests for all enabled functionalities and extensions using the PHP
   17: CLI binary.
   18:  To execute test scripts, you must build PHP with some SAPI, then you
   19: type "make test" to execute all or some test scripts saved under
   20: "tests" directory under source root directory.
   21: 
   22: Usage:
   23: make test
   24: 
   25:  "make test" basically executes "run-tests.php" script
   26: under the source root (parallel builds will not work). Therefore you
   27: can execute the script as follows:
   28: 
   29: TEST_PHP_EXECUTABLE=sapi/cli/php \
   30: sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
   31: 
   32: 
   33: [Which "php" executable "make test" look for]
   34: ---------------------------------------------
   35: If you are running the run-tests.php script from the command line (as above)
   36: you must set the TEST_PHP_EXECUTABLE environment variable to explicitly
   37: select the PHP executable that is to be tested, that is, used to run the test scripts.
   38: 
   39: If you run the tests using make test, the PHP CLI and CGI executables are 
   40: automatically set for you. "make test" executes "run-tests.php" script with the CLI binary.  Some
   41: test scripts such as session must be executed by CGI SAPI. Therefore,
   42: you must build PHP with CGI SAPI to perform all tests.
   43: 
   44: NOTE: PHP binary executing "run-tests.php" and php binary used for
   45: executing test scripts may differ. If you use different PHP binary for
   46: executing "run-tests.php" script, you may get errors.
   47: 
   48: 
   49: [Which php.ini is used]
   50: -----------------------
   51:  "make test" uses the same php.ini file as it would once installed.
   52: The tests have been written to be independent of that php.ini file,
   53: so if you find a test that is affected by a setting, please report
   54: this, so we can address the issue.
   55: 
   56: 
   57: [Which test scripts are executed]
   58: ---------------------------------
   59:  "run-tests.php" ("make test"), without any arguments executes all
   60: test scripts by extracting all directories named "tests"
   61: from the source root and any subdirectories below. If there are files,
   62: which have a "phpt" extension, "run-tests.php" looks at the sections
   63: in these files, determines whether it should run it, by evaluating
   64: the 'SKIP' section. If the test is eligible for execution, the 'FILE'
   65: section is extracted into a ".php" file (with the same name besides 
   66: the extension) and gets executed.
   67: When an argument is given or TESTS environment variable is set, the
   68: GLOB is expanded by the shell and any file with extension "*.phpt" is
   69: regarded as a test file.
   70: 
   71:  Tester can easily execute tests selectively with as follows.
   72: 
   73: Examples:
   74: ./sapi/cli/php run-tests.php ext/mbstring/*
   75: ./sapi/cli/php run-tests.php ext/mbstring/020.phpt
   76: 
   77: 
   78: [Test results]
   79: --------------
   80:  Test results are printed to standard output. If there is a failed test, 
   81: the "run-tests.php" script saves the result, the expected result and the
   82: code executed to the test script directory. For example, if 
   83: ext/myext/tests/myext.phpt fails to pass, the following files are created:
   84: 
   85: ext/myext/tests/myext.php   - actual test file executed
   86: ext/myext/tests/myext.log   - log of test execution (L)
   87: ext/myext/tests/myext.exp   - expected output (E)
   88: ext/myext/tests/myext.out   - output from test script (O)
   89: ext/myext/tests/myext.diff  - diff of .out and .exp (D)
   90: 
   91:  Failed tests are always bugs. Either the test is bugged or not considering
   92: factors applying to the tester's environment, or there is a bug in PHP.
   93: If this is a known bug, we strive to provide bug numbers, in either the
   94: test name or the file name. You can check the status of such a bug, by
   95: going to: http://bugs.php.net/12345 where 12345 is the bug number.
   96: For clarity and automated processing, bug numbers are prefixed by a hash
   97: sign '#' in test names and/or test cases are named bug12345.phpt.
   98: 
   99: NOTE: The files generated by tests can be selected by setting the
  100: environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
  101: generated use the character in brackets as shown above (default is LEOD).
  102: The php file will be generated always.
  103: 
  104: NOTE: You can set environment variable TEST_PHP_DETAILED to enable
  105: detailed test information.
  106: 
  107: [Automated testing]
  108:  If you like to keep up to speed, with latest developments and quality
  109: assurance, setting the environment variable NO_INTERACTION to 1, will not
  110: prompt the tester for any user input.
  111: 
  112: Normally, the exit status of "make test" is zero, regardless of the results
  113: of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
  114: and "make test" will set the exit status ("$?") to non-zero, when an
  115: individual test has failed.
  116: 
  117: Example script to be run by cron(1):
  118: ========== qa-test.sh =============
  119: #!/bin/sh
  120: 
  121: CO_DIR=$HOME/cvs/php5
  122: MYMAIL=qa-test@domain.com
  123: TMPDIR=/var/tmp
  124: TODAY=`date +"%Y%m%d"`
  125: 
  126: # Make sure compilation enviroment is correct
  127: CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
  128: export MAKE=gmake
  129: export CC=gcc
  130: 
  131: # Set test environment
  132: export NO_INTERACTION=1
  133: export REPORT_EXIT_STATUS=1
  134: 
  135: cd $CO_DIR
  136: cvs update . >>$TMPDIR/phpqatest.$TODAY
  137: ./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
  138: $MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
  139: if test $? -gt 0
  140: then
  141:         cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
  142: fi
  143: ========== end of qa-test.sh =============
  144: 
  145: NOTE: the exit status of run-tests.php will be 1 when
  146: REPORT_EXIT_STATUS is set. The result of "make test" may be higher
  147: than that. At present, gmake 3.79.1 returns 2, so it is
  148: advised to test for non-zero, rather then a specific value.
  149: 
  150: 
  151: [Creating new test files]
  152: -------------------------
  153:  Writing test file is very easy if you are used to PHP. 
  154: See the HOWTO at http://qa.php.net/write-test.php
  155: 
  156: 
  157: [How to help us]
  158: ----------------
  159:  If you find bug in PHP, you can submit bug report AND test script 
  160: for us. You don't have to write complete script, just give us test
  161: script with following format. Please test the script and make sure
  162: you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
  163: submit.
  164: 
  165: <?php
  166: /* 
  167: Bug #12345
  168: substr() bug. Do not return expected string.
  169: 
  170: ACTUAL OUTPUT
  171: XYXA
  172: 
  173: EXPECTED OUTPUT
  174: ABCD
  175: */
  176: 
  177: $str = "XYZABCD";
  178: echo substr($str,3,7);
  179: 
  180: ?>

FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>