Annotation of embedaddon/curl/tests/README, revision 1.1
1.1 ! misho 1: _ _ ____ _
! 2: ___| | | | _ \| |
! 3: / __| | | | |_) | |
! 4: | (__| |_| | _ <| |___
! 5: \___|\___/|_| \_\_____|
! 6:
! 7: The curl Test Suite
! 8:
! 9: 1. Running
! 10: 1.1 Requires to run
! 11: 1.2 Port numbers used by test servers
! 12: 1.3 Test servers
! 13: 1.4 Run
! 14: 1.5 Shell startup scripts
! 15: 1.6 Memory test
! 16: 1.7 Debug
! 17: 1.8 Logs
! 18: 1.9 Test input files
! 19: 1.10 Code coverage
! 20: 1.11 Remote testing
! 21:
! 22: 2. Numbering
! 23: 2.1 Test case numbering
! 24:
! 25: 3. Write tests
! 26: 3.1 test data
! 27: 3.2 curl tests
! 28: 3.3 libcurl tests
! 29: 3.4 unit tests
! 30:
! 31: 4. TODO
! 32: 4.1 More protocols
! 33: 4.2 SOCKS auth
! 34:
! 35: ==============================================================================
! 36:
! 37: 1. Running
! 38:
! 39: 1.1 Requires to run
! 40:
! 41: perl (and a unix-style shell)
! 42: python (and a unix-style shell, for SMB and TELNET tests)
! 43: python-impacket (for SMB tests)
! 44: diff (when a test fails, a diff is shown)
! 45: stunnel (for HTTPS and FTPS tests)
! 46: OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
! 47: nghttpx (for HTTP/2 tests)
! 48: nroff (for --manual tests)
! 49:
! 50: 1.1.1 Installation of python-impacket
! 51:
! 52: The Python-based test servers support both recent Python 2 and 3.
! 53: You can figure out your default Python interpreter with python -V
! 54:
! 55: Please install python-impacket in the correct Python environment.
! 56: You can use pip or your OS' package manager to install 'impacket'.
! 57:
! 58: On Debian/Ubuntu the package names are:
! 59: Python 2: 'python-impacket'
! 60: Python 3: 'python3-impacket'
! 61:
! 62: On FreeBSD the package names are:
! 63: Python 2: 'py27-impacket'
! 64: Python 3: 'py37-impacket'
! 65:
! 66: On any system where pip is available:
! 67: Python 2: 'pip2 install impacket'
! 68: Python 3: 'pip3 install impacket'
! 69:
! 70: You may also need to manually install the Python package 'six'
! 71: as that may be a missing requirement for impacket on Python 3.
! 72:
! 73: 1.2 Port numbers used by test servers
! 74:
! 75: Tests are written to use as few fixed fixed port numbers as possible and all
! 76: tests should be written to use suitable variables instead of port numbers so
! 77: that test cases continue to work independent on what port numbers the test
! 78: servers actually use.
! 79:
! 80: The remaining fixed-port test servers that are still used, use the port
! 81: range 8890 - 8904 by default, but can be moved with runtests' -b option.
! 82:
! 83: See the FILEFORMAT for a listing of existing port number variables.
! 84:
! 85: 1.3 Test servers
! 86:
! 87: The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
! 88: servers on the ports listed above to which it makes requests. For SSL tests,
! 89: it runs stunnel to handle encryption to the regular servers. For SSH, it
! 90: runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
! 91: the SOCKS functionality and requires a SSH client and server.
! 92:
! 93: The base port number (8990), which all the individual port numbers are
! 94: indexed from, can be set explicitly using runtests.pl' -b option to allow
! 95: running more than one instance of the test suite simultaneously on one
! 96: machine, or just move the servers in case you have local services on any of
! 97: those ports.
! 98:
! 99: The HTTP server supports listening on a Unix domain socket, the default
! 100: location is 'http.sock'.
! 101:
! 102: 1.4 Run
! 103:
! 104: './configure && make && make test'. This builds the test suite support code
! 105: and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
! 106: variables of that script in case you have some specific needs, or run the
! 107: script manually (after the support code has been built).
! 108:
! 109: The script breaks on the first test that doesn't do OK. Use -a to prevent
! 110: the script from aborting on the first error. Run the script with -v for more
! 111: verbose output. Use -d to run the test servers with debug output enabled as
! 112: well. Specifying -k keeps all the log files generated by the test intact.
! 113:
! 114: Use -s for shorter output, or pass test numbers to run specific tests only
! 115: (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
! 116: ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
! 117: 3 to 9. Any test numbers starting with ! are disabled, as are any test
! 118: numbers found in the files data/DISABLED or data/DISABLED.local (one per
! 119: line). The latter is meant for local temporary disables and will be ignored
! 120: by git.
! 121:
! 122: When -s is not present, each successful test will display on one line the
! 123: test number and description and on the next line a set of flags, the test
! 124: result, current test sequence, total number of tests to be run and an
! 125: estimated amount of time to complete the test run. The flags consist of
! 126: these letters describing what is checked in this test:
! 127:
! 128: s stdout
! 129: d data
! 130: u upload
! 131: p protocol
! 132: o output
! 133: e exit code
! 134: m memory
! 135: v valgrind
! 136:
! 137: 1.5 Shell startup scripts
! 138:
! 139: Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
! 140: influenced by the output of system wide or user specific shell startup
! 141: scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
! 142: output text messages or escape sequences on user login. When these shell
! 143: startup messages or escape sequences are output they might corrupt the
! 144: expected stream of data which flows to the sftp-server or from the ssh
! 145: client which can result in bad test behaviour or even prevent the test
! 146: server from running.
! 147:
! 148: If the test suite ssh or sftp server fails to start up and logs the message
! 149: 'Received message too long' then you are certainly suffering the unwanted
! 150: output of a shell startup script. Locate, cleanup or adjust the shell
! 151: script.
! 152:
! 153: 1.6 Memory test
! 154:
! 155: The test script will check that all allocated memory is freed properly IF
! 156: curl has been built with the CURLDEBUG define set. The script will
! 157: automatically detect if that is the case, and it will use the
! 158: 'memanalyze.pl' script to analyze the memory debugging output.
! 159:
! 160: Also, if you run tests on a machine where valgrind is found, the script will
! 161: use valgrind to run the test with (unless you use -n) to further verify
! 162: correctness.
! 163:
! 164: runtests.pl's -t option will enable torture testing mode, which runs each
! 165: test many times and makes each different memory allocation fail on each
! 166: successive run. This tests the out of memory error handling code to ensure
! 167: that memory leaks do not occur even in those situations. It can help to
! 168: compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
! 169: ensure that the memory log file is properly written even if curl crashes.
! 170:
! 171: 1.7 Debug
! 172:
! 173: If a test case fails, you can conveniently get the script to invoke the
! 174: debugger (gdb) for you with the server running and the exact same command
! 175: line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
! 176: then just type 'run' in the debugger to perform the command through the
! 177: debugger.
! 178:
! 179: 1.8 Logs
! 180:
! 181: All logs are generated in the log/ subdirectory (it is emptied first in the
! 182: runtests.pl script). Use runtests.pl -k to force it to keep the temporary
! 183: files after the test run since successful runs will clean it up otherwise.
! 184:
! 185: 1.9 Test input files
! 186:
! 187: All test cases are put in the data/ subdirectory. Each test is stored in the
! 188: file named according to the test number.
! 189:
! 190: See FILEFORMAT for the description of the test case files.
! 191:
! 192: 1.10 Code coverage
! 193:
! 194: gcc provides a tool that can determine the code coverage figures for
! 195: the test suite. To use it, configure curl with
! 196: CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
! 197: and torture tests to get more full coverage, i.e. do:
! 198:
! 199: make test
! 200: make test-torture
! 201:
! 202: The graphical tool ggcov can be used to browse the source and create
! 203: coverage reports on *NIX hosts:
! 204:
! 205: ggcov -r lib src
! 206:
! 207: The text mode tool gcov may also be used, but it doesn't handle object files
! 208: in more than one directory very well.
! 209:
! 210: 1.11 Remote testing
! 211:
! 212: The runtests.pl script provides some hooks to allow curl to be tested on a
! 213: machine where perl can not be run. The test framework in this case runs on
! 214: a workstation where perl is available, while curl itself is run on a remote
! 215: system using ssh or some other remote execution method. See the comments at
! 216: the beginning of runtests.pl for details.
! 217:
! 218: 2. Numbering
! 219:
! 220: 2.1 Test case numbering
! 221:
! 222: Test cases used to be numbered by category, but the ranges filled
! 223: up. Subsets of tests can now be selected by passing keywords to the
! 224: runtests.pl script via the make TFLAGS variable.
! 225:
! 226: New tests should now be added by finding a free number in
! 227: tests/data/Makefile.inc.
! 228:
! 229: 3. Write tests
! 230:
! 231: Here's a quick description on writing test cases. We basically have three
! 232: kinds of tests: the ones that test the curl tool, the ones that build small
! 233: applications and test libcurl directly and the unit tests that test
! 234: individual (possibly internal) functions.
! 235:
! 236: 3.1 test data
! 237:
! 238: Each test has a master file that controls all the test data. What to read,
! 239: what the protocol exchange should look like, what exit code to expect and
! 240: what command line arguments to use etc.
! 241:
! 242: These files are tests/data/test[num] where [num] is described in section 2
! 243: of this document, and the XML-like file format of them is described in the
! 244: separate tests/FILEFORMAT document.
! 245:
! 246: 3.2 curl tests
! 247:
! 248: A test case that runs the curl tool and verifies that it gets the correct
! 249: data, it sends the correct data, it uses the correct protocol primitives
! 250: etc.
! 251:
! 252: 3.3 libcurl tests
! 253:
! 254: The libcurl tests are identical to the curl ones, except that they use a
! 255: specific and dedicated custom-built program to run instead of "curl". This
! 256: tool is built from source code placed in tests/libtest and if you want to
! 257: make a new libcurl test that is where you add your code.
! 258:
! 259: 3.4 unit tests
! 260:
! 261: Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
! 262: There's a tests/unit/README describing the specific set of checks and macros
! 263: that may be used when writing tests that verify behaviors of specific
! 264: individual functions.
! 265:
! 266: The unit tests depend on curl being built with debug enabled.
! 267:
! 268: 4. TODO
! 269:
! 270: 4.1 More protocols
! 271:
! 272: Add tests for TELNET, LDAP, DICT...
! 273:
! 274: 4.2 SOCKS auth
! 275:
! 276: SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
! 277: test mechanism) doesn't support them
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / tests