1: _ _ ____ _
2: ___| | | | _ \| |
3: / __| | | | |_) | |
4: | (__| |_| | _ <| |___
5: \___|\___/|_| \_\_____|
6:
7: The curl Test Suite
8:
9: 1. Running
10: 1.1 Requires to run
11: 1.2 Port numbers used by test servers
12: 1.3 Test servers
13: 1.4 Run
14: 1.5 Shell startup scripts
15: 1.6 Memory test
16: 1.7 Debug
17: 1.8 Logs
18: 1.9 Test input files
19: 1.10 Code coverage
20: 1.11 Remote testing
21:
22: 2. Numbering
23: 2.1 Test case numbering
24:
25: 3. Write tests
26: 3.1 test data
27: 3.2 curl tests
28: 3.3 libcurl tests
29: 3.4 unit tests
30:
31: 4. TODO
32: 4.1 More protocols
33: 4.2 SOCKS auth
34:
35: ==============================================================================
36:
37: 1. Running
38:
39: 1.1 Requires to run
40:
41: perl (and a unix-style shell)
42: python (and a unix-style shell, for SMB and TELNET tests)
43: python-impacket (for SMB tests)
44: diff (when a test fails, a diff is shown)
45: stunnel (for HTTPS and FTPS tests)
46: OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests)
47: nghttpx (for HTTP/2 tests)
48: nroff (for --manual tests)
49:
50: 1.1.1 Installation of python-impacket
51:
52: The Python-based test servers support both recent Python 2 and 3.
53: You can figure out your default Python interpreter with python -V
54:
55: Please install python-impacket in the correct Python environment.
56: You can use pip or your OS' package manager to install 'impacket'.
57:
58: On Debian/Ubuntu the package names are:
59: Python 2: 'python-impacket'
60: Python 3: 'python3-impacket'
61:
62: On FreeBSD the package names are:
63: Python 2: 'py27-impacket'
64: Python 3: 'py37-impacket'
65:
66: On any system where pip is available:
67: Python 2: 'pip2 install impacket'
68: Python 3: 'pip3 install impacket'
69:
70: You may also need to manually install the Python package 'six'
71: as that may be a missing requirement for impacket on Python 3.
72:
73: 1.2 Port numbers used by test servers
74:
75: Tests are written to use as few fixed fixed port numbers as possible and all
76: tests should be written to use suitable variables instead of port numbers so
77: that test cases continue to work independent on what port numbers the test
78: servers actually use.
79:
80: The remaining fixed-port test servers that are still used, use the port
81: range 8890 - 8904 by default, but can be moved with runtests' -b option.
82:
83: See the FILEFORMAT for a listing of existing port number variables.
84:
85: 1.3 Test servers
86:
87: The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone
88: servers on the ports listed above to which it makes requests. For SSL tests,
89: it runs stunnel to handle encryption to the regular servers. For SSH, it
90: runs a standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform
91: the SOCKS functionality and requires a SSH client and server.
92:
93: The base port number (8990), which all the individual port numbers are
94: indexed from, can be set explicitly using runtests.pl' -b option to allow
95: running more than one instance of the test suite simultaneously on one
96: machine, or just move the servers in case you have local services on any of
97: those ports.
98:
99: The HTTP server supports listening on a Unix domain socket, the default
100: location is 'http.sock'.
101:
102: 1.4 Run
103:
104: './configure && make && make test'. This builds the test suite support code
105: and invokes the 'runtests.pl' perl script to run all the tests. Edit the top
106: variables of that script in case you have some specific needs, or run the
107: script manually (after the support code has been built).
108:
109: The script breaks on the first test that doesn't do OK. Use -a to prevent
110: the script from aborting on the first error. Run the script with -v for more
111: verbose output. Use -d to run the test servers with debug output enabled as
112: well. Specifying -k keeps all the log files generated by the test intact.
113:
114: Use -s for shorter output, or pass test numbers to run specific tests only
115: (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case
116: ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from
117: 3 to 9. Any test numbers starting with ! are disabled, as are any test
118: numbers found in the files data/DISABLED or data/DISABLED.local (one per
119: line). The latter is meant for local temporary disables and will be ignored
120: by git.
121:
122: When -s is not present, each successful test will display on one line the
123: test number and description and on the next line a set of flags, the test
124: result, current test sequence, total number of tests to be run and an
125: estimated amount of time to complete the test run. The flags consist of
126: these letters describing what is checked in this test:
127:
128: s stdout
129: d data
130: u upload
131: p protocol
132: o output
133: e exit code
134: m memory
135: v valgrind
136:
137: 1.5 Shell startup scripts
138:
139: Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly
140: influenced by the output of system wide or user specific shell startup
141: scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which
142: output text messages or escape sequences on user login. When these shell
143: startup messages or escape sequences are output they might corrupt the
144: expected stream of data which flows to the sftp-server or from the ssh
145: client which can result in bad test behaviour or even prevent the test
146: server from running.
147:
148: If the test suite ssh or sftp server fails to start up and logs the message
149: 'Received message too long' then you are certainly suffering the unwanted
150: output of a shell startup script. Locate, cleanup or adjust the shell
151: script.
152:
153: 1.6 Memory test
154:
155: The test script will check that all allocated memory is freed properly IF
156: curl has been built with the CURLDEBUG define set. The script will
157: automatically detect if that is the case, and it will use the
158: 'memanalyze.pl' script to analyze the memory debugging output.
159:
160: Also, if you run tests on a machine where valgrind is found, the script will
161: use valgrind to run the test with (unless you use -n) to further verify
162: correctness.
163:
164: runtests.pl's -t option will enable torture testing mode, which runs each
165: test many times and makes each different memory allocation fail on each
166: successive run. This tests the out of memory error handling code to ensure
167: that memory leaks do not occur even in those situations. It can help to
168: compile curl with CPPFLAGS=-DMEMDEBUG_LOG_SYNC when using this option, to
169: ensure that the memory log file is properly written even if curl crashes.
170:
171: 1.7 Debug
172:
173: If a test case fails, you can conveniently get the script to invoke the
174: debugger (gdb) for you with the server running and the exact same command
175: line parameters that failed. Just invoke 'runtests.pl <test number> -g' and
176: then just type 'run' in the debugger to perform the command through the
177: debugger.
178:
179: 1.8 Logs
180:
181: All logs are generated in the log/ subdirectory (it is emptied first in the
182: runtests.pl script). Use runtests.pl -k to force it to keep the temporary
183: files after the test run since successful runs will clean it up otherwise.
184:
185: 1.9 Test input files
186:
187: All test cases are put in the data/ subdirectory. Each test is stored in the
188: file named according to the test number.
189:
190: See FILEFORMAT for the description of the test case files.
191:
192: 1.10 Code coverage
193:
194: gcc provides a tool that can determine the code coverage figures for
195: the test suite. To use it, configure curl with
196: CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal
197: and torture tests to get more full coverage, i.e. do:
198:
199: make test
200: make test-torture
201:
202: The graphical tool ggcov can be used to browse the source and create
203: coverage reports on *NIX hosts:
204:
205: ggcov -r lib src
206:
207: The text mode tool gcov may also be used, but it doesn't handle object files
208: in more than one directory very well.
209:
210: 1.11 Remote testing
211:
212: The runtests.pl script provides some hooks to allow curl to be tested on a
213: machine where perl can not be run. The test framework in this case runs on
214: a workstation where perl is available, while curl itself is run on a remote
215: system using ssh or some other remote execution method. See the comments at
216: the beginning of runtests.pl for details.
217:
218: 2. Numbering
219:
220: 2.1 Test case numbering
221:
222: Test cases used to be numbered by category, but the ranges filled
223: up. Subsets of tests can now be selected by passing keywords to the
224: runtests.pl script via the make TFLAGS variable.
225:
226: New tests should now be added by finding a free number in
227: tests/data/Makefile.inc.
228:
229: 3. Write tests
230:
231: Here's a quick description on writing test cases. We basically have three
232: kinds of tests: the ones that test the curl tool, the ones that build small
233: applications and test libcurl directly and the unit tests that test
234: individual (possibly internal) functions.
235:
236: 3.1 test data
237:
238: Each test has a master file that controls all the test data. What to read,
239: what the protocol exchange should look like, what exit code to expect and
240: what command line arguments to use etc.
241:
242: These files are tests/data/test[num] where [num] is described in section 2
243: of this document, and the XML-like file format of them is described in the
244: separate tests/FILEFORMAT document.
245:
246: 3.2 curl tests
247:
248: A test case that runs the curl tool and verifies that it gets the correct
249: data, it sends the correct data, it uses the correct protocol primitives
250: etc.
251:
252: 3.3 libcurl tests
253:
254: The libcurl tests are identical to the curl ones, except that they use a
255: specific and dedicated custom-built program to run instead of "curl". This
256: tool is built from source code placed in tests/libtest and if you want to
257: make a new libcurl test that is where you add your code.
258:
259: 3.4 unit tests
260:
261: Unit tests are tests in the 13xx sequence and they are placed in tests/unit.
262: There's a tests/unit/README describing the specific set of checks and macros
263: that may be used when writing tests that verify behaviors of specific
264: individual functions.
265:
266: The unit tests depend on curl being built with debug enabled.
267:
268: 4. TODO
269:
270: 4.1 More protocols
271:
272: Add tests for TELNET, LDAP, DICT...
273:
274: 4.2 SOCKS auth
275:
276: SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the
277: test mechanism) doesn't support them
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to README CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / tests