Annotation of embedaddon/curl/tests/FILEFORMAT.md, revision 1.1
1.1 ! misho 1: # curl test suite file format
! 2:
! 3: The curl test suite's file format is very simple and extensible, closely
! 4: resembling XML. All data for a single test case resides in a single ASCII
! 5: file. Labels mark the beginning and the end of all sections, and each label
! 6: must be written in its own line. Comments are either XML-style (enclosed with
! 7: `<!--` and `-->`) or shell script style (beginning with `#`) and must appear
! 8: on their own lines and not alongside actual test data. Most test data files
! 9: are syntactically valid XML, although a few files are not (lack of support for
! 10: character entities and the preservation of CR/LF characters at the end of
! 11: lines are the biggest differences).
! 12:
! 13: Each test case exists as a file matching the format `tests/data/testNUM`,
! 14: where NUM is considered the unique test number.
! 15:
! 16: The file begins with a 'testcase' tag, which encompasses the remainder of the
! 17: file.
! 18:
! 19: # `<testcase>`
! 20:
! 21: Each test is always within the testcase tag. Each test case is split up in
! 22: four main sections: `info`, `reply`, `client` and `verify`.
! 23:
! 24: - **info** provides information about the test case
! 25:
! 26: - **reply** is used for the server to know what to send as a reply for the
! 27: requests curl sends
! 28:
! 29: - **client** defines how the client should behave
! 30:
! 31: - **verify** defines how to verify that the data stored after a command has
! 32: been run ended up correctly
! 33:
! 34: Each main section has a number of available subsections that can be specified,
! 35: that will be checked/used if specified.
! 36:
! 37: ## `<info>`
! 38:
! 39: ### `<keywords>`
! 40: A newline-separated list of keywords describing what this test case uses and
! 41: tests. Try to use an already used keyword. These keywords will be used for
! 42: statistical/informational purposes and for choosing or skipping classes
! 43: of tests. "Keywords" must begin with an alphabetic character, "-", "["
! 44: or "{" and may actually consist of multiple words separated by spaces
! 45: which are treated together as a single identifier.
! 46:
! 47: ## `<reply>`
! 48:
! 49: ### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"]>`
! 50:
! 51: data to be sent to the client on its request and later verified that it
! 52: arrived safely. Set `nocheck="yes"` to prevent the test script from verifying
! 53: the arrival of this data.
! 54:
! 55: If the data contains `swsclose` anywhere within the start and end tag, and
! 56: this is a HTTP test, then the connection will be closed by the server after
! 57: this response is sent. If not, the connection will be kept persistent.
! 58:
! 59: If the data contains `swsbounce` anywhere within the start and end tag, the
! 60: HTTP server will detect if this is a second request using the same test and
! 61: part number and will then increase the part number with one. This is useful
! 62: for auth tests and similar.
! 63:
! 64: `sendzero=yes` means that the (FTP) server will "send" the data even if the
! 65: size is zero bytes. Used to verify curl's behaviour on zero bytes transfers.
! 66:
! 67: `base64=yes` means that the data provided in the test-file is a chunk of data
! 68: encoded with base64. It is the only way a test case can contain binary
! 69: data. (This attribute can in fact be used on any section, but it doesn't make
! 70: much sense for other sections than "data").
! 71:
! 72: `hex=yes` means that the data is a sequence of hex pairs. It will get decoded
! 73: and used as "raw" data.
! 74:
! 75: For FTP file listings, the `<data>` section will be used *only* if you make
! 76: sure that there has been a CWD done first to a directory named `test-[num]`
! 77: where [num] is the test case number. Otherwise the ftp server can't know from
! 78: which test file to load the list content.
! 79:
! 80: ### `<dataNUM>`
! 81:
! 82: Send back this contents instead of the <data> one. The num is set by:
! 83:
! 84: - The test number in the request line is >10000 and this is the remainder
! 85: of [test case number]%10000.
! 86: - The request was HTTP and included digest details, which adds 1000 to NUM
! 87: - If a HTTP request is NTLM type-1, it adds 1001 to num
! 88: - If a HTTP request is NTLM type-3, it adds 1002 to num
! 89: - If a HTTP request is Basic and num is already >=1000, it adds 1 to num
! 90: - If a HTTP request is Negotiate, num gets incremented by one for each
! 91: request with Negotiate authorization header on the same test case.
! 92:
! 93: Dynamically changing num in this way allows the test harness to be used to
! 94: test authentication negotiation where several different requests must be sent
! 95: to complete a transfer. The response to each request is found in its own data
! 96: section. Validating the entire negotiation sequence can be done by specifying
! 97: a datacheck section.
! 98:
! 99: ### `<connect>`
! 100: The connect section is used instead of the 'data' for all CONNECT
! 101: requests. The remainder of the rules for the data section then apply but with
! 102: a connect prefix.
! 103:
! 104: ### `<datacheck [mode="text"] [nonewline="yes"]>`
! 105: if the data is sent but this is what should be checked afterwards. If
! 106: `nonewline=yes` is set, runtests will cut off the trailing newline from the
! 107: data before comparing with the one actually received by the client.
! 108:
! 109: Use the `mode="text"` attribute if the output is in text mode on platforms
! 110: that have a text/binary difference.
! 111:
! 112: ### `<datacheckNUM [nonewline="yes"] [mode="text"]>`
! 113: The contents of numbered datacheck sections are appended to the non-numbered
! 114: one.
! 115:
! 116: ### `<size>`
! 117: number to return on a ftp SIZE command (set to -1 to make this command fail)
! 118:
! 119: ### `<mdtm>`
! 120: what to send back if the client sends a (FTP) MDTM command, set to -1 to
! 121: have it return that the file doesn't exist
! 122:
! 123: ### `<postcmd>`
! 124: special purpose server-command to control its behavior *after* the
! 125: reply is sent
! 126: For HTTP/HTTPS, these are supported:
! 127:
! 128: `wait [secs]` - Pause for the given time
! 129:
! 130: ### `<servercmd>`
! 131: Special-commands for the server.
! 132:
! 133: The first line of this file will always be set to `Testnum [number]` by the
! 134: test script, to allow servers to read that to know what test the client is
! 135: about to issue.
! 136:
! 137: #### For FTP/SMTP/POP/IMAP
! 138:
! 139: - `REPLY [command] [return value] [response string]` - Changes how the server
! 140: responds to the [command]. [response string] is evaluated as a perl string,
! 141: so it can contain embedded \r\n, for example. There's a special [command]
! 142: named "welcome" (without quotes) which is the string sent immediately on
! 143: connect as a welcome.
! 144: - `REPLYLF` (like above but sends the response terminated with LF-only and not
! 145: CRLF)
! 146: - `COUNT [command] [num]` - Do the `REPLY` change for `[command]` only `[num]`
! 147: times and then go back to the built-in approach
! 148: - `DELAY [command] [secs]` - Delay responding to this command for the given
! 149: time
! 150: - `RETRWEIRDO` - Enable the "weirdo" RETR case when multiple response lines
! 151: appear at once when a file is transferred
! 152: - `RETRNOSIZE` - Make sure the RETR response doesn't contain the size of the
! 153: file
! 154: - `NOSAVE` - Don't actually save what is received
! 155: - `SLOWDOWN` - Send FTP responses with 0.01 sec delay between each byte
! 156: - `PASVBADIP` - makes PASV send back an illegal IP in its 227 response
! 157: - `CAPA [capabilities]` - Enables support for and specifies a list of space
! 158: separated capabilities to return to the client for the IMAP `CAPABILITY`,
! 159: POP3 `CAPA` and SMTP `EHLO` commands
! 160: - `AUTH [mechanisms]` - Enables support for SASL authentication and specifies
! 161: a list of space separated mechanisms for IMAP, POP3 and SMTP
! 162:
! 163: #### For HTTP/HTTPS
! 164:
! 165: - `auth_required` if this is set and a POST/PUT is made without auth, the
! 166: server will NOT wait for the full request body to get sent
! 167: - `idle` - do nothing after receiving the request, just "sit idle"
! 168: - `stream` - continuously send data to the client, never-ending
! 169: - `writedelay: [secs]` delay this amount between reply packets
! 170: - `skip: [num]` - instructs the server to ignore reading this many bytes from
! 171: a PUT or POST request
! 172: - `rtp: part [num] channel [num] size [num]` - stream a fake RTP packet for
! 173: the given part on a chosen channel with the given payload size
! 174: - `connection-monitor` - When used, this will log `[DISCONNECT]` to the
! 175: `server.input` log when the connection is disconnected.
! 176: - `upgrade` - when an HTTP upgrade header is found, the server will upgrade to
! 177: http2
! 178: - `swsclose` - instruct server to close connection after response
! 179: - `no-expect` - don't read the request body if Expect: is present
! 180:
! 181: #### For TFTP
! 182: `writedelay: [secs]` delay this amount between reply packets (each packet
! 183: being 512 bytes payload)
! 184:
! 185: ## `<client>`
! 186:
! 187: ### `<server>`
! 188: What server(s) this test case requires/uses. Available servers:
! 189:
! 190: - `file`
! 191: - `ftp-ipv6`
! 192: - `ftp`
! 193: - `ftps`
! 194: - `http-ipv6`
! 195: - `http-proxy`
! 196: - `http-unix`
! 197: - `http/2`
! 198: - `http`
! 199: - `https`
! 200: - `httptls+srp-ipv6`
! 201: - `httptls+srp`
! 202: - `imap`
! 203: - `mqtt`
! 204: - `none`
! 205: - `pop3`
! 206: - `rtsp-ipv6`
! 207: - `rtsp`
! 208: - `scp`
! 209: - `sftp`
! 210: - `smtp`
! 211: - `socks4`
! 212: - `socks5`
! 213:
! 214: Give only one per line. This subsection is mandatory.
! 215:
! 216: ### `<features>`
! 217: A list of features that MUST be present in the client/library for this test to
! 218: be able to run. If a required feature is not present then the test will be
! 219: SKIPPED.
! 220:
! 221: Alternatively a feature can be prefixed with an exclamation mark to indicate a
! 222: feature is NOT required. If the feature is present then the test will be
! 223: SKIPPED.
! 224:
! 225: Features testable here are:
! 226:
! 227: - `alt-svc`
! 228: - `crypto`
! 229: - `debug`
! 230: - `getrlimit`
! 231: - `GnuTLS`
! 232: - `GSS-API`
! 233: - `http/2`
! 234: - `idn`
! 235: - `ipv6`
! 236: - `Kerberos`
! 237: - `large_file`
! 238: - `ld_preload`
! 239: - `libz`
! 240: - `manual`
! 241: - `Metalink`
! 242: - `NSS`
! 243: - `NTLM`
! 244: - `OpenSSL`
! 245: - `PSL`
! 246: - `socks`
! 247: - `SPNEGO`
! 248: - `SSL`
! 249: - `SSLpinning`
! 250: - `SSPI`
! 251: - `threaded-resolver`
! 252: - `TLS-SRP`
! 253: - `TrackMemory`
! 254: - `unittest`
! 255: - `unix-sockets`
! 256: - `win32`
! 257: - `WinSSL`
! 258:
! 259: as well as each protocol that curl supports. A protocol only needs to be
! 260: specified if it is different from the server (useful when the server
! 261: is `none`).
! 262:
! 263: ### `<killserver>`
! 264: Using the same syntax as in `<server>` but when mentioned here these servers
! 265: are explicitly KILLED when this test case is completed. Only use this if there
! 266: is no other alternatives. Using this of course requires subsequent tests to
! 267: restart servers.
! 268:
! 269: ### `<precheck>`
! 270: A command line that if set gets run by the test script before the test. If an
! 271: output is displayed by the command or if the return code is non-zero, the test
! 272: will be skipped and the (single-line) output will be displayed as reason for
! 273: not running the test. Variables are substituted as in the `<command>`
! 274: section.
! 275:
! 276: ### `<postcheck>`
! 277: A command line that if set gets run by the test script after the test. If
! 278: the command exists with a non-zero status code, the test will be considered
! 279: to have failed. Variables are substituted as in the `<command>` section.
! 280:
! 281: ### `<tool>`
! 282: Name of tool to invoke instead of "curl". This tool must be built and exist
! 283: either in the libtest/ directory (if the tool name starts with 'lib') or in
! 284: the unit/ directory (if the tool name starts with 'unit').
! 285:
! 286: ### `<name>`
! 287: Brief test case description, shown when the test runs.
! 288:
! 289: ### `<setenv>`
! 290: variable1=contents1
! 291: variable2=contents2
! 292:
! 293: Set the given environment variables to the specified value before the actual
! 294: command is run. They are cleared again after the command has been run.
! 295: Variables are first substituted as in the `<command>` section.
! 296: ### `<command [option="no-output/no-include/force-output/binary-trace"] [timeout="secs"][delay="secs"][type="perl"]>`
! 297: Command line to run. There's a bunch of %variables that get replaced
! 298: accordingly.
! 299:
! 300: Note that the URL that gets passed to the server actually controls what data
! 301: that is returned. The last slash in the URL must be followed by a number. That
! 302: number (N) will be used by the test-server to load test case N and return the
! 303: data that is defined within the `<reply><data></data></reply>` section.
! 304:
! 305: If there's no test number found above, the HTTP test server will use the
! 306: number following the last dot in the given hostname (made so that a CONNECT
! 307: can still pass on test number) so that "foo.bar.123" gets treated as test case
! 308: 123. Alternatively, if an IPv6 address is provided to CONNECT, the last
! 309: hexadecimal group in the address will be used as the test number! For example
! 310: the address "[1234::ff]" would be treated as test case 255.
! 311:
! 312: Set `type="perl"` to write the test case as a perl script. It implies that
! 313: there's no memory debugging and valgrind gets shut off for this test.
! 314:
! 315: Set `option="no-output"` to prevent the test script to slap on the `--output`
! 316: argument that directs the output to a file. The `--output` is also not added
! 317: if the verify/stdout section is used.
! 318:
! 319: Set `option="force-output"` to make use of `--output` even when the test is
! 320: otherwise written to verify stdout.
! 321:
! 322: Set `option="no-include"` to prevent the test script to slap on the
! 323: `--include` argument.
! 324:
! 325: Set `option="binary-trace"` to use `--trace` instead of `--trace-ascii` for
! 326: tracing. Suitable for binary-oriented protocols such as MQTT.
! 327:
! 328: Set `timeout="secs"` to override default server logs advisor read lock
! 329: timeout. This timeout is used by the test harness, once that the command has
! 330: completed execution, to wait for the test server to write out server side log
! 331: files and remove the lock that advised not to read them. The "secs" parameter
! 332: is the not negative integer number of seconds for the timeout. This `timeout`
! 333: attribute is documented for completeness sake, but is deep test harness stuff
! 334: and only needed for very singular and specific test cases. Avoid using it.
! 335:
! 336: Set `delay="secs"` to introduce a time delay once that the command has
! 337: completed execution and before the `<postcheck>` section runs. The "secs"
! 338: parameter is the not negative integer number of seconds for the delay. This
! 339: 'delay' attribute is intended for very specific test cases, and normally not
! 340: needed.
! 341:
! 342: Available substitute variables include:
! 343:
! 344: - `%CLIENT6IP` - IPv6 address of the client running curl
! 345: - `%CLIENTIP` - IPv4 address of the client running curl
! 346: - `%CURL` - Path to the curl executable
! 347: - `%FILE_PWD` - Current directory, on windows prefixed with a slash
! 348: - `%FTP2PORT` - Port number of the FTP server 2
! 349: - `%FTP6PORT` - IPv6 port number of the FTP server
! 350: - `%FTPPORT` - Port number of the FTP server
! 351: - `%FTPSPORT` - Port number of the FTPS server
! 352: - `%FTPTIME2` - Timeout in seconds that should be just sufficient to receive a response from the test FTP server
! 353: - `%FTPTIME3` - Even longer than %FTPTIME2
! 354: - `%GOPHER6PORT` - IPv6 port number of the Gopher server
! 355: - `%GOPHERPORT` - Port number of the Gopher server
! 356: - `%HOST6IP` - IPv6 address of the host running this test
! 357: - `%HOSTIP` - IPv4 address of the host running this test
! 358: - `%HTTP6PORT` - IPv6 port number of the HTTP server
! 359: - `%HTTPPORT` - Port number of the HTTP server
! 360: - `%HTTPSPORT` - Port number of the HTTPS server
! 361: - `%HTTPTLS6PORT` - IPv6 port number of the HTTP TLS server
! 362: - `%HTTPTLSPORT` - Port number of the HTTP TLS server
! 363: - `%HTTPUNIXPATH` - Path to the Unix socket of the HTTP server
! 364: - `%IMAP6PORT` - IPv6 port number of the IMAP server
! 365: - `%IMAPPORT` - Port number of the IMAP server
! 366: - `%MQTTPORT` - Port number of the MQTT server
! 367: - `%NEGTELNETPORT` - Port number of the telnet server
! 368: - `%NOLISTENPORT` - Port number where no service is listening
! 369: - `%POP36PORT` - IPv6 port number of the POP3 server
! 370: - `%POP3PORT` - Port number of the POP3 server
! 371: - `%POSIX_PWD` - Current directory somewhat mingw friendly
! 372: - `%PROXYPORT` - Port number of the HTTP proxy
! 373: - `%PWD` - Current directory
! 374: - `%RTSP6PORT` - IPv6 port number of the RTSP server
! 375: - `%RTSPPORT` - Port number of the RTSP server
! 376: - `%SMBPORT` - Port number of the SMB server
! 377: - `%SMBSPORT` - Port number of the SMBS server
! 378: - `%SMTP6PORT` - IPv6 port number of the SMTP server
! 379: - `%SMTPPORT` - Port number of the SMTP server
! 380: - `%SOCKSPORT` - Port number of the SOCKS4/5 server
! 381: - `%SRCDIR` - Full path to the source dir
! 382: - `%SSHPORT` - Port number of the SCP/SFTP server
! 383: - `%SSHSRVMD5` - MD5 of SSH server's public key
! 384: - `%TFTP6PORT` - IPv6 port number of the TFTP server
! 385: - `%TFTPPORT` - Port number of the TFTP server
! 386: - `%USER` - Login ID of the user running the test
! 387:
! 388: ### `<file name="log/filename">`
! 389: This creates the named file with this content before the test case is run,
! 390: which is useful if the test case needs a file to act on. Variables are
! 391: substituted on the contents of the file as in the `<command>` section.
! 392:
! 393: ### `<stdin [nonewline="yes"]>`
! 394: Pass this given data on stdin to the tool.
! 395:
! 396: If 'nonewline' is set, we will cut off the trailing newline of this given data
! 397: before comparing with the one actually received by the client
! 398:
! 399: ## `<verify>`
! 400: ### `<errorcode>`
! 401: numerical error code curl is supposed to return. Specify a list of accepted
! 402: error codes by separating multiple numbers with comma. See test 237 for an
! 403: example.
! 404:
! 405: ### `<strip>`
! 406: One regex per line that is removed from the protocol dumps before the
! 407: comparison is made. This is very useful to remove dependencies on dynamically
! 408: changing protocol data such as port numbers or user-agent strings.
! 409:
! 410: ### `<strippart>`
! 411: One perl op per line that operates on the protocol dump. This is pretty
! 412: advanced. Example: `s/^EPRT .*/EPRT stripped/`.
! 413:
! 414: ### `<protocol [nonewline="yes"]>`
! 415:
! 416: the protocol dump curl should transmit, if 'nonewline' is set, we will cut off
! 417: the trailing newline of this given data before comparing with the one actually
! 418: sent by the client Variables are substituted as in the `<command>` section.
! 419: The `<strip>` and `<strippart>` rules are applied before comparisons are made.
! 420:
! 421: ### `<proxy [nonewline="yes"]>`
! 422:
! 423: The protocol dump curl should transmit to a HTTP proxy (when the http-proxy
! 424: server is used), if 'nonewline' is set, we will cut off the trailing newline
! 425: of this given data before comparing with the one actually sent by the client
! 426: Variables are substituted as in the `<command>` section. The `<strip>` and
! 427: `<strippart>` rules are applied before comparisons are made.
! 428:
! 429: ### `<stdout [mode="text"] [nonewline="yes"]>`
! 430: This verifies that this data was passed to stdout. Variables are
! 431: substituted as in the `<command>` section.
! 432:
! 433: Use the mode="text" attribute if the output is in text mode on platforms that
! 434: have a text/binary difference.
! 435:
! 436: If 'nonewline' is set, we will cut off the trailing newline of this given data
! 437: before comparing with the one actually received by the client
! 438:
! 439: ### `<file name="log/filename" [mode="text"]>`
! 440: The file's contents must be identical to this after the test is complete. Use
! 441: the mode="text" attribute if the output is in text mode on platforms that have
! 442: a text/binary difference. Variables are substituted as in the `<command>`
! 443: section.
! 444:
! 445: ### `<file1>`
! 446: 1 to 4 can be appended to 'file' to compare more files.
! 447:
! 448: ### `<file2>`
! 449:
! 450: ### `<file3>`
! 451:
! 452: ### `<file4>`
! 453:
! 454: ### `<stripfile>`
! 455: One perl op per line that operates on the output file or stdout before being
! 456: compared with what is stored in the test file. This is pretty
! 457: advanced. Example: "s/^EPRT .*/EPRT stripped/"
! 458:
! 459: ### `<stripfile1>`
! 460: 1 to 4 can be appended to 'stripfile' to strip the corresponding <fileN>
! 461: content
! 462:
! 463: ### `<stripfile2>`
! 464:
! 465: ### `<stripfile3>`
! 466:
! 467: ### `<stripfile4>`
! 468:
! 469: ### `<upload>`
! 470: the contents of the upload data curl should have sent
! 471:
! 472: ### `<valgrind>`
! 473: disable - disables the valgrind log check for this test
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to FILEFORMAT.md CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / tests