Return to FILEFORMAT.md CVS log

Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / tests

curl

    1: # curl test suite file format
    2: 
    3: The curl test suite's file format is very simple and extensible, closely
    4: resembling XML. All data for a single test case resides in a single ASCII
    5: file. Labels mark the beginning and the end of all sections, and each label
    6: must be written in its own line.  Comments are either XML-style (enclosed with
    7: `<!--` and `-->`) or shell script style (beginning with `#`) and must appear
    8: on their own lines and not alongside actual test data.  Most test data files
    9: are syntactically valid XML, although a few files are not (lack of support for
   10: character entities and the preservation of CR/LF characters at the end of
   11: lines are the biggest differences).
   12: 
   13: Each test case exists as a file matching the format `tests/data/testNUM`,
   14: where NUM is considered the unique test number.
   15: 
   16: The file begins with a 'testcase' tag, which encompasses the remainder of the
   17: file.
   18: 
   19: # `<testcase>`
   20: 
   21: Each test is always within the testcase tag. Each test case is split up in
   22: four main sections: `info`, `reply`, `client` and `verify`.
   23: 
   24: - **info** provides information about the test case
   25: 
   26: - **reply** is used for the server to know what to send as a reply for the
   27: requests curl sends
   28: 
   29: - **client** defines how the client should behave
   30: 
   31: - **verify** defines how to verify that the data stored after a command has
   32: been run ended up correctly
   33: 
   34: Each main section has a number of available subsections that can be specified,
   35: that will be checked/used if specified.
   36: 
   37: ## `<info>`
   38: 
   39: ### `<keywords>`
   40: A newline-separated list of keywords describing what this test case uses and
   41: tests. Try to use an already used keyword.  These keywords will be used for
   42: statistical/informational purposes and for choosing or skipping classes
   43: of tests.  "Keywords" must begin with an alphabetic character, "-", "["
   44: or "{" and may actually consist of multiple words separated by spaces
   45: which are treated together as a single identifier.
   46: 
   47: ## `<reply>`
   48: 
   49: ### `<data [nocheck="yes"] [sendzero="yes"] [base64="yes"] [hex="yes"]>`
   50: 
   51: data to be sent to the client on its request and later verified that it
   52: arrived safely. Set `nocheck="yes"` to prevent the test script from verifying
   53: the arrival of this data.
   54: 
   55: If the data contains `swsclose` anywhere within the start and end tag, and
   56: this is a HTTP test, then the connection will be closed by the server after
   57: this response is sent. If not, the connection will be kept persistent.
   58: 
   59: If the data contains `swsbounce` anywhere within the start and end tag, the
   60: HTTP server will detect if this is a second request using the same test and
   61: part number and will then increase the part number with one. This is useful
   62: for auth tests and similar.
   63: 
   64: `sendzero=yes` means that the (FTP) server will "send" the data even if the
   65: size is zero bytes. Used to verify curl's behaviour on zero bytes transfers.
   66: 
   67: `base64=yes` means that the data provided in the test-file is a chunk of data
   68: encoded with base64. It is the only way a test case can contain binary
   69: data. (This attribute can in fact be used on any section, but it doesn't make
   70: much sense for other sections than "data").
   71: 
   72: `hex=yes` means that the data is a sequence of hex pairs. It will get decoded
   73: and used as "raw" data.
   74: 
   75: For FTP file listings, the `<data>` section will be used *only* if you make
   76: sure that there has been a CWD done first to a directory named `test-[num]`
   77: where [num] is the test case number. Otherwise the ftp server can't know from
   78: which test file to load the list content.
   79: 
   80: ### `<dataNUM>`
   81: 
   82: Send back this contents instead of the <data> one. The num is set by:
   83: 
   84:  - The test number in the request line is >10000 and this is the remainder
   85:    of [test case number]%10000.
   86:  - The request was HTTP and included digest details, which adds 1000 to NUM
   87:  - If a HTTP request is NTLM type-1, it adds 1001 to num
   88:  - If a HTTP request is NTLM type-3, it adds 1002 to num
   89:  - If a HTTP request is Basic and num is already >=1000, it adds 1 to num
   90:  - If a HTTP request is Negotiate, num gets incremented by one for each
   91:    request with Negotiate authorization header on the same test case.
   92: 
   93: Dynamically changing num in this way allows the test harness to be used to
   94: test authentication negotiation where several different requests must be sent
   95: to complete a transfer. The response to each request is found in its own data
   96: section.  Validating the entire negotiation sequence can be done by specifying
   97: a datacheck section.
   98: 
   99: ### `<connect>`
  100: The connect section is used instead of the 'data' for all CONNECT
  101: requests. The remainder of the rules for the data section then apply but with
  102: a connect prefix.
  103: 
  104: ### `<datacheck [mode="text"] [nonewline="yes"]>`
  105: if the data is sent but this is what should be checked afterwards. If
  106: `nonewline=yes` is set, runtests will cut off the trailing newline from the
  107: data before comparing with the one actually received by the client.
  108: 
  109: Use the `mode="text"` attribute if the output is in text mode on platforms
  110: that have a text/binary difference.
  111: 
  112: ### `<datacheckNUM [nonewline="yes"] [mode="text"]>`
  113: The contents of numbered datacheck sections are appended to the non-numbered
  114: one.
  115: 
  116: ### `<size>`
  117: number to return on a ftp SIZE command (set to -1 to make this command fail)
  118: 
  119: ### `<mdtm>`
  120: what to send back if the client sends a (FTP) MDTM command, set to -1 to
  121: have it return that the file doesn't exist
  122: 
  123: ### `<postcmd>`
  124: special purpose server-command to control its behavior *after* the
  125: reply is sent
  126: For HTTP/HTTPS, these are supported:
  127: 
  128: `wait [secs]` - Pause for the given time
  129: 
  130: ### `<servercmd>`
  131: Special-commands for the server.
  132: 
  133: The first line of this file will always be set to `Testnum [number]` by the
  134: test script, to allow servers to read that to know what test the client is
  135: about to issue.
  136: 
  137: #### For FTP/SMTP/POP/IMAP
  138: 
  139: - `REPLY [command] [return value] [response string]` - Changes how the server
  140:   responds to the [command]. [response string] is evaluated as a perl string,
  141:   so it can contain embedded \r\n, for example. There's a special [command]
  142:   named "welcome" (without quotes) which is the string sent immediately on
  143:   connect as a welcome.
  144: - `REPLYLF` (like above but sends the response terminated with LF-only and not
  145:    CRLF)
  146: - `COUNT [command] [num]` - Do the `REPLY` change for `[command]` only `[num]`
  147:   times and then go back to the built-in approach
  148: - `DELAY [command] [secs]` - Delay responding to this command for the given
  149:   time
  150: - `RETRWEIRDO` - Enable the "weirdo" RETR case when multiple response lines
  151:    appear at once when a file is transferred
  152: - `RETRNOSIZE` - Make sure the RETR response doesn't contain the size of the
  153:   file
  154: - `NOSAVE` - Don't actually save what is received
  155: - `SLOWDOWN` - Send FTP responses with 0.01 sec delay between each byte
  156: - `PASVBADIP` - makes PASV send back an illegal IP in its 227 response
  157: - `CAPA [capabilities]` - Enables support for and specifies a list of space
  158:    separated capabilities to return to the client for the IMAP `CAPABILITY`,
  159:    POP3 `CAPA` and SMTP `EHLO` commands
  160: - `AUTH [mechanisms]` - Enables support for SASL authentication and specifies
  161:    a list of space separated mechanisms for IMAP, POP3 and SMTP
  162: 
  163: #### For HTTP/HTTPS
  164: 
  165: - `auth_required` if this is set and a POST/PUT is made without auth, the
  166:   server will NOT wait for the full request body to get sent
  167: - `idle` - do nothing after receiving the request, just "sit idle"
  168: - `stream` - continuously send data to the client, never-ending
  169: - `writedelay: [secs]` delay this amount between reply packets
  170: - `skip: [num]` - instructs the server to ignore reading this many bytes from
  171:   a PUT or POST request
  172: - `rtp: part [num] channel [num] size [num]` - stream a fake RTP packet for
  173:   the given part on a chosen channel with the given payload size
  174: - `connection-monitor` - When used, this will log `[DISCONNECT]` to the
  175:   `server.input` log when the connection is disconnected.
  176: - `upgrade` - when an HTTP upgrade header is found, the server will upgrade to
  177:   http2
  178: - `swsclose` - instruct server to close connection after response
  179: - `no-expect` - don't read the request body if Expect: is present
  180: 
  181: #### For TFTP
  182: `writedelay: [secs]` delay this amount between reply packets (each packet
  183:   being 512 bytes payload)
  184: 
  185: ## `<client>`
  186: 
  187: ### `<server>`
  188: What server(s) this test case requires/uses. Available servers:
  189: 
  190: - `file`
  191: - `ftp-ipv6`
  192: - `ftp`
  193: - `ftps`
  194: - `http-ipv6`
  195: - `http-proxy`
  196: - `http-unix`
  197: - `http/2`
  198: - `http`
  199: - `https`
  200: - `httptls+srp-ipv6`
  201: - `httptls+srp`
  202: - `imap`
  203: - `mqtt`
  204: - `none`
  205: - `pop3`
  206: - `rtsp-ipv6`
  207: - `rtsp`
  208: - `scp`
  209: - `sftp`
  210: - `smtp`
  211: - `socks4`
  212: - `socks5`
  213: 
  214: Give only one per line.  This subsection is mandatory.
  215: 
  216: ### `<features>`
  217: A list of features that MUST be present in the client/library for this test to
  218: be able to run. If a required feature is not present then the test will be
  219: SKIPPED.
  220: 
  221: Alternatively a feature can be prefixed with an exclamation mark to indicate a
  222: feature is NOT required. If the feature is present then the test will be
  223: SKIPPED.
  224: 
  225: Features testable here are:
  226: 
  227: - `alt-svc`
  228: - `crypto`
  229: - `debug`
  230: - `getrlimit`
  231: - `GnuTLS`
  232: - `GSS-API`
  233: - `http/2`
  234: - `idn`
  235: - `ipv6`
  236: - `Kerberos`
  237: - `large_file`
  238: - `ld_preload`
  239: - `libz`
  240: - `manual`
  241: - `Metalink`
  242: - `NSS`
  243: - `NTLM`
  244: - `OpenSSL`
  245: - `PSL`
  246: - `socks`
  247: - `SPNEGO`
  248: - `SSL`
  249: - `SSLpinning`
  250: - `SSPI`
  251: - `threaded-resolver`
  252: - `TLS-SRP`
  253: - `TrackMemory`
  254: - `unittest`
  255: - `unix-sockets`
  256: - `win32`
  257: - `WinSSL`
  258: 
  259: as well as each protocol that curl supports.  A protocol only needs to be
  260: specified if it is different from the server (useful when the server
  261: is `none`).
  262: 
  263: ### `<killserver>`
  264: Using the same syntax as in `<server>` but when mentioned here these servers
  265: are explicitly KILLED when this test case is completed. Only use this if there
  266: is no other alternatives. Using this of course requires subsequent tests to
  267: restart servers.
  268: 
  269: ### `<precheck>`
  270: A command line that if set gets run by the test script before the test. If an
  271: output is displayed by the command or if the return code is non-zero, the test
  272: will be skipped and the (single-line) output will be displayed as reason for
  273: not running the test.  Variables are substituted as in the `<command>`
  274:   section.
  275: 
  276: ### `<postcheck>`
  277: A command line that if set gets run by the test script after the test. If
  278: the command exists with a non-zero status code, the test will be considered
  279: to have failed. Variables are substituted as in the `<command>` section.
  280: 
  281: ### `<tool>`
  282: Name of tool to invoke instead of "curl". This tool must be built and exist
  283: either in the libtest/ directory (if the tool name starts with 'lib') or in
  284: the unit/ directory (if the tool name starts with 'unit').
  285: 
  286: ### `<name>`
  287: Brief test case description, shown when the test runs.
  288: 
  289: ### `<setenv>`
  290:     variable1=contents1
  291:     variable2=contents2
  292: 
  293: Set the given environment variables to the specified value before the actual
  294: command is run. They are cleared again after the command has been run.
  295: Variables are first substituted as in the `<command>` section.
  296: ### `<command [option="no-output/no-include/force-output/binary-trace"] [timeout="secs"][delay="secs"][type="perl"]>`
  297: Command line to run. There's a bunch of %variables that get replaced
  298: accordingly.
  299: 
  300: Note that the URL that gets passed to the server actually controls what data
  301: that is returned. The last slash in the URL must be followed by a number. That
  302: number (N) will be used by the test-server to load test case N and return the
  303: data that is defined within the `<reply><data></data></reply>` section.
  304: 
  305: If there's no test number found above, the HTTP test server will use the
  306: number following the last dot in the given hostname (made so that a CONNECT
  307: can still pass on test number) so that "foo.bar.123" gets treated as test case
  308: 123. Alternatively, if an IPv6 address is provided to CONNECT, the last
  309: hexadecimal group in the address will be used as the test number! For example
  310: the address "[1234::ff]" would be treated as test case 255.
  311: 
  312: Set `type="perl"` to write the test case as a perl script. It implies that
  313: there's no memory debugging and valgrind gets shut off for this test.
  314: 
  315: Set `option="no-output"` to prevent the test script to slap on the `--output`
  316: argument that directs the output to a file. The `--output` is also not added
  317: if the verify/stdout section is used.
  318: 
  319: Set `option="force-output"` to make use of `--output` even when the test is
  320: otherwise written to verify stdout.
  321: 
  322: Set `option="no-include"` to prevent the test script to slap on the
  323: `--include` argument.
  324: 
  325: Set `option="binary-trace"` to use `--trace` instead of `--trace-ascii` for
  326: tracing.  Suitable for binary-oriented protocols such as MQTT.
  327: 
  328: Set `timeout="secs"` to override default server logs advisor read lock
  329: timeout.  This timeout is used by the test harness, once that the command has
  330: completed execution, to wait for the test server to write out server side log
  331: files and remove the lock that advised not to read them. The "secs" parameter
  332: is the not negative integer number of seconds for the timeout. This `timeout`
  333: attribute is documented for completeness sake, but is deep test harness stuff
  334: and only needed for very singular and specific test cases. Avoid using it.
  335: 
  336: Set `delay="secs"` to introduce a time delay once that the command has
  337: completed execution and before the `<postcheck>` section runs. The "secs"
  338: parameter is the not negative integer number of seconds for the delay. This
  339: 'delay' attribute is intended for very specific test cases, and normally not
  340: needed.
  341: 
  342: Available substitute variables include:
  343: 
  344: - `%CLIENT6IP` - IPv6 address of the client running curl
  345: - `%CLIENTIP` - IPv4 address of the client running curl
  346: - `%CURL` - Path to the curl executable
  347: - `%FILE_PWD` - Current directory, on windows prefixed with a slash
  348: - `%FTP2PORT` - Port number of the FTP server 2
  349: - `%FTP6PORT` - IPv6 port number of the FTP server
  350: - `%FTPPORT` - Port number of the FTP server
  351: - `%FTPSPORT` - Port number of the FTPS server
  352: - `%FTPTIME2` - Timeout in seconds that should be just sufficient to receive a response from the test FTP server
  353: - `%FTPTIME3` - Even longer than %FTPTIME2
  354: - `%GOPHER6PORT` - IPv6 port number of the Gopher server
  355: - `%GOPHERPORT` - Port number of the Gopher server
  356: - `%HOST6IP` - IPv6 address of the host running this test
  357: - `%HOSTIP` - IPv4 address of the host running this test
  358: - `%HTTP6PORT` - IPv6 port number of the HTTP server
  359: - `%HTTPPORT` - Port number of the HTTP server
  360: - `%HTTPSPORT` - Port number of the HTTPS server
  361: - `%HTTPTLS6PORT` - IPv6 port number of the HTTP TLS server
  362: - `%HTTPTLSPORT` - Port number of the HTTP TLS server
  363: - `%HTTPUNIXPATH` - Path to the Unix socket of the HTTP server
  364: - `%IMAP6PORT` - IPv6 port number of the IMAP server
  365: - `%IMAPPORT` - Port number of the IMAP server
  366: - `%MQTTPORT` - Port number of the MQTT server
  367: - `%NEGTELNETPORT` - Port number of the telnet server
  368: - `%NOLISTENPORT` - Port number where no service is listening
  369: - `%POP36PORT` - IPv6 port number of the POP3 server
  370: - `%POP3PORT` - Port number of the POP3 server
  371: - `%POSIX_PWD` - Current directory somewhat mingw friendly
  372: - `%PROXYPORT` - Port number of the HTTP proxy
  373: - `%PWD` - Current directory
  374: - `%RTSP6PORT` - IPv6 port number of the RTSP server
  375: - `%RTSPPORT` - Port number of the RTSP server
  376: - `%SMBPORT` - Port number of the SMB server
  377: - `%SMBSPORT` - Port number of the SMBS server
  378: - `%SMTP6PORT` - IPv6 port number of the SMTP server
  379: - `%SMTPPORT` - Port number of the SMTP server
  380: - `%SOCKSPORT` - Port number of the SOCKS4/5 server
  381: - `%SRCDIR` - Full path to the source dir
  382: - `%SSHPORT` - Port number of the SCP/SFTP server
  383: - `%SSHSRVMD5` - MD5 of SSH server's public key
  384: - `%TFTP6PORT` - IPv6 port number of the TFTP server
  385: - `%TFTPPORT` - Port number of the TFTP server
  386: - `%USER` - Login ID of the user running the test
  387: 
  388: ### `<file name="log/filename">`
  389: This creates the named file with this content before the test case is run,
  390: which is useful if the test case needs a file to act on.  Variables are
  391: substituted on the contents of the file as in the `<command>` section.
  392: 
  393: ### `<stdin [nonewline="yes"]>`
  394: Pass this given data on stdin to the tool.
  395: 
  396: If 'nonewline' is set, we will cut off the trailing newline of this given data
  397: before comparing with the one actually received by the client
  398: 
  399: ## `<verify>`
  400: ### `<errorcode>`
  401: numerical error code curl is supposed to return. Specify a list of accepted
  402: error codes by separating multiple numbers with comma. See test 237 for an
  403: example.
  404: 
  405: ### `<strip>`
  406: One regex per line that is removed from the protocol dumps before the
  407: comparison is made. This is very useful to remove dependencies on dynamically
  408: changing protocol data such as port numbers or user-agent strings.
  409: 
  410: ### `<strippart>`
  411: One perl op per line that operates on the protocol dump. This is pretty
  412: advanced. Example: `s/^EPRT .*/EPRT stripped/`.
  413: 
  414: ### `<protocol [nonewline="yes"]>`
  415: 
  416: the protocol dump curl should transmit, if 'nonewline' is set, we will cut off
  417: the trailing newline of this given data before comparing with the one actually
  418: sent by the client Variables are substituted as in the `<command>` section.
  419: The `<strip>` and `<strippart>` rules are applied before comparisons are made.
  420: 
  421: ### `<proxy [nonewline="yes"]>`
  422: 
  423: The protocol dump curl should transmit to a HTTP proxy (when the http-proxy
  424: server is used), if 'nonewline' is set, we will cut off the trailing newline
  425: of this given data before comparing with the one actually sent by the client
  426: Variables are substituted as in the `<command>` section. The `<strip>` and
  427: `<strippart>` rules are applied before comparisons are made.
  428: 
  429: ### `<stdout [mode="text"] [nonewline="yes"]>`
  430: This verifies that this data was passed to stdout.  Variables are
  431: substituted as in the `<command>` section.
  432: 
  433: Use the mode="text" attribute if the output is in text mode on platforms that
  434: have a text/binary difference.
  435: 
  436: If 'nonewline' is set, we will cut off the trailing newline of this given data
  437: before comparing with the one actually received by the client
  438: 
  439: ### `<file name="log/filename" [mode="text"]>`
  440: The file's contents must be identical to this after the test is complete.  Use
  441: the mode="text" attribute if the output is in text mode on platforms that have
  442: a text/binary difference.  Variables are substituted as in the `<command>`
  443: section.
  444: 
  445: ### `<file1>`
  446: 1 to 4 can be appended to 'file' to compare more files.
  447: 
  448: ### `<file2>`
  449: 
  450: ### `<file3>`
  451: 
  452: ### `<file4>`
  453: 
  454: ### `<stripfile>`
  455: One perl op per line that operates on the output file or stdout before being
  456: compared with what is stored in the test file. This is pretty
  457: advanced. Example: "s/^EPRT .*/EPRT stripped/"
  458: 
  459: ### `<stripfile1>`
  460: 1 to 4 can be appended to 'stripfile' to strip the corresponding <fileN>
  461: content
  462: 
  463: ### `<stripfile2>`
  464: 
  465: ### `<stripfile3>`
  466: 
  467: ### `<stripfile4>`
  468: 
  469: ### `<upload>`
  470: the contents of the upload data curl should have sent
  471: 
  472: ### `<valgrind>`
  473: disable - disables the valgrind log check for this test