1: curl internals
2: ==============
3:
4: - [Intro](#intro)
5: - [git](#git)
6: - [Portability](#Portability)
7: - [Windows vs Unix](#winvsunix)
8: - [Library](#Library)
9: - [`Curl_connect`](#Curl_connect)
10: - [`multi_do`](#multi_do)
11: - [`Curl_readwrite`](#Curl_readwrite)
12: - [`multi_done`](#multi_done)
13: - [`Curl_disconnect`](#Curl_disconnect)
14: - [HTTP(S)](#http)
15: - [FTP](#ftp)
16: - [Kerberos](#kerberos)
17: - [TELNET](#telnet)
18: - [FILE](#file)
19: - [SMB](#smb)
20: - [LDAP](#ldap)
21: - [E-mail](#email)
22: - [General](#general)
23: - [Persistent Connections](#persistent)
24: - [multi interface/non-blocking](#multi)
25: - [SSL libraries](#ssl)
26: - [Library Symbols](#symbols)
27: - [Return Codes and Informationals](#returncodes)
28: - [AP/ABI](#abi)
29: - [Client](#client)
30: - [Memory Debugging](#memorydebug)
31: - [Test Suite](#test)
32: - [Asynchronous name resolves](#asyncdns)
33: - [c-ares](#cares)
34: - [`curl_off_t`](#curl_off_t)
35: - [curlx](#curlx)
36: - [Content Encoding](#contentencoding)
37: - [`hostip.c` explained](#hostip)
38: - [Track Down Memory Leaks](#memoryleak)
39: - [`multi_socket`](#multi_socket)
40: - [Structs in libcurl](#structs)
41: - [Curl_easy](#Curl_easy)
42: - [connectdata](#connectdata)
43: - [Curl_multi](#Curl_multi)
44: - [Curl_handler](#Curl_handler)
45: - [conncache](#conncache)
46: - [Curl_share](#Curl_share)
47: - [CookieInfo](#CookieInfo)
48:
49: <a name="intro"></a>
50: Intro
51: =====
52:
53: This project is split in two. The library and the client. The client part
54: uses the library, but the library is designed to allow other applications to
55: use it.
56:
57: The largest amount of code and complexity is in the library part.
58:
59:
60: <a name="git"></a>
61: git
62: ===
63:
64: All changes to the sources are committed to the git repository as soon as
65: they're somewhat verified to work. Changes shall be committed as independently
66: as possible so that individual changes can be easily spotted and tracked
67: afterwards.
68:
69: Tagging shall be used extensively, and by the time we release new archives we
70: should tag the sources with a name similar to the released version number.
71:
72: <a name="Portability"></a>
73: Portability
74: ===========
75:
76: We write curl and libcurl to compile with C89 compilers. On 32-bit and up
77: machines. Most of libcurl assumes more or less POSIX compliance but that's
78: not a requirement.
79:
80: We write libcurl to build and work with lots of third party tools, and we
81: want it to remain functional and buildable with these and later versions
82: (older versions may still work but is not what we work hard to maintain):
83:
84: Dependencies
85: ------------
86:
87: - OpenSSL 0.9.7
88: - GnuTLS 3.1.10
89: - zlib 1.1.4
90: - libssh2 0.16
91: - c-ares 1.6.0
92: - libidn2 2.0.0
93: - wolfSSL 2.0.0
94: - openldap 2.0
95: - MIT Kerberos 1.2.4
96: - GSKit V5R3M0
97: - NSS 3.14.x
98: - Heimdal ?
99: - nghttp2 1.12.0
100:
101: Operating Systems
102: -----------------
103:
104: On systems where configure runs, we aim at working on them all - if they have
105: a suitable C compiler. On systems that don't run configure, we strive to keep
106: curl running correctly on:
107:
108: - Windows 98
109: - AS/400 V5R3M0
110: - Symbian 9.1
111: - Windows CE ?
112: - TPF ?
113:
114: Build tools
115: -----------
116:
117: When writing code (mostly for generating stuff included in release tarballs)
118: we use a few "build tools" and we make sure that we remain functional with
119: these versions:
120:
121: - GNU Libtool 1.4.2
122: - GNU Autoconf 2.57
123: - GNU Automake 1.7
124: - GNU M4 1.4
125: - perl 5.004
126: - roffit 0.5
127: - groff ? (any version that supports `groff -Tps -man [in] [out]`)
128: - ps2pdf (gs) ?
129:
130: <a name="winvsunix"></a>
131: Windows vs Unix
132: ===============
133:
134: There are a few differences in how to program curl the Unix way compared to
135: the Windows way. Perhaps the four most notable details are:
136:
137: 1. Different function names for socket operations.
138:
139: In curl, this is solved with defines and macros, so that the source looks
140: the same in all places except for the header file that defines them. The
141: macros in use are `sclose()`, `sread()` and `swrite()`.
142:
143: 2. Windows requires a couple of init calls for the socket stuff.
144:
145: That's taken care of by the `curl_global_init()` call, but if other libs
146: also do it etc there might be reasons for applications to alter that
147: behaviour.
148:
149: 3. The file descriptors for network communication and file operations are
150: not as easily interchangeable as in Unix.
151:
152: We avoid this by not trying any funny tricks on file descriptors.
153:
154: 4. When writing data to stdout, Windows makes end-of-lines the DOS way, thus
155: destroying binary data, although you do want that conversion if it is
156: text coming through... (sigh)
157:
158: We set stdout to binary under windows
159:
160: Inside the source code, We make an effort to avoid `#ifdef [Your OS]`. All
161: conditionals that deal with features *should* instead be in the format
162: `#ifdef HAVE_THAT_WEIRD_FUNCTION`. Since Windows can't run configure scripts,
163: we maintain a `curl_config-win32.h` file in lib directory that is supposed to
164: look exactly like a `curl_config.h` file would have looked like on a Windows
165: machine!
166:
167: Generally speaking: always remember that this will be compiled on dozens of
168: operating systems. Don't walk on the edge!
169:
170: <a name="Library"></a>
171: Library
172: =======
173:
174: (See [Structs in libcurl](#structs) for the separate section describing all
175: major internal structs and their purposes.)
176:
177: There are plenty of entry points to the library, namely each publicly defined
178: function that libcurl offers to applications. All of those functions are
179: rather small and easy-to-follow. All the ones prefixed with `curl_easy` are
180: put in the `lib/easy.c` file.
181:
182: `curl_global_init()` and `curl_global_cleanup()` should be called by the
183: application to initialize and clean up global stuff in the library. As of
184: today, it can handle the global SSL initing if SSL is enabled and it can init
185: the socket layer on windows machines. libcurl itself has no "global" scope.
186:
187: All printf()-style functions use the supplied clones in `lib/mprintf.c`. This
188: makes sure we stay absolutely platform independent.
189:
190: [ `curl_easy_init()`][2] allocates an internal struct and makes some
191: initializations. The returned handle does not reveal internals. This is the
192: `Curl_easy` struct which works as an "anchor" struct for all `curl_easy`
193: functions. All connections performed will get connect-specific data allocated
194: that should be used for things related to particular connections/requests.
195:
196: [`curl_easy_setopt()`][1] takes three arguments, where the option stuff must
197: be passed in pairs: the parameter-ID and the parameter-value. The list of
198: options is documented in the man page. This function mainly sets things in
199: the `Curl_easy` struct.
200:
201: `curl_easy_perform()` is just a wrapper function that makes use of the multi
202: API. It basically calls `curl_multi_init()`, `curl_multi_add_handle()`,
203: `curl_multi_wait()`, and `curl_multi_perform()` until the transfer is done
204: and then returns.
205:
206: Some of the most important key functions in `url.c` are called from
207: `multi.c` when certain key steps are to be made in the transfer operation.
208:
209: <a name="Curl_connect"></a>
210: Curl_connect()
211: --------------
212:
213: Analyzes the URL, it separates the different components and connects to the
214: remote host. This may involve using a proxy and/or using SSL. The
215: `Curl_resolv()` function in `lib/hostip.c` is used for looking up host
216: names (it does then use the proper underlying method, which may vary
217: between platforms and builds).
218:
219: When `Curl_connect` is done, we are connected to the remote site. Then it
220: is time to tell the server to get a document/file. `Curl_do()` arranges
221: this.
222:
223: This function makes sure there's an allocated and initiated `connectdata`
224: struct that is used for this particular connection only (although there may
225: be several requests performed on the same connect). A bunch of things are
226: inited/inherited from the `Curl_easy` struct.
227:
228: <a name="multi_do"></a>
229: multi_do()
230: ---------
231:
232: `multi_do()` makes sure the proper protocol-specific function is called.
233: The functions are named after the protocols they handle.
234:
235: The protocol-specific functions of course deal with protocol-specific
236: negotiations and setup. They have access to the `Curl_sendf()` (from
237: `lib/sendf.c`) function to send printf-style formatted data to the remote
238: host and when they're ready to make the actual file transfer they call the
239: `Curl_setup_transfer()` function (in `lib/transfer.c`) to setup the
240: transfer and returns.
241:
242: If this DO function fails and the connection is being re-used, libcurl will
243: then close this connection, setup a new connection and re-issue the DO
244: request on that. This is because there is no way to be perfectly sure that
245: we have discovered a dead connection before the DO function and thus we
246: might wrongly be re-using a connection that was closed by the remote peer.
247:
248: <a name="Curl_readwrite"></a>
249: Curl_readwrite()
250: ----------------
251:
252: Called during the transfer of the actual protocol payload.
253:
254: During transfer, the progress functions in `lib/progress.c` are called at
255: frequent intervals (or at the user's choice, a specified callback might get
256: called). The speedcheck functions in `lib/speedcheck.c` are also used to
257: verify that the transfer is as fast as required.
258:
259: <a name="multi_done"></a>
260: multi_done()
261: -----------
262:
263: Called after a transfer is done. This function takes care of everything
264: that has to be done after a transfer. This function attempts to leave
265: matters in a state so that `multi_do()` should be possible to call again on
266: the same connection (in a persistent connection case). It might also soon
267: be closed with `Curl_disconnect()`.
268:
269: <a name="Curl_disconnect"></a>
270: Curl_disconnect()
271: -----------------
272:
273: When doing normal connections and transfers, no one ever tries to close any
274: connections so this is not normally called when `curl_easy_perform()` is
275: used. This function is only used when we are certain that no more transfers
276: are going to be made on the connection. It can be also closed by force, or
277: it can be called to make sure that libcurl doesn't keep too many
278: connections alive at the same time.
279:
280: This function cleans up all resources that are associated with a single
281: connection.
282:
283: <a name="http"></a>
284: HTTP(S)
285: =======
286:
287: HTTP offers a lot and is the protocol in curl that uses the most lines of
288: code. There is a special file `lib/formdata.c` that offers all the
289: multipart post functions.
290:
291: base64-functions for user+password stuff (and more) is in `lib/base64.c`
292: and all functions for parsing and sending cookies are found in
293: `lib/cookie.c`.
294:
295: HTTPS uses in almost every case the same procedure as HTTP, with only two
296: exceptions: the connect procedure is different and the function used to read
297: or write from the socket is different, although the latter fact is hidden in
298: the source by the use of `Curl_read()` for reading and `Curl_write()` for
299: writing data to the remote server.
300:
301: `http_chunks.c` contains functions that understands HTTP 1.1 chunked transfer
302: encoding.
303:
304: An interesting detail with the HTTP(S) request, is the `Curl_add_buffer()`
305: series of functions we use. They append data to one single buffer, and when
306: the building is finished the entire request is sent off in one single write.
307: This is done this way to overcome problems with flawed firewalls and lame
308: servers.
309:
310: <a name="ftp"></a>
311: FTP
312: ===
313:
314: The `Curl_if2ip()` function can be used for getting the IP number of a
315: specified network interface, and it resides in `lib/if2ip.c`.
316:
317: `Curl_ftpsendf()` is used for sending FTP commands to the remote server. It
318: was made a separate function to prevent us programmers from forgetting that
319: they must be CRLF terminated. They must also be sent in one single `write()`
320: to make firewalls and similar happy.
321:
322: <a name="kerberos"></a>
323: Kerberos
324: ========
325:
326: Kerberos support is mainly in `lib/krb5.c` and `lib/security.c` but also
327: `curl_sasl_sspi.c` and `curl_sasl_gssapi.c` for the email protocols and
328: `socks_gssapi.c` and `socks_sspi.c` for SOCKS5 proxy specifics.
329:
330: <a name="telnet"></a>
331: TELNET
332: ======
333:
334: Telnet is implemented in `lib/telnet.c`.
335:
336: <a name="file"></a>
337: FILE
338: ====
339:
340: The `file://` protocol is dealt with in `lib/file.c`.
341:
342: <a name="smb"></a>
343: SMB
344: ===
345:
346: The `smb://` protocol is dealt with in `lib/smb.c`.
347:
348: <a name="ldap"></a>
349: LDAP
350: ====
351:
352: Everything LDAP is in `lib/ldap.c` and `lib/openldap.c`.
353:
354: <a name="email"></a>
355: E-mail
356: ======
357:
358: The e-mail related source code is in `lib/imap.c`, `lib/pop3.c` and
359: `lib/smtp.c`.
360:
361: <a name="general"></a>
362: General
363: =======
364:
365: URL encoding and decoding, called escaping and unescaping in the source code,
366: is found in `lib/escape.c`.
367:
368: While transferring data in `Transfer()` a few functions might get used.
369: `curl_getdate()` in `lib/parsedate.c` is for HTTP date comparisons (and
370: more).
371:
372: `lib/getenv.c` offers `curl_getenv()` which is for reading environment
373: variables in a neat platform independent way. That's used in the client, but
374: also in `lib/url.c` when checking the proxy environment variables. Note that
375: contrary to the normal unix `getenv()`, this returns an allocated buffer that
376: must be `free()`ed after use.
377:
378: `lib/netrc.c` holds the `.netrc` parser.
379:
380: `lib/timeval.c` features replacement functions for systems that don't have
381: `gettimeofday()` and a few support functions for timeval conversions.
382:
383: A function named `curl_version()` that returns the full curl version string
384: is found in `lib/version.c`.
385:
386: <a name="persistent"></a>
387: Persistent Connections
388: ======================
389:
390: The persistent connection support in libcurl requires some considerations on
391: how to do things inside of the library.
392:
393: - The `Curl_easy` struct returned in the [`curl_easy_init()`][2] call
394: must never hold connection-oriented data. It is meant to hold the root data
395: as well as all the options etc that the library-user may choose.
396:
397: - The `Curl_easy` struct holds the "connection cache" (an array of
398: pointers to `connectdata` structs).
399:
400: - This enables the 'curl handle' to be reused on subsequent transfers.
401:
402: - When libcurl is told to perform a transfer, it first checks for an already
403: existing connection in the cache that we can use. Otherwise it creates a
404: new one and adds that to the cache. If the cache is full already when a new
405: connection is added, it will first close the oldest unused one.
406:
407: - When the transfer operation is complete, the connection is left
408: open. Particular options may tell libcurl not to, and protocols may signal
409: closure on connections and then they won't be kept open, of course.
410:
411: - When `curl_easy_cleanup()` is called, we close all still opened connections,
412: unless of course the multi interface "owns" the connections.
413:
414: The curl handle must be re-used in order for the persistent connections to
415: work.
416:
417: <a name="multi"></a>
418: multi interface/non-blocking
419: ============================
420:
421: The multi interface is a non-blocking interface to the library. To make that
422: interface work as well as possible, no low-level functions within libcurl
423: must be written to work in a blocking manner. (There are still a few spots
424: violating this rule.)
425:
426: One of the primary reasons we introduced c-ares support was to allow the name
427: resolve phase to be perfectly non-blocking as well.
428:
429: The FTP and the SFTP/SCP protocols are examples of how we adapt and adjust
430: the code to allow non-blocking operations even on multi-stage command-
431: response protocols. They are built around state machines that return when
432: they would otherwise block waiting for data. The DICT, LDAP and TELNET
433: protocols are crappy examples and they are subject for rewrite in the future
434: to better fit the libcurl protocol family.
435:
436: <a name="ssl"></a>
437: SSL libraries
438: =============
439:
440: Originally libcurl supported SSLeay for SSL/TLS transports, but that was then
441: extended to its successor OpenSSL but has since also been extended to several
442: other SSL/TLS libraries and we expect and hope to further extend the support
443: in future libcurl versions.
444:
445: To deal with this internally in the best way possible, we have a generic SSL
446: function API as provided by the `vtls/vtls.[ch]` system, and they are the only
447: SSL functions we must use from within libcurl. vtls is then crafted to use
448: the appropriate lower-level function calls to whatever SSL library that is in
449: use. For example `vtls/openssl.[ch]` for the OpenSSL library.
450:
451: <a name="symbols"></a>
452: Library Symbols
453: ===============
454:
455: All symbols used internally in libcurl must use a `Curl_` prefix if they're
456: used in more than a single file. Single-file symbols must be made static.
457: Public ("exported") symbols must use a `curl_` prefix. (There are exceptions,
458: but they are to be changed to follow this pattern in future versions.) Public
459: API functions are marked with `CURL_EXTERN` in the public header files so
460: that all others can be hidden on platforms where this is possible.
461:
462: <a name="returncodes"></a>
463: Return Codes and Informationals
464: ===============================
465:
466: I've made things simple. Almost every function in libcurl returns a CURLcode,
467: that must be `CURLE_OK` if everything is OK or otherwise a suitable error
468: code as the `curl/curl.h` include file defines. The very spot that detects an
469: error must use the `Curl_failf()` function to set the human-readable error
470: description.
471:
472: In aiding the user to understand what's happening and to debug curl usage, we
473: must supply a fair number of informational messages by using the
474: `Curl_infof()` function. Those messages are only displayed when the user
475: explicitly asks for them. They are best used when revealing information that
476: isn't otherwise obvious.
477:
478: <a name="abi"></a>
479: API/ABI
480: =======
481:
482: We make an effort to not export or show internals or how internals work, as
483: that makes it easier to keep a solid API/ABI over time. See docs/libcurl/ABI
484: for our promise to users.
485:
486: <a name="client"></a>
487: Client
488: ======
489:
490: `main()` resides in `src/tool_main.c`.
491:
492: `src/tool_hugehelp.c` is automatically generated by the `mkhelp.pl` perl
493: script to display the complete "manual" and the `src/tool_urlglob.c` file
494: holds the functions used for the URL-"globbing" support. Globbing in the
495: sense that the `{}` and `[]` expansion stuff is there.
496:
497: The client mostly sets up its `config` struct properly, then
498: it calls the `curl_easy_*()` functions of the library and when it gets back
499: control after the `curl_easy_perform()` it cleans up the library, checks
500: status and exits.
501:
502: When the operation is done, the `ourWriteOut()` function in `src/writeout.c`
503: may be called to report about the operation. That function is using the
504: `curl_easy_getinfo()` function to extract useful information from the curl
505: session.
506:
507: It may loop and do all this several times if many URLs were specified on the
508: command line or config file.
509:
510: <a name="memorydebug"></a>
511: Memory Debugging
512: ================
513:
514: The file `lib/memdebug.c` contains debug-versions of a few functions.
515: Functions such as `malloc()`, `free()`, `fopen()`, `fclose()`, etc that
516: somehow deal with resources that might give us problems if we "leak" them.
517: The functions in the memdebug system do nothing fancy, they do their normal
518: function and then log information about what they just did. The logged data
519: can then be analyzed after a complete session,
520:
521: `memanalyze.pl` is the perl script present in `tests/` that analyzes a log
522: file generated by the memory tracking system. It detects if resources are
523: allocated but never freed and other kinds of errors related to resource
524: management.
525:
526: Internally, definition of preprocessor symbol `DEBUGBUILD` restricts code
527: which is only compiled for debug enabled builds. And symbol `CURLDEBUG` is
528: used to differentiate code which is _only_ used for memory
529: tracking/debugging.
530:
531: Use `-DCURLDEBUG` when compiling to enable memory debugging, this is also
532: switched on by running configure with `--enable-curldebug`. Use
533: `-DDEBUGBUILD` when compiling to enable a debug build or run configure with
534: `--enable-debug`.
535:
536: `curl --version` will list 'Debug' feature for debug enabled builds, and
537: will list 'TrackMemory' feature for curl debug memory tracking capable
538: builds. These features are independent and can be controlled when running
539: the configure script. When `--enable-debug` is given both features will be
540: enabled, unless some restriction prevents memory tracking from being used.
541:
542: <a name="test"></a>
543: Test Suite
544: ==========
545:
546: The test suite is placed in its own subdirectory directly off the root in the
547: curl archive tree, and it contains a bunch of scripts and a lot of test case
548: data.
549:
550: The main test script is `runtests.pl` that will invoke test servers like
551: `httpserver.pl` and `ftpserver.pl` before all the test cases are performed.
552: The test suite currently only runs on Unix-like platforms.
553:
554: You'll find a description of the test suite in the `tests/README` file, and
555: the test case data files in the `tests/FILEFORMAT` file.
556:
557: The test suite automatically detects if curl was built with the memory
558: debugging enabled, and if it was, it will detect memory leaks, too.
559:
560: <a name="asyncdns"></a>
561: Asynchronous name resolves
562: ==========================
563:
564: libcurl can be built to do name resolves asynchronously, using either the
565: normal resolver in a threaded manner or by using c-ares.
566:
567: <a name="cares"></a>
568: [c-ares][3]
569: ------
570:
571: ### Build libcurl to use a c-ares
572:
573: 1. ./configure --enable-ares=/path/to/ares/install
574: 2. make
575:
576: ### c-ares on win32
577:
578: First I compiled c-ares. I changed the default C runtime library to be the
579: single-threaded rather than the multi-threaded (this seems to be required to
580: prevent linking errors later on). Then I simply build the areslib project
581: (the other projects adig/ahost seem to fail under MSVC).
582:
583: Next was libcurl. I opened `lib/config-win32.h` and I added a:
584: `#define USE_ARES 1`
585:
586: Next thing I did was I added the path for the ares includes to the include
587: path, and the libares.lib to the libraries.
588:
589: Lastly, I also changed libcurl to be single-threaded rather than
590: multi-threaded, again this was to prevent some duplicate symbol errors. I'm
591: not sure why I needed to change everything to single-threaded, but when I
592: didn't I got redefinition errors for several CRT functions (`malloc()`,
593: `stricmp()`, etc.)
594:
595: <a name="curl_off_t"></a>
596: `curl_off_t`
597: ==========
598:
599: `curl_off_t` is a data type provided by the external libcurl include
600: headers. It is the type meant to be used for the [`curl_easy_setopt()`][1]
601: options that end with LARGE. The type is 64-bit large on most modern
602: platforms.
603:
604: <a name="curlx"></a>
605: curlx
606: =====
607:
608: The libcurl source code offers a few functions by source only. They are not
609: part of the official libcurl API, but the source files might be useful for
610: others so apps can optionally compile/build with these sources to gain
611: additional functions.
612:
613: We provide them through a single header file for easy access for apps:
614: `curlx.h`
615:
616: `curlx_strtoofft()`
617: -------------------
618: A macro that converts a string containing a number to a `curl_off_t` number.
619: This might use the `curlx_strtoll()` function which is provided as source
620: code in strtoofft.c. Note that the function is only provided if no
621: `strtoll()` (or equivalent) function exist on your platform. If `curl_off_t`
622: is only a 32-bit number on your platform, this macro uses `strtol()`.
623:
624: Future
625: ------
626:
627: Several functions will be removed from the public `curl_` name space in a
628: future libcurl release. They will then only become available as `curlx_`
629: functions instead. To make the transition easier, we already today provide
630: these functions with the `curlx_` prefix to allow sources to be built
631: properly with the new function names. The concerned functions are:
632:
633: - `curlx_getenv`
634: - `curlx_strequal`
635: - `curlx_strnequal`
636: - `curlx_mvsnprintf`
637: - `curlx_msnprintf`
638: - `curlx_maprintf`
639: - `curlx_mvaprintf`
640: - `curlx_msprintf`
641: - `curlx_mprintf`
642: - `curlx_mfprintf`
643: - `curlx_mvsprintf`
644: - `curlx_mvprintf`
645: - `curlx_mvfprintf`
646:
647: <a name="contentencoding"></a>
648: Content Encoding
649: ================
650:
651: ## About content encodings
652:
653: [HTTP/1.1][4] specifies that a client may request that a server encode its
654: response. This is usually used to compress a response using one (or more)
655: encodings from a set of commonly available compression techniques. These
656: schemes include `deflate` (the zlib algorithm), `gzip`, `br` (brotli) and
657: `compress`. A client requests that the server perform an encoding by including
658: an `Accept-Encoding` header in the request document. The value of the header
659: should be one of the recognized tokens `deflate`, ... (there's a way to
660: register new schemes/tokens, see sec 3.5 of the spec). A server MAY honor
661: the client's encoding request. When a response is encoded, the server
662: includes a `Content-Encoding` header in the response. The value of the
663: `Content-Encoding` header indicates which encodings were used to encode the
664: data, in the order in which they were applied.
665:
666: It's also possible for a client to attach priorities to different schemes so
667: that the server knows which it prefers. See sec 14.3 of RFC 2616 for more
668: information on the `Accept-Encoding` header. See sec
669: [3.1.2.2 of RFC 7231][15] for more information on the `Content-Encoding`
670: header.
671:
672: ## Supported content encodings
673:
674: The `deflate`, `gzip` and `br` content encodings are supported by libcurl.
675: Both regular and chunked transfers work fine. The zlib library is required
676: for the `deflate` and `gzip` encodings, while the brotli decoding library is
677: for the `br` encoding.
678:
679: ## The libcurl interface
680:
681: To cause libcurl to request a content encoding use:
682:
683: [`curl_easy_setopt`][1](curl, [`CURLOPT_ACCEPT_ENCODING`][5], string)
684:
685: where string is the intended value of the `Accept-Encoding` header.
686:
687: Currently, libcurl does support multiple encodings but only
688: understands how to process responses that use the `deflate`, `gzip` and/or
689: `br` content encodings, so the only values for [`CURLOPT_ACCEPT_ENCODING`][5]
690: that will work (besides `identity`, which does nothing) are `deflate`,
691: `gzip` and `br`. If a response is encoded using the `compress` or methods,
692: libcurl will return an error indicating that the response could
693: not be decoded. If `<string>` is NULL no `Accept-Encoding` header is
694: generated. If `<string>` is a zero-length string, then an `Accept-Encoding`
695: header containing all supported encodings will be generated.
696:
697: The [`CURLOPT_ACCEPT_ENCODING`][5] must be set to any non-NULL value for
698: content to be automatically decoded. If it is not set and the server still
699: sends encoded content (despite not having been asked), the data is returned
700: in its raw form and the `Content-Encoding` type is not checked.
701:
702: ## The curl interface
703:
704: Use the [`--compressed`][6] option with curl to cause it to ask servers to
705: compress responses using any format supported by curl.
706:
707: <a name="hostip"></a>
708: `hostip.c` explained
709: ====================
710:
711: The main compile-time defines to keep in mind when reading the `host*.c`
712: source file are these:
713:
714: ## `CURLRES_IPV6`
715:
716: this host has `getaddrinfo()` and family, and thus we use that. The host may
717: not be able to resolve IPv6, but we don't really have to take that into
718: account. Hosts that aren't IPv6-enabled have `CURLRES_IPV4` defined.
719:
720: ## `CURLRES_ARES`
721:
722: is defined if libcurl is built to use c-ares for asynchronous name
723: resolves. This can be Windows or \*nix.
724:
725: ## `CURLRES_THREADED`
726:
727: is defined if libcurl is built to use threading for asynchronous name
728: resolves. The name resolve will be done in a new thread, and the supported
729: asynch API will be the same as for ares-builds. This is the default under
730: (native) Windows.
731:
732: If any of the two previous are defined, `CURLRES_ASYNCH` is defined too. If
733: libcurl is not built to use an asynchronous resolver, `CURLRES_SYNCH` is
734: defined.
735:
736: ## `host*.c` sources
737:
738: The `host*.c` sources files are split up like this:
739:
740: - `hostip.c` - method-independent resolver functions and utility functions
741: - `hostasyn.c` - functions for asynchronous name resolves
742: - `hostsyn.c` - functions for synchronous name resolves
743: - `asyn-ares.c` - functions for asynchronous name resolves using c-ares
744: - `asyn-thread.c` - functions for asynchronous name resolves using threads
745: - `hostip4.c` - IPv4 specific functions
746: - `hostip6.c` - IPv6 specific functions
747:
748: The `hostip.h` is the single united header file for all this. It defines the
749: `CURLRES_*` defines based on the `config*.h` and `curl_setup.h` defines.
750:
751: <a name="memoryleak"></a>
752: Track Down Memory Leaks
753: =======================
754:
755: ## Single-threaded
756:
757: Please note that this memory leak system is not adjusted to work in more
758: than one thread. If you want/need to use it in a multi-threaded app. Please
759: adjust accordingly.
760:
761: ## Build
762:
763: Rebuild libcurl with `-DCURLDEBUG` (usually, rerunning configure with
764: `--enable-debug` fixes this). `make clean` first, then `make` so that all
765: files are actually rebuilt properly. It will also make sense to build
766: libcurl with the debug option (usually `-g` to the compiler) so that
767: debugging it will be easier if you actually do find a leak in the library.
768:
769: This will create a library that has memory debugging enabled.
770:
771: ## Modify Your Application
772:
773: Add a line in your application code:
774:
775: `curl_dbg_memdebug("dump");`
776:
777: This will make the malloc debug system output a full trace of all resource
778: using functions to the given file name. Make sure you rebuild your program
779: and that you link with the same libcurl you built for this purpose as
780: described above.
781:
782: ## Run Your Application
783:
784: Run your program as usual. Watch the specified memory trace file grow.
785:
786: Make your program exit and use the proper libcurl cleanup functions etc. So
787: that all non-leaks are returned/freed properly.
788:
789: ## Analyze the Flow
790:
791: Use the `tests/memanalyze.pl` perl script to analyze the dump file:
792:
793: tests/memanalyze.pl dump
794:
795: This now outputs a report on what resources that were allocated but never
796: freed etc. This report is very fine for posting to the list!
797:
798: If this doesn't produce any output, no leak was detected in libcurl. Then
799: the leak is mostly likely to be in your code.
800:
801: <a name="multi_socket"></a>
802: `multi_socket`
803: ==============
804:
805: Implementation of the `curl_multi_socket` API
806:
807: The main ideas of this API are simply:
808:
809: 1. The application can use whatever event system it likes as it gets info
810: from libcurl about what file descriptors libcurl waits for what action
811: on. (The previous API returns `fd_sets` which is very
812: `select()`-centric).
813:
814: 2. When the application discovers action on a single socket, it calls
815: libcurl and informs that there was action on this particular socket and
816: libcurl can then act on that socket/transfer only and not care about
817: any other transfers. (The previous API always had to scan through all
818: the existing transfers.)
819:
820: The idea is that [`curl_multi_socket_action()`][7] calls a given callback
821: with information about what socket to wait for what action on, and the
822: callback only gets called if the status of that socket has changed.
823:
824: We also added a timer callback that makes libcurl call the application when
825: the timeout value changes, and you set that with [`curl_multi_setopt()`][9]
826: and the [`CURLMOPT_TIMERFUNCTION`][10] option. To get this to work,
827: Internally, there's an added struct to each easy handle in which we store
828: an "expire time" (if any). The structs are then "splay sorted" so that we
829: can add and remove times from the linked list and yet somewhat swiftly
830: figure out both how long there is until the next nearest timer expires
831: and which timer (handle) we should take care of now. Of course, the upside
832: of all this is that we get a [`curl_multi_timeout()`][8] that should also
833: work with old-style applications that use [`curl_multi_perform()`][11].
834:
835: We created an internal "socket to easy handles" hash table that given
836: a socket (file descriptor) returns the easy handle that waits for action on
837: that socket. This hash is made using the already existing hash code
838: (previously only used for the DNS cache).
839:
840: To make libcurl able to report plain sockets in the socket callback, we had
841: to re-organize the internals of the [`curl_multi_fdset()`][12] etc so that
842: the conversion from sockets to `fd_sets` for that function is only done in
843: the last step before the data is returned. I also had to extend c-ares to
844: get a function that can return plain sockets, as that library too returned
845: only `fd_sets` and that is no longer good enough. The changes done to c-ares
846: are available in c-ares 1.3.1 and later.
847:
848: <a name="structs"></a>
849: Structs in libcurl
850: ==================
851:
852: This section should cover 7.32.0 pretty accurately, but will make sense even
853: for older and later versions as things don't change drastically that often.
854:
855: <a name="Curl_easy"></a>
856: ## Curl_easy
857:
858: The `Curl_easy` struct is the one returned to the outside in the external API
859: as a `CURL *`. This is usually known as an easy handle in API documentations
860: and examples.
861:
862: Information and state that is related to the actual connection is in the
863: `connectdata` struct. When a transfer is about to be made, libcurl will
864: either create a new connection or re-use an existing one. The particular
865: connectdata that is used by this handle is pointed out by
866: `Curl_easy->easy_conn`.
867:
868: Data and information that regard this particular single transfer is put in
869: the `SingleRequest` sub-struct.
870:
871: When the `Curl_easy` struct is added to a multi handle, as it must be in
872: order to do any transfer, the `->multi` member will point to the `Curl_multi`
873: struct it belongs to. The `->prev` and `->next` members will then be used by
874: the multi code to keep a linked list of `Curl_easy` structs that are added to
875: that same multi handle. libcurl always uses multi so `->multi` *will* point
876: to a `Curl_multi` when a transfer is in progress.
877:
878: `->mstate` is the multi state of this particular `Curl_easy`. When
879: `multi_runsingle()` is called, it will act on this handle according to which
880: state it is in. The mstate is also what tells which sockets to return for a
881: specific `Curl_easy` when [`curl_multi_fdset()`][12] is called etc.
882:
883: The libcurl source code generally use the name `data` for the variable that
884: points to the `Curl_easy`.
885:
886: When doing multiplexed HTTP/2 transfers, each `Curl_easy` is associated with
887: an individual stream, sharing the same connectdata struct. Multiplexing
888: makes it even more important to keep things associated with the right thing!
889:
890: <a name="connectdata"></a>
891: ## connectdata
892:
893: A general idea in libcurl is to keep connections around in a connection
894: "cache" after they have been used in case they will be used again and then
895: re-use an existing one instead of creating a new as it creates a significant
896: performance boost.
897:
898: Each `connectdata` identifies a single physical connection to a server. If
899: the connection can't be kept alive, the connection will be closed after use
900: and then this struct can be removed from the cache and freed.
901:
902: Thus, the same `Curl_easy` can be used multiple times and each time select
903: another `connectdata` struct to use for the connection. Keep this in mind,
904: as it is then important to consider if options or choices are based on the
905: connection or the `Curl_easy`.
906:
907: Functions in libcurl will assume that `connectdata->data` points to the
908: `Curl_easy` that uses this connection (for the moment).
909:
910: As a special complexity, some protocols supported by libcurl require a
911: special disconnect procedure that is more than just shutting down the
912: socket. It can involve sending one or more commands to the server before
913: doing so. Since connections are kept in the connection cache after use, the
914: original `Curl_easy` may no longer be around when the time comes to shut down
915: a particular connection. For this purpose, libcurl holds a special dummy
916: `closure_handle` `Curl_easy` in the `Curl_multi` struct to use when needed.
917:
918: FTP uses two TCP connections for a typical transfer but it keeps both in
919: this single struct and thus can be considered a single connection for most
920: internal concerns.
921:
922: The libcurl source code generally use the name `conn` for the variable that
923: points to the connectdata.
924:
925: <a name="Curl_multi"></a>
926: ## Curl_multi
927:
928: Internally, the easy interface is implemented as a wrapper around multi
929: interface functions. This makes everything multi interface.
930:
931: `Curl_multi` is the multi handle struct exposed as `CURLM *` in external
932: APIs.
933:
934: This struct holds a list of `Curl_easy` structs that have been added to this
935: handle with [`curl_multi_add_handle()`][13]. The start of the list is
936: `->easyp` and `->num_easy` is a counter of added `Curl_easy`s.
937:
938: `->msglist` is a linked list of messages to send back when
939: [`curl_multi_info_read()`][14] is called. Basically a node is added to that
940: list when an individual `Curl_easy`'s transfer has completed.
941:
942: `->hostcache` points to the name cache. It is a hash table for looking up
943: name to IP. The nodes have a limited life time in there and this cache is
944: meant to reduce the time for when the same name is wanted within a short
945: period of time.
946:
947: `->timetree` points to a tree of `Curl_easy`s, sorted by the remaining time
948: until it should be checked - normally some sort of timeout. Each `Curl_easy`
949: has one node in the tree.
950:
951: `->sockhash` is a hash table to allow fast lookups of socket descriptor for
952: which `Curl_easy` uses that descriptor. This is necessary for the
953: `multi_socket` API.
954:
955: `->conn_cache` points to the connection cache. It keeps track of all
956: connections that are kept after use. The cache has a maximum size.
957:
958: `->closure_handle` is described in the `connectdata` section.
959:
960: The libcurl source code generally use the name `multi` for the variable that
961: points to the `Curl_multi` struct.
962:
963: <a name="Curl_handler"></a>
964: ## Curl_handler
965:
966: Each unique protocol that is supported by libcurl needs to provide at least
967: one `Curl_handler` struct. It defines what the protocol is called and what
968: functions the main code should call to deal with protocol specific issues.
969: In general, there's a source file named `[protocol].c` in which there's a
970: `struct Curl_handler Curl_handler_[protocol]` declared. In `url.c` there's
971: then the main array with all individual `Curl_handler` structs pointed to
972: from a single array which is scanned through when a URL is given to libcurl
973: to work with.
974:
975: `->scheme` is the URL scheme name, usually spelled out in uppercase. That's
976: "HTTP" or "FTP" etc. SSL versions of the protocol need their own
977: `Curl_handler` setup so HTTPS separate from HTTP.
978:
979: `->setup_connection` is called to allow the protocol code to allocate
980: protocol specific data that then gets associated with that `Curl_easy` for
981: the rest of this transfer. It gets freed again at the end of the transfer.
982: It will be called before the `connectdata` for the transfer has been
983: selected/created. Most protocols will allocate its private
984: `struct [PROTOCOL]` here and assign `Curl_easy->req.protop` to point to it.
985:
986: `->connect_it` allows a protocol to do some specific actions after the TCP
987: connect is done, that can still be considered part of the connection phase.
988:
989: Some protocols will alter the `connectdata->recv[]` and
990: `connectdata->send[]` function pointers in this function.
991:
992: `->connecting` is similarly a function that keeps getting called as long as
993: the protocol considers itself still in the connecting phase.
994:
995: `->do_it` is the function called to issue the transfer request. What we call
996: the DO action internally. If the DO is not enough and things need to be kept
997: getting done for the entire DO sequence to complete, `->doing` is then
998: usually also provided. Each protocol that needs to do multiple commands or
999: similar for do/doing need to implement their own state machines (see SCP,
1000: SFTP, FTP). Some protocols (only FTP and only due to historical reasons) has
1001: a separate piece of the DO state called `DO_MORE`.
1002:
1003: `->doing` keeps getting called while issuing the transfer request command(s)
1004:
1005: `->done` gets called when the transfer is complete and DONE. That's after the
1006: main data has been transferred.
1007:
1008: `->do_more` gets called during the `DO_MORE` state. The FTP protocol uses
1009: this state when setting up the second connection.
1010:
1011: `->proto_getsock`
1012: `->doing_getsock`
1013: `->domore_getsock`
1014: `->perform_getsock`
1015: Functions that return socket information. Which socket(s) to wait for which
1016: action(s) during the particular multi state.
1017:
1018: `->disconnect` is called immediately before the TCP connection is shutdown.
1019:
1020: `->readwrite` gets called during transfer to allow the protocol to do extra
1021: reads/writes
1022:
1023: `->defport` is the default report TCP or UDP port this protocol uses
1024:
1025: `->protocol` is one or more bits in the `CURLPROTO_*` set. The SSL versions
1026: have their "base" protocol set and then the SSL variation. Like
1027: "HTTP|HTTPS".
1028:
1029: `->flags` is a bitmask with additional information about the protocol that will
1030: make it get treated differently by the generic engine:
1031:
1032: - `PROTOPT_SSL` - will make it connect and negotiate SSL
1033:
1034: - `PROTOPT_DUAL` - this protocol uses two connections
1035:
1036: - `PROTOPT_CLOSEACTION` - this protocol has actions to do before closing the
1037: connection. This flag is no longer used by code, yet still set for a bunch
1038: of protocol handlers.
1039:
1040: - `PROTOPT_DIRLOCK` - "direction lock". The SSH protocols set this bit to
1041: limit which "direction" of socket actions that the main engine will
1042: concern itself with.
1043:
1044: - `PROTOPT_NONETWORK` - a protocol that doesn't use network (read `file:`)
1045:
1046: - `PROTOPT_NEEDSPWD` - this protocol needs a password and will use a default
1047: one unless one is provided
1048:
1049: - `PROTOPT_NOURLQUERY` - this protocol can't handle a query part on the URL
1050: (?foo=bar)
1051:
1052: <a name="conncache"></a>
1053: ## conncache
1054:
1055: Is a hash table with connections for later re-use. Each `Curl_easy` has a
1056: pointer to its connection cache. Each multi handle sets up a connection
1057: cache that all added `Curl_easy`s share by default.
1058:
1059: <a name="Curl_share"></a>
1060: ## Curl_share
1061:
1062: The libcurl share API allocates a `Curl_share` struct, exposed to the
1063: external API as `CURLSH *`.
1064:
1065: The idea is that the struct can have a set of its own versions of caches and
1066: pools and then by providing this struct in the `CURLOPT_SHARE` option, those
1067: specific `Curl_easy`s will use the caches/pools that this share handle
1068: holds.
1069:
1070: Then individual `Curl_easy` structs can be made to share specific things
1071: that they otherwise wouldn't, such as cookies.
1072:
1073: The `Curl_share` struct can currently hold cookies, DNS cache and the SSL
1074: session cache.
1075:
1076: <a name="CookieInfo"></a>
1077: ## CookieInfo
1078:
1079: This is the main cookie struct. It holds all known cookies and related
1080: information. Each `Curl_easy` has its own private `CookieInfo` even when
1081: they are added to a multi handle. They can be made to share cookies by using
1082: the share API.
1083:
1084:
1085: [1]: https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
1086: [2]: https://curl.haxx.se/libcurl/c/curl_easy_init.html
1087: [3]: https://c-ares.haxx.se/
1088: [4]: https://tools.ietf.org/html/rfc7230 "RFC 7230"
1089: [5]: https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
1090: [6]: https://curl.haxx.se/docs/manpage.html#--compressed
1091: [7]: https://curl.haxx.se/libcurl/c/curl_multi_socket_action.html
1092: [8]: https://curl.haxx.se/libcurl/c/curl_multi_timeout.html
1093: [9]: https://curl.haxx.se/libcurl/c/curl_multi_setopt.html
1094: [10]: https://curl.haxx.se/libcurl/c/CURLMOPT_TIMERFUNCTION.html
1095: [11]: https://curl.haxx.se/libcurl/c/curl_multi_perform.html
1096: [12]: https://curl.haxx.se/libcurl/c/curl_multi_fdset.html
1097: [13]: https://curl.haxx.se/libcurl/c/curl_multi_add_handle.html
1098: [14]: https://curl.haxx.se/libcurl/c/curl_multi_info_read.html
1099: [15]: https://tools.ietf.org/html/rfc7231#section-3.1.2.2
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to INTERNALS.md CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / docs