1: _ _ ____ _
2: ___| | | | _ \| |
3: / __| | | | |_) | |
4: | (__| |_| | _ <| |___
5: \___|\___/|_| \_\_____|
6:
7:
8: The Art Of Scripting HTTP Requests Using Curl
9:
10: 1. HTTP Scripting
11: 1.1 Background
12: 1.2 The HTTP Protocol
13: 1.3 See the Protocol
14: 1.4 See the Timing
15: 1.5 See the Response
16: 2. URL
17: 2.1 Spec
18: 2.2 Host
19: 2.3 Port number
20: 2.4 User name and password
21: 2.5 Path part
22: 3. Fetch a page
23: 3.1 GET
24: 3.2 HEAD
25: 3.3 Multiple URLs in a single command line
26: 3.4 Multiple HTTP methods in a single command line
27: 4. HTML forms
28: 4.1 Forms explained
29: 4.2 GET
30: 4.3 POST
31: 4.4 File Upload POST
32: 4.5 Hidden Fields
33: 4.6 Figure Out What A POST Looks Like
34: 5. HTTP upload
35: 5.1 PUT
36: 6. HTTP Authentication
37: 6.1 Basic Authentication
38: 6.2 Other Authentication
39: 6.3 Proxy Authentication
40: 6.4 Hiding credentials
41: 7. More HTTP Headers
42: 7.1 Referer
43: 7.2 User Agent
44: 8. Redirects
45: 8.1 Location header
46: 8.2 Other redirects
47: 9. Cookies
48: 9.1 Cookie Basics
49: 9.2 Cookie options
50: 10. HTTPS
51: 10.1 HTTPS is HTTP secure
52: 10.2 Certificates
53: 11. Custom Request Elements
54: 11.1 Modify method and headers
55: 11.2 More on changed methods
56: 12. Web Login
57: 12.1 Some login tricks
58: 13. Debug
59: 13.1 Some debug tricks
60: 14. References
61: 14.1 Standards
62: 14.2 Sites
63:
64: ==============================================================================
65:
66: 1. HTTP Scripting
67:
68: 1.1 Background
69:
70: This document assumes that you're familiar with HTML and general networking.
71:
72: The increasing amount of applications moving to the web has made "HTTP
73: Scripting" more frequently requested and wanted. To be able to automatically
74: extract information from the web, to fake users, to post or upload data to
75: web servers are all important tasks today.
76:
77: Curl is a command line tool for doing all sorts of URL manipulations and
78: transfers, but this particular document will focus on how to use it when
79: doing HTTP requests for fun and profit. I'll assume that you know how to
80: invoke 'curl --help' or 'curl --manual' to get basic information about it.
81:
82: Curl is not written to do everything for you. It makes the requests, it gets
83: the data, it sends data and it retrieves the information. You probably need
84: to glue everything together using some kind of script language or repeated
85: manual invokes.
86:
87: 1.2 The HTTP Protocol
88:
89: HTTP is the protocol used to fetch data from web servers. It is a very simple
90: protocol that is built upon TCP/IP. The protocol also allows information to
91: get sent to the server from the client using a few different methods, as will
92: be shown here.
93:
94: HTTP is plain ASCII text lines being sent by the client to a server to
95: request a particular action, and then the server replies a few text lines
96: before the actual requested content is sent to the client.
97:
98: The client, curl, sends a HTTP request. The request contains a method (like
99: GET, POST, HEAD etc), a number of request headers and sometimes a request
100: body. The HTTP server responds with a status line (indicating if things went
101: well), response headers and most often also a response body. The "body" part
102: is the plain data you requested, like the actual HTML or the image etc.
103:
104: 1.3 See the Protocol
105:
106: Using curl's option --verbose (-v as a short option) will display what kind
107: of commands curl sends to the server, as well as a few other informational
108: texts.
109:
110: --verbose is the single most useful option when it comes to debug or even
111: understand the curl<->server interaction.
112:
113: Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer
114: even more details as they show EVERYTHING curl sends and receives. Use it
115: like this:
116:
117: curl --trace-ascii debugdump.txt http://www.example.com/
118:
119: 1.4 See the Timing
120:
121: Many times you may wonder what exactly is taking all the time, or you just
122: want to know the amount of milliseconds between two points in a
123: transfer. For those, and other similar situations, the --trace-time option
124: is what you need. It'll prepend the time to each trace output line:
125:
126: curl --trace-ascii d.txt --trace-time http://example.com/
127:
128: 1.5 See the Response
129:
130: By default curl sends the response to stdout. You need to redirect it
131: somewhere to avoid that, most often that is done with -o or -O.
132:
133: 2. URL
134:
135: 2.1 Spec
136:
137: The Uniform Resource Locator format is how you specify the address of a
138: particular resource on the Internet. You know these, you've seen URLs like
139: https://curl.haxx.se or https://yourbank.com a million times. RFC 3986 is the
140: canonical spec. And yeah, the formal name is not URL, it is URI.
141:
142: 2.2 Host
143:
144: The host name is usually resolved using DNS or your /etc/hosts file to an IP
145: address and that's what curl will communicate with. Alternatively you specify
146: the IP address directly in the URL instead of a name.
147:
148: For development and other trying out situations, you can point to a different
149: IP address for a host name than what would otherwise be used, by using curl's
150: --resolve option:
151:
152: curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/
153:
154: 2.3 Port number
155:
156: Each protocol curl supports operates on a default port number, be it over TCP
157: or in some cases UDP. Normally you don't have to take that into
158: consideration, but at times you run test servers on other ports or
159: similar. Then you can specify the port number in the URL with a colon and a
160: number immediately following the host name. Like when doing HTTP to port
161: 1234:
162:
163: curl http://www.example.org:1234/
164:
165: The port number you specify in the URL is the number that the server uses to
166: offer its services. Sometimes you may use a local proxy, and then you may
167: need to specify that proxy's port number separately for what curl needs to
168: connect to locally. Like when using a HTTP proxy on port 4321:
169:
170: curl --proxy http://proxy.example.org:4321 http://remote.example.org/
171:
172: 2.4 User name and password
173:
174: Some services are setup to require HTTP authentication and then you need to
175: provide name and password which is then transferred to the remote site in
176: various ways depending on the exact authentication protocol used.
177:
178: You can opt to either insert the user and password in the URL or you can
179: provide them separately:
180:
181: curl http://user:password@example.org/
182:
183: or
184:
185: curl -u user:password http://example.org/
186:
187: You need to pay attention that this kind of HTTP authentication is not what
188: is usually done and requested by user-oriented web sites these days. They
189: tend to use forms and cookies instead.
190:
191: 2.5 Path part
192:
193: The path part is just sent off to the server to request that it sends back
194: the associated response. The path is what is to the right side of the slash
195: that follows the host name and possibly port number.
196:
197: 3. Fetch a page
198:
199: 3.1 GET
200:
201: The simplest and most common request/operation made using HTTP is to GET a
202: URL. The URL could itself refer to a web page, an image or a file. The client
203: issues a GET request to the server and receives the document it asked for.
204: If you issue the command line
205:
206: curl https://curl.haxx.se
207:
208: you get a web page returned in your terminal window. The entire HTML document
209: that that URL holds.
210:
211: All HTTP replies contain a set of response headers that are normally hidden,
212: use curl's --include (-i) option to display them as well as the rest of the
213: document.
214:
215: 3.2 HEAD
216:
217: You can ask the remote server for ONLY the headers by using the --head (-I)
218: option which will make curl issue a HEAD request. In some special cases
219: servers deny the HEAD method while others still work, which is a particular
220: kind of annoyance.
221:
222: The HEAD method is defined and made so that the server returns the headers
223: exactly the way it would do for a GET, but without a body. It means that you
224: may see a Content-Length: in the response headers, but there must not be an
225: actual body in the HEAD response.
226:
227: 3.3 Multiple URLs in a single command line
228:
229: A single curl command line may involve one or many URLs. The most common case
230: is probably to just use one, but you can specify any amount of URLs. Yes
231: any. No limits. You'll then get requests repeated over and over for all the
232: given URLs.
233:
234: Example, send two GETs:
235:
236: curl http://url1.example.com http://url2.example.com
237:
238: If you use --data to POST to the URL, using multiple URLs means that you send
239: that same POST to all the given URLs.
240:
241: Example, send two POSTs:
242:
243: curl --data name=curl http://url1.example.com http://url2.example.com
244:
245:
246: 3.4 Multiple HTTP methods in a single command line
247:
248: Sometimes you need to operate on several URLs in a single command line and do
249: different HTTP methods on each. For this, you'll enjoy the --next option. It
250: is basically a separator that separates a bunch of options from the next. All
251: the URLs before --next will get the same method and will get all the POST
252: data merged into one.
253:
254: When curl reaches the --next on the command line, it'll sort of reset the
255: method and the POST data and allow a new set.
256:
257: Perhaps this is best shown with a few examples. To send first a HEAD and then
258: a GET:
259:
260: curl -I http://example.com --next http://example.com
261:
262: To first send a POST and then a GET:
263:
264: curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html
265:
266:
267: 4. HTML forms
268:
269: 4.1 Forms explained
270:
271: Forms are the general way a web site can present a HTML page with fields for
272: the user to enter data in, and then press some kind of 'OK' or 'Submit'
273: button to get that data sent to the server. The server then typically uses
274: the posted data to decide how to act. Like using the entered words to search
275: in a database, or to add the info in a bug tracking system, display the entered
276: address on a map or using the info as a login-prompt verifying that the user
277: is allowed to see what it is about to see.
278:
279: Of course there has to be some kind of program on the server end to receive
280: the data you send. You cannot just invent something out of the air.
281:
282: 4.2 GET
283:
284: A GET-form uses the method GET, as specified in HTML like:
285:
286: <form method="GET" action="junk.cgi">
287: <input type=text name="birthyear">
288: <input type=submit name=press value="OK">
289: </form>
290:
291: In your favorite browser, this form will appear with a text box to fill in
292: and a press-button labeled "OK". If you fill in '1905' and press the OK
293: button, your browser will then create a new URL to get for you. The URL will
294: get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the
295: previous URL.
296:
297: If the original form was seen on the page "www.hotmail.com/when/birth.html",
298: the second page you'll get will become
299: "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK".
300:
301: Most search engines work this way.
302:
303: To make curl do the GET form post for you, just enter the expected created
304: URL:
305:
306: curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK"
307:
308: 4.3 POST
309:
310: The GET method makes all input field names get displayed in the URL field of
311: your browser. That's generally a good thing when you want to be able to
312: bookmark that page with your given data, but it is an obvious disadvantage
313: if you entered secret information in one of the fields or if there are a
314: large amount of fields creating a very long and unreadable URL.
315:
316: The HTTP protocol then offers the POST method. This way the client sends the
317: data separated from the URL and thus you won't see any of it in the URL
318: address field.
319:
320: The form would look very similar to the previous one:
321:
322: <form method="POST" action="junk.cgi">
323: <input type=text name="birthyear">
324: <input type=submit name=press value=" OK ">
325: </form>
326:
327: And to use curl to post this form with the same data filled in as before, we
328: could do it like:
329:
330: curl --data "birthyear=1905&press=%20OK%20" \
331: http://www.example.com/when.cgi
332:
333: This kind of POST will use the Content-Type
334: application/x-www-form-urlencoded and is the most widely used POST kind.
335:
336: The data you send to the server MUST already be properly encoded, curl will
337: not do that for you. For example, if you want the data to contain a space,
338: you need to replace that space with %20 etc. Failing to comply with this
339: will most likely cause your data to be received wrongly and messed up.
340:
341: Recent curl versions can in fact url-encode POST data for you, like this:
342:
343: curl --data-urlencode "name=I am Daniel" http://www.example.com
344:
345: If you repeat --data several times on the command line, curl will
346: concatenate all the given data pieces - and put a '&' symbol between each
347: data segment.
348:
349: 4.4 File Upload POST
350:
351: Back in late 1995 they defined an additional way to post data over HTTP. It
352: is documented in the RFC 1867, why this method sometimes is referred to as
353: RFC1867-posting.
354:
355: This method is mainly designed to better support file uploads. A form that
356: allows a user to upload a file could be written like this in HTML:
357:
358: <form method="POST" enctype='multipart/form-data' action="upload.cgi">
359: <input type=file name=upload>
360: <input type=submit name=press value="OK">
361: </form>
362:
363: This clearly shows that the Content-Type about to be sent is
364: multipart/form-data.
365:
366: To post to a form like this with curl, you enter a command line like:
367:
368: curl --form upload=@localfilename --form press=OK [URL]
369:
370: 4.5 Hidden Fields
371:
372: A very common way for HTML based applications to pass state information
373: between pages is to add hidden fields to the forms. Hidden fields are
374: already filled in, they aren't displayed to the user and they get passed
375: along just as all the other fields.
376:
377: A similar example form with one visible field, one hidden field and one
378: submit button could look like:
379:
380: <form method="POST" action="foobar.cgi">
381: <input type=text name="birthyear">
382: <input type=hidden name="person" value="daniel">
383: <input type=submit name="press" value="OK">
384: </form>
385:
386: To POST this with curl, you won't have to think about if the fields are
387: hidden or not. To curl they're all the same:
388:
389: curl --data "birthyear=1905&press=OK&person=daniel" [URL]
390:
391: 4.6 Figure Out What A POST Looks Like
392:
393: When you're about fill in a form and send to a server by using curl instead
394: of a browser, you're of course very interested in sending a POST exactly the
395: way your browser does.
396:
397: An easy way to get to see this, is to save the HTML page with the form on
398: your local disk, modify the 'method' to a GET, and press the submit button
399: (you could also change the action URL if you want to).
400:
401: You will then clearly see the data get appended to the URL, separated with a
402: '?'-letter as GET forms are supposed to.
403:
404: 5. HTTP upload
405:
406: 5.1 PUT
407:
408: Perhaps the best way to upload data to a HTTP server is to use PUT. Then
409: again, this of course requires that someone put a program or script on the
410: server end that knows how to receive a HTTP PUT stream.
411:
412: Put a file to a HTTP server with curl:
413:
414: curl --upload-file uploadfile http://www.example.com/receive.cgi
415:
416: 6. HTTP Authentication
417:
418: 6.1 Basic Authentication
419:
420: HTTP Authentication is the ability to tell the server your username and
421: password so that it can verify that you're allowed to do the request you're
422: doing. The Basic authentication used in HTTP (which is the type curl uses by
423: default) is *plain* *text* based, which means it sends username and password
424: only slightly obfuscated, but still fully readable by anyone that sniffs on
425: the network between you and the remote server.
426:
427: To tell curl to use a user and password for authentication:
428:
429: curl --user name:password http://www.example.com
430:
431: 6.2 Other Authentication
432:
433: The site might require a different authentication method (check the headers
434: returned by the server), and then --ntlm, --digest, --negotiate or even
435: --anyauth might be options that suit you.
436:
437: 6.3 Proxy Authentication
438:
439: Sometimes your HTTP access is only available through the use of a HTTP
440: proxy. This seems to be especially common at various companies. A HTTP proxy
441: may require its own user and password to allow the client to get through to
442: the Internet. To specify those with curl, run something like:
443:
444: curl --proxy-user proxyuser:proxypassword curl.haxx.se
445:
446: If your proxy requires the authentication to be done using the NTLM method,
447: use --proxy-ntlm, if it requires Digest use --proxy-digest.
448:
449: If you use any one of these user+password options but leave out the password
450: part, curl will prompt for the password interactively.
451:
452: 6.4 Hiding credentials
453:
454: Do note that when a program is run, its parameters might be possible to see
455: when listing the running processes of the system. Thus, other users may be
456: able to watch your passwords if you pass them as plain command line
457: options. There are ways to circumvent this.
458:
459: It is worth noting that while this is how HTTP Authentication works, very
460: many web sites will not use this concept when they provide logins etc. See
461: the Web Login chapter further below for more details on that.
462:
463: 7. More HTTP Headers
464:
465: 7.1 Referer
466:
467: A HTTP request may include a 'referer' field (yes it is misspelled), which
468: can be used to tell from which URL the client got to this particular
469: resource. Some programs/scripts check the referer field of requests to verify
470: that this wasn't arriving from an external site or an unknown page. While
471: this is a stupid way to check something so easily forged, many scripts still
472: do it. Using curl, you can put anything you want in the referer-field and
473: thus more easily be able to fool the server into serving your request.
474:
475: Use curl to set the referer field with:
476:
477: curl --referer http://www.example.come http://www.example.com
478:
479: 7.2 User Agent
480:
481: Very similar to the referer field, all HTTP requests may set the User-Agent
482: field. It names what user agent (client) that is being used. Many
483: applications use this information to decide how to display pages. Silly web
484: programmers try to make different pages for users of different browsers to
485: make them look the best possible for their particular browsers. They usually
486: also do different kinds of javascript, vbscript etc.
487:
488: At times, you will see that getting a page with curl will not return the same
489: page that you see when getting the page with your browser. Then you know it
490: is time to set the User Agent field to fool the server into thinking you're
491: one of those browsers.
492:
493: To make curl look like Internet Explorer 5 on a Windows 2000 box:
494:
495: curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]
496:
497: Or why not look like you're using Netscape 4.73 on an old Linux box:
498:
499: curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
500:
501: 8. Redirects
502:
503: 8.1 Location header
504:
505: When a resource is requested from a server, the reply from the server may
506: include a hint about where the browser should go next to find this page, or a
507: new page keeping newly generated output. The header that tells the browser
508: to redirect is Location:.
509:
510: Curl does not follow Location: headers by default, but will simply display
511: such pages in the same manner it displays all HTTP replies. It does however
512: feature an option that will make it attempt to follow the Location: pointers.
513:
514: To tell curl to follow a Location:
515:
516: curl --location http://www.example.com
517:
518: If you use curl to POST to a site that immediately redirects you to another
519: page, you can safely use --location (-L) and --data/--form together. Curl will
520: only use POST in the first request, and then revert to GET in the following
521: operations.
522:
523: 8.2 Other redirects
524:
525: Browser typically support at least two other ways of redirects that curl
526: doesn't: first the html may contain a meta refresh tag that asks the browser
527: to load a specific URL after a set number of seconds, or it may use
528: javascript to do it.
529:
530: 9. Cookies
531:
532: 9.1 Cookie Basics
533:
534: The way the web browsers do "client side state control" is by using
535: cookies. Cookies are just names with associated contents. The cookies are
536: sent to the client by the server. The server tells the client for what path
537: and host name it wants the cookie sent back, and it also sends an expiration
538: date and a few more properties.
539:
540: When a client communicates with a server with a name and path as previously
541: specified in a received cookie, the client sends back the cookies and their
542: contents to the server, unless of course they are expired.
543:
544: Many applications and servers use this method to connect a series of requests
545: into a single logical session. To be able to use curl in such occasions, we
546: must be able to record and send back cookies the way the web application
547: expects them. The same way browsers deal with them.
548:
549: 9.2 Cookie options
550:
551: The simplest way to send a few cookies to the server when getting a page with
552: curl is to add them on the command line like:
553:
554: curl --cookie "name=Daniel" http://www.example.com
555:
556: Cookies are sent as common HTTP headers. This is practical as it allows curl
557: to record cookies simply by recording headers. Record cookies with curl by
558: using the --dump-header (-D) option like:
559:
560: curl --dump-header headers_and_cookies http://www.example.com
561:
562: (Take note that the --cookie-jar option described below is a better way to
563: store cookies.)
564:
565: Curl has a full blown cookie parsing engine built-in that comes in use if you
566: want to reconnect to a server and use cookies that were stored from a
567: previous connection (or hand-crafted manually to fool the server into
568: believing you had a previous connection). To use previously stored cookies,
569: you run curl like:
570:
571: curl --cookie stored_cookies_in_file http://www.example.com
572:
573: Curl's "cookie engine" gets enabled when you use the --cookie option. If you
574: only want curl to understand received cookies, use --cookie with a file that
575: doesn't exist. Example, if you want to let curl understand cookies from a
576: page and follow a location (and thus possibly send back cookies it received),
577: you can invoke it like:
578:
579: curl --cookie nada --location http://www.example.com
580:
581: Curl has the ability to read and write cookie files that use the same file
582: format that Netscape and Mozilla once used. It is a convenient way to share
583: cookies between scripts or invokes. The --cookie (-b) switch automatically
584: detects if a given file is such a cookie file and parses it, and by using the
585: --cookie-jar (-c) option you'll make curl write a new cookie file at the end
586: of an operation:
587:
588: curl --cookie cookies.txt --cookie-jar newcookies.txt \
589: http://www.example.com
590:
591: 10. HTTPS
592:
593: 10.1 HTTPS is HTTP secure
594:
595: There are a few ways to do secure HTTP transfers. By far the most common
596: protocol for doing this is what is generally known as HTTPS, HTTP over
597: SSL. SSL encrypts all the data that is sent and received over the network and
598: thus makes it harder for attackers to spy on sensitive information.
599:
600: SSL (or TLS as the latest version of the standard is called) offers a
601: truckload of advanced features to allow all those encryptions and key
602: infrastructure mechanisms encrypted HTTP requires.
603:
604: Curl supports encrypted fetches when built to use a TLS library and it can be
605: built to use one out of a fairly large set of libraries - "curl -V" will show
606: which one your curl was built to use (if any!). To get a page from a HTTPS
607: server, simply run curl like:
608:
609: curl https://secure.example.com
610:
611: 10.2 Certificates
612:
613: In the HTTPS world, you use certificates to validate that you are the one
614: you claim to be, as an addition to normal passwords. Curl supports client-
615: side certificates. All certificates are locked with a pass phrase, which you
616: need to enter before the certificate can be used by curl. The pass phrase
617: can be specified on the command line or if not, entered interactively when
618: curl queries for it. Use a certificate with curl on a HTTPS server like:
619:
620: curl --cert mycert.pem https://secure.example.com
621:
622: curl also tries to verify that the server is who it claims to be, by
623: verifying the server's certificate against a locally stored CA cert
624: bundle. Failing the verification will cause curl to deny the connection. You
625: must then use --insecure (-k) in case you want to tell curl to ignore that
626: the server can't be verified.
627:
628: More about server certificate verification and ca cert bundles can be read
629: in the SSLCERTS document, available online here:
630:
631: https://curl.haxx.se/docs/sslcerts.html
632:
633: At times you may end up with your own CA cert store and then you can tell
634: curl to use that to verify the server's certificate:
635:
636: curl --cacert ca-bundle.pem https://example.com/
637:
638:
639: 11. Custom Request Elements
640:
641: 11.1 Modify method and headers
642:
643: Doing fancy stuff, you may need to add or change elements of a single curl
644: request.
645:
646: For example, you can change the POST request to a PROPFIND and send the data
647: as "Content-Type: text/xml" (instead of the default Content-Type) like this:
648:
649: curl --data "<xml>" --header "Content-Type: text/xml" \
650: --request PROPFIND url.com
651:
652: You can delete a default header by providing one without content. Like you
653: can ruin the request by chopping off the Host: header:
654:
655: curl --header "Host:" http://www.example.com
656:
657: You can add headers the same way. Your server may want a "Destination:"
658: header, and you can add it:
659:
660: curl --header "Destination: http://nowhere" http://example.com
661:
662: 11.2 More on changed methods
663:
664: It should be noted that curl selects which methods to use on its own
665: depending on what action to ask for. -d will do POST, -I will do HEAD and so
666: on. If you use the --request / -X option you can change the method keyword
667: curl selects, but you will not modify curl's behavior. This means that if you
668: for example use -d "data" to do a POST, you can modify the method to a
669: PROPFIND with -X and curl will still think it sends a POST. You can change
670: the normal GET to a POST method by simply adding -X POST in a command line
671: like:
672:
673: curl -X POST http://example.org/
674:
675: ... but curl will still think and act as if it sent a GET so it won't send any
676: request body etc.
677:
678:
679: 12. Web Login
680:
681: 12.1 Some login tricks
682:
683: While not strictly just HTTP related, it still causes a lot of people problems
684: so here's the executive run-down of how the vast majority of all login forms
685: work and how to login to them using curl.
686:
687: It can also be noted that to do this properly in an automated fashion, you
688: will most certainly need to script things and do multiple curl invokes etc.
689:
690: First, servers mostly use cookies to track the logged-in status of the
691: client, so you will need to capture the cookies you receive in the
692: responses. Then, many sites also set a special cookie on the login page (to
693: make sure you got there through their login page) so you should make a habit
694: of first getting the login-form page to capture the cookies set there.
695:
696: Some web-based login systems feature various amounts of javascript, and
697: sometimes they use such code to set or modify cookie contents. Possibly they
698: do that to prevent programmed logins, like this manual describes how to...
699: Anyway, if reading the code isn't enough to let you repeat the behavior
700: manually, capturing the HTTP requests done by your browsers and analyzing the
701: sent cookies is usually a working method to work out how to shortcut the
702: javascript need.
703:
704: In the actual <form> tag for the login, lots of sites fill-in random/session
705: or otherwise secretly generated hidden tags and you may need to first capture
706: the HTML code for the login form and extract all the hidden fields to be able
707: to do a proper login POST. Remember that the contents need to be URL encoded
708: when sent in a normal POST.
709:
710: 13. Debug
711:
712: 13.1 Some debug tricks
713:
714: Many times when you run curl on a site, you'll notice that the site doesn't
715: seem to respond the same way to your curl requests as it does to your
716: browser's.
717:
718: Then you need to start making your curl requests more similar to your
719: browser's requests:
720:
721: * Use the --trace-ascii option to store fully detailed logs of the requests
722: for easier analyzing and better understanding
723:
724: * Make sure you check for and use cookies when needed (both reading with
725: --cookie and writing with --cookie-jar)
726:
727: * Set user-agent to one like a recent popular browser does
728:
729: * Set referer like it is set by the browser
730:
731: * If you use POST, make sure you send all the fields and in the same order as
732: the browser does it.
733:
734: A very good helper to make sure you do this right, is the LiveHTTPHeader tool
735: that lets you view all headers you send and receive with Mozilla/Firefox
736: (even when using HTTPS). Chrome features similar functionality out of the box
737: among the developer's tools.
738:
739: A more raw approach is to capture the HTTP traffic on the network with tools
740: such as ethereal or tcpdump and check what headers that were sent and
741: received by the browser. (HTTPS makes this technique inefficient.)
742:
743: 14. References
744:
745: 14.1 Standards
746:
747: RFC 7230 is a must to read if you want in-depth understanding of the HTTP
748: protocol
749:
750: RFC 3986 explains the URL syntax
751:
752: RFC 1867 defines the HTTP post upload format
753:
754: RFC 6525 defines how HTTP cookies work
755:
756: 14.2 Sites
757:
758: https://curl.haxx.se is the home of the curl project
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to TheArtOfHttpScripting CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / curl / docs