Codebase list clj-http-clojure / bdc1e6c
New upstream version 2.3.0 Apollon Oikonomopoulos 6 years ago
30 changed file(s) with 6570 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
+40
-0
.gitignore less more
0 # leiningen .gitignore defaults
1 /target
2 /classes
3 /checkouts
4 pom.xml
5 pom.xml.asc
6 *.jar
7 *.class
8 /.lein-*
9 /.nrepl-port
10
11 # custom from here on out
12 build
13 lib
14 *.dot
15
16 # use glob syntax.
17 syntax: glob
18 creds.clj
19 Manifest.txt
20 aws.clj
21 *.ser
22 *~
23 *.bak
24 *.off
25 *.old
26 .DS_Store
27 *.#*
28 *#*
29 *.classpath
30 *.project
31 *.settings
32 *.pyc
33 docs/*
34 doc
35 http.log
36
37 # Intellij Idea
38 /*.iml
39 /.idea
+9
-0
.travis.yml less more
0 language: clojure
1 lein: lein2
2 script: lein2 all do clean, test
3 branches:
4 only:
5 - master
6 jdk:
7 - openjdk7
8 - oraclejdk7
+21
-0
LICENSE less more
0 The MIT License (MIT)
1
2 Copyright (c) 2013 M. Lee Hinman
3
4 Permission is hereby granted, free of charge, to any person obtaining a copy
5 of this software and associated documentation files (the "Software"), to deal
6 in the Software without restriction, including without limitation the rights
7 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
8 copies of the Software, and to permit persons to whom the Software is
9 furnished to do so, subject to the following conditions:
10
11 The above copyright notice and this permission notice shall be included in
12 all copies or substantial portions of the Software.
13
14 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
20 THE SOFTWARE.
+1038
-0
README.org less more
0 #+TITLE: clj-http documentation
1 #+AUTHOR: Lee Hinman
2 #+STARTUP: align fold nodlcheck lognotestate showall
3 #+OPTIONS: H:4 num:nil toc:t \n:nil @:t ::t |:t ^:{} -:t f:t *:t
4 #+OPTIONS: skip:nil d:(HIDE) tags:not-in-toc auto-id:t
5 #+PROPERTY: header-args :results code :exports both :noweb yes
6 #+HTML_HEAD: <style type="text/css"> body {margin-right:15%; margin-left:15%;} </style>
7 #+LANGUAGE: en
8
9 * Table of Contents :TOC:
10 :PROPERTIES:
11 :CUSTOM_ID: h:84c64317-1dfa-4955-bfa7-180745c31546
12 :END:
13 - [[#introduction][Introduction]]
14 - [[#overview][Overview]]
15 - [[#philosophy-][Philosophy ]]
16 - [[#installation][Installation]]
17 - [[#quickstart][Quickstart]]
18 - [[#head][HEAD]]
19 - [[#get][GET]]
20 - [[#put][PUT]]
21 - [[#post][POST]]
22 - [[#delete][DELETE]]
23 - [[#coercions][Coercions]]
24 - [[#headers][Headers]]
25 - [[#meta-tag-headers][Meta Tag Headers]]
26 - [[#link-headers][Link Headers]]
27 - [[#redirects-][Redirects ]]
28 - [[#cookies][Cookies]]
29 - [[#exceptions][Exceptions]]
30 - [[#decompression][Decompression]]
31 - [[#debugging][Debugging]]
32 - [[#authentication][Authentication]]
33 - [[#basic-auth][Basic Auth]]
34 - [[#digest-auth][Digest Auth]]
35 - [[#oauth2][oAuth2]]
36 - [[#advanced-usage][Advanced Usage]]
37 - [[#raw-request][Raw Request]]
38 - [[#persistent-connections][Persistent Connections]]
39 - [[#proxies][Proxies]]
40 - [[#custom-middleware][Custom Middleware]]
41 - [[#development][Development]]
42 - [[#faking-responses][Faking Responses]]
43 - [[#optional-dependencies][Optional Dependencies]]
44 - [[#clj-http-lite][clj-http-lite]]
45 - [[#troubleshooting][Troubleshooting]]
46 - [[#tests][Tests]]
47 - [[#testimonials][Testimonials]]
48 - [[#license][License]]
49
50 * Introduction
51 :PROPERTIES:
52 :CUSTOM_ID: h:5caf5111-96b3-401b-bba3-6b66cc625cbd
53 :END:
54
55 ** Overview
56 :PROPERTIES:
57 :CUSTOM_ID: h:301e4e08-cd19-4066-888d-166f35d3f696
58 :END:
59
60 clj-http is an HTTP library wrapping the [[http://hc.apache.org/][Apache HttpComponents]] client. This
61 library has taken over from mmcgrana's clj-http.
62
63 [[https://secure.travis-ci.org/dakrone/clj-http.png]]
64
65 ** Philosophy
66 :PROPERTIES:
67 :CUSTOM_ID: h:b8fdaffd-1f9f-4d08-93f7-ef50392e7af1
68 :END:
69
70 The design of =clj-http= is inspired by the [[http://github.com/mmcgrana/ring][Ring]] protocol for Clojure HTTP
71 server applications.
72
73 The client in =clj-http.core= makes HTTP requests according to a given Ring
74 request map and returns [[https://github.com/ring-clojure/ring/blob/master/SPEC][Ring response maps]] corresponding to the resulting HTTP
75 response. The function =clj-http.client/request= uses Ring-style middleware to
76 layer functionality over the core HTTP request/response implementation. Methods
77 like =clj-http.client/get= are sugar over this =clj-http.client/request=
78 function.
79
80 * Installation
81 :PROPERTIES:
82 :CUSTOM_ID: h:280b3315-2b20-484d-962b-7f7132d20840
83 :END:
84
85 =clj-http= is available as a Maven artifact from [[http://clojars.org/clj-http][Clojars]].
86
87 With Leiningen/Boot:
88
89 #+BEGIN_SRC clojure
90 [clj-http "2.3.0"]
91 #+END_SRC
92
93 The previous major versions is available as:
94
95 #+BEGIN_SRC clojure
96 [clj-http "1.1.2"]
97 #+END_SRC
98
99 clj-http supports clojure 1.6.0 and higher.
100
101 * Quickstart
102 :PROPERTIES:
103 :CUSTOM_ID: h:67644772-7a82-451b-91b8-6cab871445b6
104 :END:
105
106 The main HTTP client functionality is provided by the =clj-http.client= namespace.
107
108 First, require it in the REPL:
109
110 #+BEGIN_SRC clojure
111 (require '[clj-http.client :as client])
112 #+END_SRC
113
114 Or in your application:
115
116 #+BEGIN_SRC clojure
117 (ns my-app.core
118 (:require [clj-http.client :as client]))
119 #+END_SRC
120
121 The client supports simple =get=, =head=, =put=, =post=, =delete=, =copy=,
122 =move=, =patch=, and =options= requests. Response are returned as [[https://github.com/ring-clojure/ring/blob/master/SPEC][Ring-style
123 response maps]]:
124
125 ** HEAD
126 :PROPERTIES:
127 :CUSTOM_ID: h:5db60716-9834-4658-9256-63732e69bed6
128 :END:
129
130 #+BEGIN_SRC clojure
131
132 (client/head "http://example.com/resource")
133
134 (client/head "http://site.com/resource" {:accept :json})
135
136 #+END_SRC
137
138 ** GET
139 :PROPERTIES:
140 :CUSTOM_ID: h:b2c26b5c-a36a-4f65-9c70-5d9921a2390d
141 :END:
142
143 Example requests:
144
145 #+BEGIN_SRC clojure
146
147 (client/get "http://site.com/resources/id")
148
149 (client/get "http://site.com/resources/3" {:accept :json})
150
151 ;; Specifying headers as either a string or collection:
152 (client/get "http://example.com"
153 {:headers {"foo" ["bar" "baz"], "eggplant" "quux"}})
154
155 ;; Using either string or keyword header names:
156 (client/get "http://example.com"
157 {:headers {:foo ["bar" "baz"], :eggplant "quux"}})
158
159 ;; Set any specific client parameters manually:
160 (client/post "http://example.com"
161 {:client-params {"http.protocol.allow-circular-redirects" false
162 "http.protocol.version" HttpVersion/HTTP_1_0
163 "http.useragent" "clj-http"}})
164
165 ;; Set your own cookie policy
166 (client/post "http://example.com"
167 {:client-params {:cookie-policy (fn [cookie origin] (your-validation cookie origin))}})
168
169 ;; Completely ignore cookies:
170 (client/post "http://example.com"
171 {:client-params {:cookie-policy (constantly nil)}})
172
173 ;; Need to contact a server with an untrusted SSL cert?
174 (client/get "https://alioth.debian.org" {:insecure? true})
175
176 ;; If you don't want to follow-redirects automatically:
177 (client/get "http://site.come/redirects-somewhere" {:follow-redirects false})
178
179 ;; Only follow a certain number of redirects:
180 (client/get "http://site.come/redirects-somewhere" {:max-redirects 5})
181
182 ;; Throw an exception if redirected too many times:
183 (client/get "http://site.come/redirects-somewhere" {:max-redirects 5 :throw-exceptions true})
184
185 ;; Throw an exception if the get takes too long. Timeouts in milliseconds.
186 (client/get "http://site.come/redirects-somewhere" {:socket-timeout 1000 :conn-timeout 1000})
187
188 ;; Query parameters
189 (client/get "http://site.com/search" {:query-params {"q" "foo, bar"}})
190
191 ;; "Nested" query parameters
192 ;; (this yields a query string of `a[e][f]=6&a[b][c]=5`)
193 (client/get "http://site.com/search" {:query-params {:a {:b {:c 5} :e {:f 6})
194
195 ;; Provide cookies — uses same schema as :cookies returned in responses
196 ;; (see the cookie store option for easy cross-request maintenance of cookies)
197 (client/get "http://site.com"
198 {:cookies {"ring-session" {:discard true, :path "/", :value "", :version 0}}})
199
200 ;; Tell clj-http not to decode cookies from the response header
201 (client/get "http://example.com" {:decode-cookies false})
202
203 ;; Support for IPv6!
204 (client/get "http://[2001:62f5:9006:e472:cabd:c8ff:fee3:8ddf]")
205
206 #+END_SRC
207
208 The client will also follow redirects on the appropriate =30*= status codes.
209
210 The client transparently accepts and decompresses the =gzip= and =deflate=
211 content encodings.
212
213 =:trace-redirects= will contain the chain of the redirections followed.
214
215 ** PUT
216 :PROPERTIES:
217 :CUSTOM_ID: h:f28939e5-24af-4e0d-ac3d-81c52a271418
218 :END:
219
220 #+BEGIN_SRC clojure
221
222 (client/put "http://example.com/api" {:body "my PUT body"})
223
224 #+END_SRC
225
226 ** POST
227 :PROPERTIES:
228 :CUSTOM_ID: h:f1454284-011f-4426-8cfc-c116da4301e0
229 :END:
230
231 #+BEGIN_SRC clojure
232
233 ;; Various options:
234 (client/post "http://site.com/api"
235 {:basic-auth ["user" "pass"]
236 :body "{\"json\": \"input\"}"
237 :headers {"X-Api-Version" "2"}
238 :content-type :json
239 :socket-timeout 1000 ;; in milliseconds
240 :conn-timeout 1000 ;; in milliseconds
241 :accept :json})
242
243 ;; Send form params as a urlencoded body (POST or PUT)
244 (client/post "http://site.com" {:form-params {:foo "bar"}})
245
246 ;; Send form params as a json encoded body (POST or PUT)
247 (client/post "http://site.com" {:form-params {:foo "bar"} :content-type :json})
248
249 ;; Send form params as a json encoded body (POST or PUT) with options
250 (client/post "http://site.com" {:form-params {:foo "bar"}
251 :content-type :json
252 :json-opts {:date-format "yyyy-MM-dd"}})
253
254 ;; You can also specify the encoding of form parameters
255 (client/post "http://site.com" {:form-params {:foo "bar"}
256 :form-param-encoding "ISO-8859-1"})
257
258 ;; Send form params as a Transit encoded JSON body (POST or PUT) with options
259 (client/post "http://site.com" {:form-params {:foo "bar"}
260 :content-type :transit+json
261 :transit-opts
262 {:encode {:handlers {}}
263 :decode {:handlers {}}}})
264
265 ;; Send form params as a Transit encoded MessagePack body (POST or PUT) with options
266 (client/post "http://site.com" {:form-params {:foo "bar"}
267 :content-type :transit+msgpack
268 :transit-opts
269 {:encode {:handlers {}}
270 :decode {:handlers {}}}})
271
272 ;; Multipart form uploads/posts
273 ;; takes a vector of maps, to preserve the order of entities, :name
274 ;; will be used as the part name unless :part-name is specified
275 (client/post "http://example.org" {:multipart [{:name "title" :content "My Awesome Picture"}
276 {:name "Content/type" :content "image/jpeg"}
277 {:name "foo.txt" :part-name "eggplant" :content "Eggplants"}
278 {:name "file" :content (clojure.java.io/file "pic.jpg")}]})
279
280 ;; Multipart :content values can be one of the following:
281 ;; String, InputStream, File, a byte-array, or an instance of org.apache.http.entity.mime.content.ContentBody
282 ;; Some Multipart bodies can also support more keys (like :encoding
283 ;; and :mime-type), check src/clj-http/multipart.clj to see all flags
284
285 ;; Apache's http client automatically retries on IOExceptions, if you
286 ;; would like to handle these retries yourself, you can specify a
287 ;; :retry-handler. Return true to retry, false to stop trying:
288 (client/post "http://example.org" {:multipart [["title" "Foo"]
289 ["Content/type" "text/plain"]
290 ["file" (clojure.java.io/file "/tmp/missing-file")]]
291 :retry-handler (fn [ex try-count http-context]
292 (println "Got:" ex)
293 (if (> try-count 4) false true))})
294
295 #+END_SRC
296
297 ** DELETE
298 :PROPERTIES:
299 :CUSTOM_ID: h:23225b05-4fc1-48f1-995c-8daeaf4c7c90
300 :END:
301
302 #+BEGIN_SRC clojure
303
304 (client/delete "http://example.com/resource")
305
306 #+END_SRC
307
308 ** Coercions
309 :PROPERTIES:
310 :CUSTOM_ID: h:44b36885-b8ea-4e78-9e04-5141995e6771
311 :END:
312 *** Input coercion
313 :PROPERTIES:
314 :CUSTOM_ID: h:b72eb0a0-5546-440b-95ea-ff10aa631fd8
315 :END:
316
317 #+BEGIN_SRC clojure
318 ;; body as a byte-array
319 (client/post "http://site.com/resources" {:body my-byte-array})
320
321 ;; body as a string
322 (client/post "http://site.com/resources" {:body "string"})
323
324 ;; :body-encoding is optional and defaults to "UTF-8"
325 (client/post "http://site.com/resources"
326 {:body "string" :body-encoding "UTF-8"})
327
328 ;; body as a file
329 (client/post "http://site.com/resources"
330 {:body (clojure.java.io/file "/tmp/foo") :body-encoding "UTF-8"})
331
332 ;; :length is optional for passing in an InputStream; if not
333 ;; supplied it will default to -1 to signal to HttpClient to use
334 ;; chunked encoding
335 (client/post "http://site.com/resources"
336 {:body (clojure.java.io/input-stream "/tmp/foo")})
337
338 (client/post "http://site.com/resources"
339 {:body (clojure.java.io/input-stream "/tmp/foo") :length 1000})
340 #+END_SRC
341
342 *** Output coercion
343 :PROPERTIES:
344 :CUSTOM_ID: h:f617095a-dbda-40a8-8662-db62d0389fd5
345 :END:
346
347 #+BEGIN_SRC clojure
348 ;; The default output is a string body
349 (client/get "http://site.com/foo.txt")
350
351 ;; Coerce as a byte-array
352 (client/get "http://site.com/favicon.ico" {:as :byte-array})
353
354 ;; Coerce as something other than UTF-8 string
355 (client/get "http://site.com/string.txt" {:as "UTF-16"})
356
357 ;; Coerce as json
358 (client/get "http://site.com/foo.json" {:as :json})
359 (client/get "http://site.com/foo.json" {:as :json-strict})
360 (client/get "http://site.com/foo.json" {:as :json-string-keys})
361 (client/get "http://site.com/foo.json" {:as :json-strict-string-keys})
362
363 ;; Coerce as Transit encoded JSON or MessagePack
364 (client/get "http://site.com/foo" {:as :transit+json})
365 (client/get "http://site.com/foo" {:as :transit+msgpack})
366
367 ;; Coerce as a clojure datastructure
368 (client/get "http://site.com/foo.clj" {:as :clojure})
369
370 ;; Coerce as x-www-form-urlencoded
371 (client/post "http://site.com/foo" {:as :x-www-form-urlencoded})
372
373 ;; Try to automatically coerce the output based on the content-type
374 ;; header (this is currently a BETA feature!). Currently supports
375 ;; text, json and clojure (with automatic charset detection)
376 ;; clojure coercion requires "application/clojure" or
377 ;; "application/edn" in the content-type header
378 (client/get "http://site.com/foo.json" {:as :auto})
379
380 ;; Return the body as a stream
381 (client/get "http://site.com/bigrequest.html" {:as :stream})
382 ;; Note that the connection to the server will NOT be closed until the
383 ;; stream has been read
384 #+END_SRC
385
386 JSON coercion defaults to only an "unexceptional" statuses, meaning status codes
387 in the #{200 201 202 203 204 205 206 207 300 301 302 303 307} range. If you
388 would like to change this, you can send the =:coerce= option, which can be set
389 to:
390
391 #+BEGIN_SRC clojure
392 :always ;; always json decode the body
393 :unexceptional ;; only json decode when not an HTTP error response
394 :exceptional ;; only json decode when it IS an HTTP error response
395 #+END_SRC
396
397 The =:coerce= setting defaults to =:unexceptional=.
398
399 ** Headers
400 :PROPERTIES:
401 :CUSTOM_ID: h:5f5e2c8b-e9da-40ea-a5aa-2afc9fa4f2df
402 :END:
403
404 clj-http's treatment of headers is a little more permissive than the [[https://github.com/ring-clojure/ring/blob/master/SPEC][ring spec]]
405 specifies.
406
407 Rather than forcing all request headers to be lowercase strings,
408 clj-http allows strings or keywords of any case. Keywords will be
409 transformed into their canonical representation, so the :content-md5
410 header will be sent to the server as "Content-MD5", for instance.
411 String keys in request headers, however, will be sent to the server
412 with their casing unchanged.
413
414 Response headers can be read as keywords or strings of any case. If
415 the server responds with a "Date" header, you could access the value
416 of that header as :date, "date", "Date", etc.
417
418 If for some reason you require access to the original header name that
419 the server specified, it is available by invoking (keys ...) on the
420 header map.
421
422 This special treatment of headers is implemented in the
423 wrap-header-map middleware, which (like any middleware) can be
424 disabled by using with-middleware to specify different behavior.
425
426 ** Query-string parameters
427 :PROPERTIES:
428 :CUSTOM_ID: h:dd49992c-a516-4af0-9735-4f4340773361
429 :END:
430
431 There are three different ways that query string parameters for array values can
432 be generated, depending on what the resulting query string should look like,
433 they are:
434
435 - A repeating parameter (default)
436 - Array style
437 - Indexed array style
438
439 Here is an example of the input and output for the ~:query_string~ parameter,
440 controlled by the ~:multi-param-style~ option:
441
442 #+BEGIN_SRC clojure
443 ;; default style, with :multi-param-style unset
444 :a [1 2 3] => "a=1&a=2&a=3"
445 ;; with :multi-param-style :array, a repeating param with array suffix
446 ;; (PHP-style):
447 :a [1 2 3] => "a[]=1&a[]=2&a[]=3"
448 ;; with :multi-param-style :indexed, a repeating param with array suffix and
449 ;; index (Rails-style):
450 :a [1 2 3] => "a[0]=1&a[1]=2&a[2]=3"
451 #+END_SRC
452
453 ** Meta Tag Headers
454 :PROPERTIES:
455 :CUSTOM_ID: h:1f1a4258-849f-4324-8687-d066c15de09b
456 :END:
457
458 HTML 4.01 allows using the tag ~<meta http-equiv="..." />~ and HTML 5 allows
459 using the tag ~<meta charset="..." />~ to specify a header that should be
460 treated as an HTTP response header. By default, clj-http will ignore the body of
461 the response (other than the regular output coercion), but if you need clj-http
462 to parse the headers out of the body, you can use the =:decode-body-headers=
463 option:
464
465 #+BEGIN_SRC clojure
466 ;; without decoding body headers (defaults to off):
467 (:headers (client/get "http://www.yomiuri.co.jp/"))
468 => {"server" "Apache",
469 "content-encoding" "gzip",
470 "content-type" "text/html",
471 "date" "Tue, 09 Oct 2012 18:02:41 GMT",
472 "cache-control" "max-age=0, no-cache",
473 "expires" "Tue, 09 Oct 2012 18:02:41 GMT",
474 "etag" "\"1dfb-2686-4cba2686fb8b1\"",
475 "pragma" "no-cache",
476 "connection" "close"}
477
478 ;; with decoding body headers, notice the content-type,
479 ;; content-style-type and content-script-type headers:
480 (:headers (client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true}))
481 => {"server" "Apache",
482 "content-encoding" "gzip",
483 "content-script-type" "text/javascript",
484 "content-style-type" "text/css",
485 "content-type" "text/html; charset=Shift_JIS",
486 "date" "Tue, 09 Oct 2012 18:02:59 GMT",
487 "cache-control" "max-age=0, no-cache",
488 "expires" "Tue, 09 Oct 2012 18:02:59 GMT",
489 "etag" "\"1dfb-2686-4cba2686fb8b1\"",
490 "pragma" "no-cache",
491 "connection" "close"}
492 #+END_SRC
493
494 This can be used to have clj-http correctly interpret the body's charset by
495 using:
496
497 #+BEGIN_SRC clojure
498 (client/get "http://www.yomiuri.co.jp/" {:decode-body-headers true :as :auto})
499 => ;; correctly formatted :body (Shift_JIS charset instead of UTF-8)
500 #+END_SRC
501
502 Note that this feature is currently beta and uses [[https://github.com/weavejester/crouton][Crouton]] to parse the body of
503 the request. If you do not want to use this feature, you can include Crouton in
504 addition to clj-http as a dependency like so:
505
506 #+BEGIN_SRC clojure
507 (defproject foo "0.1.0-SNAPSHOT"
508 :dependencies [[org.clojure/clojure "1.3.0"]
509 [clj-http "0.6.0"]
510 [crouton "1.0.0"]])
511 #+END_SRC
512
513 Note also that HEAD requests will not return a body, in which case this setting will have no effect.
514
515 clj-http will automatically disable the =:decode-body-headers= option.
516
517 ** Link Headers
518 :PROPERTIES:
519 :CUSTOM_ID: h:338ed551-06d7-4889-91cd-b21aec21d15f
520 :END:
521
522 clj-http parses any [[http://tools.ietf.org/html/rfc5988][link headers]] returned in the response, and adds them to the
523 =:links= key on the response map. This is particularly useful for paging RESTful
524 APIs:
525
526 #+BEGIN_SRC clojure
527 (:links (client/get "https://api.github.com/gists"))
528 => {:next {:href "https://api.github.com/gists?page=2"}
529 :last {:href "https://api.github.com/gists?page=22884"}}
530 #+END_SRC
531
532 ** Redirects
533 :PROPERTIES:
534 :CUSTOM_ID: h:0176f085-4ddb-4dfd-9007-d27a6e598ebd
535 :END:
536
537 clj-http conforms its behaviour regarding automatic redirects to the [[https://tools.ietf.org/html/rfc2616#section-10.3][RFC]].
538
539 It means that redirects on status =301=, =302= and =307= are not redirected on
540 methods other than =GET= and =HEAD=. If you want a behaviour closer to what most
541 browser have, you can set =:force-redirects= to =true= in your request to have
542 automatic redirection work on all methods by transforming the method of the
543 request to =GET=.
544
545 ** Cookies
546 :PROPERTIES:
547 :CUSTOM_ID: h:82472506-4fbe-4c1d-9c09-b6f764647c24
548 :END:
549
550 *** Cookiestores
551 :PROPERTIES:
552 :CUSTOM_ID: h:d9887431-486f-456f-b698-3f708e46367f
553 :END:
554
555 clj-http can simplify the maintenance of cookies across requests if it is
556 provided with a _cookie store_.
557
558 #+BEGIN_SRC clojure
559 (binding [clj-http.core/*cookie-store* (clj-http.cookies/cookie-store)]
560 (client/post "http://site.com/login" {:form-params {:username "..."
561 :password "..."}})
562 (client/get "http://site.com/secured-page")
563 ...)
564 #+END_SRC
565
566 (The =clj-http.cookies/cookie-store= function returns a new empty instance of a
567 default implementation of =org.apache.http.client.CookieStore=.)
568
569 This will allow cookies to only be _written_ to the cookie store. Cookies from
570 the cookie-store will not automatically be sent with future requests.
571
572 If you would like cookies from the cookie-store to automatically be sent with
573 each request, specify the cookie-store with the =:cookie-store= option:
574
575 #+BEGIN_SRC clojure
576 (let [my-cs (clj-http.cookies/cookie-store)]
577 (client/post "http://site.com/login" {:form-params {:username "..."
578 :password "..."}
579 :cookie-store my-cs})
580 (client/post "http://site.com/update" {:body my-data
581 :cookie-store my-cs}))
582 #+END_SRC
583
584 You can also us the =get-cookies= function to retrieve the cookies
585 from a cookie store:
586
587 #+BEGIN_SRC clojure
588 (def cs (clj-http.cookies/cookie-store))
589
590 (client/get "http://google.com" {:cookie-store cs})
591
592 (clojure.pprint/pprint (clj-http.cookies/get-cookies cs))
593 {"NID"
594 {:domain ".google.com",
595 :expires #<Date Tue Oct 02 10:12:06 MDT 2012>,
596 :path "/",
597 :value
598 "58=c387....",
599 :version 0},
600 "PREF"
601 {:domain ".google.com",
602 :expires #<Date Wed Apr 02 10:12:06 MDT 2014>,
603 :path "/",
604 :value
605 "ID=3ba...:FF=0:TM=133...:LM=133...:S=_iRM...",
606 :version 0}}
607 #+END_SRC
608
609 *** Keystores, Trust-stores
610 :PROPERTIES:
611 :CUSTOM_ID: h:1546e3f1-3f9f-459a-9015-628afa22f59e
612 :END:
613
614 You can also specify your own keystore/trust-store to be used:
615
616 #+BEGIN_SRC clojure
617 (client/get "https://example.com" {:keystore "/path/to/keystore.ks"
618 :keystore-type "jks" ; default: jks
619 :keystore-pass "secretpass"
620 :trust-store "/path/to/trust-store.ks"
621 :trust-store-type "jks" ; default jks
622 :trust-store-pass "trustpass"})
623 #+END_SRC
624
625 The =:keystore/:trust-store= values may be either paths to keystore
626 files or =KeyStore= instances.
627
628 ** Exceptions
629 :PROPERTIES:
630 :CUSTOM_ID: h:dfb56fc9-a958-41ad-8de2-af15a4cd4902
631 :END:
632
633 The client will throw exceptions on, well, exceptional status codes, meaning all
634 HTTP responses other than =#{200 201 202 203 204 205 206 207 300 301 302 303
635 307}=. clj-http will throw a [[http://github.com/scgilardi/slingshot][Slingshot]] Stone that can be caught by a regular
636 =(catch Exception e ...)= or in Slingshot's =try+= block:
637
638 #+BEGIN_SRC clojure
639 (client/get "http://site.com/broken")
640 => ExceptionInfo clj-http: status 404 clj-http.client/wrap-exceptions/fn--583 (client.clj:41)
641 ;; Or, if you would like the Exception message to contain the entire response:
642 (client/get "http://site.com/broken" {:throw-entire-message? true})
643 => ExceptionInfo clj-http: status 404 {:status 404,
644 :headers {"server" "nginx/1.0.4",
645 "x-runtime" "12ms",
646 "content-encoding" "gzip",
647 "content-type" "text/html; charset=utf-8",
648 "date" "Mon, 17 Oct 2011 23:15 :36 GMT",
649 "cache-control" "no-cache",
650 "status" "404 Not Found",
651 "transfer-encoding" "chunked",
652 "connection" "close"},
653 :body "...body here..."}
654 clj-http.client/wrap-exceptions/fn--584 (client.clj:42
655
656 ;; You can also ignore HTTP-status-code exceptions and handle them yourself:
657 (client/get "http://site.com/broken" {:throw-exceptions false})
658 ;; Or ignore an unknown host (methods return 'nil' if this is set to
659 ;; true and the host does not exist:
660 (client/get "http://aoeuntahuf89o.com" {:ignore-unknown-host? true})
661 #+END_SRC
662
663 (spacing added by me to be human readable)
664
665 How to use with Slingshot:
666
667 #+BEGIN_SRC
668 ; Response map is thrown as exception obj.
669 ; We filter out by status codes
670 (try+
671 (client/get "http://some-site.com/broken")
672 (catch [:status 403] {:keys [request-time headers body]}
673 (log/warn "403" request-time headers))
674 (catch [:status 404] {:keys [request-time headers body]}
675 (log/warn "NOT Found 404" request-time headers body))
676 (catch Object _
677 (log/error (:throwable &throw-context) "unexpected error")
678 (throw+)))
679 #+END_SRC
680
681 ** Decompression
682 :PROPERTIES:
683 :CUSTOM_ID: h:1ff48808-dc42-46de-8cef-12983c446d80
684 :END:
685
686 By default, clj-http will add the ={"Accept-Encoding" "gzip, deflate"}= header
687 to requests, and automatically decompress the resulting gzip or deflate stream
688 if the =Content-Encoding= header is found on the response. If this is undesired,
689 the ={:decompress-body false}= option can be specified:
690
691 #+BEGIN_SRC clojure
692 ;; Auto-decompression used: (google requires a user-agent to send gzip data)
693 (def h {"User-Agent" "Mozilla/5.0 (Windows NT 6.1;) Gecko/20100101 Firefox/13.0.1"})
694 (def resp (client/get "http://google.com" {:headers h}))
695 (:orig-content-encoding resp)
696 => "gzip" ;; <= google sent response gzipped
697
698 ;; and without decompression:
699 (def resp2 (client/get "http://google.com" {:headers h :decompress-body false})
700 (:orig-content-encoding resp2)
701 => nil
702 #+END_SRC
703
704 If clj-http decompresses something, the "content-encoding" header is removed
705 from the headers (because the encoding is no longer true). This allows clj-http
706 to be used as a pass-through proxy with ring. The original content-encoding is
707 available as =:orig-content-encoding= in the response map if auto-decompression
708 is enabled.
709
710 ** Debugging
711 :PROPERTIES:
712 :CUSTOM_ID: h:f86f4daa-356e-40ca-b87e-bf347ec1f38e
713 :END:
714
715 There are four debugging methods you can use:
716
717 #+BEGIN_SRC clojure
718 ;; print request info to *out*:
719 (client/get "http://example.org" {:debug true})
720
721 ;; print request info to *out*, including request body:
722 (client/post "http://example.org" {:debug true :debug-body true :body "..."})
723
724 ;; save the request that was sent in a :request key in the response:
725 (client/get "http://example.org" {:save-request? true})
726
727 ;; save the request that was sent in a :request key in the response,
728 ;; including the body content:
729 (client/get "http://example.org" {:save-request? true :debug-body true})
730
731 ;; add an HttpResponseInterceptor to the request. This callback
732 ;; is called for each redirects with the following args:
733 ;; ^HttpResponse resp, HttpContext^ ctx
734 ;; this allows low level debugging + access to socket.
735 ;; see http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/HttpResponseInterceptor.html
736 (client/get "http://example.org" {:response-interceptor (fn [resp ctx] (println ctx))})
737 #+END_SRC
738
739 * Authentication
740 :PROPERTIES:
741 :CUSTOM_ID: h:3c375d7a-7acc-45cb-a7a4-6f2bdf4cad95
742 :END:
743
744 ** Basic Auth
745 :PROPERTIES:
746 :CUSTOM_ID: h:8ae77bcc-68c6-4560-affb-4bbe02c6b7a9
747 :END:
748
749 #+BEGIN_SRC
750
751 (client/get "http://site.com/protected" {:basic-auth ["user" "pass"]})
752 (client/get "http://site.com/protected" {:basic-auth "user:pass"})
753
754 #+END_SRC
755
756 ** Digest Auth
757 :PROPERTIES:
758 :CUSTOM_ID: h:47c07a03-677f-4a4f-967f-242329a8ab07
759 :END:
760
761 #+BEGIN_SRC
762
763 (client/get "http://site.com/protected" {:digest-auth ["user" "pass"]})
764
765 #+END_SRC
766
767 ** oAuth2
768 :PROPERTIES:
769 :CUSTOM_ID: h:e34482c0-15d2-483a-a183-d5e3f1f662a6
770 :END:
771
772 #+BEGIN_SRC
773
774 (client/get "http://site.com/protected" {:oauth-token "secret-token"})
775
776 #+END_SRC
777
778 * Advanced Usage
779 :PROPERTIES:
780 :CUSTOM_ID: h:e6aed224-7ed5-4340-bc4e-7874eefadd87
781 :END:
782
783 ** Raw Request
784 :PROPERTIES:
785 :CUSTOM_ID: h:71bf84d3-2ff0-44a8-99aa-214d339cf7d2
786 :END:
787
788 A more general =request= function is also available, which is useful as a
789 primitive for building higher-level interfaces:
790
791 #+BEGIN_SRC clojure
792 (defn api-action [method path & [opts]]
793 (client/request
794 (merge {:method method :url (str "http://site.com/" path)} opts)))
795 #+END_SRC
796
797 *** Boolean options
798 :PROPERTIES:
799 :CUSTOM_ID: h:4a5870a5-693f-441d-a69c-da96eebbbb6e
800 :END:
801
802 Since 0.9.0, all boolean options can be expressed as either ={:debug true}= or
803 ={:debug? true}=, with or without the question mark.
804
805 ** Persistent Connections
806 :PROPERTIES:
807 :CUSTOM_ID: h:5f755de8-c106-4b89-aa0a-3074ef96efc4
808 :END:
809
810 clj-http can use persistent connections to speed up connections if multiple
811 connections are being used:
812
813 #+BEGIN_SRC clojure
814 (with-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
815 (get "http://aoeu.com/1")
816 (post "http://aoeu.com/2")
817 (get "http://aoeu.com/3")
818 ...
819 (get "http://aoeu.com/999"))
820 #+END_SRC
821
822 This is MUCH faster than sequentially performing all requests, because a
823 persistent connection can be used instead creating a new connection for each
824 request.
825
826 If you would prefer to handle managing the connection manager yourself, you can
827 create a connection manager yourself and specify it for each request:
828
829 #+BEGIN_SRC clojure
830 (def cm (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 2 :threads 3}))
831 (def cm2 (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 10 :threads 1}))
832
833 (get "http://aoeu.com/1" {:connection-manager cm2})
834 (post "http://aoeu.com/2" {:connection-manager cm})
835 (get "http://aoeu.com/3" {:connection-manager cm2})
836
837 ;; Don't forget to shut it down when you're done!
838 (clj-http.conn-mgr/shutdown-manager cm)
839 (clj-http.conn-mgr/shutdown-manager cm2)
840 #+END_SRC
841
842 See the docstring on =make-reusable-conn-manager= for options and default
843 values.
844
845 ** Proxies
846 :PROPERTIES:
847 :CUSTOM_ID: h:b5007a6f-f0bf-4d98-9ab9-b23fcebfa49a
848 :END:
849
850 A proxy can be specified by setting the Java properties: =<scheme>.proxyHost=
851 and =<scheme>.proxyPort= where =<scheme>= is the client scheme used (normally
852 'http' or 'https'). =http.nonProxyHosts= allows you to specify a pattern for
853 hostnames which do not require proxy routing - this is shared for all schemes.
854 Additionally, per-request proxies can be specified with the =proxy-host= and
855 =proxy-port= options (this overrides =http.nonProxyHosts= too):
856
857 #+BEGIN_SRC clojure
858 (client/get "http://foo.com" {:proxy-host "127.0.0.1" :proxy-port 8118})
859 #+END_SRC
860
861 You can also specify the =proxy-ignore-hosts= parameter with a list of
862 hosts where the proxy should be ignored. By default this list is
863 =#{"localhost" "127.0.0.1"}=.
864
865 A SOCKS proxy can be used by creating a proxied connection manager with
866 =clj-http.conn-mgr/make-socks-proxied-conn-manager=. Then using that connection
867 manager in the request.
868
869 For example if you wanted to connect to a local socks proxy on port =8081= you
870 would:
871
872 #+BEGIN_SRC clojure
873 (ns foo.bar
874 (:require [clj-http.client :as client]
875 [clj-http.conn-mgr :as conn-mgr]))
876
877 (client/get "https://google.com"
878 {:connection-manager
879 (conn-mgr/make-socks-proxied-conn-manager "localhost" 8081)})
880 #+END_SRC
881
882 You can also store the proxied connection manager and reuse it later.
883
884 ** Custom Middleware
885 :PROPERTIES:
886 :CUSTOM_ID: h:afec8fd4-580a-4a82-9521-628f8fa4fbd8
887 :END:
888
889 Sometime it is desirable to run a request with some middleware enabled and some
890 left out, the =with-middleware= method provides this functionality:
891
892 #+BEGIN_SRC clojure
893 (with-middleware [#'clj-http.client/wrap-method
894 #'clj-http.client/wrap-url
895 #'clj-http.client/wrap-exceptions]
896 (get "http://example.com")
897 (post "http://example.com/foo" {:body (.getBytes "foo")}))
898 #+END_SRC
899
900 To see available middleware, check the =clj-http.client/default-middleware= var,
901 which is a vector of the default middleware that clj-http uses.
902 =clj-http.client/*current-middleware*= is bound to the current list of
903 middleware during request time.
904
905 * Development
906 :PROPERTIES:
907 :CUSTOM_ID: h:0fb11882-d060-4e2e-85f7-3bf18dc9051b
908 :END:
909
910 Please send a pull request or open an issue if you have any problems.
911
912 ** Faking Responses
913 :PROPERTIES:
914 :CUSTOM_ID: h:8bacf773-9af1-4098-907e-f96a780d3fca
915 :END:
916
917 If you need to fake clj-http responses (for things like testing and such), check
918 out the [[https://github.com/myfreeweb/clj-http-fake][clj-http-fake]] library.
919
920 ** Optional Dependencies
921 :PROPERTIES:
922 :CUSTOM_ID: h:a847429d-741f-4a5a-8d27-916ea1017461
923 :END:
924
925 In 2.0.0+ clj-http's optional dependencies at excluded by default, in order to
926 use the features you will need to add them to your =project.clj= file.
927
928 clj-http currently has four optional dependencies, =cheshire=, =crouton=,
929 =tools.reader= and =ring/ring-codec=. Any number of them may be included by
930 adding them with the clj-http dependency in your project.clj:
931
932 #+BEGIN_SRC clojure
933 ;; optional dependencies
934 [cheshire] ;; for :as :json
935 [crouton] ;; for :decode-body-headers
936 [org.clojure/tools.reader] ;; for :as :clojure
937 [ring/ring-codec] ;; for :as :x-www-form-urlencoded
938 #+END_SRC
939
940 Prior to 2.0.0, you can /exclude/ the dependencies and clj-http will work
941 without them.
942
943 ** clj-http-lite
944 :PROPERTIES:
945 :CUSTOM_ID: h:472e4fba-3c50-4eb6-95fb-95d0d9afdbad
946 :END:
947
948 Like clj-http but need something more lightweight without as many external
949 dependencies? Check out [[https://github.com/hiredman/clj-http-lite][clj-http-lite]] for a project that can be used as a
950 drop-in replacement for clj-http.
951
952 ** Troubleshooting
953 :PROPERTIES:
954 :CUSTOM_ID: h:e97ba275-3324-4102-9beb-9bcbc483ad15
955 :END:
956 *** VerifyError class org.codehaus.jackson.smile.SmileParser overrides final method getBinaryValue...
957 :PROPERTIES:
958 :CUSTOM_ID: h:048d8994-647a-4325-ab5d-f96fa12d5798
959 :END:
960
961 This is actually caused by your project attempting to use [[https://github.com/mmcgrana/clj-json/][clj-json]] and [[https://github.com/dakrone/cheshire][cheshire]]
962 in the same classloader. You can fix the issue by either not using clj-json (and
963 thus choosing cheshire), or specifying an exclusion for clj-http in your project
964 like this:
965
966 #+BEGIN_SRC clojure
967 (defproject foo "0.1.0-SNAPSHOT"
968 :dependencies [[org.clojure/clojure "1.3.0"]
969 [clj-http "0.3.4" :exclusions [cheshire]]])
970 #+END_SRC
971
972 Note that if you exclude cheshire, json decoding of response bodies
973 and json encoding of form-params cannot happen, you are responsible
974 for your own encoding/decoding.
975
976 As of clj-http 0.3.5, you should no longer see this, as Cheshire 3.1.0
977 and clj-json can now live together without causing problems.
978
979 *** NoHttpResponseException ... due to stale connections**
980 :PROPERTIES:
981 :CUSTOM_ID: h:c2a2ea17-d402-43ba-b57b-2a5bc75a6750
982 :END:
983
984 Persistent connections kept alive by the connection manager become stale: the
985 target server shuts down the connection on its end without HttpClient being able
986 to react to that event, while the connection is being idle, thus rendering the
987 connection half-closed or 'stale'.
988
989 This can be solved by using (with-connection-pool) as described in the
990 'Using Persistent Connection' section above.
991
992 * Tests
993 :PROPERTIES:
994 :CUSTOM_ID: h:34bd658a-26a2-4731-9b7d-dd93bce8c35a
995 :END:
996
997 To run the tests:
998
999 #+BEGIN_SRC
1000 $ lein deps
1001 $ lein test
1002
1003 Run all tests (including integration):
1004 $ lein test :all
1005
1006 Run tests against 1.2.1, 1.3 and 1.4
1007 $ lein all test
1008 $ lein all test :all
1009 #+END_SRC
1010
1011 * Testimonials
1012 :PROPERTIES:
1013 :CUSTOM_ID: h:3e19427d-7d2b-465e-8fa4-1d59f9555924
1014 :END:
1015
1016 With close to a [million](https://clojars.org/clj-http) downloads, clj-http is a
1017 widely used, battle-tested clojure library. It is also included in other
1018 libraries (like database clients) as a low-level http wrapper.
1019
1020 Libraries using clj-http:
1021
1022 - [[https://github.com/mattrepl/clj-oauth] [clj-oauth]]
1023 - [[[[https://github.com/clojurewerkz/elastisch]]] [elasticsearch]]
1024 - [[https://github.com/olauzon/capacitor] [influxdb]]
1025
1026 Libraries inspired by clj-http:
1027
1028 - [[https://github.com/mpenet/jet] [jet]]
1029 - [[https://github.com/hiredman/clj-http-lite] [clj-http-lite]]
1030
1031 * License
1032 :PROPERTIES:
1033 :CUSTOM_ID: h:9968d81c-ff40-40f5-be27-60bab27c64c9
1034 :END:
1035
1036 Released under the MIT License:
1037 <http://www.opensource.org/licenses/mit-license.php>
+227
-0
changelog.org less more
0 #+TITLE: clj-http changelog
1 #+AUTHOR: Lee Hinman
2 #+STARTUP: fold nodlcheck lognotestate hideall
3 #+OPTIONS: H:4 num:nil toc:t \n:nil @:t ::t |:t ^:{} -:t f:t *:t
4 #+OPTIONS: skip:nil d:(HIDE) tags:not-in-toc
5 #+PROPERTY: header-args :results code :exports both :noweb yes
6 #+HTML_HEAD: <style type="text/css"> body {margin-right:15%; margin-left:15%;} </style>
7 #+LANGUAGE: en
8
9 * Changelog
10 List of user-visible changes that have gone into each release
11
12 ** 2.0.0
13 - merged https://github.com/dakrone/clj-http/pull/274 to update Potemkin so it
14 supports Clojure 1.7.0 correctly
15 - merged https://github.com/dakrone/clj-http/pull/264 to add support for
16 coercion of urlencoded data
17 - make ALL optional dependencies opt-in, rather than opt-out
18 ** 1.1.2
19 - bumped dependencies for transit-clj and tools.reader
20 - merge https://github.com/dakrone/clj-http/pull/263 to only decode body headers
21 when the content-type is either missing or starts with "text"
22 ** 1.1.1
23 - merge https://github.com/dakrone/clj-http/pull/262 to prevent
24 NullPointerException when decoding body headers with HEAD requests
25 - merge https://github.com/dakrone/clj-http/pull/261 to decode user info from
26 URL if provided
27 - merge https://github.com/dakrone/clj-http/pull/260 to upgrade tools.reader
28 for better cljs compatibility
29 - add =304= (not modified) to the list of unexceptional responses, see #259
30 ** 1.1.0
31 - merged https://github.com/dakrone/clj-http/pull/255 to add support for Windows
32 NTLM authentication
33 - Add the `with-additional-middleware` macro
34 - Add the ability to specify form-param-encoding for encoding form parameters
35 - merged https://github.com/dakrone/clj-http/pull/248 to removed deprecated
36 cookie APIs from cookie.clj
37 - merged https://github.com/dakrone/clj-http/pull/245 to do some cleanups and
38 small import fixes
39 - merged https://github.com/dakrone/clj-http/pull/240 to implement
40 meta/with-meta for the header map
41 - merged https://github.com/dakrone/clj-http/pull/242 fixing a connection leak
42 when http-entity is null
43 - bumped all dependencies to latest versions
44 - merged https://github.com/dakrone/clj-http/pull/235 to fix wrap-nested-params
45 - merged https://github.com/dakrone/clj-http/pull/236 to clean up multipart
46 constructors and reflection
47 - merged https://github.com/dakrone/clj-http/pull/234 to allow scheme
48 customization in default connection
49 ** 1.0.1
50 - merged https://github.com/dakrone/clj-http/pull/232 to fix =empty= on
51 header-map
52 - fix :json-strict-string-keys
53 - exclude clojure.core/update from client ns
54 - added =:decode-cookies= option to allow skipping cookie header decode (if the
55 server sends incorrectly formatted cookies for some reason)
56 ** 1.0.0
57 - merged https://github.com/dakrone/clj-http/pull/215 to add transit support
58 - drop support for clojure 1.4.0, start testing 1.7.0
59 - merged https://github.com/dakrone/clj-http/pull/213 to allow passing in an
60 already existing keystore, not just a path
61 - merged https://github.com/dakrone/clj-http/pull/211 to detect charset encoding
62 for url-encode
63 ** 0.9.2
64 - merged https://github.com/dakrone/clj-http/pull/206 to handle null passwords
65 for keystores
66 - merged https://github.com/dakrone/clj-http/pull/201 to make :auto content type
67 parsing dispatch pluggable
68 - Bump crouton and tools.reader dependencies
69 - Merged https://github.com/dakrone/clj-http/pull/199 to add support for form
70 parameters in the PATCH method
71 - Bump dependencies and fix tests for 1.6.0 compatibility
72 ** 0.9.1
73 - automatically coerce header values to strings
74 - fix issue where :ignore-unknown-host wasn't using the =opt= function correctly
75 ** 0.9.0
76 - Bumped httpcore to 4.3.2
77 - Merged https://github.com/dakrone/clj-http/pull/190 to support file multiparts
78 with content, mime-type and name
79 - Unify all boolean operators so {:debug true} and {:debug? true} are treated
80 the same
81 - Fix :trace-redirects being [nil] when :uri is used
82 - Merged https://github.com/dakrone/clj-http/pull/184 containing a bevy of
83 changes:
84 - initial header-map implementation, allowing headers to be used case
85 insensitively
86 - drop support for clojure 1.2 and 1.3
87 - add support for clojure 1.6
88 - change all :use statements to :require statements
89 - use better docstring support for defs
90 - remove sleep calls in tests
91 - make Jetty quieter while running tests
92 - newer type hinting syntax
93 ** 0.7.9
94 - Make :decode-body-headers more reliable by using a byte array instead of
95 slurp.
96 - Merged https://github.com/dakrone/clj-http/pull/181 to fix some tests
97 - Merged https://github.com/dakrone/clj-http/pull/178 to eliminate test
98 reflection
99 - Merged https://github.com/dakrone/clj-http/pull/177 to update apache HTTP deps
100 - Merged https://github.com/dakrone/clj-http/pull/175 to add {:as :json-strict}
101 for output coercion
102 - Added {:as :json-strict-string-keys} output coercion
103 - bump dependencies to their latest
104 - Merged https://github.com/dakrone/clj-http/pull/172 to update .gitignore file
105 and clean up whitespace for new clojure-mode
106 - Merged https://github.com/dakrone/clj-http/pull/171 to support SOCKS proxies
107 * Work log
108 ** Released 2.0.0
109 ** 2015-07-18
110 - merged https://github.com/dakrone/clj-http/pull/274 to update Potemkin so it
111 supports Clojure 1.7.0 correctly
112 ** 2015-05-23
113 - merged https://github.com/dakrone/clj-http/pull/264 to add support for
114 coercion of urlencoded data
115 - make ALL optional dependencies opt-in, rather than opt-out
116 ** Released 1.1.2
117 ** 2015-05-06
118 - bumped dependencies for transit-clj and tools.reader
119 ** 2015-04-24
120 - merge https://github.com/dakrone/clj-http/pull/263 to only decode body headers
121 when the content-type is either missing or starts with "text"
122 ** Released 1.1.1
123 ** 2015-04-22
124 - merge https://github.com/dakrone/clj-http/pull/262 to prevent
125 NullPointerException when decoding body headers with HEAD requests
126 ** 2015-04-20
127 - merge https://github.com/dakrone/clj-http/pull/261 to decode user info from
128 URL if provided
129 ** 2015-04-14
130 - merge https://github.com/dakrone/clj-http/pull/260 to upgrade tools.reader
131 for better cljs compatibility
132 ** 2015-04-05
133 - add =304= (not modified) to the list of unexceptional responses, see #259
134 ** Released 1.1.0
135 ** 2015-03-03
136 - merged https://github.com/dakrone/clj-http/pull/255 to add support for Windows
137 NTLM authentication
138 ** 2015-02-08
139 - Add the `with-additional-middleware` macro
140 - Add the ability to specify form-param-encoding for encoding form parameters
141 ** 2015-01-19
142 - merged https://github.com/dakrone/clj-http/pull/248 to removed deprecated
143 cookie APIs from cookie.clj
144 - merged https://github.com/dakrone/clj-http/pull/245 to do some cleanups and
145 small import fixes
146 ** 2015-01-15
147 - merged https://github.com/dakrone/clj-http/pull/240 to implement
148 meta/with-meta for the header map
149 - merged https://github.com/dakrone/clj-http/pull/242 fixing a connection leak
150 when http-entity is null
151 - bumped all dependencies to latest versions
152 ** 2014-12-13
153 - merged https://github.com/dakrone/clj-http/pull/235 to fix wrap-nested-params
154 ** 2014-12-12
155 - merged https://github.com/dakrone/clj-http/pull/236 to clean up multipart
156 constructors and reflection
157 ** 2014-12-02
158 - merged https://github.com/dakrone/clj-http/pull/234 to allow scheme
159 customization in default connection
160 ** Released 1.0.1
161 ** 2014-10-28
162 - merged https://github.com/dakrone/clj-http/pull/232 to fix =empty= on
163 header-map
164 ** 2014-10-17
165 - fix :json-strict-string-keys
166 ** 2014-09-08
167 - exclude clojure.core/update from client ns
168 ** 2014-08-15
169 - added =:decode-cookies= option to allow skipping cookie header decode (if the
170 server sends incorrectly formatted cookies for some reason)
171 ** Released 1.0.0
172 ** 2014-08-11
173 - merged https://github.com/dakrone/clj-http/pull/215 to add transit support
174 - drop support for clojure 1.4.0, start testing 1.7.0
175 ** 2014-08-07
176 - merged https://github.com/dakrone/clj-http/pull/213 to allow passing in an
177 already existing keystore, not just a path
178 ** 2014-07-27
179 - merged https://github.com/dakrone/clj-http/pull/211 to detect charset encoding
180 for url-encode
181 ** Released 0.9.2
182 ** 2014-05-27
183 - merged https://github.com/dakrone/clj-http/pull/206 to handle null passwords
184 for keystores
185 ** 2014-05-14
186 - merged https://github.com/dakrone/clj-http/pull/201 to make :auto content type
187 parsing dispatch pluggable
188 ** 2014-04-21
189 - Bump crouton and tools.reader dependencies
190 ** 2014-04-09
191 - Merged https://github.com/dakrone/clj-http/pull/199 to add support for form
192 parameters in the PATCH method
193 ** 2014-03-26
194 - Bump dependencies and fix tests for 1.6.0 compatibility
195 ** Released 0.9.1
196 ** 2014-03-15
197 - automatically coerce header values to strings
198 ** 2014-03-05
199 - fix issue where :ignore-unknown-host wasn't using the =opt= function correctly
200 ** Released 0.9.0
201 ** 2014-02-25
202 - Bumped httpcore to 4.3.2
203 ** 2014-02-19
204 - Merged https://github.com/dakrone/clj-http/pull/190 to support file multiparts
205 with content, mime-type and name
206 ** 2014-02-16
207 - Unify all boolean operators so {:debug true} and {:debug? true} are treated
208 the same
209 ** 2014-02-09
210 - Fix :trace-redirects being [nil] when :uri is used
211 ** 2014-02-06
212 - Merged https://github.com/dakrone/clj-http/pull/184 containing a bevy of
213 changes:
214 - initial header-map implementation, allowing headers to be used case
215 insensitively
216 - drop support for clojure 1.2 and 1.3
217 - add support for clojure 1.6
218 - change all :use statements to :require statements
219 - use better docstring support for defs
220 - remove sleep calls in tests
221 - make Jetty quieter while running tests
222 - newer type hinting syntax
223 ** Released 0.7.9
224 ** 2014-02-01
225 - Make :decode-body-headers more reliable by using a byte array instead of
226 slurp.
+713
-0
changelog.org_archive less more
0
1 Archived entries from file /Users/hinmanm/src/clj/clj-http/changelog.org
2
3
4 * Archived Tasks
5
6 ** 0.7.8
7 :PROPERTIES:
8 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
9 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
10 :ARCHIVE_OLPATH: Changelog
11 :ARCHIVE_CATEGORY: changelog
12 :END:
13 - Added the `proxy-ignore-hosts` option to allow specifying a list
14 of hosts where a proxy should be ignored
15 - merged https://github.com/dakrone/clj-http/pull/166 to fix some
16 small whitespace and reflection stuff
17 - Close the body of a response in wrap-redirects since all bodies
18 are streams now. Otherwise, the body is kept open indefinitely.
19
20 ** 0.7.7
21 :PROPERTIES:
22 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
23 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
24 :ARCHIVE_OLPATH: Changelog
25 :ARCHIVE_CATEGORY: changelog
26 :END:
27 - merged https://github.com/dakrone/clj-http/pull/162 to pass
28 through json opts for form-param encoding
29 - bumped dependencies
30 - fix #159 - issue with :decode-body-headers introduced with
31 streaming bodies
32 - merged https://github.com/dakrone/clj-http/pull/156 to add
33 `:raw-headers` option to return an additional
34 untouched :raw-headers map
35 - merged https://github.com/dakrone/clj-http/pull/154 to handle
36 query-params not clobbering query-params in the URL string
37 - bump main deps
38 - merged https://github.com/dakrone/clj-http/pull/151 to prevent
39 shutting down a reusable connection manager when an error occurs
40
41 ** 0.7.6
42 :PROPERTIES:
43 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
44 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
45 :ARCHIVE_OLPATH: Changelog
46 :ARCHIVE_CATEGORY: changelog
47 :END:
48 - add logging config for local testing only
49 - remove "content-encoding" header if the body is automatically
50 decompressed to allow for pass-through. If header is removed,
51 assoc :orig-content-encoding to response map.
52 - merged https://github.com/dakrone/clj-http/pull/149 to fix
53 closing the stream when coerced to byte array
54 - merged https://github.com/dakrone/clj-http/pull/146 to correctly
55 reference parameter names
56
57 ** 0.7.5
58 :PROPERTIES:
59 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
60 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
61 :ARCHIVE_OLPATH: Changelog
62 :ARCHIVE_CATEGORY: changelog
63 :END:
64 - Only redirect if a "location" header is actually, present, avoiding an
65 NPE in the event it's missing. (fixes #145)
66
67 ** 0.7.4
68 :PROPERTIES:
69 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
70 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
71 :ARCHIVE_OLPATH: Changelog
72 :ARCHIVE_CATEGORY: changelog
73 :END:
74 - merged https://github.com/dakrone/clj-http/pull/143 for fixing some
75 weirdness around body streams and inflation
76 - streams everywhere, all bodies coming out of core.clj are now streams, so
77 {:as :stream} truly streams the response, keeping it out of memory
78 - remove some more reflection
79
80 ** 0.7.3
81 :PROPERTIES:
82 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
83 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
84 :ARCHIVE_OLPATH: Changelog
85 :ARCHIVE_CATEGORY: changelog
86 :END:
87 - correctly close single client connection manager if {:as :stream} is used, fixes #142
88 - merged https://github.com/dakrone/clj-http/pull/138 to preserve
89 http method for 307 redirect
90 - merged in parse-url parameters into follow-redirect so request
91 map is not inconsistent
92 - bumped http* deps to 4.2.5
93 - fixed cookie compact-map not to remove falsey values, only nil
94 ones
95 - merged https://github.com/dakrone/clj-http/pull/135 to fix
96 discard always defaulting to true
97 - add *current-middleware* to see available middleware during a
98 with-middleware request (for nesting)
99
100 ** 0.7.2
101 :PROPERTIES:
102 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
103 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
104 :ARCHIVE_OLPATH: Changelog
105 :ARCHIVE_CATEGORY: changelog
106 :END:
107 - merged https://github.com/dakrone/clj-http/pull/127 to allow
108 custom cookie policies
109 - allow specifying :length for mulitpart inputstream bodies to
110 avoid chunked transfer encoding
111 - bumped cheshire to 5.1.1
112 - merged https://github.com/dakrone/clj-http/pull/133 to remove
113 some reflection
114
115 ** 0.7.1
116 :PROPERTIES:
117 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
118 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
119 :ARCHIVE_OLPATH: Changelog
120 :ARCHIVE_CATEGORY: changelog
121 :END:
122 - clarify :throw-exceptions in documentation
123 - define default-middleware for use in wrap-request, remove bad
124 all-middleware method
125 - merged https://github.com/dakrone/clj-http/pull/130 to encode
126 query-params
127 - merged https://github.com/dakrone/clj-http/pull/124 to handle
128 URL-encoding invalid characters in the URI
129 - bump cheshire to 5.1.0
130 - Switch from deprecated SingleClientConnManager to BasicClientConnectionManager
131 - merged https://github.com/dakrone/clj-http/pull/126 to bump
132 httpcore version
133 - bump dependencies to latest versions
134
135 ** 0.7.0
136 :PROPERTIES:
137 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
138 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
139 :ARCHIVE_OLPATH: Changelog
140 :ARCHIVE_CATEGORY: changelog
141 :END:
142 - merged https://github.com/dakrone/clj-http/pull/122 for
143 using *data-readers* when using tools.reader to parse EDN
144 - fix an issue with 1.3 where *data-readers* is not available
145 - merged https://github.com/dakrone/clj-http/pull/121 to fix
146 auto-coercion with json
147 - support application/edn as an auto-coercion type
148 - add tools.reader as an optional dependency, edn/read will be
149 used if available, otherwise read-string with *read-eval* bound
150 to false is used. See https://github.com/dakrone/clj-http/pull/120
151 - Bump clojure to 1.5.1
152
153 ** 0.6.5
154 :PROPERTIES:
155 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
156 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
157 :ARCHIVE_OLPATH: Changelog
158 :ARCHIVE_CATEGORY: changelog
159 :END:
160 - allow json coercion for exception cases based on :coerce setting,
161 can be either :always, :exceptional or :unexceptional
162 - Update clojure to 1.5
163 - Move SingleClientConnManager shutdown into finally block
164 - bind *read-eval* to false when reading for {:as :clojure}
165 - bump cheshire to 5.0.2
166
167 ** 0.6.4
168 :PROPERTIES:
169 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
170 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
171 :ARCHIVE_OLPATH: Changelog
172 :ARCHIVE_CATEGORY: changelog
173 :END:
174 - merged https://github.com/dakrone/clj-http/pull/113 to update
175 the connection pooling code
176 - refactor pooled connection managers to allow specifying
177 the :connection-manager option
178 - merged https://github.com/dakrone/clj-http/pull/112 to allow
179 json coercion on error responses when :as :auto is used
180 - allow redirects when :url is not set in the request
181 - merged https://github.com/dakrone/clj-http/pull/110 to handle the
182 case when the server-side uses deflate incorrectly
183 - added `with-middleware` to allow running requests with a custom
184 middleware list
185 - added `all-middleware` var listing all the wrap-* middleware that
186 clj-http knows of
187 - clj-http.client/request is now marked as dynamic for rebinding
188
189 ** 0.6.3
190 :PROPERTIES:
191 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
192 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
193 :ARCHIVE_OLPATH: Changelog
194 :ARCHIVE_CATEGORY: changelog
195 :END:
196 - Remove wrap-cookie-store middleware, CookieStore headers are
197 automatically added by Apache
198 - set the SINGLE_COOKIE_HEADER value to true to ensure Apache sends
199 only one "Cookie:" header
200 - Do not add CookieStore or Cookie header if there are no cookies
201 in the cookie jar
202
203 ** 0.6.2
204 :PROPERTIES:
205 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
206 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
207 :ARCHIVE_OLPATH: Changelog
208 :ARCHIVE_CATEGORY: changelog
209 :END:
210 - merged https://github.com/dakrone/clj-http/pull/106 to remove
211 query params for redirection.
212 - whitespace fixes; fix test that wasn't working correctly
213
214 ** 0.6.1
215 :PROPERTIES:
216 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
217 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
218 :ARCHIVE_OLPATH: Changelog
219 :ARCHIVE_CATEGORY: changelog
220 :END:
221 - bump httpcore to 4.2.3
222 - Fix an issue (#105) related to the "Content-Length" header being
223 automatically added to GET requests
224
225 ** 0.6.0
226 :PROPERTIES:
227 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
228 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
229 :ARCHIVE_OLPATH: Changelog
230 :ARCHIVE_CATEGORY: changelog
231 :END:
232 (bumped to 0.6.0 since Cheshire has changed major versions)
233 - Update Cheshire to 5.0.1
234 - Add type hint for getting headers from body (michaelklishin)
235
236 ** 0.5.8
237 :PROPERTIES:
238 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
239 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
240 :ARCHIVE_OLPATH: Changelog
241 :ARCHIVE_CATEGORY: changelog
242 :END:
243 - add buffering for HttpEntity, with ability to turn off if needed,
244 fixes lein issue with repeatable requests
245
246 ** 0.5.7
247 :PROPERTIES:
248 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
249 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
250 :ARCHIVE_OLPATH: Changelog
251 :ARCHIVE_CATEGORY: changelog
252 :END:
253 - create a custom X509HostnameVerifier for the :insecure? option
254 - explicitly require httpcore instead of leaving it to a transitive dep
255 - update httpcomponents to 4.2.2
256 - implement HTML5 charset header reading from body
257
258 ** 0.5.6
259 :PROPERTIES:
260 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
261 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
262 :ARCHIVE_OLPATH: Changelog
263 :ARCHIVE_CATEGORY: changelog
264 :END:
265 - bump Crouton to 0.1.1 for faster speeds
266 - add feature to decode body headers, merging them into response
267 headers if they are present
268 - merged https://github.com/dakrone/clj-http/pull/98 to add
269 optional :default-per-route to with-connection-pool
270
271 ** 0.5.5
272 :PROPERTIES:
273 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
274 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
275 :ARCHIVE_OLPATH: Changelog
276 :ARCHIVE_CATEGORY: changelog
277 :END:
278 - bump cheshire to fix json encoding bug
279
280 ** 0.5.4
281 :PROPERTIES:
282 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
283 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
284 :ARCHIVE_OLPATH: Changelog
285 :ARCHIVE_CATEGORY: changelog
286 :END:
287 - merged https://github.com/dakrone/clj-http/pull/95 to add support
288 for setting aribtrary client params to the http client
289 - Merged https://github.com/dakrone/clj-http/pull/94 to remove some
290 reflection
291 - update cheshire dep, make clojure a dev-dependency
292 - allow overriding the multipart part name with :part-name
293
294 ** 0.5.3
295 :PROPERTIES:
296 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
297 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
298 :ARCHIVE_OLPATH: Changelog
299 :ARCHIVE_CATEGORY: changelog
300 :END:
301 - merged https://github.com/dakrone/clj-http/pull/91 to add support
302 for :digest-auth
303 - added request timing middleware to add :request-time key for
304 request timing
305 - add wrap-cookie-store to send cookie-store cookies with a request
306 automatically
307 - merged https://github.com/dakrone/clj-http/pull/90 to standardize
308 on lower-case headers for HTTP requests
309
310 ** 0.5.2
311 :PROPERTIES:
312 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
313 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
314 :ARCHIVE_OLPATH: Changelog
315 :ARCHIVE_CATEGORY: changelog
316 :END:
317 - merged https://github.com/dakrone/clj-http/pull/88 to add chunked encoding
318 support (=:length= no longer required along with input stream =:body=)
319
320 ** 0.5.1
321 :PROPERTIES:
322 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
323 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
324 :ARCHIVE_OLPATH: Changelog
325 :ARCHIVE_CATEGORY: changelog
326 :END:
327 - fix clojure 1.3's exception wrapping for some exceptions
328 - merged https://github.com/dakrone/clj-http/pull/87 to allow using
329 http.nonProxyHosts
330 - mark json-encode and json-decode dynamic, so they could be
331 rebound if desired
332 - update httpclient and httpmime to 4.2.1
333 - update commons-codec to 1.6
334 - update common-io to 2.4
335 - change body decompression to be optional, if desired
336 - make the :content-type and :character-encoding options part of
337 middleware, not the core request
338 - document all the middleware
339 - merged https://github.com/dakrone/clj-http/pull/85 to allow
340 low-level callback for debugging
341
342 ** 0.5.0
343 :PROPERTIES:
344 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
345 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
346 :ARCHIVE_OLPATH: Changelog
347 :ARCHIVE_CATEGORY: changelog
348 :END:
349 - rewrite multipart body entity creation to use different map
350 format, allowing :mime-type and :encoding keys in some cases
351
352 ** 0.4.4
353 :PROPERTIES:
354 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
355 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
356 :ARCHIVE_OLPATH: Changelog
357 :ARCHIVE_CATEGORY: changelog
358 :END:
359 - bump cheshire to 4.0.1 and slingshot to 0.10.3
360 - fix an issue where cookies were encoded and should not be
361 - merged https://github.com/dakrone/clj-http/pull/80 to allow
362 specifying the keystore type
363 - merged https://github.com/dakrone/clj-http/pull/79 to allow
364 pluggable output coercion (multimethod)
365
366 ** 0.4.3
367 :PROPERTIES:
368 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
369 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
370 :ARCHIVE_OLPATH: Changelog
371 :ARCHIVE_CATEGORY: changelog
372 :END:
373 - support custom x509 keystore/trust-stores
374
375 ** 0.4.2
376 :PROPERTIES:
377 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
378 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
379 :ARCHIVE_OLPATH: Changelog
380 :ARCHIVE_CATEGORY: changelog
381 :END:
382 - fixed an issue where multiple link headers would cause an
383 exception to be thrown
384
385 ** 0.4.1
386 :PROPERTIES:
387 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
388 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
389 :ARCHIVE_OLPATH: Changelog
390 :ARCHIVE_CATEGORY: changelog
391 :END:
392 - added :debug-body that adds plaintext body information to
393 the :debug output
394 - fix json encoded form params with nested maps
395 - fix attempted json coercion when a bad status is received
396 - merged https://github.com/dakrone/clj-http/pull/69 to add support
397 for :oauth-token authentication
398 - merged https://github.com/dakrone/clj-http/pull/70 to save the
399 apache Http object when :save-request? is true
400 - merged https://github.com/dakrone/clj-http/pull/68 to support
401 additional options/delete/copy/move HTTP methods
402 - add support for the :patch method type
403
404 ** 0.4.0
405 :PROPERTIES:
406 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
407 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
408 :ARCHIVE_OLPATH: Changelog
409 :ARCHIVE_CATEGORY: changelog
410 :END:
411 - merged https://github.com/dakrone/clj-http/pull/66 to add support
412 for 'Link' header
413 - added ability to specify your own retry-handler for IOExceptions
414 if desired
415 - bumped httpclient and httpmime to 4.1.3
416 - bump to released version of clojure (1.4)
417 - added documentation about ipv6 requests
418 - fixed https://github.com/dakrone/clj-http/issues/57 by have
419 wrap-redirects redirect according to the RFC and adding
420 the :force-redirects option to be more browser-like
421 - merged https://github.com/dakrone/clj-http/pull/61 to add support
422 for nested param maps
423
424 ** 0.3.6
425 :PROPERTIES:
426 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
427 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
428 :ARCHIVE_OLPATH: Changelog
429 :ARCHIVE_CATEGORY: changelog
430 :END:
431 - fixed an issue where urls like http://user:pass@foo.com didn't
432 work correctly for basic-auth
433 - added support for cookie stores
434 - added utility methods to retrieve cookies as a map from the
435 cookie store
436 - set the default maximum number of redirects to 20
437
438 ** 0.3.5
439 :PROPERTIES:
440 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
441 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
442 :ARCHIVE_OLPATH: Changelog
443 :ARCHIVE_CATEGORY: changelog
444 :END:
445 - same as 0.3.4, but with a newer cheshire that doesn't interfere
446 with clj-json
447
448 ** 0.3.4
449 :PROPERTIES:
450 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
451 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
452 :ARCHIVE_OLPATH: Changelog
453 :ARCHIVE_CATEGORY: changelog
454 :END:
455 - improved commit from pull/55 to make the predicate more generalized to
456 any kind of entity request
457 - make Cheshire an optional dependency, only for {:as :json} and
458 json form-params
459 - merged https://github.com/dakrone/clj-http/pull/55 to fix HEAD
460 requests with body contents
461 - merged https://github.com/dakrone/clj-http/pull/53 to add status
462 functions into the clj-http.client namespace
463 - added the ability to specify {:as :clojure} to get back a clojure
464 datastructure, or {:as :auto} with content-type=application/clojure
465 - merged https://github.com/dakrone/clj-http/pull/52 to support
466 json-encoded form params
467 - added a test for json-encoded form params as request body
468
469 ** 0.3.3
470 :PROPERTIES:
471 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
472 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
473 :ARCHIVE_OLPATH: Changelog
474 :ARCHIVE_CATEGORY: changelog
475 :END:
476 - merged https://github.com/dakrone/clj-http/pull/51 to
477 allow :form-params on PUT requests
478 - bump Cheshire and slingshot deps
479 - add the :throw-entire-message? option to include resp in
480 Exception message
481 - throw an IllegalArgumentException instead of a regulor Exception
482 on nil urls
483 - add ability to redirect to relative paths (ngrunwald)
484
485 ** 0.3.2
486 :PROPERTIES:
487 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
488 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
489 :ARCHIVE_OLPATH: Changelog
490 :ARCHIVE_CATEGORY: changelog
491 :END:
492 - merged https://github.com/dakrone/clj-http/pull/48 to fix :stream
493 bodies (to make sure they are not coerced on output)
494 - merged https://github.com/dakrone/clj-http/pull/49 to check for
495 nil URLs when using client functions
496 - switch from assertions to exceptions for nil URLs
497 - merged https://github.com/dakrone/clj-http/pull/46 to
498 add :trace-redirects to the response map
499 - merged https://github.com/dakrone/clj-http/pull/47 to allow GET
500 requests with a :body set
501 - merged https://github.com/dakrone/clj-http/pull/44 to add ability
502 to specify maximum number of redirects
503 - add tests for max-redirects
504 - merged https://github.com/dakrone/clj-http/pull/42 to allow
505 strings or keywords for :scheme in requests
506 - added test for different :schemes
507
508 ** 0.3.1
509 :PROPERTIES:
510 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
511 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
512 :ARCHIVE_OLPATH: Changelog
513 :ARCHIVE_CATEGORY: changelog
514 :END:
515 - merged https://github.com/dakrone/clj-http/pull/40 to allow
516 per-request proxy settings
517 - remove a few more reflections
518 - added ablity to return the body as a stream with {:as :stream}
519 - general code cleanup
520
521 ** 0.3.0
522 :PROPERTIES:
523 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
524 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
525 :ARCHIVE_OLPATH: Changelog
526 :ARCHIVE_CATEGORY: changelog
527 :END:
528 - add ability to ignore unknown host if desired ({:ignore-unknown-host? true})
529 - use much better Enitity's for the body, depending on type
530 - bump all dependencies
531 - test re-org to make better sense (and allow C-c t in emacs)
532 - merged https://github.com/dakrone/clj-http/pull/36 to fix
533 url-encoding of multiple query params using the same key
534 - merged https://github.com/dakrone/clj-http/pull/34 to fix
535 decoding cookies that don't follow RFC spec
536 - Add better coercion, adding {:as :json}, {:as :json-string-keys}
537 and {:as :auto}
538
539 ** 0.2.7
540 :PROPERTIES:
541 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
542 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
543 :ARCHIVE_OLPATH: Changelog
544 :ARCHIVE_CATEGORY: changelog
545 :END:
546 - merged https://github.com/dakrone/clj-http/pull/31 to remove more
547 reflection warnings
548 - some whitespace changes
549 - merged https://github.com/dakrone/clj-http/pull/30 to remove more
550 reflection warnings
551 - removed swank from dev deps
552 - bump 1.4 to alpha3 in multi deps
553
554 ** 0.2.6
555 :PROPERTIES:
556 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
557 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
558 :ARCHIVE_OLPATH: Changelog
559 :ARCHIVE_CATEGORY: changelog
560 :END:
561 - don't use :server-port unless required (fixes problem with some
562 web servers)
563 - smaller error message on exceptions (thrown object is still the same)
564 - added the :save-request? option to return the request object in
565 a :request key in the response map
566 - multiple headers with the same name are now preserved when they
567 have differing cases
568
569 ** 0.2.5
570 :PROPERTIES:
571 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
572 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
573 :ARCHIVE_OLPATH: Changelog
574 :ARCHIVE_CATEGORY: changelog
575 :END:
576 - multipart form uploads
577 - bump slingshot to 0.9.0
578
579 ** 0.2.4
580 :PROPERTIES:
581 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
582 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
583 :ARCHIVE_OLPATH: Changelog
584 :ARCHIVE_CATEGORY: changelog
585 :END:
586 - Got a functioning reusable connection method,
587 (with-connection-pool ...)
588 - upgrade slingshot to 0.8.0
589 - upgrade commons-io to 2.1
590 - merged https://github.com/dakrone/clj-http/pull/20 to
591 allow :basic-auth as a string
592
593 ** 0.2.3
594 :PROPERTIES:
595 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
596 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
597 :ARCHIVE_OLPATH: Changelog
598 :ARCHIVE_CATEGORY: changelog
599 :END:
600 - added :insecure? flag
601 - fix AOT by requiring clojure.pprint
602 - wrap-redirects now handles recursive redirects
603
604 ** 0.2.2
605 :PROPERTIES:
606 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
607 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
608 :ARCHIVE_OLPATH: Changelog
609 :ARCHIVE_CATEGORY: changelog
610 :END:
611 - wrap-exceptions now uses Slingshot to throw a much more useful
612 exception when there was a problem with the request
613 - fixed an issue when malformed server responses could NPE the
614 decompression middleware
615 - added a :debug flag to pretty-print the request map and object
616 to stdout before performing the request to aid in debugging
617
618 ** 0.2.1
619 :PROPERTIES:
620 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
621 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
622 :ARCHIVE_OLPATH: Changelog
623 :ARCHIVE_CATEGORY: changelog
624 :END:
625 - decode cookies from response into :cookies (thanks r0man)
626 - redone redirects, they can now be toggled with {:follow-redirects
627 false} in the request
628 - decompression of responses has been fixed (thanks senior)
629 - accept Content-Encoding or content-encoding from responses
630 (thanks senior)
631 - added ability to specify sending a url-encoded :body of form
632 params using {:form-params {:key value}} (thanks senior)
633
634 ** 0.2.0
635 :PROPERTIES:
636 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
637 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
638 :ARCHIVE_OLPATH: Changelog
639 :ARCHIVE_CATEGORY: changelog
640 :END:
641 - updated dependencies to be the latest versions
642 - added ability to use system proxy for connections (thanks jou4)
643 - added ability to specify socket and connection timeouts in
644 request (thanks zkim)
645
646 ** 0.1.3
647 :PROPERTIES:
648 :ARCHIVE_TIME: 2014-03-05 Wed 16:32
649 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
650 :ARCHIVE_OLPATH: Changelog
651 :ARCHIVE_CATEGORY: changelog
652 :END:
653 - see: https://github.com/mmcgrana/clj-http
654
655 ** Released 0.7.8
656 :PROPERTIES:
657 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
658 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
659 :ARCHIVE_OLPATH: Work log
660 :ARCHIVE_CATEGORY: changelog
661 :END:
662
663 ** 2014-01-03
664 :PROPERTIES:
665 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
666 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
667 :ARCHIVE_OLPATH: Work log
668 :ARCHIVE_CATEGORY: changelog
669 :END:
670 - bump dependencies to their latest
671 - Merged https://github.com/dakrone/clj-http/pull/172 to update .gitignore file
672 and clean up whitespace for new clojure-mode
673 - Merged https://github.com/dakrone/clj-http/pull/171 to support SOCKS proxies
674
675 ** 2014-01-15
676 :PROPERTIES:
677 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
678 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
679 :ARCHIVE_OLPATH: Work log
680 :ARCHIVE_CATEGORY: changelog
681 :END:
682 - Merged https://github.com/dakrone/clj-http/pull/175 to add {:as :json-strict}
683 for output coercion
684 - Added {:as :json-strict-string-keys} output coercion
685
686 ** 2014-01-21
687 :PROPERTIES:
688 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
689 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
690 :ARCHIVE_OLPATH: Work log
691 :ARCHIVE_CATEGORY: changelog
692 :END:
693 - Merged https://github.com/dakrone/clj-http/pull/177 to update apache HTTP deps
694
695 ** 2014-01-27
696 :PROPERTIES:
697 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
698 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
699 :ARCHIVE_OLPATH: Work log
700 :ARCHIVE_CATEGORY: changelog
701 :END:
702 - Merged https://github.com/dakrone/clj-http/pull/178 to eliminate test
703 reflection
704
705 ** 2014-01-28
706 :PROPERTIES:
707 :ARCHIVE_TIME: 2015-04-22 Wed 23:13
708 :ARCHIVE_FILE: ~/src/clj/clj-http/changelog.org
709 :ARCHIVE_OLPATH: Work log
710 :ARCHIVE_CATEGORY: changelog
711 :END:
712 - Merged https://github.com/dakrone/clj-http/pull/181 to fix some tests
+80
-0
examples/progress_download.clj less more
0 (ns clj-http.examples.progress-download
1 (:require [clj-http.client :as http]
2 [clojure.java.io :refer [output-stream]])
3 (:import (org.apache.commons.io.input CountingInputStream)))
4
5 (defn print-progress-bar
6 "Render a simple progress bar given the progress and total. If the total is zero
7 the progress will run as indeterminated."
8 ([progress total] (print-progress-bar progress total {}))
9 ([progress total {:keys [bar-width]
10 :or {bar-width 50}}]
11 (if (pos? total)
12 (let [pct (/ progress total)
13 render-bar (fn []
14 (let [bars (Math/floor (* pct bar-width))
15 pad (- bar-width bars)]
16 (str (clojure.string/join (repeat bars "="))
17 (clojure.string/join (repeat pad " ")))))]
18 (print (str "[" (render-bar) "] "
19 (int (* pct 100)) "% "
20 progress "/" total)))
21 (let [render-bar (fn [] (clojure.string/join (repeat bar-width "-")))]
22 (print (str "[" (render-bar) "] "
23 progress "/?"))))))
24
25 (defn insert-at [v idx val]
26 "Addes value into a vector at an specific index."
27 (-> (subvec v 0 idx)
28 (conj val)
29 (concat (subvec v idx))))
30
31 (defn insert-after [v needle val]
32 "Finds an item into a vector and adds val just after it.
33 If needle is not found, the input vector will be returned."
34 (let [index (.indexOf v needle)]
35 (if (neg? index)
36 v
37 (insert-at v (inc index) val))))
38
39 (defn wrap-downloaded-bytes-counter
40 "Middleware that provides an CountingInputStream wrapping the stream output"
41 [client]
42 (fn [req]
43 (let [resp (client req)
44 counter (CountingInputStream. (:body resp))]
45 (merge resp {:body counter
46 :downloaded-bytes-counter counter}))))
47
48 (defn download-with-progress [url target]
49 (http/with-middleware
50 (-> http/default-middleware
51 (insert-after http/wrap-redirects wrap-downloaded-bytes-counter)
52 (conj http/wrap-lower-case-headers))
53 (let [request (http/get url {:as :stream})
54 length (Integer. (get-in request [:headers "content-length"] 0))
55 buffer-size (* 1024 10)]
56 (println)
57 (with-open [input (:body request)
58 output (output-stream target)]
59 (let [buffer (make-array Byte/TYPE buffer-size)
60 counter (:downloaded-bytes-counter request)]
61 (loop []
62 (let [size (.read input buffer)]
63 (when (pos? size)
64 (.write output buffer 0 size)
65 (print "\r")
66 (print-progress-bar (.getByteCount counter) length)
67 (recur))))))
68 (println))))
69
70 ;; Example of progress bar output (sample steps)
71 ;;
72 ;; [=== ] 7% 2094930/26572400
73 ;; [============================== ] 60% 16062930/26572400
74 ;; [========================================= ] 83% 22290930/26572400
75 ;; [==================================================] 100% 26572400/26572400
76 ;;
77 ;; In case the content-length is unknown, the bar will be displayed as:
78 ;;
79 ;; [--------------------------------------------------] 4211440/?
+35
-0
project.clj less more
0 (defproject clj-http "2.3.0"
1 :description "A Clojure HTTP library wrapping the Apache HttpComponents client."
2 :url "https://github.com/dakrone/clj-http/"
3 :license {:name "The MIT License"
4 :url "http://opensource.org/licenses/mit-license.php"
5 :distribution :repo}
6 :global-vars {*warn-on-reflection* false}
7 :min-lein-version "2.0.0"
8 :exclusions [org.clojure/clojure]
9 :dependencies [[org.apache.httpcomponents/httpcore "4.4.5"]
10 [org.apache.httpcomponents/httpclient "4.5.2"]
11 [org.apache.httpcomponents/httpmime "4.5.2"]
12 [commons-codec "1.10"]
13 [commons-io "2.5"]
14 [slingshot "0.12.2"]
15 [potemkin "0.4.3"]]
16 :profiles {:dev {:dependencies [;; optional deps
17 [cheshire "5.6.3"]
18 [crouton "0.1.2"]
19 [org.clojure/tools.reader "0.10.0"]
20 [com.cognitect/transit-clj "0.8.288"]
21 [ring/ring-codec "1.0.1"]
22 ;; other (testing) deps
23 [org.clojure/clojure "1.8.0"]
24 [org.clojure/tools.logging "0.3.1"]
25 [log4j "1.2.17"]
26 [ring/ring-jetty-adapter "1.5.0"]
27 [ring/ring-devel "1.5.0"]]}
28 :1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]}
29 :1.7 {:dependencies [[org.clojure/clojure "1.7.0"]]}}
30 :aliases {"all" ["with-profile" "dev,1.6:dev,1.7:dev"]}
31 :plugins [[codox "0.6.4"]]
32 :test-selectors {:default #(not (:integration %))
33 :integration :integration
34 :all (constantly true)})
+1027
-0
src/clj_http/client.clj less more
0 (ns clj-http.client
1 "Batteries-included HTTP client."
2 (:require [clj-http.conn-mgr :as conn]
3 [clj-http.cookies :refer [wrap-cookies]]
4 [clj-http.core :as core]
5 [clj-http.headers :refer [wrap-header-map]]
6 [clj-http.links :refer [wrap-links]]
7 [clj-http.util :refer [opt] :as util]
8 [clojure.stacktrace :refer [root-cause]]
9 [clojure.string :as str]
10 [clojure.walk :refer [keywordize-keys prewalk]]
11 [slingshot.slingshot :refer [throw+]])
12 (:import (java.io InputStream File ByteArrayOutputStream ByteArrayInputStream)
13 (java.net URL UnknownHostException)
14 (org.apache.http.entity BufferedHttpEntity ByteArrayEntity
15 InputStreamEntity FileEntity StringEntity)
16 (org.apache.http.impl.conn PoolingClientConnectionManager))
17 (:refer-clojure :exclude [get update]))
18
19 ;; Cheshire is an optional dependency, so we check for it at compile time.
20 (def json-enabled?
21 (try
22 (require 'cheshire.core)
23 true
24 (catch Throwable _ false)))
25
26 ;; Crouton is an optional dependency, so we check for it at compile time.
27 (def crouton-enabled?
28 (try
29 (require 'crouton.html)
30 true
31 (catch Throwable _ false)))
32
33 ;; tools.reader is an optional dependency, so check at compile time.
34 (def edn-enabled?
35 (try
36 (require 'clojure.tools.reader.edn)
37 true
38 (catch Throwable _ false)))
39
40 ;; Transit is an optional dependency, so check at compile time.
41 (def transit-enabled?
42 (try
43 (require 'cognitect.transit)
44 true
45 (catch Throwable _ false)))
46
47 ;; ring-codec is an optional dependency, so we check for it at compile time.
48 (def ring-codec-enabled?
49 (try
50 (require 'ring.util.codec)
51 true
52 (catch Throwable _ false)))
53
54 (defn ^:dynamic parse-edn
55 "Resolve and apply tool.reader's EDN parsing."
56 [& args]
57 {:pre [edn-enabled?]}
58 (apply (ns-resolve (symbol "clojure.tools.reader.edn")
59 (symbol "read-string"))
60 {:readers @(or (resolve '*data-readers*) (atom {}))} args))
61
62 (defn ^:dynamic parse-html
63 "Resolve and apply crouton's HTML parsing."
64 [& args]
65 {:pre [crouton-enabled?]}
66 (apply (ns-resolve (symbol "crouton.html") (symbol "parse")) args))
67
68 (defn- transit-opts-by-type
69 "Return the Transit options by type."
70 [opts type class-name]
71 {:pre [transit-enabled?]}
72 (cond
73 (empty? opts)
74 opts
75 (contains? opts type)
76 (clojure.core/get opts type)
77 :else
78 (let [class (Class/forName class-name)]
79 (println "Deprecated use of :transit-opts found.")
80 (update-in opts [:handlers]
81 (fn [handlers]
82 (->> handlers
83 (filter #(instance? class (second %)))
84 (into {})))))))
85
86 (defn- transit-read-opts
87 "Return the Transit read options."
88 [opts]
89 {:pre [transit-enabled?]}
90 (transit-opts-by-type opts :decode "com.cognitect.transit.ReadHandler"))
91
92 (defn- transit-write-opts
93 "Return the Transit write options."
94 [opts]
95 {:pre [transit-enabled?]}
96 (transit-opts-by-type opts :encode "com.cognitect.transit.WriteHandler"))
97
98 (defn ^:dynamic parse-transit
99 "Resolve and apply Transit's JSON/MessagePack decoding."
100 [in type & [opts]]
101 {:pre [transit-enabled?]}
102 (let [reader (ns-resolve 'cognitect.transit 'reader)
103 read (ns-resolve 'cognitect.transit 'read)]
104 (read (reader in type (transit-read-opts opts)))))
105
106 (defn ^:dynamic transit-encode
107 "Resolve and apply Transit's JSON/MessagePack encoding."
108 [out type & [opts]]
109 {:pre [transit-enabled?]}
110 (let [output (ByteArrayOutputStream.)
111 writer (ns-resolve 'cognitect.transit 'writer)
112 write (ns-resolve 'cognitect.transit 'write)]
113 (write (writer output type (transit-write-opts opts)) out)
114 (.toByteArray output)))
115
116 (defn ^:dynamic json-encode
117 "Resolve and apply cheshire's json encoding dynamically."
118 [& args]
119 {:pre [json-enabled?]}
120 (apply (ns-resolve (symbol "cheshire.core") (symbol "encode")) args))
121
122 (defn ^:dynamic json-decode
123 "Resolve and apply cheshire's json decoding dynamically."
124 [& args]
125 {:pre [json-enabled?]}
126 (apply (ns-resolve (symbol "cheshire.core") (symbol "decode")) args))
127
128 (defn ^:dynamic json-decode-strict
129 "Resolve and apply cheshire's json decoding dynamically (with lazy parsing
130 disabled)."
131 [& args]
132 {:pre [json-enabled?]}
133 (apply (ns-resolve (symbol "cheshire.core") (symbol "decode-strict")) args))
134
135 (defn ^:dynamic form-decode
136 "Resolve and apply ring-codec's form decoding dynamically."
137 [& args]
138 {:pre [ring-codec-enabled?]}
139 (apply (ns-resolve (symbol "ring.util.codec") (symbol "form-decode")) args))
140
141 (defn update [m k f & args]
142 (assoc m k (apply f (m k) args)))
143
144 (defn when-pos [v]
145 (when (and v (pos? v)) v))
146
147 (defn dissoc-in
148 "Dissociates an entry from a nested associative structure returning a new
149 nested structure. keys is a sequence of keys. Any empty maps that result
150 will not be present in the new structure."
151 [m [k & ks :as keys]]
152 (if ks
153 (if-let [nextmap (clojure.core/get m k)]
154 (let [newmap (dissoc-in nextmap ks)]
155 (if (seq newmap)
156 (assoc m k newmap)
157 (dissoc m k)))
158 m)
159 (dissoc m k)))
160
161 (defn url-encode-illegal-characters
162 "Takes a raw url path or query and url-encodes any illegal characters.
163 Minimizes ambiguity by encoding space to %20."
164 [path-or-query]
165 (when path-or-query
166 (-> path-or-query
167 (str/replace " " "%20")
168 (str/replace #"[^a-zA-Z0-9\.\-\_\~\!\$\&\'\(\)\*\+\,\;\=\:\@\/\%\?]"
169 util/url-encode))))
170
171 (defn parse-url
172 "Parse a URL string into a map of interesting parts."
173 [url]
174 (let [url-parsed (URL. url)]
175 {:scheme (keyword (.getProtocol url-parsed))
176 :server-name (.getHost url-parsed)
177 :server-port (when-pos (.getPort url-parsed))
178 :uri (url-encode-illegal-characters (.getPath url-parsed))
179 :user-info (if-let [user-info (.getUserInfo url-parsed)]
180 (util/url-decode user-info))
181 :query-string (url-encode-illegal-characters (.getQuery url-parsed))}))
182
183 ;; Statuses for which clj-http will not throw an exception
184 (def unexceptional-status?
185 #{200 201 202 203 204 205 206 207 300 301 302 303 304 307})
186
187 ;; helper methods to determine realm of a response
188 (defn success?
189 [{:keys [status]}]
190 (<= 200 status 299))
191
192 (defn missing?
193 [{:keys [status]}]
194 (= status 404))
195
196 (defn conflict?
197 [{:keys [status]}]
198 (= status 409))
199
200 (defn redirect?
201 [{:keys [status]}]
202 (<= 300 status 399))
203
204 (defn client-error?
205 [{:keys [status]}]
206 (<= 400 status 499))
207
208 (defn server-error?
209 [{:keys [status]}]
210 (<= 500 status 599))
211
212 (defn wrap-exceptions
213 "Middleware that throws a slingshot exception if the response is not a
214 regular response. If :throw-entire-message? is set to true, the entire
215 response is used as the message, instead of just the status number."
216 [client]
217 (fn [req]
218 (let [{:keys [status] :as resp} (client req)]
219 (if (unexceptional-status? status)
220 resp
221 (if (false? (opt req :throw-exceptions))
222 resp
223 (if (opt req :throw-entire-message)
224 (throw+ resp "clj-http: status %d %s" (:status %) resp)
225 (throw+ resp "clj-http: status %s" (:status %))))))))
226
227 (declare wrap-redirects)
228
229 (defn follow-redirect
230 "Attempts to follow the redirects from the \"location\" header, if no such
231 header exists (bad server!), returns the response without following the
232 request."
233 [client {:keys [uri url scheme server-name server-port] :as req}
234 {:keys [trace-redirects ^InputStream body] :as resp}]
235 (let [url (or url (str (name scheme) "://" server-name
236 (when server-port (str ":" server-port)) uri))]
237 (if-let [raw-redirect (get-in resp [:headers "location"])]
238 (let [redirect (str (URL. (URL. url) raw-redirect))]
239 (try (.close body) (catch Exception _))
240 ((wrap-redirects client) (-> req
241 (merge (parse-url redirect))
242 (dissoc :query-params)
243 (assoc :url redirect)
244 (assoc :trace-redirects trace-redirects))))
245 ;; Oh well, we tried, but if no location is set, return the response
246 resp)))
247
248 (defn wrap-redirects
249 "Middleware that follows redirects in the response. A slingshot exception is
250 thrown if too many redirects occur. Options
251
252 :follow-redirects - default:true, whether to follow redirects
253 :max-redirects - default:20, maximum number of redirects to follow
254 :force-redirects - default:false, force redirecting methods to GET requests
255
256 In the response:
257
258 :redirects-count - number of redirects
259 :trace-redirects - vector of sites the request was redirected from"
260 [client]
261 (fn [{:keys [request-method max-redirects redirects-count trace-redirects url]
262 :or {redirects-count 1 trace-redirects []
263 ;; max-redirects default taken from Firefox
264 max-redirects 20}
265 :as req}]
266 (let [{:keys [status] :as resp} (client req)
267 resp-r (assoc resp :trace-redirects
268 (if url
269 (conj trace-redirects url)
270 trace-redirects))]
271 (cond
272 (false? (opt req :follow-redirects))
273 resp
274 (not (redirect? resp-r))
275 resp-r
276 (and max-redirects (> redirects-count max-redirects))
277 (if (opt req :throw-exceptions)
278 (throw+ resp-r "Too many redirects: %s" redirects-count)
279 resp-r)
280 (= 303 status)
281 (follow-redirect client (assoc req :request-method :get
282 :redirects-count (inc redirects-count))
283 resp-r)
284 (#{301 302} status)
285 (cond
286 (#{:get :head} request-method)
287 (follow-redirect client (assoc req :redirects-count
288 (inc redirects-count)) resp-r)
289 (opt req :force-redirects)
290 (follow-redirect client (assoc req
291 :request-method :get
292 :redirects-count (inc redirects-count))
293 resp-r)
294 :else
295 resp-r)
296 (= 307 status)
297 (if (or (#{:get :head} request-method)
298 (opt req :force-redirects))
299 (follow-redirect client (assoc req :redirects-count
300 (inc redirects-count)) resp-r)
301 resp-r)
302 :else
303 resp-r))))
304
305 ;; Multimethods for Content-Encoding dispatch automatically
306 ;; decompressing response bodies
307 (defmulti decompress-body
308 (fn [resp] (get-in resp [:headers "content-encoding"])))
309
310 (defmethod decompress-body "gzip"
311 [resp]
312 (-> resp
313 (update :body util/gunzip)
314 (assoc :orig-content-encoding (get-in resp [:headers "content-encoding"]))
315 (dissoc-in [:headers "content-encoding"])))
316
317 (defmethod decompress-body "deflate"
318 [resp]
319 (-> resp
320 (update :body util/inflate)
321 (assoc :orig-content-encoding (get-in resp [:headers "content-encoding"]))
322 (dissoc-in [:headers "content-encoding"])))
323
324 (defmethod decompress-body :default [resp]
325 (assoc resp
326 :orig-content-encoding
327 (get-in resp [:headers "content-encoding"])))
328
329 (defn wrap-decompression
330 "Middleware handling automatic decompression of responses from web servers. If
331 :decompress-body is set to false, does not automatically set `Accept-Encoding`
332 header or decompress body."
333 [client]
334 (fn [req]
335 (if (false? (opt req :decompress-body))
336 (client req)
337 (let [req-c (update req :headers assoc "accept-encoding" "gzip, deflate")
338 resp-c (client req-c)]
339 (decompress-body resp-c)))))
340
341 ;; Multimethods for coercing body type to the :as key
342 (defmulti coerce-response-body (fn [req _] (:as req)))
343
344 (defmethod coerce-response-body :byte-array [_ resp]
345 (assoc resp :body (util/force-byte-array (:body resp))))
346
347 (defmethod coerce-response-body :stream [_ resp]
348 (let [body (:body resp)]
349 (cond (instance? InputStream body) resp
350 ;; This shouldn't happen, but we plan for it anyway
351 (instance? (Class/forName "[B") body)
352 (assoc resp :body (ByteArrayInputStream. body)))))
353
354 (defn coerce-json-body
355 [{:keys [coerce]} {:keys [body status] :as resp} keyword? strict? & [charset]]
356 (let [^String charset (or charset (-> resp :content-type-params :charset)
357 "UTF-8")
358 body (util/force-byte-array body)
359 decode-func (if strict? json-decode-strict json-decode)]
360 (if json-enabled?
361 (cond
362 (= coerce :always)
363 (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
364
365 (and (unexceptional-status? status)
366 (or (nil? coerce) (= coerce :unexceptional)))
367 (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
368
369 (and (not (unexceptional-status? status)) (= coerce :exceptional))
370 (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
371
372 :else (assoc resp :body (String. ^"[B" body charset)))
373 (assoc resp :body (String. ^"[B" body charset)))))
374
375 (defn coerce-clojure-body
376 [request {:keys [body] :as resp}]
377 (let [^String charset (or (-> resp :content-type-params :charset) "UTF-8")
378 body (util/force-byte-array body)]
379 (if edn-enabled?
380 (assoc resp :body (parse-edn (String. ^"[B" body charset)))
381 (binding [*read-eval* false]
382 (assoc resp :body (read-string (String. ^"[B" body charset)))))))
383
384 (defn coerce-transit-body
385 [{:keys [transit-opts] :as request} {:keys [body] :as resp} type]
386 (if transit-enabled?
387 (assoc resp :body (parse-transit body type transit-opts))
388 resp))
389
390 (defn coerce-form-urlencoded-body
391 [request {:keys [body] :as resp}]
392 (let [^String charset (or (-> resp :content-type-params :charset) "UTF-8")
393 body-bytes (util/force-byte-array body)]
394 (if ring-codec-enabled?
395 (assoc resp :body (-> (String. ^"[B" body-bytes charset)
396 form-decode keywordize-keys))
397 (assoc resp :body (String. ^"[B" body-bytes charset)))))
398
399 (defmulti coerce-content-type (fn [req resp] (:content-type resp)))
400
401 (defmethod coerce-content-type :application/clojure [req resp]
402 (coerce-clojure-body req resp))
403
404 (defmethod coerce-content-type :application/edn [req resp]
405 (coerce-clojure-body req resp))
406
407 (defmethod coerce-content-type :application/json [req resp]
408 (coerce-json-body req resp true false))
409
410 (defmethod coerce-content-type :application/transit+json [req resp]
411 (coerce-transit-body req resp :json))
412
413 (defmethod coerce-content-type :application/transit+msgpack [req resp]
414 (coerce-transit-body req resp :msgpack))
415
416 (defmethod coerce-content-type :application/x-www-form-urlencoded [req resp]
417 (coerce-form-urlencoded-body req resp))
418
419 (defmethod coerce-content-type :default [req resp]
420 (if-let [charset (-> resp :content-type-params :charset)]
421 (coerce-response-body {:as charset} resp)
422 (coerce-response-body {:as :default} resp)))
423
424 (defmethod coerce-response-body :auto [request resp]
425 (let [header (get-in resp [:headers "content-type"])]
426 (->> (merge resp (util/parse-content-type header))
427 (coerce-content-type request))))
428
429 (defmethod coerce-response-body :json [req resp]
430 (coerce-json-body req resp true false))
431
432 (defmethod coerce-response-body :json-strict [req resp]
433 (coerce-json-body req resp true true))
434
435 (defmethod coerce-response-body :json-strict-string-keys [req resp]
436 (coerce-json-body req resp false true))
437
438 (defmethod coerce-response-body :json-string-keys [req resp]
439 (coerce-json-body req resp false false))
440
441 (defmethod coerce-response-body :clojure [req resp]
442 (coerce-clojure-body req resp))
443
444 (defmethod coerce-response-body :transit+json [req resp]
445 (coerce-transit-body req resp :json))
446
447 (defmethod coerce-response-body :transit+msgpack [req resp]
448 (coerce-transit-body req resp :msgpack))
449
450 (defmethod coerce-response-body :x-www-form-urlencoded [req resp]
451 (coerce-form-urlencoded-body req resp))
452
453 (defmethod coerce-response-body :default
454 [{:keys [as]} {:keys [body] :as resp}]
455 (let [body-bytes (util/force-byte-array body)]
456 (cond
457 (string? as) (assoc resp :body (String. ^"[B" body-bytes ^String as))
458 :else (assoc resp :body (String. ^"[B" body-bytes "UTF-8")))))
459
460 (defn wrap-output-coercion
461 "Middleware converting a response body from a byte-array to a different
462 object. Defaults to a String if no :as key is specified, the
463 `coerce-response-body` multimethod may be extended to add
464 additional coercions."
465 [client]
466 (fn [req]
467 (let [{:keys [body] :as resp} (client req)]
468 (if body
469 (coerce-response-body req resp)
470 resp))))
471
472 (defn maybe-wrap-entity
473 "Wrap an HttpEntity in a BufferedHttpEntity if warranted."
474 [{:keys [entity-buffering]} entity]
475 (if (and entity-buffering (not= BufferedHttpEntity (class entity)))
476 (BufferedHttpEntity. entity)
477 entity))
478
479 (defn wrap-input-coercion
480 "Middleware coercing the :body of a request from a number of formats into an
481 Apache Entity. Currently supports Strings, Files, InputStreams
482 and byte-arrays."
483 [client]
484 (fn [{:keys [body body-encoding length]
485 :or {^String body-encoding "UTF-8"} :as req}]
486 (if body
487 (cond
488 (string? body)
489 (client (-> req (assoc :body (maybe-wrap-entity
490 req (StringEntity. ^String body
491 ^String body-encoding))
492 :character-encoding (or body-encoding
493 "UTF-8"))))
494 (instance? File body)
495 (client (-> req (assoc :body
496 (maybe-wrap-entity
497 req (FileEntity. ^File body
498 ^String body-encoding)))))
499
500 ;; A length of -1 instructs HttpClient to use chunked encoding.
501 (instance? InputStream body)
502 (client (-> req
503 (assoc :body
504 (if length
505 (InputStreamEntity.
506 ^InputStream body (long length))
507 (maybe-wrap-entity
508 req
509 (InputStreamEntity. ^InputStream body -1))))))
510
511 (instance? (Class/forName "[B") body)
512 (client (-> req (assoc :body (maybe-wrap-entity
513 req (ByteArrayEntity. body)))))
514
515 :else
516 (client req))
517 (client req))))
518
519 (defn get-headers-from-body
520 "Given a map of body content, return a map of header-name to header-value."
521 [body-map]
522 (let [;; parse out HTML content
523 h (or (:content body-map)
524 (:content (first (filter #(= (:tag %) :html) body-map))))
525 ;; parse out <head> tags
526 heads (:content (first (filter #(= (:tag %) :head) h)))
527 ;; parse out attributes of 'meta' head tags
528 attrs (map :attrs (filter #(= (:tag %) :meta) heads))
529 ;; parse out the 'http-equiv' meta head tags
530 http-attrs (filter :http-equiv attrs)
531 ;; parse out HTML5 charset meta tags
532 html5-charset (filter :charset attrs)
533 ;; convert http-attributes into map of headers (lowercased)
534 headers (apply merge (map (fn [{:keys [http-equiv content]}]
535 {(.toLowerCase ^String http-equiv) content})
536 http-attrs))
537 ;; merge in html5 charset setting
538 headers (merge headers
539 (when-let [cs (:charset (first html5-charset))]
540 {"content-type" (str "text/html; charset=" cs)}))]
541 headers))
542
543 (defn wrap-additional-header-parsing
544 "Middleware that parses additional http headers from the body of a web page,
545 adding them into the headers map of the response if any are found. Only looks
546 at the body if the :decode-body-headers option is set to a truthy value. Will
547 be silently disabled if crouton is excluded from clj-http's dependencies. Will
548 do nothing if no body is returned, e.g. HEAD requests"
549 [client]
550 (fn [req]
551 (let [resp (client req)]
552 (if (and (opt req :decode-body-headers)
553 crouton-enabled?
554 (:body resp)
555 (let [content-type (get-in resp [:headers "content-type"])]
556 (or (str/blank? content-type)
557 (.startsWith content-type "text"))))
558 (let [body-bytes (util/force-byte-array (:body resp))
559 body-stream1 (java.io.ByteArrayInputStream. body-bytes)
560 body-map (parse-html body-stream1)
561 additional-headers (get-headers-from-body body-map)
562 body-stream2 (java.io.ByteArrayInputStream. body-bytes)]
563 (assoc resp
564 :headers (merge (:headers resp) additional-headers)
565 :body body-stream2))
566 resp))))
567
568 (defn content-type-value [type]
569 (if (keyword? type)
570 (str "application/" (name type))
571 type))
572
573 (defn wrap-content-type
574 "Middleware converting a `:content-type <keyword>` option to the formal
575 application/<name> format and adding it as a header."
576 [client]
577 (fn [{:keys [content-type character-encoding] :as req}]
578 (if content-type
579 (let [ctv (content-type-value content-type)
580 ct (if character-encoding
581 (str ctv "; charset=" character-encoding)
582 ctv)]
583 (client (update-in req [:headers] assoc "content-type" ct)))
584 (client req))))
585
586 (defn wrap-accept
587 "Middleware converting the :accept key in a request to application/<type>"
588 [client]
589 (fn [{:keys [accept] :as req}]
590 (if accept
591 (client (-> req (dissoc :accept)
592 (assoc-in [:headers "accept"]
593 (content-type-value accept))))
594 (client req))))
595
596 (defn accept-encoding-value [accept-encoding]
597 (str/join ", " (map name accept-encoding)))
598
599 (defn wrap-accept-encoding
600 "Middleware converting the :accept-encoding option to an acceptable
601 Accept-Encoding header in the request."
602 [client]
603 (fn [{:keys [accept-encoding] :as req}]
604 (if accept-encoding
605 (client (-> req (dissoc :accept-encoding)
606 (assoc-in [:headers "accept-encoding"]
607 (accept-encoding-value accept-encoding))))
608 (client req))))
609
610 (defn detect-charset
611 "Given a charset header, detect the charset, returns UTF-8 if not found."
612 [content-type]
613 (or
614 (when-let [found (when content-type
615 (re-find #"(?i)charset\s*=\s*([^\s]+)" content-type))]
616 (second found))
617 "UTF-8"))
618
619 (defn- multi-param-suffix [index multi-param-style]
620 (case multi-param-style
621 :indexed (str "[" index "]")
622 :array "[]"
623 ""))
624
625 (defn generate-query-string-with-encoding [params encoding multi-param-style]
626 (str/join "&"
627 (mapcat (fn [[k v]]
628 (if (sequential? v)
629 (map-indexed #(str (util/url-encode (name k) encoding)
630 (multi-param-suffix %1 multi-param-style)
631 "="
632 (util/url-encode (str %2) encoding)) v)
633 [(str (util/url-encode (name k) encoding)
634 "="
635 (util/url-encode (str v) encoding))]))
636 params)))
637
638 (defn generate-query-string [params & [content-type multi-param-style]]
639 (let [encoding (detect-charset content-type)]
640 (generate-query-string-with-encoding params encoding multi-param-style)))
641
642 (defn wrap-query-params
643 "Middleware converting the :query-params option to a querystring on
644 the request."
645 [client]
646 (fn [{:keys [query-params content-type multi-param-style]
647 :or {content-type :x-www-form-urlencoded}
648 :as req}]
649 (if query-params
650 (client (-> req (dissoc :query-params)
651 (update-in [:query-string]
652 (fn [old-query-string new-query-string]
653 (if-not (empty? old-query-string)
654 (str old-query-string "&" new-query-string)
655 new-query-string))
656 (generate-query-string
657 query-params
658 (content-type-value content-type)
659 multi-param-style))))
660 (client req))))
661
662 (defn basic-auth-value [basic-auth]
663 (let [basic-auth (if (string? basic-auth)
664 basic-auth
665 (str (first basic-auth) ":" (second basic-auth)))]
666 (str "Basic " (util/base64-encode (util/utf8-bytes basic-auth)))))
667
668 (defn wrap-basic-auth
669 "Middleware converting the :basic-auth option into an Authorization header."
670 [client]
671 (fn [req]
672 (if-let [basic-auth (:basic-auth req)]
673 (client (-> req (dissoc :basic-auth)
674 (assoc-in [:headers "authorization"]
675 (basic-auth-value basic-auth))))
676 (client req))))
677
678 (defn wrap-oauth
679 "Middleware converting the :oauth-token option into an Authorization header."
680 [client]
681 (fn [req]
682 (if-let [oauth-token (:oauth-token req)]
683 (client (-> req (dissoc :oauth-token)
684 (assoc-in [:headers "authorization"]
685 (str "Bearer " oauth-token))))
686 (client req))))
687
688
689 (defn parse-user-info [user-info]
690 (when user-info
691 (str/split user-info #":")))
692
693 (defn wrap-user-info
694 "Middleware converting the :user-info option into a :basic-auth option"
695 [client]
696 (fn [req]
697 (if-let [[user password] (parse-user-info (:user-info req))]
698 (client (assoc req :basic-auth [user password]))
699 (client req))))
700
701 (defn wrap-method
702 "Middleware converting the :method option into the :request-method option"
703 [client]
704 (fn [req]
705 (if-let [m (:method req)]
706 (client (-> req (dissoc :method)
707 (assoc :request-method m)))
708 (client req))))
709
710 (defmulti coerce-form-params
711 (fn [req] (keyword (content-type-value (:content-type req)))))
712
713 (defmethod coerce-form-params :application/edn
714 [{:keys [form-params]}]
715 (pr-str form-params))
716
717 (defn- coerce-transit-form-params [type {:keys [form-params transit-opts]}]
718 (when-not transit-enabled?
719 (throw (ex-info (format (str "Can't encode form params as "
720 "\"application/transit+%s\". "
721 "Transit dependency not loaded.")
722 (name type))
723 {:type :transit-not-loaded
724 :form-params form-params
725 :transit-opts transit-opts
726 :transit-type type})))
727 (transit-encode form-params type transit-opts))
728
729 (defmethod coerce-form-params :application/transit+json [req]
730 (coerce-transit-form-params :json req))
731
732 (defmethod coerce-form-params :application/transit+msgpack [req]
733 (coerce-transit-form-params :msgpack req))
734
735 (defmethod coerce-form-params :application/json
736 [{:keys [form-params json-opts]}]
737 (when-not json-enabled?
738 (throw (ex-info (str "Can't encode form params as \"application/json\". "
739 "Cheshire dependency not loaded.")
740 {:type :cheshire-not-loaded
741 :form-params form-params
742 :json-opts json-opts})))
743 (json-encode form-params json-opts))
744
745 (defmethod coerce-form-params :default [{:keys [content-type
746 multi-param-style
747 form-params
748 form-param-encoding]}]
749 (if form-param-encoding
750 (generate-query-string-with-encoding form-params form-param-encoding multi-param-style)
751 (generate-query-string form-params (content-type-value content-type) multi-param-style)))
752
753 (defn wrap-form-params
754 "Middleware wrapping the submission or form parameters."
755 [client]
756 (fn [{:keys [form-params content-type request-method]
757 :or {content-type :x-www-form-urlencoded}
758 :as req}]
759 (if (and form-params (#{:post :put :patch} request-method))
760 (client (-> req
761 (dissoc :form-params)
762 (assoc :content-type (content-type-value content-type)
763 :body (coerce-form-params req))))
764 (client req))))
765
766 (defn- nest-params
767 [request param-key]
768 (if-let [params (request param-key)]
769 (assoc request param-key (prewalk
770 #(if (and (vector? %) (map? (second %)))
771 (let [[fk m] %]
772 (reduce
773 (fn [m [sk v]]
774 (assoc m (str (name fk)
775 \[ (name sk) \]) v))
776 {}
777 m))
778 %)
779 params))
780 request))
781
782 (defn wrap-nested-params
783 "Middleware wrapping nested parameters for query strings."
784 [client]
785 (fn [{:keys [content-type]
786 :as req}]
787 (if (or (nil? content-type)
788 (= content-type :x-www-form-urlencoded))
789 (client (reduce
790 nest-params
791 req
792 [:query-params :form-params]))
793 (client req))))
794
795 (defn wrap-url
796 "Middleware wrapping request URL parsing."
797 [client]
798 (fn [req]
799 (if-let [url (:url req)]
800 (client (-> req (dissoc :url) (merge (parse-url url))))
801 (client req))))
802
803 (defn wrap-unknown-host
804 "Middleware ignoring unknown hosts when the :ignore-unknown-host? option
805 is set."
806 [client]
807 (fn [req]
808 (try
809 (client req)
810 (catch Exception e
811 (if (= (type (root-cause e)) UnknownHostException)
812 (when-not (opt req :ignore-unknown-host)
813 (throw (root-cause e)))
814 (throw (root-cause e)))))))
815
816 (defn wrap-lower-case-headers
817 "Middleware lowercasing all headers, as per RFC (case-insensitive) and
818 Ring spec."
819 [client]
820 (let [lower-case-headers
821 #(if-let [headers (:headers %1)]
822 (assoc %1 :headers (util/lower-case-keys headers))
823 %1)]
824 (fn [req]
825 (-> (client (lower-case-headers req))
826 (lower-case-headers)))))
827
828 (defn wrap-request-timing
829 "Middleware that times the request, putting the total time (in milliseconds)
830 of the request into the :request-time key in the response."
831 [client]
832 (fn [req]
833 (let [start (System/currentTimeMillis)
834 resp (client req)]
835 (assoc resp :request-time (- (System/currentTimeMillis) start)))))
836
837 (def default-middleware
838 "The default list of middleware clj-http uses for wrapping requests."
839 [wrap-request-timing
840 wrap-header-map
841 wrap-query-params
842 wrap-basic-auth
843 wrap-oauth
844 wrap-user-info
845 wrap-url
846 wrap-redirects
847 wrap-decompression
848 wrap-input-coercion
849 ;; put this before output-coercion, so additional charset
850 ;; headers can be used if desired
851 wrap-additional-header-parsing
852 wrap-output-coercion
853 wrap-exceptions
854 wrap-accept
855 wrap-accept-encoding
856 wrap-content-type
857 wrap-form-params
858 wrap-nested-params
859 wrap-method
860 wrap-cookies
861 wrap-links
862 wrap-unknown-host])
863
864 (def ^:dynamic
865 *current-middleware*
866 "Available at any time to retrieve the middleware being used.
867 Automatically bound when `with-middleware` is used."
868 default-middleware)
869
870 (defn wrap-request
871 "Returns a batteries-included HTTP request function corresponding to the given
872 core client. See default-middleware for the middleware wrappers that are used
873 by default"
874 [request]
875 (reduce (fn wrap-request* [request middleware]
876 (middleware request))
877 request
878 default-middleware))
879
880 (def ^:dynamic request
881 "Executes the HTTP request corresponding to the given map and returns
882 the response map for corresponding to the resulting HTTP response.
883
884 In addition to the standard Ring request keys, the following keys are also
885 recognized:
886 * :url
887 * :method
888 * :query-params
889 * :basic-auth
890 * :content-type
891 * :accept
892 * :accept-encoding
893 * :as
894
895 The following additional behaviors over also automatically enabled:
896 * Exceptions are thrown for status codes other than 200-207, 300-303, or 307
897 * Gzip and deflate responses are accepted and decompressed
898 * Input and output bodies are coerced as required and indicated by the :as
899 option."
900 (wrap-request #'core/request))
901
902 ;; Inline function to throw a slightly more readable exception when
903 ;; the URL is nil
904 (definline check-url! [url]
905 `(when (nil? ~url)
906 (throw (IllegalArgumentException. "Host URL cannot be nil"))))
907
908 (defn get
909 "Like #'request, but sets the :method and :url as appropriate."
910 [url & [req]]
911 (check-url! url)
912 (request (merge req {:method :get :url url})))
913
914 (defn head
915 "Like #'request, but sets the :method and :url as appropriate."
916 [url & [req]]
917 (check-url! url)
918 (request (merge req {:method :head :url url})))
919
920 (defn post
921 "Like #'request, but sets the :method and :url as appropriate."
922 [url & [req]]
923 (check-url! url)
924 (request (merge req {:method :post :url url})))
925
926 (defn put
927 "Like #'request, but sets the :method and :url as appropriate."
928 [url & [req]]
929 (check-url! url)
930 (request (merge req {:method :put :url url})))
931
932 (defn delete
933 "Like #'request, but sets the :method and :url as appropriate."
934 [url & [req]]
935 (check-url! url)
936 (request (merge req {:method :delete :url url})))
937
938 (defn options
939 "Like #'request, but sets the :method and :url as appropriate."
940 [url & [req]]
941 (check-url! url)
942 (request (merge req {:method :options :url url})))
943
944 (defn copy
945 "Like #'request, but sets the :method and :url as appropriate."
946 [url & [req]]
947 (check-url! url)
948 (request (merge req {:method :copy :url url})))
949
950 (defn move
951 "Like #'request, but sets the :method and :url as appropriate."
952 [url & [req]]
953 (check-url! url)
954 (request (merge req {:method :move :url url})))
955
956 (defn patch
957 "Like #'request, but sets the :method and :url as appropriate."
958 [url & [req]]
959 (check-url! url)
960 (request (merge req {:method :patch :url url})))
961
962 (defmacro with-middleware
963 "Perform the body of the macro with a custom middleware list.
964
965 It is highly recommended to at least include:
966 clj-http.client/wrap-url
967 clj-http.client/wrap-method
968
969 Unless you really know what you are doing."
970 [middleware & body]
971 `(let [m# ~middleware]
972 (binding [*current-middleware* m#
973 clj-http.client/request (reduce #(%2 %1)
974 clj-http.core/request
975 m#)]
976 ~@body)))
977
978 (defmacro with-additional-middleware
979 "Perform the body of the macro with a list of additional middleware.
980
981 The given `middleware-seq' is concatenated to the beginning of the
982 `*current-middleware*' sequence."
983 [middleware-seq & body]
984 `(with-middleware (concat ~middleware-seq *current-middleware*)
985 ~@body))
986
987 (defmacro with-connection-pool
988 "Macro to execute the body using a connection manager. Creates a
989 PoolingClientConnectionManager to use for all requests within the body of
990 the expression. An option map is allowed to set options for the connection
991 manager.
992
993 The following options are supported:
994
995 :timeout - Time that connections are left open before automatically closing
996 default: 5
997 :threads - Maximum number of threads that will be used for connecting
998 default: 4
999 :default-per-route - Maximum number of simultaneous connections per host
1000 default: 2
1001 :insecure? - Boolean flag to specify allowing insecure HTTPS connections
1002 default: false
1003
1004 :keystore - keystore file path or KeyStore instance to be used for
1005 connection manager
1006 :keystore-pass - keystore password
1007 :trust-store - trust store file path or KeyStore instance to be used for
1008 connection manager
1009 :trust-store-pass - trust store password
1010
1011 Note that :insecure? and :keystore/:trust-store options are mutually exclusive
1012
1013 If the value 'nil' is specified or the value is not set, the default value
1014 will be used."
1015 [opts & body]
1016 ;; I'm leaving the connection bindable for now because in the
1017 ;; future I'm toying with the idea of managing the connection
1018 ;; manager yourself and passing it into the request
1019 `(let [cm# (conn/make-reusable-conn-manager ~opts)]
1020 (binding [conn/*connection-manager* cm#]
1021 (try
1022 ~@body
1023 (finally
1024 (.shutdown
1025 ^PoolingClientConnectionManager
1026 conn/*connection-manager*))))))
+199
-0
src/clj_http/conn_mgr.clj less more
0 (ns clj-http.conn-mgr
1 "Utility methods for Scheme registries and HTTP connection managers"
2 (:require [clj-http.util :refer [opt]]
3 [clojure.java.io :as io])
4 (:import (java.net Socket Proxy Proxy$Type InetSocketAddress)
5 (java.security KeyStore)
6 (java.security.cert X509Certificate)
7 (javax.net.ssl SSLSession SSLSocket)
8 (org.apache.http.conn ClientConnectionManager)
9 (org.apache.http.conn.params ConnPerRouteBean)
10 (org.apache.http.conn.ssl SSLSocketFactory TrustStrategy
11 X509HostnameVerifier SSLContexts)
12 (org.apache.http.conn.scheme PlainSocketFactory
13 SchemeRegistry Scheme)
14 (org.apache.http.impl.conn BasicClientConnectionManager
15 PoolingClientConnectionManager
16 SchemeRegistryFactory
17 SingleClientConnManager)))
18
19 (def ^SSLSocketFactory insecure-socket-factory
20 (SSLSocketFactory. (reify TrustStrategy
21 (isTrusted [_ _ _] true))
22 (reify X509HostnameVerifier
23 (^void verify [this ^String host ^SSLSocket sock]
24 ;; for some strange reason, only TLSv1 really
25 ;; works here, if you know why, tell me.
26 (.setEnabledProtocols
27 sock (into-array String ["TLSv1"]))
28 (.setWantClientAuth sock false)
29 (let [session (.getSession sock)]
30 (when-not session
31 (.startHandshake sock))
32 (aget (.getPeerCertificates session) 0)
33 ;; normally you'd want to verify the cert
34 ;; here, but since this is an insecure
35 ;; socketfactory, we don't
36 nil))
37 (^void verify [_ ^String _ ^X509Certificate _]
38 nil)
39 (^void verify [_ ^String _ ^"[Ljava.lang.String;" _
40 ^"[Ljava.lang.String;" _]
41 nil)
42 (^boolean verify [_ ^String _ ^SSLSession _]
43 true))))
44
45 (def ^SSLSocketFactory secure-ssl-socket-factory
46 (doto (SSLSocketFactory/getSocketFactory)
47 (.setHostnameVerifier SSLSocketFactory/STRICT_HOSTNAME_VERIFIER)))
48
49 ;; New Generic Socket Factories that can support socks proxy
50 (defn ^SSLSocketFactory SSLGenericSocketFactory
51 "Given a function that returns a new socket, create an SSLSocketFactory that
52 will use that socket."
53 [socket-factory]
54 (proxy [SSLSocketFactory] [(SSLContexts/createDefault)]
55 (connectSocket [socket remoteAddress localAddress params]
56 (let [^SSLSocketFactory this this] ;; avoid reflection
57 (proxy-super connectSocket (socket-factory)
58 remoteAddress localAddress params)))))
59
60 (defn ^PlainSocketFactory PlainGenericSocketFactory
61 "Given a Function that returns a new socket, create a PlainSocketFactory that
62 will use that socket."
63 [socket-factory]
64 (proxy [PlainSocketFactory] []
65 (createSocket [params]
66 (socket-factory))))
67
68 (defn socks-proxied-socket
69 "Create a Socket proxied through socks, using the given hostname and port"
70 [^String hostname ^Integer port]
71 (Socket. (Proxy. Proxy$Type/SOCKS (InetSocketAddress. hostname port))))
72
73 (defn make-socks-proxied-conn-manager
74 "Given an optional hostname and a port, create a connection manager that's
75 proxied using a SOCKS proxy."
76 [^String hostname ^Integer port]
77 (let [socket-factory #(socks-proxied-socket hostname port)
78 reg (doto (SchemeRegistry.)
79 (.register
80 (Scheme. "https" 443 (SSLGenericSocketFactory socket-factory)))
81 (.register
82 (Scheme. "http" 80 (PlainGenericSocketFactory socket-factory))))]
83 (PoolingClientConnectionManager. reg)))
84
85 (def insecure-scheme-registry
86 (doto (SchemeRegistry.)
87 (.register (Scheme. "http" 80 (PlainSocketFactory/getSocketFactory)))
88 (.register (Scheme. "https" 443 insecure-socket-factory))))
89
90 (def regular-scheme-registry
91 (doto (SchemeRegistry.)
92 (.register (Scheme. "http" 80 (PlainSocketFactory/getSocketFactory)))
93 (.register (Scheme. "https" 443 secure-ssl-socket-factory))))
94
95 (defn ^KeyStore get-keystore*
96 [keystore-file keystore-type ^String keystore-pass]
97 (when keystore-file
98 (let [keystore (KeyStore/getInstance (or keystore-type
99 (KeyStore/getDefaultType)))]
100 (with-open [is (io/input-stream keystore-file)]
101 (.load keystore is (when keystore-pass (.toCharArray keystore-pass)))
102 keystore))))
103
104 (defn ^KeyStore get-keystore [keystore & args]
105 (if (instance? KeyStore keystore)
106 keystore
107 (apply get-keystore* keystore args)))
108
109 (defn ^SchemeRegistry get-keystore-scheme-registry
110 [{:keys [keystore keystore-type keystore-pass keystore-instance
111 trust-store trust-store-type trust-store-pass]
112 :as req}]
113 (let [ks (get-keystore keystore keystore-type keystore-pass)
114 ts (get-keystore trust-store trust-store-type trust-store-pass)
115 factory (SSLSocketFactory. ks keystore-pass ts)]
116 (if (opt req :insecure)
117 (.setHostnameVerifier factory
118 SSLSocketFactory/ALLOW_ALL_HOSTNAME_VERIFIER))
119 (doto (SchemeRegistryFactory/createDefault)
120 (.register (Scheme. "https" 443 factory)))))
121
122 (defn ^BasicClientConnectionManager make-regular-conn-manager
123 [{:keys [keystore trust-store] :as req}]
124 (cond
125 (or keystore trust-store)
126 (BasicClientConnectionManager. (get-keystore-scheme-registry req))
127
128 (opt req :insecure) (BasicClientConnectionManager. insecure-scheme-registry)
129
130 :else (BasicClientConnectionManager. regular-scheme-registry)))
131
132 ;; need the fully qualified class name because this fn is later used in a
133 ;; macro from a different ns
134 (defn ^org.apache.http.impl.conn.PoolingClientConnectionManager
135 make-reusable-conn-manager*
136 "Given an timeout and optional insecure? flag, create a
137 PoolingClientConnectionManager with <timeout> seconds set as the
138 timeout value."
139 [{:keys [timeout keystore trust-store] :as config}]
140 (let [registry (cond
141 (opt config :insecure) insecure-scheme-registry
142
143 (or keystore trust-store)
144 (get-keystore-scheme-registry config)
145
146 :else regular-scheme-registry)]
147 (PoolingClientConnectionManager.
148 registry timeout java.util.concurrent.TimeUnit/SECONDS)))
149
150 (def dmcpr ConnPerRouteBean/DEFAULT_MAX_CONNECTIONS_PER_ROUTE)
151
152 (defn reusable? [^ClientConnectionManager conn-mgr]
153 (not (or (instance? SingleClientConnManager conn-mgr)
154 (instance? BasicClientConnectionManager conn-mgr))))
155
156 (defn ^PoolingClientConnectionManager make-reusable-conn-manager
157 "Creates a default pooling connection manager with the specified options.
158
159 The following options are supported:
160
161 :timeout - Time that connections are left open before automatically closing
162 default: 5
163 :threads - Maximum number of threads that will be used for connecting
164 default: 4
165 :default-per-route - Maximum number of simultaneous connections per host
166 default: 2
167 :insecure? - Boolean flag to specify allowing insecure HTTPS connections
168 default: false
169
170 :keystore - keystore file to be used for connection manager
171 :keystore-pass - keystore password
172 :trust-store - trust store file to be used for connection manager
173 :trust-store-pass - trust store password
174
175 Note that :insecure? and :keystore/:trust-store options are mutually exclusive
176
177 If the value 'nil' is specified or the value is not set, the default value
178 will be used."
179 [opts]
180 (let [timeout (or (:timeout opts) 5)
181 threads (or (:threads opts) 4)
182 default-per-route (or (:default-per-route opts) dmcpr)
183 insecure? (opt opts :insecure)
184 leftovers (dissoc opts :timeout :threads :insecure? :insecure)]
185 (doto (make-reusable-conn-manager* (merge {:timeout timeout
186 :insecure? insecure?}
187 leftovers))
188 (.setMaxTotal threads)
189 (.setDefaultMaxPerRoute default-per-route))))
190
191 (defn shutdown-manager
192 "Shut down the given connection manager, if it is not nil"
193 [^ClientConnectionManager manager]
194 (and manager (.shutdown manager)))
195
196 (def ^:dynamic *connection-manager*
197 "connection manager to be rebound during request execution"
198 nil)
+139
-0
src/clj_http/cookies.clj less more
0 (ns clj-http.cookies
1 "Namespace dealing with HTTP cookies"
2 (:require [clj-http.util :refer [opt]]
3 [clojure.string :refer [blank? join lower-case]])
4 (:import (org.apache.http.client.params ClientPNames CookiePolicy)
5 (org.apache.http.cookie ClientCookie CookieOrigin CookieSpec)
6 (org.apache.http.params BasicHttpParams)
7 (org.apache.http.impl.cookie BasicClientCookie2)
8 (org.apache.http.impl.cookie BrowserCompatSpecFactory)
9 (org.apache.http.message BasicHeader)
10 org.apache.http.client.CookieStore
11 (org.apache.http.impl.client BasicCookieStore)
12 (org.apache.http Header)
13 (org.apache.http.protocol BasicHttpContext)))
14
15 (defn cookie-spec ^CookieSpec []
16 (.create
17 (BrowserCompatSpecFactory.)
18 (BasicHttpContext.)))
19
20 (defn compact-map
21 "Removes all map entries where value is nil."
22 [m]
23 (reduce (fn [newm k]
24 (if (not (nil? (get m k)))
25 (assoc newm k (get m k))
26 newm))
27 (sorted-map) (sort (keys m))))
28
29 (defn to-cookie
30 "Converts a ClientCookie object into a tuple where the first item is
31 the name of the cookie and the second item the content of the
32 cookie."
33 [^ClientCookie cookie]
34 [(.getName cookie)
35 (compact-map
36 {:comment (.getComment cookie)
37 :comment-url (.getCommentURL cookie)
38 :discard (not (.isPersistent cookie))
39 :domain (.getDomain cookie)
40 :expires (when (.getExpiryDate cookie) (.getExpiryDate cookie))
41 :path (.getPath cookie)
42 :ports (when (.getPorts cookie) (seq (.getPorts cookie)))
43 :secure (.isSecure cookie)
44 :value (.getValue cookie)
45 :version (.getVersion cookie)})])
46
47 (defn ^BasicClientCookie2
48 to-basic-client-cookie
49 "Converts a cookie seq into a BasicClientCookie2."
50 [[cookie-name cookie-content]]
51 (doto (BasicClientCookie2. (name cookie-name)
52 (name (:value cookie-content)))
53 (.setComment (:comment cookie-content))
54 (.setCommentURL (:comment-url cookie-content))
55 (.setDiscard (:discard cookie-content true))
56 (.setDomain (:domain cookie-content))
57 (.setExpiryDate (:expires cookie-content))
58 (.setPath (:path cookie-content))
59 (.setPorts (int-array (:ports cookie-content)))
60 (.setSecure (:secure cookie-content false))
61 (.setVersion (:version cookie-content 0))))
62
63 (defn decode-cookie
64 "Decode the Set-Cookie string into a cookie seq."
65 [set-cookie-str]
66 (if-not (blank? set-cookie-str)
67 ;; I just want to parse a cookie without providing origin. How?
68 (let [domain (lower-case (str (gensym)))
69 origin (CookieOrigin. domain 80 "/" false)
70 [cookie-name cookie-content] (-> (cookie-spec)
71 (.parse (BasicHeader.
72 "set-cookie"
73 set-cookie-str)
74 origin)
75 first
76 to-cookie)]
77 [cookie-name
78 (if (= domain (:domain cookie-content))
79 (dissoc cookie-content :domain) cookie-content)])))
80
81 (defn decode-cookies
82 "Converts a cookie string or seq of strings into a cookie map."
83 [cookies]
84 (reduce #(assoc %1 (first %2) (second %2)) {}
85 (map decode-cookie (if (sequential? cookies) cookies [cookies]))))
86
87 (defn decode-cookie-header
88 "Decode the Set-Cookie header into the cookies key."
89 [response]
90 (if-let [cookies (get (:headers response) "set-cookie")]
91 (assoc response
92 :cookies (decode-cookies cookies)
93 :headers (dissoc (:headers response) "set-cookie"))
94 response))
95
96 (defn encode-cookie
97 "Encode the cookie into a string used by the Cookie header."
98 [cookie]
99 (when-let [header (-> (cookie-spec)
100 (.formatCookies [(to-basic-client-cookie cookie)])
101 first)]
102 (.getValue ^Header header)))
103
104 (defn encode-cookies
105 "Encode the cookie map into a string."
106 [cookie-map] (join ";" (map encode-cookie (seq cookie-map))))
107
108 (defn encode-cookie-header
109 "Encode the :cookies key of the request into a Cookie header."
110 [request]
111 (if (:cookies request)
112 (-> request
113 (assoc-in [:headers "Cookie"] (encode-cookies (:cookies request)))
114 (dissoc :cookies))
115 request))
116
117 (defn wrap-cookies
118 "Middleware wrapping cookie handling. Handles converting
119 the :cookies request parameter into the 'Cookies' header for an HTTP
120 request."
121 [client]
122 (fn [request]
123 (let [response (client (encode-cookie-header request))]
124 (if (= false (opt request :decode-cookies))
125 response
126 (decode-cookie-header response)))))
127
128 (defn cookie-store
129 "Returns a new, empty instance of the default implementation of the
130 org.apache.http.client.CookieStore interface."
131 []
132 (BasicCookieStore.))
133
134 (defn get-cookies
135 "Given a cookie-store, return a map of cookie name to a map of cookie values."
136 [^CookieStore cookie-store]
137 (when cookie-store
138 (into {} (map to-cookie (.getCookies cookie-store)))))
+335
-0
src/clj_http/core.clj less more
0 (ns clj-http.core
1 "Core HTTP request/response implementation."
2 (:require [clj-http.conn-mgr :as conn]
3 [clj-http.headers :as headers]
4 [clj-http.multipart :as mp]
5 [clj-http.util :refer [opt]]
6 [clojure.pprint])
7 (:import (java.io ByteArrayOutputStream FilterInputStream InputStream)
8
9 (org.apache.http HeaderIterator HttpEntity
10 HttpEntityEnclosingRequest
11 HttpResponse Header HttpHost
12 HttpRequestInterceptor HttpResponseInterceptor)
13 (org.apache.http.auth UsernamePasswordCredentials AuthScope
14 NTCredentials)
15 (org.apache.http.params CoreConnectionPNames)
16 (org.apache.http.client HttpClient HttpRequestRetryHandler)
17 (org.apache.http.client.methods HttpDelete
18 HttpEntityEnclosingRequestBase
19 HttpGet HttpHead HttpOptions
20 HttpPatch HttpPost HttpPut
21 HttpUriRequest)
22 (org.apache.http.client.params CookiePolicy ClientPNames)
23 (org.apache.http.conn ClientConnectionManager)
24 (org.apache.http.conn.routing HttpRoute)
25 (org.apache.http.conn.params ConnRoutePNames)
26 (org.apache.http.cookie CookieSpecFactory)
27 (org.apache.http.cookie.params CookieSpecPNames)
28 (org.apache.http.entity ByteArrayEntity StringEntity)
29
30 (org.apache.http.impl.client DefaultHttpClient)
31 (org.apache.http.impl.conn ProxySelectorRoutePlanner)
32 (org.apache.http.impl.cookie BrowserCompatSpec)
33 (java.net URI)))
34
35 (defn parse-headers
36 "Takes a HeaderIterator and returns a map of names to values.
37
38 If a name appears more than once (like `set-cookie`) then the value
39 will be a vector containing the values in the order they appeared
40 in the headers."
41 [^HeaderIterator headers & [use-header-maps-in-response?]]
42 (if-not use-header-maps-in-response?
43 (->> (headers/header-iterator-seq headers)
44 (map (fn [[k v]]
45 [(.toLowerCase ^String k) v]))
46 (reduce (fn [hs [k v]]
47 (headers/assoc-join hs k v))
48 {}))
49 (->> (headers/header-iterator-seq headers)
50 (reduce (fn [hs [k v]]
51 (headers/assoc-join hs k v))
52 (headers/header-map)))))
53
54 (defn set-client-param [^HttpClient client key val]
55 (when-not (nil? val)
56 (-> client
57 (.getParams)
58 (.setParameter key val))))
59
60 (defn make-proxy-method-with-body
61 [method]
62 (fn [^String url]
63 (doto (proxy [HttpEntityEnclosingRequestBase] []
64 (getMethod [] (.toUpperCase (name method))))
65 (.setURI (URI. url)))))
66
67 (def proxy-delete-with-body (make-proxy-method-with-body :delete))
68 (def proxy-get-with-body (make-proxy-method-with-body :get))
69 (def proxy-copy-with-body (make-proxy-method-with-body :copy))
70 (def proxy-move-with-body (make-proxy-method-with-body :move))
71 (def proxy-patch-with-body (make-proxy-method-with-body :patch))
72
73 (def ^:dynamic *cookie-store* nil)
74
75 (defn- set-routing
76 "Use ProxySelectorRoutePlanner to choose proxy sensible based on
77 http.nonProxyHosts"
78 [^DefaultHttpClient client]
79 (.setRoutePlanner client
80 (ProxySelectorRoutePlanner.
81 (.. client getConnectionManager getSchemeRegistry) nil))
82 client)
83
84 (defn maybe-force-proxy [^DefaultHttpClient client
85 ^HttpEntityEnclosingRequestBase request
86 proxy-host proxy-port proxy-ignore-hosts]
87 (let [uri (.getURI request)]
88 (when (and (nil? ((set proxy-ignore-hosts) (.getHost uri))) proxy-host)
89 (let [target (HttpHost. (.getHost uri) (.getPort uri) (.getScheme uri))
90 route (HttpRoute. target nil (HttpHost. ^String proxy-host
91 (int proxy-port))
92 (.. client getConnectionManager getSchemeRegistry
93 (getScheme target) isLayered))]
94 (set-client-param client ConnRoutePNames/FORCED_ROUTE route)))
95 request))
96
97 (defn cookie-spec
98 "Create an instance of a
99 org.apache.http.impl.cookie.BrowserCompatSpec with a validate
100 function that you pass in. This function takes two parameters, a
101 cookie and an origin."
102 [f]
103 (proxy [BrowserCompatSpec] []
104 (validate [cookie origin] (f cookie origin))))
105
106 (defn cookie-spec-factory
107 "Create an instance of a org.apache.http.cookie.CookieSpecFactory
108 with a newInstance implementation that returns a cookie
109 specification with a validate function that you pass in. The
110 function takes two parameters: cookie and origin."
111 [f]
112 (proxy
113 [CookieSpecFactory] []
114 (newInstance [params] (cookie-spec f))))
115
116 (defn add-client-params!
117 "Add various client params to the http-client object, if needed."
118 [^DefaultHttpClient http-client kvs]
119 (let [cookie-policy (:cookie-policy kvs)
120 cookie-policy-name (str (type cookie-policy))
121 kvs (dissoc kvs :cookie-policy)]
122 (when cookie-policy
123 (-> http-client
124 .getCookieSpecs
125 (.register cookie-policy-name (cookie-spec-factory cookie-policy))))
126 (doto http-client
127 (set-client-param ClientPNames/COOKIE_POLICY
128 (if cookie-policy
129 cookie-policy-name
130 CookiePolicy/BROWSER_COMPATIBILITY))
131 (set-client-param CookieSpecPNames/SINGLE_COOKIE_HEADER true)
132 (set-client-param ClientPNames/HANDLE_REDIRECTS false))
133
134 (doseq [[k v] kvs]
135 (set-client-param http-client
136 k (cond
137 (and (not= ClientPNames/CONN_MANAGER_TIMEOUT k)
138 (instance? Long v))
139 (Integer. ^Long v)
140 true v)))))
141
142 (defn- coerce-body-entity
143 "Coerce the http-entity from an HttpResponse to either a byte-array, or a
144 stream that closes itself and the connection manager when closed."
145 [{:keys [as]} ^HttpEntity http-entity ^ClientConnectionManager conn-mgr]
146 (if http-entity
147 (proxy [FilterInputStream]
148 [^InputStream (.getContent http-entity)]
149 (close []
150 (try
151 ;; Eliminate the reflection warning from proxy-super
152 (let [^InputStream this this]
153 (proxy-super close))
154 (finally
155 (when-not (conn/reusable? conn-mgr)
156 (conn/shutdown-manager conn-mgr))))))
157 (when-not (conn/reusable? conn-mgr)
158 (conn/shutdown-manager conn-mgr))))
159
160 (defn- print-debug!
161 "Print out debugging information to *out* for a given request."
162 [{:keys [debug-body body] :as req} http-req]
163 (println "Request:" (type body))
164 (clojure.pprint/pprint
165 (assoc req
166 :body (if (opt req :debug-body)
167 (cond
168 (isa? (type body) String)
169 body
170
171 (isa? (type body) HttpEntity)
172 (let [baos (ByteArrayOutputStream.)]
173 (.writeTo ^HttpEntity body baos)
174 (.toString baos "UTF-8"))
175
176 :else nil)
177 (if (isa? (type body) String)
178 (format "... %s bytes ..."
179 (count body))
180 (and body (bean body))))
181 :body-type (type body)))
182 (println "HttpRequest:")
183 (clojure.pprint/pprint (bean http-req)))
184
185 (defn http-request-for
186 "Provides the HttpRequest object for a particular request-method and url"
187 [request-method ^String http-url body]
188 (case request-method
189 :get (if body
190 (proxy-get-with-body http-url)
191 (HttpGet. http-url))
192 :head (HttpHead. http-url)
193 :put (HttpPut. http-url)
194 :post (HttpPost. http-url)
195 :options (HttpOptions. http-url)
196 :delete (if body
197 (proxy-delete-with-body http-url)
198 (HttpDelete. http-url))
199 :copy (proxy-copy-with-body http-url)
200 :move (proxy-move-with-body http-url)
201 :patch (if body
202 (proxy-patch-with-body http-url)
203 (HttpPatch. http-url))
204 (throw (IllegalArgumentException.
205 (str "Invalid request method " request-method)))))
206
207 (defn request
208 "Executes the HTTP request corresponding to the given Ring request map and
209 returns the Ring response map corresponding to the resulting HTTP response.
210
211 Note that where Ring uses InputStreams for the request and response bodies,
212 the clj-http uses ByteArrays for the bodies."
213 [{:keys [request-method scheme server-name server-port uri query-string
214 headers body multipart socket-timeout conn-timeout proxy-host
215 proxy-ignore-hosts proxy-port proxy-user proxy-pass as cookie-store
216 retry-handler request-interceptor response-interceptor
217 digest-auth ntlm-auth connection-manager client-params]
218 :as req}]
219 (let [^ClientConnectionManager conn-mgr
220 (or connection-manager
221 conn/*connection-manager*
222 (conn/make-regular-conn-manager req))
223 ^DefaultHttpClient http-client (set-routing
224 (DefaultHttpClient. conn-mgr))
225 scheme (name scheme)]
226 (when-let [cookie-store (or cookie-store *cookie-store*)]
227 (.setCookieStore http-client cookie-store))
228 (when retry-handler
229 (.setHttpRequestRetryHandler
230 http-client
231 (proxy [HttpRequestRetryHandler] []
232 (retryRequest [e cnt context]
233 (retry-handler e cnt context)))))
234 (add-client-params!
235 http-client
236 ;; merge in map of specified timeouts, to
237 ;; support backward compatiblity.
238 (merge {CoreConnectionPNames/SO_TIMEOUT socket-timeout
239 CoreConnectionPNames/CONNECTION_TIMEOUT conn-timeout}
240 client-params))
241
242 (when-let [[user pass] digest-auth]
243 (.setCredentials
244 (.getCredentialsProvider http-client)
245 (AuthScope. nil -1 nil)
246 (UsernamePasswordCredentials. user pass)))
247 (when-let [[user password host domain] ntlm-auth]
248 (.setCredentials
249 (.getCredentialsProvider http-client)
250 (AuthScope. nil -1 nil)
251 (NTCredentials. user password host domain)))
252
253 (when (and proxy-user proxy-pass)
254 (let [authscope (AuthScope. proxy-host proxy-port)
255 creds (UsernamePasswordCredentials. proxy-user proxy-pass)]
256 (.setCredentials (.getCredentialsProvider http-client)
257 authscope creds)))
258 (let [http-url (str scheme "://" server-name
259 (when server-port (str ":" server-port))
260 uri
261 (when query-string (str "?" query-string)))
262 req (assoc req :http-url http-url)
263 proxy-ignore-hosts (or proxy-ignore-hosts
264 #{"localhost" "127.0.0.1"})
265 ^HttpUriRequest http-req (maybe-force-proxy
266 http-client
267 (http-request-for request-method
268 http-url body)
269 proxy-host
270 proxy-port
271 proxy-ignore-hosts)]
272 (when request-interceptor
273 (.addRequestInterceptor
274 http-client
275 (proxy [HttpRequestInterceptor] []
276 (process [req ctx]
277 (request-interceptor req ctx)))))
278 (when response-interceptor
279 (.addResponseInterceptor
280 http-client
281 (proxy [HttpResponseInterceptor] []
282 (process [resp ctx]
283 (response-interceptor resp ctx)))))
284 (when-not (conn/reusable? conn-mgr)
285 (.addHeader http-req "Connection" "close"))
286 (doseq [[header-n header-v] headers]
287 (if (coll? header-v)
288 (doseq [header-vth header-v]
289 (.addHeader http-req header-n header-vth))
290 (.addHeader http-req header-n (str header-v))))
291 (if multipart
292 (.setEntity ^HttpEntityEnclosingRequest http-req
293 (mp/create-multipart-entity multipart))
294 (when (and body (instance? HttpEntityEnclosingRequest http-req))
295 (if (instance? HttpEntity body)
296 (.setEntity ^HttpEntityEnclosingRequest http-req body)
297 (.setEntity ^HttpEntityEnclosingRequest http-req
298 (if (string? body)
299 (StringEntity. ^String body "UTF-8")
300 (ByteArrayEntity. body))))))
301 (when (opt req :debug) (print-debug! req http-req))
302 (try
303 (let [http-resp (.execute http-client http-req)
304 http-entity (.getEntity http-resp)
305 resp {:status (.getStatusCode (.getStatusLine http-resp))
306 :headers (parse-headers
307 (.headerIterator http-resp)
308 (opt req :use-header-maps-in-response))
309 :body (coerce-body-entity req http-entity conn-mgr)}]
310 (if (opt req :save-request)
311 (-> resp
312 (assoc :request req)
313 (assoc-in [:request :body-type] (type body))
314 (update-in [:request]
315 #(if (opt req :debug-body)
316 (assoc % :body-content
317 (cond
318 (isa? (type (:body %)) String)
319 (:body %)
320
321 (isa? (type (:body %)) HttpEntity)
322 (let [baos (ByteArrayOutputStream.)]
323 (.writeTo ^HttpEntity (:body %) baos)
324 (.toString baos "UTF-8"))
325
326 :else nil))
327 %))
328 (assoc-in [:request :http-req] http-req)
329 (dissoc :save-request?))
330 resp))
331 (catch Throwable e
332 (when-not (conn/reusable? conn-mgr)
333 (conn/shutdown-manager conn-mgr))
334 (throw e))))))
+143
-0
src/clj_http/headers.clj less more
0 (ns clj-http.headers
1 "Provides wrap-header-map, which is middleware allows headers to be
2 specified more flexibly. In requests and responses, headers can be
3 accessed as strings or keywords of any case. In requests, string
4 header names will be sent to the server with their casing unchanged,
5 while keyword header names will be transformed into their canonical
6 HTTP representation (e.g. :accept-encoding will become
7 \"Accept-Encoding\")."
8 (:require [clojure.string :as s]
9 [potemkin :as potemkin])
10 (:import (java.util Locale)
11 (org.apache.http Header HeaderIterator)))
12
13 (def special-cases
14 "A collection of HTTP headers that do not follow the normal
15 Looks-Like-This casing."
16 ["Content-MD5"
17 "DNT"
18 "ETag"
19 "P3P"
20 "TE"
21 "WWW-Authenticate"
22 "X-ATT-DeviceId"
23 "X-UA-Compatible"
24 "X-WebKit-CSP"
25 "X-XSS-Protection"])
26
27 (defn special-case
28 "Returns the special-case capitalized version of a string if that
29 string is a special case, otherwise returns the string unchanged."
30 [^String s]
31 (or (first (filter #(.equalsIgnoreCase ^String % s) special-cases))
32 s))
33
34 (defn ^String lower-case
35 "Converts a string to all lower-case, using the root locale.
36
37 Warning: This is not a general purpose lower-casing function -- it
38 is useful for case-insensitive comparisons of strings, not for
39 converting a string into something that's useful for humans."
40 [^CharSequence s]
41 (when s
42 (.toLowerCase (.toString s) Locale/ROOT)))
43
44 (defn title-case
45 "Converts a character to titlecase."
46 [^Character c]
47 (when c
48 (Character/toTitleCase c)))
49
50 (defn canonicalize
51 "Transforms a keyword header name into its canonical string
52 representation.
53
54 The canonical string representation is title-cased words separated
55 by dashes, like so: :date -> \"Date\", :DATE -> \"Date\", and
56 :foo-bar -> \"Foo-Bar\".
57
58 However, there is special-casing for some common headers, so: :p3p
59 -> \"P3P\", and :content-md5 -> \"Content-MD5\"."
60 [k]
61 (when k
62 (-> (name k)
63 (lower-case)
64 (s/replace #"(?:^.|-.)"
65 (fn [s]
66 (if (next s)
67 (str (first s)
68 (title-case (second s)))
69 (str (title-case (first s))))))
70 (special-case))))
71
72 (defn normalize
73 "Turns a string or keyword into normalized form, which is a
74 lowercase string."
75 [k]
76 (when k
77 (lower-case (name k))))
78
79 (defn header-iterator-seq
80 "Takes a HeaderIterator and returns a seq of vectors of name/value
81 pairs of headers."
82 [^HeaderIterator headers]
83 (for [^Header h (iterator-seq headers)]
84 [(.getName h) (.getValue h)]))
85
86 (defn assoc-join
87 "Like assoc, but will join multiple values into a vector if the
88 given key is already present into the map."
89 [headers name value]
90 (update-in headers [name]
91 (fn [existing]
92 (cond (vector? existing)
93 (conj existing value)
94 (nil? existing)
95 value
96 :else
97 [existing value]))))
98
99 ;; a map implementation that stores both the original (or canonical)
100 ;; key and value for each key/value pair, but performs lookups and
101 ;; other operations using the normalized -- this allows a value to be
102 ;; looked up by many similar keys, and not just the exact precise key
103 ;; it was originally stored with.
104 (potemkin/def-map-type HeaderMap [m mta]
105 (get [_ k v]
106 (second (get m (normalize k) [nil v])))
107 (assoc [_ k v]
108 (HeaderMap. (assoc m (normalize k) [(if (keyword? k)
109 (canonicalize k)
110 k)
111 v])
112 mta))
113 (dissoc [_ k]
114 (HeaderMap. (dissoc m (normalize k)) mta))
115 (keys [_]
116 (map first (vals m)))
117 (meta [_]
118 mta)
119 (with-meta [_ mta]
120 (HeaderMap. m mta))
121 clojure.lang.Associative
122 (containsKey [_ k]
123 (contains? m (normalize k)))
124 (empty [_]
125 (HeaderMap. {} nil)))
126
127 (defn header-map
128 "Returns a new header map with supplied mappings."
129 [& keyvals]
130 (into (HeaderMap. {} nil)
131 (apply array-map keyvals)))
132
133 (defn wrap-header-map
134 "Middleware that converts headers from a map into a header-map."
135 [client]
136 (fn [req]
137 (let [req-headers (:headers req)
138 req (if req-headers
139 (-> req (assoc :headers (into (header-map) req-headers)
140 :use-header-maps-in-response? true))
141 req)]
142 (client req))))
+59
-0
src/clj_http/links.clj less more
0 (ns clj-http.links
1 "Namespace dealing with HTTP link headers")
2
3 (def ^:private quoted-string
4 #"\"((?:[^\"]|\\\")*)\"")
5
6 (def ^:private token
7 #"([^,\";]*)")
8
9 (def ^:private link-param
10 (re-pattern (str "(\\w+)=(?:" quoted-string "|" token ")")))
11
12 (def ^:private uri-reference
13 #"<([^>]*)>")
14
15 (def ^:private link-value
16 (re-pattern (str uri-reference "((?:\\s*;\\s*" link-param ")*)")))
17
18 (def ^:private link-header
19 (re-pattern (str "(?:\\s*(" link-value ")\\s*,?\\s*)")))
20
21 (defn read-link-params [params]
22 (into {}
23 (for [[_ name quot tok] (re-seq link-param params)]
24 [(keyword name) (or quot tok)])))
25
26 (defn read-link-value [value]
27 (let [[_ uri params] (re-matches link-value value)
28 param-map (read-link-params params)]
29 [(keyword (:rel param-map))
30 (-> param-map
31 (assoc :href uri)
32 (dissoc :rel))]))
33
34 (defn read-link-headers [header]
35 (->> (re-seq link-header header)
36 (map second)
37 (map read-link-value)
38 (into {})))
39
40 (defn wrap-links
41 "Add a :links key to the response map that contains parsed Link headers. The
42 links will be represented as a map, with the 'rel' value being the key. The
43 URI is placed under the 'href' key, to mimic the HTML link element.
44
45 e.g. Link: <http://example.com/page2.html>; rel=next; title=\"Page 2\"
46 => {:links {:next {:href \"http://example.com/page2.html\"
47 :title \"Page 2\"}}}"
48 [client]
49 (fn [request]
50 (let [response (client request)]
51 (if-let [link-headers (get-in response [:headers "link"])]
52 (let [link-headers (if (coll? link-headers)
53 link-headers
54 [link-headers])]
55 (assoc response
56 :links
57 (into {} (map read-link-headers link-headers))))
58 response))))
+127
-0
src/clj_http/multipart.clj less more
0 (ns clj-http.multipart
1 "Namespace used for clj-http to create multipart entities and bodies."
2 (:import (java.io File InputStream)
3 (org.apache.http.entity ContentType)
4 (org.apache.http.entity.mime MultipartEntity)
5 (org.apache.http.entity.mime.content ContentBody
6 ByteArrayBody
7 FileBody
8 InputStreamBody
9 StringBody)
10 (org.apache.http Consts)))
11
12 ;; we don't need to make a fake byte-array every time, only once
13 (def byte-array-type (type (byte-array 0)))
14
15 (defmulti
16 make-multipart-body
17 "Create a body object from the given map, dispatching on the type of its content.
18 By default supported content body types are:
19 - String
20 - byte array (requires providing name)
21 - InputStream (requires providing name)
22 - File
23 - org.apache.http.entity.mime.content.ContentBody (which is just returned)"
24 (fn [multipart] (type (:content multipart))))
25
26 (defmethod make-multipart-body nil
27 [multipart]
28 (throw (Exception. "Multipart content cannot be nil")))
29
30 (defmethod make-multipart-body :default
31 [multipart]
32 (throw (Exception. (str "Unsupported type for multipart content: " (type (:content multipart))))))
33
34 (defmethod make-multipart-body File
35 ; Create a FileBody object from the given map, requiring at least :content
36 [{:keys [^String name ^String mime-type ^File content ^String encoding]}]
37 (cond
38 (and name mime-type content encoding)
39 (FileBody. content (ContentType/create mime-type encoding) name)
40
41 (and mime-type content encoding)
42 (FileBody. content (ContentType/create mime-type encoding))
43
44 (and name mime-type content)
45 (FileBody. content (ContentType/create mime-type) name)
46
47 (and mime-type content)
48 (FileBody. content (ContentType/create mime-type))
49
50 content
51 (FileBody. content)
52
53 :else
54 (throw (Exception. "Multipart file body must contain at least :content"))))
55
56 (defmethod make-multipart-body InputStream
57 ; Create an InputStreamBody object from the given map, requiring at least
58 ; :content and :name. If no :length is specified, clj-http will use
59 ; chunked transfer-encoding, if :length is specified, clj-http will
60 ; workaround things be proxying the InputStreamBody to return a length.
61 [{:keys [^String name ^String mime-type ^InputStream content length]}]
62 (cond
63 (and content name length)
64 (if mime-type
65 (proxy [InputStreamBody] [content (ContentType/create mime-type) name]
66 (getContentLength []
67 length))
68 (proxy [InputStreamBody] [content name]
69 (getContentLength []
70 length)))
71
72 (and content mime-type name)
73 (InputStreamBody. content (ContentType/create mime-type) name)
74
75 (and content name)
76 (InputStreamBody. content name)
77
78 :else
79 (throw (Exception. (str "Multipart input stream body must contain "
80 "at least :content and :name")))))
81
82 (defmethod make-multipart-body byte-array-type
83 ; Create a ByteArrayBody object from the given map, requiring at least :content
84 ; and :name.
85 [{:keys [^String name ^String mime-type ^bytes content]}]
86 (cond
87 (and content name mime-type)
88 (ByteArrayBody. content (ContentType/create mime-type) name)
89
90 (and content name)
91 (ByteArrayBody. content name)
92
93 :else
94 (throw (Exception. (str "Multipart byte array body must contain "
95 "at least :content and :name")))))
96
97 (defmethod make-multipart-body String
98 ; Create a StringBody object from the given map, requiring at least :content.
99 ; If :encoding is specified, it will be created using the Charset for
100 ; that encoding.
101 [{:keys [mime-type ^String content encoding]}]
102 (cond
103 (and content mime-type encoding)
104 (StringBody. content (ContentType/create mime-type encoding))
105
106 (and content encoding)
107 (StringBody. content (ContentType/create "text/plain" encoding))
108
109 content
110 (StringBody. content (ContentType/create "text/plain" Consts/ASCII))))
111
112 (defmethod make-multipart-body ContentBody
113 ; Use provided org.apache.http.entity.mime.content.ContentBody directly
114 [{:keys [^ContentBody content]}]
115 content)
116
117 (defn create-multipart-entity
118 "Takes a multipart vector of maps and creates a MultipartEntity with each
119 map added as a part, depending on the type of content."
120 [multipart]
121 (let [mp-entity (MultipartEntity.)]
122 (doseq [m multipart]
123 (let [name (or (:part-name m) (:name m))
124 part (make-multipart-body m)]
125 (.addPart mp-entity name part)))
126 mp-entity))
+127
-0
src/clj_http/util.clj less more
0 (ns clj-http.util
1 "Helper functions for the HTTP client."
2 (:require [clojure.string :refer [blank? lower-case split trim]]
3 [clojure.walk :refer [postwalk]])
4 (:import (org.apache.commons.codec.binary Base64)
5 (org.apache.commons.io IOUtils)
6 (java.io BufferedInputStream ByteArrayInputStream
7 ByteArrayOutputStream)
8 (java.net URLEncoder URLDecoder)
9 (java.util.zip InflaterInputStream DeflaterInputStream
10 GZIPInputStream GZIPOutputStream)))
11
12 (defn utf8-bytes
13 "Returns the encoding's bytes corresponding to the given string. If no
14 encoding is specified, UTF-8 is used."
15 [^String s & [^String encoding]]
16 (.getBytes s (or encoding "UTF-8")))
17
18 (defn utf8-string
19 "Returns the String corresponding to the given encoding's decoding of the
20 given bytes. If no encoding is specified, UTF-8 is used."
21 [^"[B" b & [^String encoding]]
22 (String. b (or encoding "UTF-8")))
23
24 (defn url-decode
25 "Returns the form-url-decoded version of the given string, using either a
26 specified encoding or UTF-8 by default."
27 [encoded & [encoding]]
28 (URLDecoder/decode encoded (or encoding "UTF-8")))
29
30 (defn url-encode
31 "Returns an UTF-8 URL encoded version of the given string."
32 [unencoded & [encoding]]
33 (URLEncoder/encode unencoded (or encoding "UTF-8")))
34
35 (defn base64-encode
36 "Encode an array of bytes into a base64 encoded string."
37 [unencoded]
38 (utf8-string (Base64/encodeBase64 unencoded)))
39
40 (defn gunzip
41 "Returns a gunzip'd version of the given byte array."
42 [b]
43 (when b
44 (cond
45 (instance? java.io.InputStream b)
46 (GZIPInputStream. b)
47 :else
48 (IOUtils/toByteArray (GZIPInputStream. (ByteArrayInputStream. b))))))
49
50 (defn gzip
51 "Returns a gzip'd version of the given byte array."
52 [b]
53 (when b
54 (let [baos (ByteArrayOutputStream.)
55 gos (GZIPOutputStream. baos)]
56 (IOUtils/copy (ByteArrayInputStream. b) gos)
57 (.close gos)
58 (.toByteArray baos))))
59
60 (defn force-byte-array
61 "force b as byte array if it is an InputStream, also close the stream"
62 ^bytes [b]
63 (if (instance? java.io.InputStream b)
64 (try (IOUtils/toByteArray ^java.io.InputStream b)
65 (finally (.close ^java.io.InputStream b)))
66 b))
67
68 (defn inflate
69 "Returns a zlib inflate'd version of the given byte array or InputStream."
70 [b]
71 (when b
72 ;; This weirdness is because HTTP servers lie about what kind of deflation
73 ;; they're using, so we try one way, then if that doesn't work, reset and
74 ;; try the other way
75 (let [stream (BufferedInputStream. (if (instance? java.io.InputStream b)
76 b
77 (ByteArrayInputStream. b)))
78 _ (.mark stream 512)
79 iis (InflaterInputStream. stream)
80 readable? (try (.read iis) true
81 (catch java.util.zip.ZipException _ false))]
82 (.reset stream)
83 (if readable?
84 (InflaterInputStream. stream)
85 (InflaterInputStream. stream (java.util.zip.Inflater. true))))))
86
87 (defn deflate
88 "Returns a deflate'd version of the given byte array."
89 [b]
90 (when b
91 (IOUtils/toByteArray (DeflaterInputStream. (ByteArrayInputStream. b)))))
92
93 (defn lower-case-keys
94 "Recursively lower-case all map keys that are strings."
95 [m]
96 (let [f (fn [[k v]] (if (string? k) [(lower-case k) v] [k v]))]
97 (postwalk (fn [x] (if (map? x) (into {} (map f x)) x)) m)))
98
99 (defn opt
100 "Check the request parameters for a keyword boolean option, with or without
101 the ?
102
103 Returns false if either of the values are false, or the value of
104 (or key1 key2) otherwise (truthy)"
105 [req param]
106 (let [param-? (keyword (str (name param) "?"))
107 v1 (clojure.core/get req param)
108 v2 (clojure.core/get req param-?)]
109 (if (false? v1)
110 false
111 (if (false? v2)
112 false
113 (or v1 v2)))))
114
115 (defn parse-content-type
116 "Parse `s` as an RFC 2616 media type."
117 [s]
118 (if-let [m (re-matches #"\s*(([^/]+)/([^ ;]+))\s*(\s*;.*)?" (str s))]
119 {:content-type (keyword (nth m 1))
120 :content-type-params
121 (->> (split (str (nth m 4)) #"\s*;\s*")
122 (identity)
123 (remove blank?)
124 (map #(split % #"="))
125 (mapcat (fn [[k v]] [(keyword (lower-case k)) (trim v)]))
126 (apply hash-map))}))
+856
-0
test/clj_http/test/client.clj less more
0 (ns clj-http.test.client
1 (:require [cheshire.core :as json]
2 [clj-http.client :as client]
3 [clj-http.conn-mgr :as conn]
4 [clj-http.test.core :refer [run-server]]
5 [clj-http.util :as util]
6 [clojure.string :as str]
7 [clojure.java.io :refer [resource]]
8 [clojure.test :refer :all]
9 [cognitect.transit :as transit]
10 [ring.util.codec :refer [form-decode-str]]
11 [ring.middleware.nested-params :refer [parse-nested-keys]])
12 (:import (java.net UnknownHostException)
13 (java.io ByteArrayInputStream)
14 (org.apache.http HttpEntity)))
15
16 (def base-req
17 {:scheme :http
18 :server-name "localhost"
19 :server-port 18080})
20
21 (defn request [req]
22 (client/request (merge base-req req)))
23
24 (defn parse-form-params [s]
25 (->> (str/split (form-decode-str s) #"&")
26 (map #(str/split % #"="))
27 (map #(vector
28 (map keyword (parse-nested-keys (first %)))
29 (second %)))
30 (reduce (fn [m [ks v]]
31 (assoc-in m ks v)) {})))
32
33 (deftest ^:integration roundtrip
34 (run-server)
35 ;; roundtrip with scheme as a keyword
36 (let [resp (request {:uri "/get" :method :get})]
37 (is (= 200 (:status resp)))
38 (is (= "close" (get-in resp [:headers "connection"])))
39 (is (= "get" (:body resp))))
40 ;; roundtrip with scheme as a string
41 (let [resp (request {:uri "/get" :method :get
42 :scheme "http"})]
43 (is (= 200 (:status resp)))
44 (is (= "close" (get-in resp [:headers "connection"])))
45 (is (= "get" (:body resp))))
46 (let [params {:a "1" :b {:c "2"}}]
47 (doseq [[content-type read-fn]
48 [[nil (comp parse-form-params slurp)]
49 [:x-www-form-urlencoded (comp parse-form-params slurp)]
50 [:edn (comp read-string slurp)]
51 [:transit+json #(client/parse-transit % :json)]
52 [:transit+msgpack #(client/parse-transit % :msgpack)]]]
53 (let [resp (request {:uri "/post"
54 :as :stream
55 :method :post
56 :content-type content-type
57 :form-params params})]
58 (is (= 200 (:status resp)))
59 (is (= "close" (get-in resp [:headers "connection"])))
60 (is (= params (read-fn (:body resp))))))))
61
62 (deftest ^:integration nil-input
63 (is (thrown-with-msg? Exception #"Host URL cannot be nil"
64 (client/get nil)))
65 (is (thrown-with-msg? Exception #"Host URL cannot be nil"
66 (client/post nil)))
67 (is (thrown-with-msg? Exception #"Host URL cannot be nil"
68 (client/put nil)))
69 (is (thrown-with-msg? Exception #"Host URL cannot be nil"
70 (client/delete nil))))
71
72 (defn is-passed [middleware req]
73 (let [client (middleware identity)]
74 (is (= req (client req)))))
75
76 (defn is-applied [middleware req-in req-out]
77 (let [client (middleware identity)]
78 (is (= req-out (client req-in)))))
79
80 (deftest redirect-on-get
81 (let [client (fn [req]
82 (if (= "foo.com" (:server-name req))
83 {:status 302
84 :headers {"location" "http://bar.com/bat"}}
85 {:status 200
86 :req req}))
87 r-client (-> client client/wrap-url client/wrap-redirects)
88 resp (r-client {:server-name "foo.com" :url "http://foo.com"
89 :request-method :get})]
90 (is (= 200 (:status resp)))
91 (is (= :get (:request-method (:req resp))))
92 (is (= :http (:scheme (:req resp))))
93 (is (= ["http://foo.com" "http://bar.com/bat"] (:trace-redirects resp)))
94 (is (= "/bat" (:uri (:req resp))))))
95
96 (deftest relative-redirect-on-get
97 (let [client (fn [req]
98 (if (:redirects-count req)
99 {:status 200
100 :req req}
101 {:status 302
102 :headers {"location" "/bat"}}))
103 r-client (-> client client/wrap-url client/wrap-redirects)
104 resp (r-client {:server-name "foo.com" :url "http://foo.com"
105 :request-method :get})]
106 (is (= 200 (:status resp)))
107 (is (= :get (:request-method (:req resp))))
108 (is (= :http (:scheme (:req resp))))
109 (is (= ["http://foo.com" "http://foo.com/bat"] (:trace-redirects resp)))
110 (is (= "/bat" (:uri (:req resp))))))
111
112 (deftest trace-redirects-using-uri
113 (let [client (fn [req] {:status 200 :req req})
114 r-client (-> client client/wrap-redirects)
115 resp (r-client {:scheme :http :server-name "foo.com" :uri "/"
116 :request-method :get})]
117 (is (= 200 (:status resp)))
118 (is (= :get (:request-method (:req resp))))
119 (is (= :http (:scheme (:req resp))))
120 (is (= [] (:trace-redirects resp)))))
121
122 (deftest redirect-without-location-header
123 (let [client (fn [req]
124 {:status 302 :body "no redirection here"})
125 r-client (-> client client/wrap-url client/wrap-redirects)
126 resp (r-client {:server-name "foo.com" :url "http://foo.com"
127 :request-method :get})]
128 (is (= 302 (:status resp)))
129 (is (= ["http://foo.com"] (:trace-redirects resp)))
130 (is (= "no redirection here" (:body resp)))))
131
132 (deftest redirect-with-query-string
133 (let [client (fn [req]
134 (if (= "foo.com" (:server-name req))
135 {:status 302
136 :headers {"location" "http://bar.com/bat?x=y"}}
137 {:status 200
138 :req req}))
139 r-client (-> client client/wrap-url client/wrap-redirects)
140 resp (r-client {:server-name "foo.com" :url "http://foo.com"
141 :request-method :get :query-params {:x "z"}})]
142 (is (= 200 (:status resp)))
143 (is (= :get (:request-method (:req resp))))
144 (is (= :http (:scheme (:req resp))))
145 (is (= ["http://foo.com" "http://bar.com/bat?x=y"] (:trace-redirects resp)))
146 (is (= "/bat" (:uri (:req resp))))
147 (is (= "x=y" (:query-string (:req resp))))
148 (is (nil? (:query-params (:req resp))))))
149
150 (deftest max-redirects
151 (let [client (fn [req]
152 (if (= "foo.com" (:server-name req))
153 {:status 302
154 :headers {"location" "http://bar.com/bat"}}
155 {:status 200
156 :req req}))
157 r-client (-> client client/wrap-url client/wrap-redirects)
158 resp (r-client {:server-name "foo.com" :url "http://foo.com"
159 :request-method :get :max-redirects 0})]
160 (is (= 302 (:status resp)))
161 (is (= ["http://foo.com"] (:trace-redirects resp)))
162 (is (= "http://bar.com/bat" (get (:headers resp) "location")))))
163
164 (deftest redirect-303-to-get-on-any-method
165 (doseq [method [:get :head :post :delete :put :option]]
166 (let [client (fn [req]
167 (if (= "foo.com" (:server-name req))
168 {:status 303
169 :headers {"location" "http://bar.com/bat"}}
170 {:status 200
171 :req req}))
172 r-client (-> client client/wrap-url client/wrap-redirects)
173 resp (r-client {:server-name "foo.com" :url "http://foo.com"
174 :request-method method})]
175 (is (= 200 (:status resp)))
176 (is (= :get (:request-method (:req resp))))
177 (is (= :http (:scheme (:req resp))))
178 (is (= ["http://foo.com" "http://bar.com/bat"] (:trace-redirects resp)))
179 (is (= "/bat" (:uri (:req resp)))))))
180
181 (deftest pass-on-non-redirect
182 (let [client (fn [req] {:status 200 :body (:body req)})
183 r-client (client/wrap-redirects client)
184 resp (r-client {:body "ok" :url "http://foo.com"})]
185 (is (= 200 (:status resp)))
186 (is (= ["http://foo.com"] (:trace-redirects resp)))
187 (is (= "ok" (:body resp)))))
188
189 (deftest pass-on-non-redirectable-methods
190 (doseq [method [:put :post :delete]
191 status [301 302 307]]
192 (let [client (fn [req] {:status status :body (:body req)
193 :headers {"location" "http://foo.com/bat"}})
194 r-client (client/wrap-redirects client)
195 resp (r-client {:body "ok" :url "http://foo.com"
196 :request-method method})]
197 (is (= status (:status resp)))
198 (is (= ["http://foo.com"] (:trace-redirects resp)))
199 (is (= {"location" "http://foo.com/bat"} (:headers resp)))
200 (is (= "ok" (:body resp))))))
201
202 (deftest force-redirects-on-non-redirectable-methods
203 (doseq [method [:put :post :delete]
204 [status expected-method] [[301 :get] [302 :get] [307 method]]]
205 (let [client (fn [{:keys [trace-redirects body] :as req}]
206 (if trace-redirects
207 {:status 200 :body body :trace-redirects trace-redirects
208 :req req}
209 {:status status :body body :req req
210 :headers {"location" "http://foo.com/bat"}}))
211 r-client (client/wrap-redirects client)
212 resp (r-client {:body "ok" :url "http://foo.com"
213 :request-method method
214 :force-redirects true})]
215 (is (= 200 (:status resp)))
216 (is (= ["http://foo.com" "http://foo.com/bat"] (:trace-redirects resp)))
217 (is (= "ok" (:body resp)))
218 (is (= expected-method (:request-method (:req resp)))))))
219
220 (deftest pass-on-follow-redirects-false
221 (let [client (fn [req] {:status 302 :body (:body req)})
222 r-client (client/wrap-redirects client)
223 resp (r-client {:body "ok" :follow-redirects false})]
224 (is (= 302 (:status resp)))
225 (is (= "ok" (:body resp)))
226 (is (nil? (:trace-redirects resp)))))
227
228 (deftest throw-on-exceptional
229 (let [client (fn [req] {:status 500})
230 e-client (client/wrap-exceptions client)]
231 (is (thrown-with-msg? Exception #"500"
232 (e-client {}))))
233 (let [client (fn [req] {:status 500 :body "foo"})
234 e-client (client/wrap-exceptions client)]
235 (is (thrown-with-msg? Exception #":body"
236 (e-client {:throw-entire-message? true})))))
237
238 (deftest pass-on-non-exceptional
239 (let [client (fn [req] {:status 200})
240 e-client (client/wrap-exceptions client)
241 resp (e-client {})]
242 (is (= 200 (:status resp)))))
243
244 (deftest pass-on-exceptional-when-surpressed
245 (let [client (fn [req] {:status 500})
246 e-client (client/wrap-exceptions client)
247 resp (e-client {:throw-exceptions false})]
248 (is (= 500 (:status resp)))))
249
250 (deftest apply-on-compressed
251 (let [client (fn [req]
252 (is (= "gzip, deflate"
253 (get-in req [:headers "accept-encoding"])))
254 {:body (util/gzip (util/utf8-bytes "foofoofoo"))
255 :headers {"content-encoding" "gzip"}})
256 c-client (client/wrap-decompression client)
257 resp (c-client {})]
258 (is (= "foofoofoo" (util/utf8-string (:body resp))))
259 (is (= "gzip" (:orig-content-encoding resp)))
260 (is (= nil (get-in resp [:headers "content-encoding"])))))
261
262 (deftest apply-on-deflated
263 (let [client (fn [req]
264 (is (= "gzip, deflate"
265 (get-in req [:headers "accept-encoding"])))
266 {:body (util/deflate (util/utf8-bytes "barbarbar"))
267 :headers {"content-encoding" "deflate"}})
268 c-client (client/wrap-decompression client)
269 resp (c-client {})]
270 (is (= "barbarbar" (-> resp :body util/force-byte-array util/utf8-string))
271 "string correctly inflated")
272 (is (= "deflate" (:orig-content-encoding resp)))
273 (is (= nil (get-in resp [:headers "content-encoding"])))))
274
275 (deftest t-disabled-body-decompression
276 (let [client (fn [req]
277 (is (not= "gzip, deflate"
278 (get-in req [:headers "accept-encoding"])))
279 {:body (util/deflate (util/utf8-bytes "barbarbar"))
280 :headers {"content-encoding" "deflate"}})
281 c-client (client/wrap-decompression client)
282 resp (c-client {:decompress-body false})]
283 (is (= (slurp (util/inflate (util/deflate (util/utf8-bytes "barbarbar"))))
284 (slurp (util/inflate (-> resp :body util/force-byte-array))))
285 "string not inflated")
286 (is (= nil (:orig-content-encoding resp)))
287 (is (= "deflate" (get-in resp [:headers "content-encoding"])))))
288
289 (deftest t-weird-non-known-compression
290 (let [client (fn [req]
291 (is (= "gzip, deflate"
292 (get-in req [:headers "accept-encoding"])))
293 {:body (util/utf8-bytes "foofoofoo")
294 :headers {"content-encoding" "pig-latin"}})
295 c-client (client/wrap-decompression client)
296 resp (c-client {})]
297 (is (= "foofoofoo" (util/utf8-string (:body resp))))
298 (is (= "pig-latin" (:orig-content-encoding resp)))
299 (is (= "pig-latin" (get-in resp [:headers "content-encoding"])))))
300
301 (deftest pass-on-non-compressed
302 (let [c-client (client/wrap-decompression (fn [req] {:body "foo"}))
303 resp (c-client {:uri "/foo"})]
304 (is (= "foo" (:body resp)))))
305
306 (deftest apply-on-accept
307 (is-applied client/wrap-accept
308 {:accept :json}
309 {:headers {"accept" "application/json"}})
310 (is-applied client/wrap-accept
311 {:accept :transit+json}
312 {:headers {"accept" "application/transit+json"}})
313 (is-applied client/wrap-accept
314 {:accept :transit+msgpack}
315 {:headers {"accept" "application/transit+msgpack"}}))
316
317 (deftest pass-on-no-accept
318 (is-passed client/wrap-accept
319 {:uri "/foo"}))
320
321 (deftest apply-on-accept-encoding
322 (is-applied client/wrap-accept-encoding
323 {:accept-encoding [:identity :gzip]}
324 {:headers {"accept-encoding" "identity, gzip"}}))
325
326 (deftest pass-on-no-accept-encoding
327 (is-passed client/wrap-accept-encoding
328 {:uri "/foo"}))
329
330 (deftest apply-on-output-coercion
331 (let [client (fn [req] {:body (util/utf8-bytes "foo")})
332 o-client (client/wrap-output-coercion client)
333 resp (o-client {:uri "/foo"})]
334 (is (= "foo" (:body resp)))))
335
336 (deftest pass-on-no-output-coercion
337 (let [client (fn [req] {:body nil})
338 o-client (client/wrap-output-coercion client)
339 resp (o-client {:uri "/foo"})]
340 (is (nil? (:body resp))))
341 (let [the-stream (ByteArrayInputStream. (byte-array []))
342 client (fn [req] {:body the-stream})
343 o-client (client/wrap-output-coercion client)
344 resp (o-client {:uri "/foo" :as :stream})]
345 (is (= the-stream (:body resp))))
346 (let [client (fn [req] {:body :thebytes})
347 o-client (client/wrap-output-coercion client)
348 resp (o-client {:uri "/foo" :as :byte-array})]
349 (is (= :thebytes (:body resp)))))
350
351 (deftest apply-on-input-coercion
352 (let [i-client (client/wrap-input-coercion identity)
353 resp (i-client {:body "foo"})
354 resp2 (i-client {:body "foo2" :body-encoding "ASCII"})
355 data (slurp (.getContent ^HttpEntity (:body resp)))
356 data2 (slurp (.getContent ^HttpEntity (:body resp2)))]
357 (is (= "UTF-8" (:character-encoding resp)))
358 (is (= "foo" data))
359 (is (= "ASCII" (:character-encoding resp2)))
360 (is (= "foo2" data2))))
361
362 (deftest pass-on-no-input-coercion
363 (is-passed client/wrap-input-coercion
364 {:body nil}))
365
366 (deftest no-length-for-input-stream
367 (let [i-client (client/wrap-input-coercion identity)
368 resp1 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))})
369 resp2 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))
370 :length 3})
371 ^HttpEntity body1 (:body resp1)
372 ^HttpEntity body2 (:body resp2)]
373 (is (= -1 (.getContentLength body1)))
374 (is (= 3 (.getContentLength body2)))))
375
376 (deftest apply-on-content-type
377 (is-applied client/wrap-content-type
378 {:content-type :json}
379 {:headers {"content-type" "application/json"}
380 :content-type :json})
381 (is-applied client/wrap-content-type
382 {:content-type :json :character-encoding "UTF-8"}
383 {:headers {"content-type" "application/json; charset=UTF-8"}
384 :content-type :json :character-encoding "UTF-8"})
385 (is-applied client/wrap-content-type
386 {:content-type :transit+json}
387 {:headers {"content-type" "application/transit+json"}
388 :content-type :transit+json})
389 (is-applied client/wrap-content-type
390 {:content-type :transit+msgpack}
391 {:headers {"content-type" "application/transit+msgpack"}
392 :content-type :transit+msgpack}))
393
394 (deftest pass-on-no-content-type
395 (is-passed client/wrap-content-type
396 {:uri "/foo"}))
397
398 (deftest apply-on-query-params
399 (is-applied client/wrap-query-params
400 {:query-params {"foo" "bar" "dir" "<<"}}
401 {:query-string "foo=bar&dir=%3C%3C"})
402 (is-applied client/wrap-query-params
403 {:query-string "foo=1"
404 :query-params {"foo" ["2" "3"]}}
405 {:query-string "foo=1&foo=2&foo=3"}))
406
407 (deftest pass-on-no-query-params
408 (is-passed client/wrap-query-params
409 {:uri "/foo"}))
410
411 (deftest apply-on-basic-auth
412 (is-applied client/wrap-basic-auth
413 {:basic-auth ["Aladdin" "open sesame"]}
414 {:headers {"authorization"
415 "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="}}))
416
417 (deftest pass-on-no-basic-auth
418 (is-passed client/wrap-basic-auth
419 {:uri "/foo"}))
420
421 (deftest apply-on-oauth
422 (is-applied client/wrap-oauth
423 {:oauth-token "my-token"}
424 {:headers {"authorization"
425 "Bearer my-token"}}))
426
427 (deftest pass-on-no-oauth
428 (is-passed client/wrap-oauth
429 {:uri "/foo"}))
430
431 (deftest apply-on-method
432 (let [m-client (client/wrap-method identity)
433 echo (m-client {:key :val :method :post})]
434 (is (= :val (:key echo)))
435 (is (= :post (:request-method echo)))
436 (is (not (:method echo)))))
437
438 (deftest pass-on-no-method
439 (let [m-client (client/wrap-method identity)
440 echo (m-client {:key :val})]
441 (is (= :val (:key echo)))
442 (is (not (:request-method echo)))))
443
444 (deftest apply-on-url
445 (let [u-client (client/wrap-url identity)
446 resp (u-client {:url "http://google.com:8080/baz foo?bar=bat bit?"})]
447 (is (= :http (:scheme resp)))
448 (is (= "google.com" (:server-name resp)))
449 (is (= 8080 (:server-port resp)))
450 (is (= "/baz%20foo" (:uri resp)))
451 (is (= "bar=bat%20bit?" (:query-string resp)))))
452
453 (deftest pass-on-no-url
454 (let [u-client (client/wrap-url identity)
455 resp (u-client {:uri "/foo"})]
456 (is (= "/foo" (:uri resp)))))
457
458 (deftest provide-default-port
459 (is (= nil (-> "http://example.com/" client/parse-url :server-port)))
460 (is (= 8080 (-> "http://example.com:8080/" client/parse-url :server-port)))
461 (is (= nil (-> "https://example.com/" client/parse-url :server-port)))
462 (is (= 8443 (-> "https://example.com:8443/" client/parse-url :server-port))))
463
464 (deftest decode-credentials-from-url
465 (is (= "fred's diner:fred's password"
466 (-> "http://fred%27s%20diner:fred%27s%20password@example.com/foo"
467 client/parse-url
468 :user-info))))
469
470 (defrecord Point [x y])
471
472 (def write-point
473 "Write a point in Transit format."
474 (transit/write-handler
475 (constantly "point")
476 (fn [point] [(:x point) (:y point)])
477 (constantly nil)))
478
479 (def read-point
480 "Read a point in Transit format."
481 (transit/read-handler
482 (fn [[x y]]
483 (->Point x y))))
484
485 (def transit-opts
486 "Transit read and write options."
487 {:encode {:handlers {Point write-point}}
488 :decode {:handlers {"point" read-point}}})
489
490 (def transit-opts-deprecated
491 "Deprecated Transit read and write options."
492 {:handlers {Point write-point "point" read-point}})
493
494 (deftest apply-on-form-params
495 (testing "With form params"
496 (let [param-client (client/wrap-form-params identity)
497 resp (param-client {:request-method :post
498 :form-params (sorted-map :param1 "value1"
499 :param2 "value2")})]
500 (is (= "param1=value1&param2=value2" (:body resp)))
501 (is (= "application/x-www-form-urlencoded" (:content-type resp)))
502 (is (not (contains? resp :form-params))))
503 (let [param-client (client/wrap-form-params identity)
504 resp (param-client {:request-method :put
505 :form-params (sorted-map :param1 "value1"
506 :param2 "value2")})]
507 (is (= "param1=value1&param2=value2" (:body resp)))
508 (is (= "application/x-www-form-urlencoded" (:content-type resp)))
509 (is (not (contains? resp :form-params)))))
510
511 (testing "With json form params"
512 (let [param-client (client/wrap-form-params identity)
513 params {:param1 "value1" :param2 "value2"}
514 resp (param-client {:request-method :post
515 :content-type :json
516 :form-params params})]
517 (is (= (json/encode params) (:body resp)))
518 (is (= "application/json" (:content-type resp)))
519 (is (not (contains? resp :form-params))))
520 (let [param-client (client/wrap-form-params identity)
521 params {:param1 "value1" :param2 "value2"}
522 resp (param-client {:request-method :put
523 :content-type :json
524 :form-params params})]
525 (is (= (json/encode params) (:body resp)))
526 (is (= "application/json" (:content-type resp)))
527 (is (not (contains? resp :form-params))))
528 (let [param-client (client/wrap-form-params identity)
529 params {:param1 "value1" :param2 "value2"}
530 resp (param-client {:request-method :patch
531 :content-type :json
532 :form-params params})]
533 (is (= (json/encode params) (:body resp)))
534 (is (= "application/json" (:content-type resp)))
535 (is (not (contains? resp :form-params))))
536 (let [param-client (client/wrap-form-params identity)
537 params {:param1 (java.util.Date. (long 0))}
538 resp (param-client {:request-method :put
539 :content-type :json
540 :form-params params
541 :json-opts {:date-format "yyyy-MM-dd"}})]
542 (is (= (json/encode params {:date-format "yyyy-MM-dd"}) (:body resp)))
543 (is (= "application/json" (:content-type resp)))
544 (is (not (contains? resp :form-params)))))
545
546 (testing "With EDN form params"
547 (doseq [method [:post :put :patch]]
548 (let [param-client (client/wrap-form-params identity)
549 params {:param1 "value1" :param2 (Point. 1 2)}
550 resp (param-client {:request-method method
551 :content-type :edn
552 :form-params params})]
553 (is (= (pr-str params) (:body resp)))
554 (is (= "application/edn" (:content-type resp)))
555 (is (not (contains? resp :form-params))))))
556
557 (testing "With Transit/JSON form params"
558 (doseq [method [:post :put :patch]]
559 (let [param-client (client/wrap-form-params identity)
560 params {:param1 "value1" :param2 (Point. 1 2)}
561 resp (param-client {:request-method method
562 :content-type :transit+json
563 :form-params params
564 :transit-opts transit-opts})]
565 (is (= params (client/parse-transit
566 (ByteArrayInputStream. (:body resp))
567 :json transit-opts)))
568 (is (= "application/transit+json" (:content-type resp)))
569 (is (not (contains? resp :form-params))))))
570
571 (testing "With Transit/MessagePack form params"
572 (doseq [method [:post :put :patch]]
573 (let [param-client (client/wrap-form-params identity)
574 params {:param1 "value1" :param2 "value2"}
575 resp (param-client {:request-method method
576 :content-type :transit+msgpack
577 :form-params params
578 :transit-opts transit-opts})]
579 (is (= params (client/parse-transit
580 (ByteArrayInputStream. (:body resp))
581 :msgpack transit-opts)))
582 (is (= "application/transit+msgpack" (:content-type resp)))
583 (is (not (contains? resp :form-params))))))
584
585 (testing "With Transit/JSON form params and deprecated options"
586 (let [param-client (client/wrap-form-params identity)
587 params {:param1 "value1" :param2 (Point. 1 2)}
588 resp (param-client {:request-method :post
589 :content-type :transit+json
590 :form-params params
591 :transit-opts transit-opts-deprecated})]
592 (is (= params (client/parse-transit
593 (ByteArrayInputStream. (:body resp))
594 :json transit-opts-deprecated)))
595 (is (= "application/transit+json" (:content-type resp)))
596 (is (not (contains? resp :form-params)))))
597
598 (testing "Ensure it does not affect GET requests"
599 (let [param-client (client/wrap-form-params identity)
600 resp (param-client {:request-method :get
601 :body "untouched"
602 :form-params {:param1 "value1"
603 :param2 "value2"}})]
604 (is (= "untouched" (:body resp)))
605 (is (not (contains? resp :content-type)))))
606
607 (testing "with no form params"
608 (let [param-client (client/wrap-form-params identity)
609 resp (param-client {:body "untouched"})]
610 (is (= "untouched" (:body resp)))
611 (is (not (contains? resp :content-type))))))
612
613 (deftest apply-on-nested-params
614 (testing "nested parameter maps"
615 (are [in out] (is-applied client/wrap-nested-params
616 {:query-params in :form-params in}
617 {:query-params out :form-params out})
618 {"foo" "bar"} {"foo" "bar"}
619 {"x" {"y" "z"}} {"x[y]" "z"}
620 {"a" {"b" {"c" "d"}}} {"a[b][c]" "d"}
621 {"a" "b", "c" "d"} {"a" "b", "c" "d"}))
622
623 (testing "not creating empty param maps"
624 (is-applied client/wrap-query-params {} {})))
625
626 (deftest t-ignore-unknown-host
627 (is (thrown? UnknownHostException (client/get "http://aorecuf892983a.com")))
628 (is (nil? (client/get "http://aorecuf892983a.com"
629 {:ignore-unknown-host? true}))))
630
631 (deftest test-status-predicates
632 (testing "2xx statuses"
633 (doseq [s (range 200 299)]
634 (is (client/success? {:status s}))
635 (is (not (client/redirect? {:status s})))
636 (is (not (client/client-error? {:status s})))
637 (is (not (client/server-error? {:status s})))))
638 (testing "3xx statuses"
639 (doseq [s (range 300 399)]
640 (is (not (client/success? {:status s})))
641 (is (client/redirect? {:status s}))
642 (is (not (client/client-error? {:status s})))
643 (is (not (client/server-error? {:status s})))))
644 (testing "4xx statuses"
645 (doseq [s (range 400 499)]
646 (is (not (client/success? {:status s})))
647 (is (not (client/redirect? {:status s})))
648 (is (client/client-error? {:status s}))
649 (is (not (client/server-error? {:status s})))))
650 (testing "5xx statuses"
651 (doseq [s (range 500 599)]
652 (is (not (client/success? {:status s})))
653 (is (not (client/redirect? {:status s})))
654 (is (not (client/client-error? {:status s})))
655 (is (client/server-error? {:status s}))))
656 (testing "409 Conflict"
657 (is (client/conflict? {:status 409}))
658 (is (not (client/conflict? {:status 201})))
659 (is (not (client/conflict? {:status 404})))))
660
661 (deftest test-wrap-lower-case-headers
662 (is (= {:status 404} ((client/wrap-lower-case-headers
663 (fn [r] r)) {:status 404})))
664 (is (= {:headers {"content-type" "application/json"}}
665 ((client/wrap-lower-case-headers
666 #(do (is (= {:headers {"accept" "application/json"}} %1))
667 {:headers {"Content-Type" "application/json"}}))
668 {:headers {"Accept" "application/json"}}))))
669
670 (deftest t-request-timing
671 (is (pos? (:request-time ((client/wrap-request-timing
672 (fn [r] (Thread/sleep 15) r)) {})))))
673
674 (deftest t-wrap-additional-header-parsing
675 (let [^String text (slurp (resource "header-test.html"))
676 client (fn [req] {:body (.getBytes text)})
677 new-client (client/wrap-additional-header-parsing client)
678 resp (new-client {:decode-body-headers true})
679 resp2 (new-client {:decode-body-headers false})
680 resp3 ((client/wrap-additional-header-parsing
681 (fn [req] {:body nil})) {:decode-body-headers true})
682 resp4 ((client/wrap-additional-header-parsing
683 (fn [req] {:headers {"content-type" "application/pdf"}
684 :body (.getBytes text)}))
685 {:decode-body-headers true})]
686 (is (= {"content-type" "text/html; charset=Shift_JIS"
687 "content-style-type" "text/css"
688 "content-script-type" "text/javascript"}
689 (:headers resp)))
690 (is (nil? (:headers resp2)))
691 (is (nil? (:headers resp3)))
692 (is (= {"content-type" "application/pdf"} (:headers resp4)))))
693
694 (deftest t-wrap-additional-header-parsing-html5
695 (let [^String text (slurp (resource "header-html5-test.html"))
696 client (fn [req] {:body (.getBytes text)})
697 new-client (client/wrap-additional-header-parsing client)
698 resp (new-client {:decode-body-headers true})]
699 (is (= {"content-type" "text/html; charset=UTF-8"}
700 (:headers resp)))))
701
702 (deftest ^:integration t-request-without-url-set
703 (run-server)
704 ;; roundtrip with scheme as a keyword
705 (let [resp (request {:uri "/redirect-to-get"
706 :method :get})]
707 (is (= 200 (:status resp)))
708 (is (= "close" (get-in resp [:headers "connection"])))
709 (is (= "get" (:body resp)))))
710
711 (deftest ^:integration t-reusable-conn-mgrs
712 (run-server)
713 (let [cm (conn/make-reusable-conn-manager {:timeout 10 :insecure? false})
714 resp1 (request {:uri "/redirect-to-get"
715 :method :get
716 :connection-manager cm})
717 resp2 (request {:uri "/redirect-to-get"
718 :method :get})]
719 (is (= 200 (:status resp1) (:status resp2)))
720 (is (nil? (get-in resp1 [:headers "connection"]))
721 "connection should remain open")
722 (is (= "close" (get-in resp2 [:headers "connection"]))
723 "connection should be closed")
724 (.shutdown cm)))
725
726 (deftest test-url-encode-path
727 (is (= (client/url-encode-illegal-characters "?foo bar+baz[]75")
728 "?foo%20bar+baz%5B%5D75"))
729 (is (= {:uri (str "/:@-._~!$&'()*+,="
730 ";"
731 ":@-._~!$&'()*+,"
732 "="
733 ":@-._~!$&'()*+,==")
734 :query-string (str "/?:@-._~!$'()*+,;"
735 "="
736 "/?:@-._~!$'()*+,;==")}
737 ;; This URL sucks, yes, it's actually a valid URL
738 (select-keys (client/parse-url
739 (str "http://example.com/:@-._~!$&'()*+,=;:@-._~!$&'()*+"
740 ",=:@-._~!$&'()*+,==?/?:@-._~!$'()*+,;=/?:@-._~!$'("
741 ")*+,;==#/?:@-._~!$&'()*+,;="))
742 [:uri :query-string])))
743 (let [all-chars (apply str (map char (range 256)))
744 all-legal (client/url-encode-illegal-characters all-chars)]
745 (is (= all-legal
746 (client/url-encode-illegal-characters all-legal)))))
747
748 (deftest t-coercion-methods
749 (let [json-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
750 auto-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
751 edn-body (ByteArrayInputStream. (.getBytes "{:foo \"bar\"}"))
752 transit-json-body (ByteArrayInputStream.
753 (.getBytes "[\"^ \",\"~:foo\",\"bar\"]"))
754 transit-msgpack-body (->> (map byte [-127 -91 126 58 102 111
755 111 -93 98 97 114])
756 (byte-array 11)
757 (ByteArrayInputStream.))
758 www-form-urlencoded-body (ByteArrayInputStream. (.getBytes "foo=bar"))
759 auto-www-form-urlencoded-body
760 (ByteArrayInputStream. (.getBytes "foo=bar"))
761 json-resp {:body json-body :status 200
762 :headers {"content-type" "application/json"}}
763 auto-resp {:body auto-body :status 200
764 :headers {"content-type" "application/json"}}
765 edn-resp {:body edn-body :status 200
766 :headers {"content-type" "application/edn"}}
767 transit-json-resp {:body transit-json-body :status 200
768 :headers {"content-type" "application/transit-json"}}
769 transit-msgpack-resp {:body transit-msgpack-body :status 200
770 :headers {"content-type"
771 "application/transit-msgpack"}}
772 www-form-urlencoded-resp
773 {:body www-form-urlencoded-body :status 200
774 :headers {"content-type"
775 "application/x-www-form-urlencoded"}}
776 auto-www-form-urlencoded-resp
777 {:body auto-www-form-urlencoded-body :status 200
778 :headers {"content-type"
779 "application/x-www-form-urlencoded"}}]
780 (is (= {:foo "bar"}
781 (:body (client/coerce-response-body {:as :json} json-resp))
782 (:body (client/coerce-response-body {:as :clojure} edn-resp))
783 (:body (client/coerce-response-body {:as :auto} auto-resp))
784 (:body (client/coerce-response-body {:as :transit+json}
785 transit-json-resp))
786 (:body (client/coerce-response-body {:as :transit+msgpack}
787 transit-msgpack-resp))
788 (:body (client/coerce-response-body {:as :auto}
789 auto-www-form-urlencoded-resp))
790 (:body (client/coerce-response-body {:as :x-www-form-urlencoded}
791 www-form-urlencoded-resp))))))
792
793 (deftest ^:integration t-with-middleware
794 (run-server)
795 (is (:request-time (request {:uri "/get" :method :get})))
796 (is (= client/*current-middleware* client/default-middleware))
797 (client/with-middleware [client/wrap-url
798 client/wrap-method
799 #'client/wrap-request-timing]
800 (is (:request-time (request {:uri "/get" :method :get})))
801 (is (= client/*current-middleware* [client/wrap-url
802 client/wrap-method
803 #'client/wrap-request-timing])))
804 (client/with-middleware (->> client/default-middleware
805 (remove #{client/wrap-request-timing}))
806 (is (not (:request-time (request {:uri "/get" :method :get}))))
807 (is (not (contains? (set client/*current-middleware*)
808 client/wrap-request-timing)))
809 (is (contains? (set client/default-middleware)
810 client/wrap-request-timing))))
811
812 (deftest t-detect-charset-by-content-type
813 (is (= "UTF-8" (client/detect-charset nil)))
814 (is (= "UTF-8"(client/detect-charset "application/json")))
815 (is (= "UTF-8"(client/detect-charset "text/html")))
816 (is (= "GBK"(client/detect-charset "application/json; charset=GBK")))
817 (is (= "ISO-8859-1" (client/detect-charset
818 "application/json; charset=ISO-8859-1")))
819 (is (= "ISO-8859-1" (client/detect-charset
820 "application/json; charset = ISO-8859-1")))
821 (is (= "GB2312" (client/detect-charset "text/html; Charset=GB2312"))))
822
823 (deftest ^:integration multi-valued-query-params
824 (run-server)
825 (testing "default (repeating) multi-valued query params"
826 (let [resp (request {:uri "/query-string"
827 :method :get
828 :query-params {:a [1 2 3]
829 :b ["x" "y" "z"]}})
830 query-string (-> resp :body form-decode-str)]
831 (is (= 200 (:status resp)))
832 (is (.contains query-string "a=1&a=2&a=3") query-string)
833 (is (.contains query-string "b=x&b=y&b=z") query-string)))
834
835 (testing "multi-valued query params in indexed-style"
836 (let [resp (request {:uri "/query-string"
837 :method :get
838 :multi-param-style :indexed
839 :query-params {:a [1 2 3]
840 :b ["x" "y" "z"]}})
841 query-string (-> resp :body form-decode-str)]
842 (is (= 200 (:status resp)))
843 (is (.contains query-string "a[0]=1&a[1]=2&a[2]=3") query-string)
844 (is (.contains query-string "b[0]=x&b[1]=y&b[2]=z") query-string)))
845
846 (testing "multi-valued query params in array-style"
847 (let [resp (request {:uri "/query-string"
848 :method :get
849 :multi-param-style :array
850 :query-params {:a [1 2 3]
851 :b ["x" "y" "z"]}})
852 query-string (-> resp :body form-decode-str)]
853 (is (= 200 (:status resp)))
854 (is (.contains query-string "a[]=1&a[]=2&a[]=3") query-string)
855 (is (.contains query-string "b[]=x&b[]=y&b[]=z") query-string))))
+95
-0
test/clj_http/test/conn_mgr.clj less more
0 (ns clj-http.test.conn-mgr
1 (:require [clj-http.conn-mgr :as conn-mgr]
2 [clj-http.core :as core]
3 [clj-http.test.core :refer [run-server]]
4 [clojure.test :refer :all]
5 [ring.adapter.jetty :as ring])
6 (:import (java.security KeyStore)
7 (org.apache.http.conn.ssl SSLSocketFactory)
8 (org.apache.http.impl.conn BasicClientConnectionManager)))
9
10 (def client-ks "test-resources/client-keystore")
11 (def client-ks-pass "keykey")
12 (def secure-request {:request-method :get :uri "/"
13 :server-port 18084 :scheme :https
14 :keystore client-ks :keystore-pass client-ks-pass
15 :trust-store client-ks :trust-store-pass client-ks-pass
16 :server-name "localhost" :insecure? true})
17
18 (defn secure-handler [req]
19 (if (nil? (:ssl-client-cert req))
20 {:status 403}
21 {:status 200}))
22
23 (deftest load-keystore
24 (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")]
25 (is (instance? KeyStore ks))
26 (is (> (.size ks) 0))))
27
28 (deftest use-existing-keystore
29 (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")
30 ks (conn-mgr/get-keystore ks)]
31 (is (instance? KeyStore ks))
32 (is (> (.size ks) 0))))
33
34 (deftest load-keystore-with-nil-pass
35 (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil nil)]
36 (is (instance? KeyStore ks))))
37
38 (deftest keystore-scheme-factory
39 (let [sr (conn-mgr/get-keystore-scheme-registry
40 {:keystore client-ks :keystore-pass client-ks-pass
41 :trust-store client-ks :trust-store-pass client-ks-pass})
42 socket-factory (.getSchemeSocketFactory (.get sr "https"))]
43 (is (instance? SSLSocketFactory socket-factory))))
44
45 (deftest ^:integration ssl-client-cert-get
46 (let [server (ring/run-jetty secure-handler
47 {:port 18083 :ssl-port 18084
48 :ssl? true
49 :join? false
50 :keystore "test-resources/keystore"
51 :key-password "keykey"
52 :client-auth :want})]
53 (try
54 (let [resp (core/request {:request-method :get :uri "/get"
55 :server-port 18084 :scheme :https
56 :insecure? true :server-name "localhost"})]
57 (is (= 403 (:status resp))))
58 (let [resp (core/request secure-request)]
59 (is (= 200 (:status resp))))
60 (finally
61 (.stop server)))))
62
63 (deftest ^:integration t-closed-conn-mgr-for-as-stream
64 (run-server)
65 (let [shutdown? (atom false)
66 cm (proxy [BasicClientConnectionManager] []
67 (shutdown []
68 (reset! shutdown? true)))]
69 (try
70 (core/request {:request-method :get :uri "/timeout"
71 :server-port 18080 :scheme :http
72 :server-name "localhost"
73 ;; timeouts forces an exception being thrown
74 :socket-timeout 1
75 :conn-timeout 1
76 :connection-manager cm
77 :as :stream})
78 (is false "request should have thrown an exception")
79 (catch Exception e))
80 (is @shutdown? "Connection manager has been shut down")))
81
82 (deftest ^:integration t-closed-conn-mgr-for-empty-body
83 (run-server)
84 (let [shutdown? (atom false)
85 cm (proxy [BasicClientConnectionManager] []
86 (shutdown []
87 (reset! shutdown? true)))
88 response (core/request {:request-method :get :uri "/unmodified-resource"
89 :server-port 18080 :scheme :http
90 :server-name "localhost"
91 :connection-manager cm })]
92 (is (nil? (:body response)) "response shouldn't have body")
93 (is (= 304 (:status response)))
94 (is @shutdown? "connection manager should be shut downed")))
+222
-0
test/clj_http/test/cookies.clj less more
0 (ns clj-http.test.cookies
1 (:require [clj-http.cookies :refer :all]
2 [clj-http.util :refer :all]
3 [clojure.test :refer :all])
4 (:import (org.apache.http.impl.cookie BasicClientCookie BasicClientCookie2)))
5
6 (defn refer-private [ns]
7 (doseq [[symbol var] (ns-interns ns)]
8 (when (:private (meta var))
9 (intern *ns* symbol var))))
10
11 (refer-private 'clj-http.cookies)
12
13 (def session (str "ltQGXSNp7cgNeFG6rPE06qzriaI+R8W7zJKFu4UOlX4=-"
14 "-lWgojFmZlDqSBnYJlUmwhqXL4OgBTkra5WXzi74v+nE="))
15
16 (deftest test-compact-map
17 (are [map expected]
18 (is (= expected (compact-map map)))
19 {:a nil :b 2 :c 3 :d nil}
20 {:b 2 :c 3}
21 {:comment nil :domain "example.com" :path "/" :ports [80 8080] :value 1}
22 {:domain "example.com" :path "/" :ports [80 8080] :value 1}))
23
24 (deftest test-decode-cookie
25 (are [set-cookie-str expected]
26 (is (= expected (decode-cookie set-cookie-str)))
27 nil nil
28 "" nil
29 "example-cookie=example-value;Path=/"
30 ["example-cookie"
31 {:discard true :path "/" :secure false
32 :value "example-value" :version 0}]
33 "example-cookie=example-value;Domain=.example.com;Path=/"
34 ["example-cookie"
35 {:discard true :domain "example.com" :secure false :path "/"
36 :value "example-value" :version 0}]))
37
38 (deftest test-decode-cookies-with-seq
39 (let [cookies (decode-cookies [(str "ring-session=" session)])]
40 (is (map? cookies))
41 (is (= 1 (count cookies)))
42 (let [cookie (get cookies "ring-session")]
43 (is (= true (:discard cookie)))
44 (is (nil? (:domain cookie)))
45 (is (= "/" (:path cookie)))
46 (is (= session (:value cookie)))
47 (is (= 0 (:version cookie))))))
48
49 (deftest test-decode-cookies-with-string
50 (let [cookies (decode-cookies
51 (str "ring-session=" session ";Path=/"))]
52 (is (map? cookies))
53 (is (= 1 (count cookies)))
54 (let [cookie (get cookies "ring-session")]
55 (is (= true (:discard cookie)))
56 (is (nil? (:domain cookie)))
57 (is (= "/" (:path cookie)))
58 (is (= session (:value cookie)))
59 (is (= 0 (:version cookie))))))
60
61 (deftest test-decode-cookie-header
62 (are [response expected]
63 (is (= expected (decode-cookie-header response)))
64 {:headers {"set-cookie" "a=1"}}
65 {:cookies {"a" {:discard true :path "/" :secure false
66 :value "1" :version 0}} :headers {}}
67 {:headers {"set-cookie"
68 (str "ring-session=" session ";Path=/")}}
69 {:cookies {"ring-session"
70 {:discard true :path "/" :secure false
71 :value session :version 0}} :headers {}}))
72
73 (deftest test-encode-cookie
74 (are [cookie expected]
75 (is (= expected (encode-cookie cookie)))
76 [:a {:value "b"}] "a=b"
77 ["a" {:value "b"}] "a=b"
78 ["example-cookie"
79 {:domain ".example.com" :path "/" :value "example-value"}]
80 "example-cookie=example-value"
81 ["ring-session" {:value session}]
82 (str "ring-session=" session)))
83
84 (deftest test-encode-cookies
85 (are [cookie expected]
86 (is (= expected (encode-cookies cookie)))
87 (sorted-map :a {:value "b"} :c {:value "d"} :e {:value "f"})
88 "a=b;c=d;e=f"
89 (sorted-map "a" {:value "b"} "c" {:value "d"} "e" {:value "f"})
90 "a=b;c=d;e=f"
91 {"example-cookie"
92 {:domain ".example.com" :path "/" :value "example-value"}}
93 "example-cookie=example-value"
94 {"example-cookie"
95 {:domain ".example.com" :path "/" :value "example-value"
96 :discard true :version 0}}
97 "example-cookie=example-value"
98 {"ring-session" {:value session}}
99 (str "ring-session=" session)))
100
101 (deftest test-encode-cookie-header
102 (are [request expected]
103 (is (= expected (encode-cookie-header request)))
104 {:cookies {"a" {:value "1"}}}
105 {:headers {"Cookie" "a=1"}}
106 {:cookies
107 {"example-cookie" {:domain ".example.com" :path "/"
108 :value "example-value"}}}
109 {:headers {"Cookie" "example-cookie=example-value"}}))
110
111 (deftest test-to-basic-client-cookie-with-simple-cookie
112 (let [cookie (to-basic-client-cookie
113 ["ring-session"
114 {:value session
115 :path "/"
116 :domain "example.com"}])]
117 (is (= "ring-session" (.getName cookie)))
118 (is (= session (.getValue cookie)))
119 (is (= "/" (.getPath cookie)))
120 (is (= "example.com" (.getDomain cookie)))
121 (is (nil? (.getComment cookie)))
122 (is (nil? (.getCommentURL cookie)))
123 (is (not (.isPersistent cookie)))
124 (is (nil? (.getExpiryDate cookie)))
125 (is (nil? (seq (.getPorts cookie))))
126 (is (not (.isSecure cookie)))
127 (is (= 0 (.getVersion cookie)))))
128
129 (deftest test-to-basic-client-cookie-with-full-cookie
130 (let [cookie (to-basic-client-cookie
131 ["ring-session"
132 {:value session
133 :path "/"
134 :domain "example.com"
135 :comment "Example Comment"
136 :comment-url "http://example.com/cookies"
137 :discard true
138 :expires (java.util.Date. (long 0))
139 :ports [80 8080]
140 :secure true
141 :version 0}])]
142 (is (= "ring-session" (.getName cookie)))
143 (is (= session (.getValue cookie)))
144 (is (= "/" (.getPath cookie)))
145 (is (= "example.com" (.getDomain cookie)))
146 (is (= "Example Comment" (.getComment cookie)))
147 (is (= "http://example.com/cookies" (.getCommentURL cookie)))
148 (is (not (.isPersistent cookie)))
149 (is (= (java.util.Date. (long 0)) (.getExpiryDate cookie)))
150 (is (= [80 8080] (seq (.getPorts cookie))))
151 (is (.isSecure cookie))
152 (is (= 0 (.getVersion cookie)))))
153
154 (deftest test-to-basic-client-cookie-with-symbol-as-name
155 (let [cookie (to-basic-client-cookie
156 [:ring-session {:value session :path "/"
157 :domain "example.com"}])]
158 (is (= "ring-session" (.getName cookie)))))
159
160 (deftest test-to-cookie-with-simple-cookie
161 (let [[name content]
162 (to-cookie
163 (doto (BasicClientCookie. "example-cookie" "example-value")
164 (.setDomain "example.com")
165 (.setPath "/")))]
166 (is (= "example-cookie" name))
167 (is (nil? (:comment content)))
168 (is (nil? (:comment-url content)))
169 (is (:discard content))
170 (is (= "example.com" (:domain content)))
171 (is (nil? (:expires content)))
172 (is (nil? (:ports content)))
173 (is (not (:secure content)))
174 (is (= 0 (:version content)))
175 (is (= "example-value" (:value content)))))
176
177 (deftest test-to-cookie-with-full-cookie
178 (let [[name content]
179 (to-cookie
180 (doto (BasicClientCookie2. "example-cookie" "example-value")
181 (.setComment "Example Comment")
182 (.setCommentURL "http://example.com/cookies")
183 (.setDiscard true)
184 (.setDomain "example.com")
185 (.setExpiryDate (java.util.Date. (long 0)))
186 (.setPath "/")
187 (.setPorts (int-array [80 8080]))
188 (.setSecure true)
189 (.setVersion 1)))]
190 (is (= "example-cookie" name))
191 (is (= "Example Comment" (:comment content)))
192 (is (= "http://example.com/cookies" (:comment-url content)))
193 (is (= true (:discard content)))
194 (is (= "example.com" (:domain content)))
195 (is (= (java.util.Date. (long 0)) (:expires content)))
196 (is (= [80 8080] (:ports content)))
197 (is (= true (:secure content)))
198 (is (= 1 (:version content)))
199 (is (= "example-value" (:value content)))))
200
201 (deftest test-wrap-cookies
202 (is (= {:cookies {"example-cookie" {:discard true :domain "example.com"
203 :path "/" :value "example-value"
204 :version 0 :secure false}} :headers {}}
205 ((wrap-cookies
206 (fn [request]
207 (is (= (get (:headers request) "Cookie") "a=1;b=2"))
208 {:headers
209 {"set-cookie"
210 "example-cookie=example-value;Domain=.example.com;Path=/"}}))
211 {:cookies (sorted-map :a {:value "1"} :b {:value "2"})})))
212 (is (= {:headers {"set-cookie"
213 "example-cookie=example-value;Domain=.example.com;Path=/"}}
214 ((wrap-cookies
215 (fn [request]
216 (is (= (get (:headers request) "Cookie") "a=1;b=2"))
217 {:headers
218 {"set-cookie"
219 "example-cookie=example-value;Domain=.example.com;Path=/"}}))
220 {:cookies (sorted-map :a {:value "1"} :b {:value "2"})
221 :decode-cookies false}))))
+589
-0
test/clj_http/test/core.clj less more
0 (ns clj-http.test.core
1 (:require [cheshire.core :as json]
2 [clj-http.client :as client]
3 [clj-http.core :as core]
4 [clj-http.util :as util]
5 [clojure.java.io :refer [file]]
6 [clojure.pprint :as pp]
7 [clojure.test :refer :all]
8 [ring.adapter.jetty :as ring])
9 (:import (java.io ByteArrayInputStream)
10 (org.apache.http.params CoreConnectionPNames CoreProtocolPNames)
11 (org.apache.http.message BasicHeader BasicHeaderIterator)
12 (org.apache.http.client.methods HttpPost)
13 (org.apache.http.client.params CookiePolicy ClientPNames)
14 (org.apache.http HttpRequest HttpResponse HttpConnection HttpInetConnection
15 HttpVersion)
16 (org.apache.http.protocol HttpContext ExecutionContext)
17 (org.apache.http.impl.client DefaultHttpClient)
18 (org.apache.http.client.params ClientPNames)
19 (java.net SocketTimeoutException)
20 (sun.security.provider.certpath SunCertPathBuilderException)))
21
22 (defn handler [req]
23 (condp = [(:request-method req) (:uri req)]
24 [:get "/get"]
25 {:status 200 :body "get"}
26 [:get "/empty"]
27 {:status 200 :body nil}
28 [:get "/clojure"]
29 {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
30 :headers {"content-type" "application/clojure"}}
31 [:get "/edn"]
32 {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
33 :headers {"content-type" "application/edn"}}
34 [:get "/clojure-bad"]
35 {:status 200 :body "{:foo \"bar\" :baz #=(+ 1 1)}"
36 :headers {"content-type" "application/clojure"}}
37 [:get "/json"]
38 {:status 200 :body "{\"foo\":\"bar\"}"
39 :headers {"content-type" "application/json"}}
40 [:get "/json-array"]
41 {:status 200 :body "[\"foo\", \"bar\"]"
42 :headers {"content-type" "application/json"}}
43 [:get "/json-bad"]
44 {:status 400 :body "{\"foo\":\"bar\"}"}
45 [:get "/redirect"]
46 {:status 302
47 :headers {"location" "http://localhost:18080/redirect"}}
48 [:get "/redirect-to-get"]
49 {:status 302
50 :headers {"location" "http://localhost:18080/get"}}
51 [:get "/unmodified-resource"]
52 {:status 304}
53 [:get "/transit-json"]
54 {:status 200 :body (str "[\"^ \",\"~:eggplant\",[\"^ \",\"~:quux\","
55 "[\"~#set\",[1,3,2]]],\"~:baz\",\"~f7\","
56 "\"~:foo\",\"bar\"]")
57 :headers {"content-type" "application/transit+json"}}
58 [:get "/transit-msgpack"]
59 {:status 200
60 :body (->> [-125 -86 126 58 101 103 103 112 108 97 110 116 -127 -90 126
61 58 113 117 117 120 -110 -91 126 35 115 101 116 -109 1 3 2
62 -91 126 58 98 97 122 -93 126 102 55 -91 126 58 102 111 111
63 -93 98 97 114]
64 (map byte)
65 (byte-array)
66 (ByteArrayInputStream.))
67 :headers {"content-type" "application/transit+msgpack"}}
68 [:head "/head"]
69 {:status 200}
70 [:get "/content-type"]
71 {:status 200 :body (:content-type req)}
72 [:get "/header"]
73 {:status 200 :body (get-in req [:headers "x-my-header"])}
74 [:post "/post"]
75 {:status 200 :body (:body req)}
76 [:get "/error"]
77 {:status 500 :body "o noes"}
78 [:get "/timeout"]
79 (do
80 (Thread/sleep 10)
81 {:status 200 :body "timeout"})
82 [:delete "/delete-with-body"]
83 {:status 200 :body "delete-with-body"}
84 [:post "/multipart"]
85 {:status 200 :body (:body req)}
86 [:get "/get-with-body"]
87 {:status 200 :body (:body req)}
88 [:options "/options"]
89 {:status 200 :body "options"}
90 [:copy "/copy"]
91 {:status 200 :body "copy"}
92 [:move "/move"]
93 {:status 200 :body "move"}
94 [:patch "/patch"]
95 {:status 200 :body "patch"}
96 [:get "/headers"]
97 {:status 200 :body (json/encode (:headers req))}
98 [:get "/query-string"]
99 {:status 200 :body (:query-string req)}))
100
101 (defn run-server
102 []
103 (defonce server
104 (ring/run-jetty #'handler {:port 18080 :join? false})))
105
106 (defn localhost [path]
107 (str "http://localhost:18080" path))
108
109 (def base-req
110 {:scheme :http
111 :server-name "localhost"
112 :server-port 18080})
113
114 (defn request [req]
115 (core/request (merge base-req req)))
116
117 (defn slurp-body [req]
118 (slurp (:body req)))
119
120 (deftest ^:integration makes-get-request
121 (run-server)
122 (let [resp (request {:request-method :get :uri "/get"})]
123 (is (= 200 (:status resp)))
124 (is (= "get" (slurp-body resp)))))
125
126 (deftest ^:integration makes-head-request
127 (run-server)
128 (let [resp (request {:request-method :head :uri "/head"})]
129 (is (= 200 (:status resp)))
130 (is (nil? (:body resp)))))
131
132 (deftest ^:integration sets-content-type-with-charset
133 (run-server)
134 (let [resp (client/request {:scheme :http
135 :server-name "localhost"
136 :server-port 18080
137 :request-method :get :uri "/content-type"
138 :content-type "text/plain"
139 :character-encoding "UTF-8"})]
140 (is (= "text/plain; charset=UTF-8" (:body resp)))))
141
142 (deftest ^:integration sets-content-type-without-charset
143 (run-server)
144 (let [resp (client/request {:scheme :http
145 :server-name "localhost"
146 :server-port 18080
147 :request-method :get :uri "/content-type"
148 :content-type "text/plain"})]
149 (is (= "text/plain" (:body resp)))))
150
151 (deftest ^:integration sets-arbitrary-headers
152 (run-server)
153 (let [resp (request {:request-method :get :uri "/header"
154 :headers {"x-my-header" "header-val"}})]
155 (is (= "header-val" (slurp-body resp)))))
156
157 (deftest ^:integration sends-and-returns-byte-array-body
158 (run-server)
159 (let [resp (request {:request-method :post :uri "/post"
160 :body (util/utf8-bytes "contents")})]
161 (is (= 200 (:status resp)))
162 (is (= "contents" (slurp-body resp)))))
163
164 (deftest ^:integration returns-arbitrary-headers
165 (run-server)
166 (let [resp (request {:request-method :get :uri "/get"})]
167 (is (string? (get-in resp [:headers "date"])))
168 (is (nil? (get-in resp [:headers "Date"])))))
169
170 (deftest ^:integration returns-status-on-exceptional-responses
171 (run-server)
172 (let [resp (request {:request-method :get :uri "/error"})]
173 (is (= 500 (:status resp)))))
174
175 (deftest ^:integration sets-socket-timeout
176 (run-server)
177 (try
178 (is (thrown? SocketTimeoutException
179 (client/request {:scheme :http
180 :server-name "localhost"
181 :server-port 18080
182 :request-method :get :uri "/timeout"
183 :socket-timeout 1})))))
184
185 (deftest ^:integration delete-with-body
186 (run-server)
187 (let [resp (request {:request-method :delete :uri "/delete-with-body"
188 :body (.getBytes "foo bar")})]
189 (is (= 200 (:status resp)))))
190
191 (deftest ^:integration self-signed-ssl-get
192 (let [server (ring/run-jetty handler
193 {:port 8081 :ssl-port 18082
194 :ssl? true
195 :join? false
196 :keystore "test-resources/keystore"
197 :key-password "keykey"})]
198 (try
199 (is (thrown? SunCertPathBuilderException
200 (client/request {:scheme :https
201 :server-name "localhost"
202 :server-port 18082
203 :request-method :get :uri "/get"})))
204 (let [resp (request {:request-method :get :uri "/get" :server-port 18082
205 :scheme :https :insecure? true})]
206 (is (= 200 (:status resp)))
207 (is (= "get" (String. (util/force-byte-array (:body resp))))))
208 (finally
209 (.stop server)))))
210
211 (deftest ^:integration multipart-form-uploads
212 (run-server)
213 (let [bytes (util/utf8-bytes "byte-test")
214 stream (ByteArrayInputStream. bytes)
215 resp (request {:request-method :post :uri "/multipart"
216 :multipart [{:name "a" :content "testFINDMEtest"
217 :encoding "UTF-8"
218 :mime-type "application/text"}
219 {:name "b" :content bytes
220 :mime-type "application/json"}
221 {:name "d"
222 :content (file "test-resources/keystore")
223 :encoding "UTF-8"
224 :mime-type "application/binary"}
225 {:name "c" :content stream
226 :mime-type "application/json"}
227 {:name "e" :part-name "eggplant"
228 :content "content"
229 :mime-type "application/text"}]})
230 resp-body (apply str (map #(try (char %) (catch Exception _ ""))
231 (util/force-byte-array (:body resp))))]
232 (is (= 200 (:status resp)))
233 (is (re-find #"testFINDMEtest" resp-body))
234 (is (re-find #"application/json" resp-body))
235 (is (re-find #"application/text" resp-body))
236 (is (re-find #"UTF-8" resp-body))
237 (is (re-find #"byte-test" resp-body))
238 (is (re-find #"name=\"c\"" resp-body))
239 (is (re-find #"name=\"d\"" resp-body))
240 (is (re-find #"name=\"eggplant\"" resp-body))
241 (is (re-find #"content" resp-body))))
242
243 (deftest ^:integration multipart-inputstream-length
244 (run-server)
245 (let [bytes (util/utf8-bytes "byte-test")
246 stream (ByteArrayInputStream. bytes)
247 resp (request {:request-method :post :uri "/multipart"
248 :multipart [{:name "c" :content stream :length 9
249 :mime-type "application/json"}]})
250 resp-body (apply str (map #(try (char %) (catch Exception _ ""))
251 (util/force-byte-array (:body resp))))]
252 (is (= 200 (:status resp)))
253 (is (re-find #"byte-test" resp-body))))
254
255 (deftest ^:integration t-save-request-obj
256 (run-server)
257 (let [resp (request {:request-method :post :uri "/post"
258 :body "foo bar"
259 :save-request? true
260 :debug-body true})]
261 (is (= 200 (:status resp)))
262 (is (= {:scheme :http
263 :http-url (localhost "/post")
264 :request-method :post
265 :save-request? true
266 :debug-body true
267 :uri "/post"
268 :server-name "localhost"
269 :server-port 18080
270 :body-content "foo bar"
271 :body-type String}
272 (dissoc (:request resp) :body :http-req)))
273 (is (instance? HttpPost (-> resp :request :http-req)))))
274
275 (deftest parse-headers
276 (are [headers expected]
277 (let [iterator (BasicHeaderIterator.
278 (into-array BasicHeader
279 (map (fn [[name value]]
280 (BasicHeader. name value))
281 headers)) nil)]
282 (is (= (core/parse-headers iterator) expected)))
283
284 [] {}
285
286 [["Set-Cookie" "one"]] {"set-cookie" "one"}
287
288 [["Set-Cookie" "one"] ["set-COOKIE" "two"]]
289 {"set-cookie" ["one" "two"]}
290
291 [["Set-Cookie" "one"] ["serVer" "some-server"] ["set-cookie" "two"]]
292 {"set-cookie" ["one" "two"] "server" "some-server"}))
293
294 (deftest ^:integration t-streaming-response
295 (run-server)
296 (let [stream (:body (request {:request-method :get :uri "/get" :as :stream}))
297 body (slurp stream)]
298 (is (= "get" body))))
299
300 (deftest throw-on-invalid-body
301 (is (thrown-with-msg? IllegalArgumentException #"Invalid request method :bad"
302 (client/request {:url "http://example.org"
303 :method :bad}))))
304
305 (deftest ^:integration throw-on-too-many-redirects
306 (run-server)
307 (let [resp (client/get (localhost "/redirect")
308 {:max-redirects 2 :throw-exceptions false})]
309 (is (= 302 (:status resp)))
310 (is (= (apply vector (repeat 3 "http://localhost:18080/redirect"))
311 (:trace-redirects resp))))
312 (is (thrown-with-msg? Exception #"Too many redirects: 3"
313 (client/get (localhost "/redirect")
314 {:max-redirects 2 :throw-exceptions true})))
315 (is (thrown-with-msg? Exception #"Too many redirects: 21"
316 (client/get (localhost "/redirect")
317 {:throw-exceptions true}))))
318
319 (deftest ^:integration get-with-body
320 (run-server)
321 (let [resp (request {:request-method :get :uri "/get-with-body"
322 :body (.getBytes "foo bar")})]
323 (is (= 200 (:status resp)))
324 (is (= "foo bar" (String. (util/force-byte-array (:body resp)))))))
325
326 (deftest ^:integration head-with-body
327 (run-server)
328 (let [resp (request {:request-method :head :uri "/head" :body "foo"})]
329 (is (= 200 (:status resp)))))
330
331 (deftest ^:integration t-clojure-output-coercion
332 (run-server)
333 (let [resp (client/get (localhost "/clojure") {:as :clojure})]
334 (is (= 200 (:status resp)))
335 (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}} (:body resp))))
336 (let [clj-resp (client/get (localhost "/clojure") {:as :auto})
337 edn-resp (client/get (localhost "/edn") {:as :auto})]
338 (is (= 200 (:status clj-resp) (:status edn-resp)))
339 (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
340 (:body clj-resp)
341 (:body edn-resp)))))
342
343 (deftest ^:integration t-transit-output-coercion
344 (run-server)
345 (let [transit-json-resp (client/get (localhost "/transit-json") {:as :auto})
346 transit-msgpack-resp (client/get (localhost "/transit-msgpack")
347 {:as :auto})]
348 (is (= 200
349 (:status transit-json-resp)
350 (:status transit-msgpack-resp)))
351 (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
352 (:body transit-json-resp)
353 (:body transit-msgpack-resp)))))
354
355 (deftest ^:integration t-json-output-coercion
356 (run-server)
357 (let [resp (client/get (localhost "/json") {:as :json})
358 resp-array (client/get (localhost "/json-array") {:as :json-strict})
359 resp-str (client/get (localhost "/json")
360 {:as :json :coerce :exceptional})
361 resp-str-keys (client/get (localhost "/json") {:as :json-string-keys})
362 resp-strict-str-keys (client/get (localhost "/json")
363 {:as :json-strict-string-keys})
364 resp-auto (client/get (localhost "/json") {:as :auto})
365 bad-resp (client/get (localhost "/json-bad")
366 {:throw-exceptions false :as :json})
367 bad-resp-json (client/get (localhost "/json-bad")
368 {:throw-exceptions false :as :json
369 :coerce :always})
370 bad-resp-json2 (client/get (localhost "/json-bad")
371 {:throw-exceptions false :as :json
372 :coerce :unexceptional})]
373 (is (= 200
374 (:status resp)
375 (:status resp-array)
376 (:status resp-str)
377 (:status resp-str-keys)
378 (:status resp-strict-str-keys)
379 (:status resp-auto)))
380 (is (= {:foo "bar"}
381 (:body resp)
382 (:body resp-auto)))
383 (is (= ["foo", "bar"]
384 (:body resp-array)))
385 (is (= {"foo" "bar"}
386 (:body resp-strict-str-keys)
387 (:body resp-str-keys)))
388 ;; '("foo" "bar") and ["foo" "bar"] compare as equal with =.
389 (is (vector? (:body resp-array)))
390 (is (= "{\"foo\":\"bar\"}" (:body resp-str)))
391 (is (= 400
392 (:status bad-resp)
393 (:status bad-resp-json)
394 (:status bad-resp-json2)))
395 (is (= "{\"foo\":\"bar\"}" (:body bad-resp))
396 "don't coerce on bad response status by default")
397 (is (= {:foo "bar"} (:body bad-resp-json)))
398 (is (= "{\"foo\":\"bar\"}" (:body bad-resp-json2)))))
399
400 (deftest ^:integration t-ipv6
401 (run-server)
402 (let [resp (client/get "http://[::1]:18080/get")]
403 (is (= 200 (:status resp)))
404 (is (= "get" (:body resp)))))
405
406 (deftest t-custom-retry-handler
407 (let [called? (atom false)]
408 (is (thrown? Exception
409 (client/post "http://localhost"
410 {:multipart [{:name "title" :content "Foo"}
411 {:name "Content/type"
412 :content "text/plain"}
413 {:name "file"
414 :content (file "/tmp/missingfile")}]
415 :retry-handler (fn [ex try-count http-context]
416 (reset! called? true)
417 false)})))
418 (is @called?)))
419
420 ;; super-basic test for methods that aren't used that often
421 (deftest ^:integration t-copy-options-move
422 (run-server)
423 (let [resp1 (client/options (localhost "/options"))
424 resp2 (client/move (localhost "/move"))
425 resp3 (client/copy (localhost "/copy"))
426 resp4 (client/patch (localhost "/patch"))]
427 (is (= #{200} (set (map :status [resp1 resp2 resp3 resp4]))))
428 (is (= "options" (:body resp1)))
429 (is (= "move" (:body resp2)))
430 (is (= "copy" (:body resp3)))
431 (is (= "patch" (:body resp4)))))
432
433 (deftest ^:integration t-json-encoded-form-params
434 (run-server)
435 (let [params {:param1 "value1" :param2 {:foo "bar"}}
436 resp (client/post (localhost "/post") {:content-type :json
437 :form-params params})]
438 (is (= 200 (:status resp)))
439 (is (= (json/encode params) (:body resp)))))
440
441 (deftest ^:integration t-request-interceptor
442 (run-server)
443 (let [req-ctx (atom [])
444 {:keys [status trace-redirects] :as resp}
445 (client/get
446 (localhost "/get")
447 {:request-interceptor
448 (fn [^HttpRequest req ^HttpContext ctx]
449 (reset! req-ctx {:method (.getMethod req) :uri (.getURI req)}))})]
450 (is (= 200 status))
451 (is (= "GET" (:method @req-ctx)))
452 (is (= "/get" (.getPath (:uri @req-ctx))))))
453
454
455 (deftest ^:integration t-response-interceptor
456 (run-server)
457 (let [saved-ctx (atom [])
458 {:keys [status trace-redirects] :as resp}
459 (client/get
460 (localhost "/redirect-to-get")
461 {:response-interceptor
462 (fn [^HttpResponse resp ^HttpContext ctx]
463 (let [^HttpInetConnection conn
464 (.getAttribute ctx ExecutionContext/HTTP_CONNECTION)]
465 (swap! saved-ctx conj {:remote-port (.getRemotePort conn)
466 :http-conn conn})))})]
467 (is (= 200 status))
468 (is (= 2 (count @saved-ctx)))
469 (is (= (count trace-redirects) (count @saved-ctx)))
470 (is (every? #(= 18080 (:remote-port %)) @saved-ctx))
471 (is (every? #(instance? HttpConnection (:http-conn %)) @saved-ctx))))
472
473 (deftest ^:integration t-send-input-stream-body
474 (run-server)
475 (let [b1 (:body (client/post "http://localhost:18080/post"
476 {:body (ByteArrayInputStream. (.getBytes "foo"))
477 :length 3}))
478 b2 (:body (client/post "http://localhost:18080/post"
479 {:body (ByteArrayInputStream.
480 (.getBytes "foo"))}))
481 b3 (:body (client/post "http://localhost:18080/post"
482 {:body (ByteArrayInputStream.
483 (.getBytes "apple"))
484 :length 2}))]
485 (is (= b1 "foo"))
486 (is (= b2 "foo"))
487 (is (= b3 "ap"))))
488
489 (deftest t-add-client-params
490 (testing "Using add-client-params!"
491 (let [ps {"http.conn-manager.timeout" 100
492 "http.socket.timeout" 250
493 "http.protocol.allow-circular-redirects" false
494 "http.protocol.version" HttpVersion/HTTP_1_0
495 "http.useragent" "clj-http"}
496 setps (.getParams (doto (DefaultHttpClient.)
497 (core/add-client-params! ps)))]
498 (doseq [[k v] ps]
499 (is (= v (.getParameter setps k)))))))
500
501 ;; Regression, get notified if something changes
502 (deftest ^:integration t-known-client-params-are-unchanged
503 (let [params ["http.socket.timeout" CoreConnectionPNames/SO_TIMEOUT
504 "http.connection.timeout"
505 CoreConnectionPNames/CONNECTION_TIMEOUT
506 "http.protocol.version" CoreProtocolPNames/PROTOCOL_VERSION
507 "http.useragent" CoreProtocolPNames/USER_AGENT
508 "http.conn-manager.timeout" ClientPNames/CONN_MANAGER_TIMEOUT
509 "http.protocol.allow-circular-redirects"
510 ClientPNames/ALLOW_CIRCULAR_REDIRECTS]]
511 (doseq [[plaintext constant] (partition 2 params)]
512 (is (= plaintext constant)))))
513
514 ;; If you don't explicitly set a :cookie-policy, use
515 ;; CookiePolicy/BROWSER_COMPATIBILITY
516 (deftest t-add-client-params-default-cookie-policy
517 (testing "Using add-client-params! to get a default cookie policy"
518 (let [setps (.getParams (doto (DefaultHttpClient.)
519 (core/add-client-params! {})))]
520 (is (= CookiePolicy/BROWSER_COMPATIBILITY
521 (.getParameter setps ClientPNames/COOKIE_POLICY))))))
522
523 ;; If you set a :cookie-policy, the name of the policy is registered
524 ;; as (str (type cookie-policy))
525 (deftest t-add-client-params-cookie-policy
526 (testing "Using add-client-params! to get an explicitly set :cookie-policy"
527 (let [setps (.getParams (doto (DefaultHttpClient.)
528 (core/add-client-params!
529 {:cookie-policy (constantly nil)})))]
530 (is (.startsWith ^String (.getParameter setps ClientPNames/COOKIE_POLICY)
531 "class ")))))
532
533
534 ;; This relies on connections to writequit.org being slower than 1ms, if this
535 ;; fails, you must have very nice internet.
536 (deftest ^:integration sets-conn-timeout
537 (run-server)
538 (try
539 (is (thrown? org.apache.http.conn.ConnectTimeoutException
540 (client/request {:scheme :http
541 :server-name "www.writequit.org"
542 :server-port 80
543 :request-method :get :uri "/"
544 :conn-timeout 1})))))
545
546 (deftest ^:integration connection-pool-timeout
547 (run-server)
548 (client/with-connection-pool {:timeout 1 :threads 1 :default-per-route 1}
549 (let [async-request #(future (client/request {:scheme :http
550 :server-name "localhost"
551 :server-port 18080
552 :request-method :get
553 :conn-timeout 1
554 :uri "/timeout"}))
555 is-pool-timeout-error?
556 (fn [req-fut]
557 (instance? org.apache.http.conn.ConnectionPoolTimeoutException
558 (try @req-fut (catch Exception e (.getCause e)))))
559 req1 (async-request)
560 req2 (async-request)
561 timeout-error1 (is-pool-timeout-error? req1)
562 timeout-error2 (is-pool-timeout-error? req2)]
563 (is (or timeout-error1 timeout-error2)))))
564
565 (deftest ^:integration t-header-collections
566 (run-server)
567 (let [headers (-> (client/get "http://localhost:18080/headers"
568 {:headers {"foo" ["bar" "baz"]
569 "eggplant" "quux"}})
570 :body
571 json/decode)]
572 (is (= {"eggplant" "quux" "foo" "bar,baz"}
573 (select-keys headers ["foo" "eggplant"])))))
574
575 (deftest ^:integration t-clojure-no-read-eval
576 (run-server)
577 (is (thrown? Exception (client/get (localhost "/clojure-bad") {:as :clojure}))
578 "Should throw an exception when reading clojure eval components"))
579
580 (deftest ^:integration t-numeric-headers
581 (run-server)
582 (client/request {:method :get :url (localhost "/get") :headers {"foo" 2}}))
583
584 ;; Currently failing, see: https://github.com/dakrone/clj-http/issues/257
585 ;; (deftest ^:integration t-empty-response-coercion
586 ;; (run-server)
587 ;; (let [resp (client/get (localhost "/empty") {:as :clojure})]
588 ;; (is (= (:body resp) ""))))
+166
-0
test/clj_http/test/headers.clj less more
0 (ns clj-http.test.headers
1 (:require [clj-http.client :as client]
2 [clj-http.headers :refer :all]
3 [clj-http.util :refer [lower-case-keys]]
4 [clojure.test :refer :all])
5 (:import (javax.servlet.http HttpServletRequest
6 HttpServletResponse)
7 (org.eclipse.jetty.server Request Server)
8 (org.eclipse.jetty.server.handler AbstractHandler)))
9
10 (deftest test-special-case
11 (are [expected given]
12 (is (= expected (special-case given)))
13 nil nil
14 "" ""
15 "foo" "foo"
16 "DNT" "dnt"
17 "P3P" "P3P"
18 "Content-MD5" "content-md5"))
19
20 (deftest test-canonicalize
21 (are [expected given]
22 (is (= expected (canonicalize given)))
23 nil nil
24 "" ""
25 "Date" :date
26 "Date" :DATE
27 "Foo-Bar-Baz" :foo-bar-baz
28 "Content-MD5" :content-md5))
29
30 (deftest test-normalize
31 (are [expected given]
32 (is (= expected (normalize given)))
33 nil nil
34 "" ""
35 "foo" "foo")
36 (is (= "foo"
37 (normalize "foo")
38 (normalize :foo)
39 (normalize "Foo")
40 (normalize :FOO))))
41
42 (deftest test-assoc-join
43 (is (= {:foo "1"} (assoc-join {} :foo "1")))
44 (is (= {:foo "1"} (assoc-join {:foo nil} :foo "1")))
45 (is (= {:foo ["1" "2"]} (assoc-join {:foo "1"} :foo "2")))
46 (is (= {:foo ["1" "2" "3"]} (assoc-join {:foo ["1" "2"]} :foo "3"))))
47
48 (deftest test-header-map
49 (let [m (header-map :foo "bar" "baz" "quux")
50 m2 (assoc m :ham "eggs")]
51 (is (= "bar"
52 (:foo m)
53 (:FOO m)
54 (m :foo)
55 (m "foo")
56 (m "FOO")
57 (get m "foo")))
58 (is (= {"baz" "quux"}
59 (dissoc m :foo)
60 (dissoc m "foo")))
61 (is (= #{"Foo" "baz"} (set (keys m))))
62 (is (= #{"Foo" "Ham" "baz"} (set (keys m2))))
63 (is (= "eggs" (m2 "ham")))
64 (is (= "nope" (get m2 "absent" "nope")))
65 (is (= "baz" (:foo (merge (header-map :foo "bar")
66 {"Foo" "baz"}))))
67 (let [m-with-meta (with-meta m {:withmeta-test true})]
68 (is (= (:withmeta-test (meta m-with-meta)) true)))))
69
70 (deftest test-empty
71 (testing "an empty header-map is a header-map"
72 (let [m (header-map :foo :bar)]
73 (is (= (class m)
74 (class (empty m)))))))
75
76 (defn ^Server header-server
77 "fixture server that copies all request headers into the response as
78 response headers"
79 []
80 ;; argh, we can't use ring for this, because it lowercases headers
81 ;; on the server side, and we explicitly want to get back the
82 ;; headers as they are. so we'll just use jetty directly, nbd.
83 (doto (Server. 18181)
84 (.setHandler (proxy [AbstractHandler] []
85 (handle [target
86 ^Request base-request
87 ^HttpServletRequest request
88 ^HttpServletResponse response]
89 (.setHandled base-request true)
90 (.setStatus response 200)
91 ;; copy over request headers verbatim
92 (doseq [n (enumeration-seq (.getHeaderNames request))]
93 (doseq [v (enumeration-seq (.getHeaders request n))]
94 ;; (println n v) ;; useful for debugging
95 (.addHeader response n v)))
96 ;; add a response header of our own in known case
97 (.addHeader response "Echo-Server" "Says Hi!")
98 (.. response getWriter (print "Echo!")))))
99 (.start)))
100
101 (deftest ^:integration test-wrap-header-map
102 (let [server (header-server)]
103 (try
104 (let [headers {:foo "bar"
105 :etag "some etag"
106 :content-md5 "some md5"
107 :multi ["value1" "value2"]
108 "MySpecialCase" "something"}
109 resp (client/get "http://localhost:18181" {:headers headers})
110 resp-headers (:headers resp)]
111 (testing "basic sanity checks"
112 (is (= "Echo!" (:body resp)))
113 (is (= "Says Hi!" (:echo-server resp-headers)))
114 ;; was everything copied over correctly
115 (doseq [[k v] headers]
116 (is (= v (resp-headers k)))))
117 (testing "foo is available as a variety of names"
118 (is (= "bar"
119 (:foo resp-headers)
120 (resp-headers "foo")
121 (resp-headers "Foo"))))
122 (testing "header case is preserved"
123 (let [resp-headers (into {} resp-headers)] ;; no more magic
124 (testing "keyword request headers are canonicalized"
125 (is (= "bar" (resp-headers "Foo")))
126 (is (= "some etag" (resp-headers "ETag")))
127 (is (= "some md5" (resp-headers "Content-MD5")))
128 (is (= ["value1" "value2"] (resp-headers "Multi"))))
129 (testing "strings are as written"
130 (is (= "something" (resp-headers "MySpecialCase")))))))
131 (finally
132 (.stop server)))))
133
134 (defmacro without-header-map [& body]
135 `(client/with-middleware '~(->> client/default-middleware
136 (list* client/wrap-lower-case-headers)
137 (remove #(= wrap-header-map %))
138 (vec))
139 ~@body))
140
141 (deftest ^:integration test-dont-wrap-header-map
142 (let [server (header-server)]
143 (try
144 (let [headers {"foo" "bar"
145 "etag" "some etag"
146 "content-md5" "some md5"
147 "multi" ["value1" "value2"]
148 "MySpecialCase" "something"}
149 resp (without-header-map
150 (client/get "http://localhost:18181" {:headers headers}))
151 resp-headers (:headers resp)]
152 (testing "basic sanity checks"
153 (is (= "Echo!" (:body resp)))
154 ;; was everything copied over correctly
155 (doseq [[k v] (lower-case-keys headers)]
156 (is (= v (resp-headers k)))))
157 (testing "header names are all lowercase"
158 (is (= "bar" (resp-headers "foo")))
159 (is (= "some etag" (resp-headers "etag")))
160 (is (= "some md5" (resp-headers "content-md5")))
161 (is (= ["value1" "value2"] (resp-headers "multi")))
162 (is (= "something" (resp-headers "myspecialcase")))
163 (is (= "Says Hi!" (resp-headers "echo-server")))))
164 (finally
165 (.stop server)))))
+38
-0
test/clj_http/test/links.clj less more
0 (ns clj-http.test.links
1 (:require [clj-http.links :refer :all]
2 [clojure.test :refer :all]))
3
4 (defn- link-handler [link-header]
5 (wrap-links (constantly {:headers {"link" link-header}})))
6
7 (deftest test-wrap-links
8 (testing "absolute link"
9 (let [handler (link-handler "<http://example.com/page2.html>; rel=next")]
10 (is (= (:links (handler {}))
11 {:next {:href "http://example.com/page2.html"}}))))
12 (testing "relative link"
13 (let [handler (link-handler "</page2.html>;rel=next")]
14 (is (= (:links (handler {}))
15 {:next {:href "/page2.html"}}))))
16 (testing "extra params"
17 (let [handler (link-handler "</page2.html>; rel=next; title=\"Page 2\"")]
18 (is (= (:links (handler {}))
19 {:next {:href "/page2.html", :title "Page 2"}}))))
20 (testing "multiple headers"
21 (let [handler (link-handler "</p1>;rel=prev, </p3>;rel=next,</>;rel=home")]
22 (is (= (:links (handler {}))
23 {:prev {:href "/p1"}
24 :next {:href "/p3"}
25 :home {:href "/"}}))))
26 (testing "no :links key if no link headers"
27 (let [handler (wrap-links (constantly {:headers {}}))
28 response (handler {})]
29 (is (not (contains? response :links))))))
30
31 (deftest t-multiple-link-headers
32 (let [handler (link-handler ["<http://tmblr.co/Zl_A>; rel=shorturl"
33 "<http://25.media.com/foo.png>; rel=icon"])
34 resp (handler {})]
35 (is (= (:links resp)
36 {:shorturl {:href "http://tmblr.co/Zl_A"}
37 :icon {:href "http://25.media.com/foo.png"}}))))
+163
-0
test/clj_http/test/multipart.clj less more
0 (ns clj-http.test.multipart
1 (:require [clj-http.multipart :refer :all]
2 [clojure.test :refer :all])
3 (:import (java.io File ByteArrayOutputStream ByteArrayInputStream)
4 (org.apache.http.entity.mime.content FileBody StringBody ContentBody ByteArrayBody InputStreamBody)
5 (java.nio.charset Charset)))
6
7 (defn body-str [^StringBody body]
8 (-> body .getReader slurp))
9
10 (defn body-bytes [^ContentBody body]
11 (let [buf (ByteArrayOutputStream.)]
12 (.writeTo body buf)
13 (.toByteArray buf)))
14
15 (defn body-charset [^ContentBody body]
16 (-> body .getContentType .getCharset))
17
18 (defn body-mime-type [^ContentBody body]
19 (-> body .getContentType .getMimeType))
20
21 (defn make-input-stream [& bytes]
22 (ByteArrayInputStream. (byte-array bytes)))
23
24 (deftest test-multipart-body
25 (testing "nil content throws exception"
26 (is (thrown-with-msg? Exception #"Multipart content cannot be nil"
27 (make-multipart-body {:content nil}))))
28
29 (testing "unsupported content type throws exception"
30 (is (thrown-with-msg? Exception #"Unsupported type for multipart content: class java.lang.Object"
31 (make-multipart-body {:content (Object.)}))))
32
33 (testing "ContentBody content direct usage"
34 (let [contentBody (StringBody. "abc")]
35 (is (identical? contentBody (make-multipart-body {:content contentBody})))))
36
37 (testing "StringBody"
38
39 (testing "can create StringBody with content only"
40 (let [body (make-multipart-body {:content "abc"})]
41 (is (instance? StringBody body))
42 (is (= "abc" (body-str body)))))
43
44 (testing "can create StringBody with content and encoding"
45 (let [body (make-multipart-body {:content "abc" :encoding "ascii"})]
46 (is (instance? StringBody body))
47 (is (= "abc" (body-str body)))
48 (is (= (Charset/forName "ascii") (body-charset body)))))
49
50 (testing "can create StringBody with content and mime-type and encoding"
51 (let [body (make-multipart-body {:content "abc" :mime-type "stream-body" :encoding "ascii"})]
52 (is (instance? StringBody body))
53 (is (= "abc" (body-str body)))
54 (is (= (Charset/forName "ascii") (body-charset body)))
55 (is (= "stream-body" (body-mime-type body))))))
56
57 (testing "ByteArrayBody"
58
59 (testing "exception thrown on missing name"
60 (is (thrown-with-msg? Exception #"Multipart byte array body must contain at least :content and :name"
61 (make-multipart-body {:content (byte-array [0 1 2])}))))
62
63 (testing "can create ByteArrayBody with name only"
64 (let [body (make-multipart-body {:content (byte-array [0 1 2]) :name "testname"})]
65 (is (instance? ByteArrayBody body))
66 (is (= "testname" (.getFilename body)))
67 (is (= [0 1 2] (vec (body-bytes body))))))
68
69 (testing "can create ByteArrayBody with name and mime-type"
70 (let [body (make-multipart-body {:content (byte-array [0 1 2])
71 :name "testname"
72 :mime-type "byte-body"})]
73 (is (instance? ByteArrayBody body))
74 (is (= "testname" (.getFilename body)))
75 (is (= "byte-body" (body-mime-type body)))
76 (is (= [0 1 2] (vec (body-bytes body)))))))
77
78 (testing "InputStreamBody"
79
80 (testing "exception thrown on missing name"
81 (is (thrown-with-msg?
82 Exception
83 #"Multipart input stream body must contain at least :content and :name"
84 (make-multipart-body {:content (ByteArrayInputStream. (byte-array [0 1 2]))}))))
85
86 (testing "can create InputStreamBody with name and content"
87 (let [input-stream (make-input-stream 1 2 3)
88 body (make-multipart-body {:content input-stream
89 :name "testname"})]
90 (is (instance? InputStreamBody body))
91 (is (= "testname" (.getFilename body)))
92 (is (identical? input-stream (.getInputStream body)))))
93
94 (testing "can create InputStreamBody with name, content and mime-type"
95 (let [input-stream (make-input-stream 1 2 3)
96 body (make-multipart-body {:content input-stream
97 :name "testname"
98 :mime-type "input-stream-body"})]
99 (is (instance? InputStreamBody body))
100 (is (= "testname" (.getFilename body)))
101 (is (= "input-stream-body" (body-mime-type body)))
102 (is (identical? input-stream (.getInputStream body)))))
103
104 (testing "can create input InputStreamBody name, content, mime-type and length"
105 (let [input-stream (make-input-stream 1 2 3)
106 body (make-multipart-body {:content input-stream
107 :name "testname"
108 :mime-type "input-stream-body"
109 :length 42})]
110 (is (instance? InputStreamBody body))
111 (is (= "testname" (.getFilename body)))
112 (is (= "input-stream-body" (body-mime-type body)))
113 (is (identical? input-stream (.getInputStream body)))
114 (is (= 42 (.getContentLength body))))))
115
116 (testing "FileBody"
117
118 (testing "can create FileBody with content only"
119 (let [test-file (File. "testfile")
120 body (make-multipart-body {:content test-file})]
121 (is (instance? FileBody body))
122 (is (= test-file (.getFile body)))))
123
124 (testing "can create FileBody with content and mime-type"
125 (let [test-file (File. "testfile")
126 body (make-multipart-body {:content test-file
127 :mime-type "file-body"})]
128 (is (instance? FileBody body))
129 (is (= "file-body" (body-mime-type body)))
130 (is (= test-file (.getFile body)))))
131
132 (testing "can create FileBody with content, mime-type and name"
133 (let [test-file (File. "testfile")
134 body (make-multipart-body {:content test-file
135 :mime-type "file-body"
136 :name "testname"})]
137 (is (instance? FileBody body))
138 (is (= "file-body" (body-mime-type body)))
139 (is (= test-file (.getFile body)))
140 (is (= "testname" (.getFilename body)))))
141
142 (testing "can create FileBody with content and mime-type and encoding"
143 (let [test-file (File. "testfile")
144 body (make-multipart-body {:content test-file
145 :mime-type "file-body"
146 :encoding "ascii"})]
147 (is (instance? FileBody body))
148 (is (= "file-body" (body-mime-type body)))
149 (is (= (Charset/forName "ascii") (body-charset body)))
150 (is (= test-file (.getFile body)))))
151
152 (testing "can create FileBody with content, mime-type, encoding and name"
153 (let [test-file (File. "testfile")
154 body (make-multipart-body {:content test-file
155 :mime-type "file-body"
156 :encoding "ascii"
157 :name "testname"})]
158 (is (instance? FileBody body))
159 (is (= "file-body" (body-mime-type body)))
160 (is (= (Charset/forName "ascii") (body-charset body)))
161 (is (= test-file (.getFile body) ))
162 (is (= "testname" (.getFilename body)))))))
+43
-0
test/clj_http/test/util.clj less more
0 (ns clj-http.test.util
1 (:require [clj-http.util :refer :all]
2 [clojure.test :refer :all]))
3
4 (deftest test-lower-case-keys
5 (are [map expected]
6 (is (= expected (lower-case-keys map)))
7 nil nil
8 {} {}
9 {"Accept" "application/json"} {"accept" "application/json"}
10 {"X" {"Y" "Z"}} {"x" {"y" "Z"}}))
11
12 (deftest t-option-retrieval
13 (is (= (opt {:thing? true :thing true} :thing) true))
14 (is (= (opt {:thing? false :thing true} :thing) false))
15 (is (= (opt {:thing? false :thing false} :thing) false))
16 (is (= (opt {:thing? true :thing nil} :thing) true))
17 (is (= (opt {:thing? nil :thing true} :thing) true))
18 (is (= (opt {:thing? false :thing nil} :thing) false))
19 (is (= (opt {:thing? nil :thing false} :thing) false))
20 (is (= (opt {:thing? nil :thing nil} :thing) nil))
21 (is (= (opt {:thing? :a :thing nil} :thing) :a)))
22
23 (deftest test-parse-content-type
24 (are [s expected]
25 (is (= expected (parse-content-type s)))
26 nil nil
27 "" nil
28 "application/json"
29 {:content-type :application/json
30 :content-type-params {}}
31 " application/json "
32 {:content-type :application/json
33 :content-type-params {}}
34 "application/json; charset=UTF-8"
35 {:content-type :application/json
36 :content-type-params {:charset "UTF-8"}}
37 " application/json; charset=UTF-8 "
38 {:content-type :application/json
39 :content-type-params {:charset "UTF-8"}}
40 "text/html; charset=ISO-8859-4"
41 {:content-type :text/html
42 :content-type-params {:charset "ISO-8859-4"}}))
+24
-0
test/header-html5-test.html less more
0 <!DOCTYPE html PUBLIC "-/W3C/DTD XHTML 1.0 Transitional/EN"
1 "http:/www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja" lang="ja">
3
4 <head>
5 <!--// title_start //-->
6 <title>titletext</title>
7 <!--// title_end //-->
8 <meta charset="UTF-8" />
9 <link rel="stylesheet" type="text/css" href="/styles/Y3home.css" />
10 <script type="text/javascript" src="/scripts/Y3.js"></script>
11 <link rel="stylesheet" type="text/css" href="/styles/common_print.css" media="print" />
12 <meta name="description" content="foo" />
13 <meta name="keywords" content="bar" />
14
15 <script type="text/javascript">foo</script>
16
17 <meta property="og:image" content="foo" />
18 <link rel="mixi-check-image" type="image/jpeg" href="bar.jpg" />
19 </head>
20 <body id="d1-news">
21 This is the body
22 </body>
23 </html>
+26
-0
test/header-test.html less more
0 <!DOCTYPE html PUBLIC "-/W3C/DTD XHTML 1.0 Transitional/EN"
1 "http:/www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
2 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="ja" lang="ja">
3
4 <head>
5 <!--// title_start //-->
6 <title>titletext</title>
7 <!--// title_end //-->
8 <meta http-equiv="Content-Type" content="text/html; charset=Shift_JIS" />
9 <meta http-equiv="Content-Style-Type" content="text/css" />
10 <meta http-equiv="Content-Script-Type" content="text/javascript" />
11 <link rel="stylesheet" type="text/css" href="/styles/Y3home.css" />
12 <script type="text/javascript" src="/scripts/Y3.js"></script>
13 <link rel="stylesheet" type="text/css" href="/styles/common_print.css" media="print" />
14 <meta name="description" content="foo" />
15 <meta name="keywords" content="bar" />
16
17 <script type="text/javascript">foo</script>
18
19 <meta property="og:image" content="foo" />
20 <link rel="mixi-check-image" type="image/jpeg" href="bar.jpg" />
21 </head>
22 <body id="d1-news">
23 This is the body
24 </body>
25 </html>
+3
-0
test/jetty-logging.properties less more
0 # quiet down jetty's logging
1 org.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog
2 org.eclipse.jetty.LEVEL=WARN
+26
-0
test/log4j.properties less more
0 #############
1 # Appenders #
2 #############
3
4 # standard out appender
5 log4j.appender.C = org.apache.log4j.ConsoleAppender
6 log4j.appender.C.layout = org.apache.log4j.PatternLayout
7 log4j.appender.C.layout.ConversionPattern = %d | ES | %-5p | [%t] | %c | %m%n
8
9 # daily rolling file appender
10 log4j.appender.F = org.apache.log4j.FileAppender
11 log4j.appender.F.File = http.log
12 log4j.appender.F.Append = true
13 log4j.appender.F.layout = org.apache.log4j.PatternLayout
14 log4j.appender.F.layout.ConversionPattern = %d | CLJ-HTTP | %-5p | [%t] | %c | %m%n
15
16 ###########
17 # Loggers #
18 ###########
19
20 # default
21 log4j.rootLogger = DEBUG, F
22
23 # Things
24 log4j.logger.org.apache.http = DEBUG
25 log4j.logger.org.apache.http.wire = INFO
test-resources/client-keystore less more
Binary diff not shown
test-resources/keystore less more
Binary diff not shown