New Upstream Snapshot - clj-http-clojure
Ready changes
Summary
Merged new upstream version: 3.12.3+git20221108.0.1cc8be6 (was: 3.12.3+git20221003.1.e3acd44).
Resulting package
Built on 2023-01-18T23:57 (took 20m25s)
The resulting binary packages can be installed (if you have the apt repository enabled) by running one of:
apt install -t fresh-snapshots libclj-http-clojure
Diff
diff --git a/.gitignore b/.gitignore
deleted file mode 100644
index 7d6f982..0000000
--- a/.gitignore
+++ /dev/null
@@ -1,40 +0,0 @@
-# leiningen .gitignore defaults
-/target
-/classes
-/checkouts
-pom.xml
-pom.xml.asc
-*.jar
-*.class
-/.lein-*
-/.nrepl-port
-
-# custom from here on out
-build
-lib
-*.dot
-
-# use glob syntax.
-syntax: glob
-creds.clj
-Manifest.txt
-aws.clj
-*.ser
-*~
-*.bak
-*.off
-*.old
-.DS_Store
-*.#*
-*#*
-*.classpath
-*.project
-*.settings
-*.pyc
-docs/*
-doc
-http.log
-
-# Intellij Idea
-/*.iml
-/.idea
diff --git a/.travis.yml b/.travis.yml
deleted file mode 100644
index d94c4af..0000000
--- a/.travis.yml
+++ /dev/null
@@ -1,9 +0,0 @@
-language: clojure
-lein: lein2
-script: lein2 all do clean, test
-branches:
- only:
- - master
-jdk:
- - openjdk7
- - oraclejdk7
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 0000000..471ab3c
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,15 @@
+# Contributing Guidelines
+
+First, thanks for the contributing! Hopefully you find it fairly painless, but
+in the interest of explanation, here are some things you might be interested in
+when contributing code:
+
+- Please run the tests locally if you submit a change, you can use `lein all
+ test :all` to ensure that they pass locally
+- If you're able, adding tests with a PR is fantastic! If not, no worries, I can
+ add those later
+- Don't hesitate to ask if you have questions, use `@dakrone` or you can email
+ me (if it's something you can't talk about publically) at `lee [at]
+ writequit.org`
+
+That's it, thanks for using and contributing to clj-http!
diff --git a/README.org b/README.org
index f5ee89d..50b98b0 100644
--- a/README.org
+++ b/README.org
@@ -7,68 +7,99 @@
#+HTML_HEAD: <style type="text/css"> body {margin-right:15%; margin-left:15%;} </style>
#+LANGUAGE: en
-* Table of Contents :TOC:
+[[https://clojars.org/clj-http][file:https://img.shields.io/clojars/v/clj-http.svg]] [[https://github.com/dakrone/clj-http/actions?query=workflow%3A%22Clojure+CI%22][file:https://github.com/dakrone/clj-http/workflows/Clojure%20CI/badge.svg]] [[https://gitter.im/clj-http/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge][file:https://badges.gitter.im/clj-http/Lobby.svg]]
+
+* Table of Contents :TOC_3:
+:PROPERTIES:
+:CUSTOM_ID: h-aaf075ea-2f0e-4a45-871a-0f89c838fb4b
+:END:
+- [[#branches][Branches]]
+- [[#introduction][Introduction]]
+ - [[#overview][Overview]]
+ - [[#philosophy][Philosophy]]
+- [[#installation][Installation]]
+- [[#quickstart][Quickstart]]
+ - [[#head][HEAD]]
+ - [[#get][GET]]
+ - [[#put][PUT]]
+ - [[#post][POST]]
+ - [[#delete][DELETE]]
+ - [[#async-http-request][Async HTTP Request]]
+ - [[#cancelling-requests][Cancelling Requests]]
+ - [[#coercions][Coercions]]
+ - [[#input-coercion][Input coercion]]
+ - [[#output-coercion][Output coercion]]
+ - [[#headers][Headers]]
+ - [[#query-string-parameters][Query-string parameters]]
+ - [[#meta-tag-headers][Meta Tag Headers]]
+ - [[#link-headers][Link Headers]]
+ - [[#redirects][Redirects]]
+ - [[#how-to-create-a-custom-redirectstrategy][How to create a custom RedirectStrategy]]
+ - [[#cookies][Cookies]]
+ - [[#cookiestores][Cookiestores]]
+ - [[#keystores-trust-stores][Keystores, Trust-stores]]
+ - [[#exceptions][Exceptions]]
+ - [[#decompression][Decompression]]
+ - [[#debugging][Debugging]]
+ - [[#logging][Logging]]
+- [[#caching][Caching]]
+- [[#authentication][Authentication]]
+ - [[#basic-auth][Basic Auth]]
+ - [[#digest-auth][Digest Auth]]
+ - [[#ntlm-auth][NTLM Auth]]
+ - [[#oauth2][oAuth2]]
+- [[#advanced-usage][Advanced Usage]]
+ - [[#raw-request][Raw Request]]
+ - [[#boolean-options][Boolean options]]
+ - [[#persistent-connections][Persistent Connections]]
+ - [[#re-using-httpclient-between-requests][Re-using =HttpClient= between requests]]
+ - [[#proxies][Proxies]]
+ - [[#custom-middleware][Custom Middleware]]
+ - [[#modifying-apache-specific-features-of-the-httpclientbuilder-and-httpasyncclientbuilder][Modifying Apache-specific features of the =HttpClientBuilder= and =HttpAsyncClientBuilder=]]
+ - [[#incrementally-json-parsing][Incrementally JSON Parsing]]
+ - [[#dns-resolution][DNS Resolution]]
+- [[#development][Development]]
+ - [[#faking-responses][Faking Responses]]
+ - [[#optional-dependencies][Optional Dependencies]]
+ - [[#clj-http-lite][clj-http-lite]]
+ - [[#troubleshooting][Troubleshooting]]
+ - [[#verifyerror-class-orgcodehausjacksonsmilesmileparser-overrides-final-method-getbinaryvalue][VerifyError class org.codehaus.jackson.smile.SmileParser overrides final method getBinaryValue...]]
+ - [[#nohttpresponseexception--due-to-stale-connections][NoHttpResponseException ... due to stale connections**]]
+- [[#tests][Tests]]
+- [[#testimonials][Testimonials]]
+- [[#other-libraries-providing-middleware][Other Libraries Providing Middleware]]
+- [[#license][License]]
+
+* Branches
:PROPERTIES:
-:CUSTOM_ID: h:84c64317-1dfa-4955-bfa7-180745c31546
+:CUSTOM_ID: h-e390585c-cbd8-4e94-b36b-4e9c27c16720
:END:
- - [[#introduction][Introduction]]
- - [[#overview][Overview]]
- - [[#philosophy-][Philosophy ]]
- - [[#installation][Installation]]
- - [[#quickstart][Quickstart]]
- - [[#head][HEAD]]
- - [[#get][GET]]
- - [[#put][PUT]]
- - [[#post][POST]]
- - [[#delete][DELETE]]
- - [[#coercions][Coercions]]
- - [[#headers][Headers]]
- - [[#meta-tag-headers][Meta Tag Headers]]
- - [[#link-headers][Link Headers]]
- - [[#redirects-][Redirects ]]
- - [[#cookies][Cookies]]
- - [[#exceptions][Exceptions]]
- - [[#decompression][Decompression]]
- - [[#debugging][Debugging]]
- - [[#authentication][Authentication]]
- - [[#basic-auth][Basic Auth]]
- - [[#digest-auth][Digest Auth]]
- - [[#oauth2][oAuth2]]
- - [[#advanced-usage][Advanced Usage]]
- - [[#raw-request][Raw Request]]
- - [[#persistent-connections][Persistent Connections]]
- - [[#proxies][Proxies]]
- - [[#custom-middleware][Custom Middleware]]
- - [[#development][Development]]
- - [[#faking-responses][Faking Responses]]
- - [[#optional-dependencies][Optional Dependencies]]
- - [[#clj-http-lite][clj-http-lite]]
- - [[#troubleshooting][Troubleshooting]]
- - [[#tests][Tests]]
- - [[#testimonials][Testimonials]]
- - [[#license][License]]
+
+There are branches for the major version numbers:
+
+- 2.x (no longer maintained except for security issues)
+- 3.x (current stable releases and the main Github branch)
+- master (which is 4.x, unreleased, based on version 5 of the Apache HTTP Client)
* Introduction
:PROPERTIES:
-:CUSTOM_ID: h:5caf5111-96b3-401b-bba3-6b66cc625cbd
+:CUSTOM_ID: h-d893078a-b20b-4086-9272-3d9c28c86846
:END:
** Overview
:PROPERTIES:
-:CUSTOM_ID: h:301e4e08-cd19-4066-888d-166f35d3f696
+:CUSTOM_ID: h-d8b17d06-124e-44fd-9c86-0399f39b0254
:END:
clj-http is an HTTP library wrapping the [[http://hc.apache.org/][Apache HttpComponents]] client. This
library has taken over from mmcgrana's clj-http.
-[[https://secure.travis-ci.org/dakrone/clj-http.png]]
-
** Philosophy
:PROPERTIES:
-:CUSTOM_ID: h:b8fdaffd-1f9f-4d08-93f7-ef50392e7af1
+:CUSTOM_ID: h-aa21d07d-333b-4ff2-93a9-ffdca31d8949
:END:
-The design of =clj-http= is inspired by the [[http://github.com/mmcgrana/ring][Ring]] protocol for Clojure HTTP
+The design of =clj-http= is inspired by the [[https://github.com/ring-clojure/ring][Ring]] protocol for Clojure HTTP
server applications.
The client in =clj-http.core= makes HTTP requests according to a given Ring
@@ -80,7 +111,7 @@ function.
* Installation
:PROPERTIES:
-:CUSTOM_ID: h:280b3315-2b20-484d-962b-7f7132d20840
+:CUSTOM_ID: h-ddfce0e2-6797-4774-add5-d5cf5bfaaa17
:END:
=clj-http= is available as a Maven artifact from [[http://clojars.org/clj-http][Clojars]].
@@ -88,20 +119,21 @@ function.
With Leiningen/Boot:
#+BEGIN_SRC clojure
-[clj-http "2.3.0"]
+[clj-http "3.12.3"]
#+END_SRC
-The previous major versions is available as:
+If you need an older version, a 2.x release is also available.
#+BEGIN_SRC clojure
-[clj-http "1.1.2"]
+[clj-http "2.3.0"]
#+END_SRC
-clj-http supports clojure 1.6.0 and higher.
+clj-http 3.x supports clojure 1.6.0 and higher.
+clj-http 4.x will support clojure 1.7.0 and higher.
* Quickstart
:PROPERTIES:
-:CUSTOM_ID: h:67644772-7a82-451b-91b8-6cab871445b6
+:CUSTOM_ID: h-65f0132e-1f96-4711-a84e-973817f37dd3
:END:
The main HTTP client functionality is provided by the =clj-http.client= namespace.
@@ -125,78 +157,128 @@ response maps]]:
** HEAD
:PROPERTIES:
-:CUSTOM_ID: h:5db60716-9834-4658-9256-63732e69bed6
+:CUSTOM_ID: h-79d1bb5f-c695-46a6-af4e-a64ca599c978
:END:
#+BEGIN_SRC clojure
(client/head "http://example.com/resource")
-(client/head "http://site.com/resource" {:accept :json})
+(client/head "http://example.com/resource" {:accept :json})
#+END_SRC
** GET
:PROPERTIES:
-:CUSTOM_ID: h:b2c26b5c-a36a-4f65-9c70-5d9921a2390d
+:CUSTOM_ID: h-89c164fb-85c2-4953-a8c4-a50867adf42a
:END:
Example requests:
#+BEGIN_SRC clojure
-(client/get "http://site.com/resources/id")
+(client/get "http://example.com/resources/id")
-(client/get "http://site.com/resources/3" {:accept :json})
+;; Setting options
+(client/get "http://example.com/resources/3" {:accept :json})
+(client/get "http://example.com/resources/3" {:accept :json :query-params {"q" "foo, bar"}})
;; Specifying headers as either a string or collection:
(client/get "http://example.com"
- {:headers {"foo" ["bar" "baz"], "eggplant" "quux"}})
+ {:headers {"foo" ["bar" "baz"], "eggplant" "quux"}})
;; Using either string or keyword header names:
(client/get "http://example.com"
- {:headers {:foo ["bar" "baz"], :eggplant "quux"}})
+ {:headers {:foo ["bar" "baz"], :eggplant "quux"}})
-;; Set any specific client parameters manually:
+;; Completely ignore cookies:
+(client/post "http://example.com" {:cookie-policy :none})
+;; There are also multiple ways to handle cookies
+(client/post "http://example.com" {:cookie-policy :default})
+(client/post "http://example.com" {:cookie-policy :netscape})
+(client/post "http://example.com" {:cookie-policy :standard})
+(client/post "http://example.com" {:cookie-policy :standard-strict})
+
+;; Cookies can be completely configurable with a custom spec by adding a
+;; function to return a cookie spec for parsing the cookie. For example, if you
+;; wanted to configure a spec provider to have a certain compatibility level:
(client/post "http://example.com"
- {:client-params {"http.protocol.allow-circular-redirects" false
- "http.protocol.version" HttpVersion/HTTP_1_0
- "http.useragent" "clj-http"}})
-
-;; Set your own cookie policy
+ {:cookie-spec
+ (fn [http-context]
+ (println "generating a new cookie spec")
+ (.create
+ (org.apache.http.impl.cookie.RFC6265CookieSpecProvider.
+ org.apache.http.impl.cookie.RFC6265CookieSpecProvider$CompatibilityLevel/IE_MEDIUM_SECURITY
+ (PublicSuffixMatcherLoader/getDefault))
+ http-context))})
+;; Or a version with relaxed compatibility
(client/post "http://example.com"
- {:client-params {:cookie-policy (fn [cookie origin] (your-validation cookie origin))}})
-
-;; Completely ignore cookies:
+ {:cookie-spec
+ (fn [http-context]
+ (println "generating a new cookie spec")
+ (.create
+ (org.apache.http.impl.cookie.RFC6265CookieSpecProvider.
+ org.apache.http.impl.cookie.RFC6265CookieSpecProvider$CompatibilityLevel/RELAXED
+ (PublicSuffixMatcherLoader/getDefault))
+ http-context))})
+
+;; Sometimes you want to do your own validation or something, which you can do
+;; by proxying the CookieSpecBase. Note that this doesn't actually return the
+;; cookies, because clj-http does its own cookie parsing. If you want to store
+;; the cookies from these methods you'll need to use a cookie store or put it in
+;; some datastructure yourself.
+(client/post "http://example.com"
+ {:cookie-spec
+ (fn [http-context]
+ (proxy [org.apache.http.impl.cookie.CookieSpecBase] []
+ ;; Version and version header
+ (getVersion [] 0)
+ (getVersionHeader [] nil)
+ ;; parse headers into cookie objects
+ (parse [header cookie-origin] (java.util.ArrayList.))
+ ;; Validate a cookie, throwing MalformedCookieException if the
+ ;; cookies isn't valid
+ (validate [cookie cookie-origin]
+ (println "validating:" cookie))
+ ;; Determine if a cookie matches the target location
+ (match [cookie cookie-origin] true)
+ ;; Format a list of cookies into a list of headers
+ (formatCookies [cookies] (java.util.ArrayList.))))})
+
+;; If you have created your own registry for cookie policies, you can provide
+;; :cookie-policy-registry to use it. See
+;; clj-http.core/create-custom-cookie-policy-registry for an example of a custom
+;; registry
(client/post "http://example.com"
- {:client-params {:cookie-policy (constantly nil)}})
+ {:cookie-policy-registry my-custom-policy-registry
+ :cookie-policy "my-policy"})
;; Need to contact a server with an untrusted SSL cert?
(client/get "https://alioth.debian.org" {:insecure? true})
;; If you don't want to follow-redirects automatically:
-(client/get "http://site.come/redirects-somewhere" {:follow-redirects false})
+(client/get "http://example.com/redirects-somewhere" {:redirect-strategy :none})
;; Only follow a certain number of redirects:
-(client/get "http://site.come/redirects-somewhere" {:max-redirects 5})
+(client/get "http://example.com/redirects-somewhere" {:max-redirects 5})
-;; Throw an exception if redirected too many times:
-(client/get "http://site.come/redirects-somewhere" {:max-redirects 5 :throw-exceptions true})
+;; Avoid throwing exceptions if redirected too many times:
+(client/get "http://example.com/redirects-somewhere" {:max-redirects 5 :redirect-strategy :graceful})
;; Throw an exception if the get takes too long. Timeouts in milliseconds.
-(client/get "http://site.come/redirects-somewhere" {:socket-timeout 1000 :conn-timeout 1000})
+(client/get "http://example.com/redirects-somewhere" {:socket-timeout 1000 :connection-timeout 1000})
;; Query parameters
-(client/get "http://site.com/search" {:query-params {"q" "foo, bar"}})
+(client/get "http://example.com/search" {:query-params {"q" "foo, bar"}})
;; "Nested" query parameters
;; (this yields a query string of `a[e][f]=6&a[b][c]=5`)
-(client/get "http://site.com/search" {:query-params {:a {:b {:c 5} :e {:f 6})
+(client/get "http://example.com/search" {:query-params {:a {:b {:c 5} :e {:f 6}}}})
;; Provide cookies — uses same schema as :cookies returned in responses
;; (see the cookie store option for easy cross-request maintenance of cookies)
-(client/get "http://site.com"
- {:cookies {"ring-session" {:discard true, :path "/", :value "", :version 0}}})
+(client/get "http://example.com"
+ {:cookies {"ring-session" {:discard true, :path "/", :value "", :version 0}}})
;; Tell clj-http not to decode cookies from the response header
(client/get "http://example.com" {:decode-cookies false})
@@ -204,6 +286,10 @@ Example requests:
;; Support for IPv6!
(client/get "http://[2001:62f5:9006:e472:cabd:c8ff:fee3:8ddf]")
+;; Super advanced, your own http-client-context and request-config
+(client/get "http://example.com/get"
+ {:http-client-context my-http-client-context
+ :http-request-config my-request-config})
#+END_SRC
The client will also follow redirects on the appropriate =30*= status codes.
@@ -215,7 +301,7 @@ content encodings.
** PUT
:PROPERTIES:
-:CUSTOM_ID: h:f28939e5-24af-4e0d-ac3d-81c52a271418
+:CUSTOM_ID: h-1582cd6e-a6e8-49c8-96e3-28eee6128c31
:END:
#+BEGIN_SRC clojure
@@ -226,57 +312,59 @@ content encodings.
** POST
:PROPERTIES:
-:CUSTOM_ID: h:f1454284-011f-4426-8cfc-c116da4301e0
+:CUSTOM_ID: h-32c8ca7a-0ef2-41b8-8158-20b0e2945e5d
:END:
#+BEGIN_SRC clojure
;; Various options:
-(client/post "http://site.com/api"
- {:basic-auth ["user" "pass"]
- :body "{\"json\": \"input\"}"
- :headers {"X-Api-Version" "2"}
- :content-type :json
- :socket-timeout 1000 ;; in milliseconds
- :conn-timeout 1000 ;; in milliseconds
- :accept :json})
+(client/post "http://example.com/api"
+ {:basic-auth ["user" "pass"]
+ :body "{\"json\": \"input\"}"
+ :headers {"X-Api-Version" "2"}
+ :content-type :json
+ :socket-timeout 1000 ;; in milliseconds
+ :connection-timeout 1000 ;; in milliseconds
+ :accept :json})
;; Send form params as a urlencoded body (POST or PUT)
-(client/post "http://site.com" {:form-params {:foo "bar"}})
+(client/post "http://example.com" {:form-params {:foo "bar"}})
;; Send form params as a json encoded body (POST or PUT)
-(client/post "http://site.com" {:form-params {:foo "bar"} :content-type :json})
+(client/post "http://example.com" {:form-params {:foo "bar"} :content-type :json})
;; Send form params as a json encoded body (POST or PUT) with options
-(client/post "http://site.com" {:form-params {:foo "bar"}
- :content-type :json
- :json-opts {:date-format "yyyy-MM-dd"}})
+(client/post "http://example.com" {:form-params {:foo "bar"}
+ :content-type :json
+ :json-opts {:date-format "yyyy-MM-dd"}})
;; You can also specify the encoding of form parameters
-(client/post "http://site.com" {:form-params {:foo "bar"}
- :form-param-encoding "ISO-8859-1"})
+(client/post "http://example.com" {:form-params {:foo "bar"}
+ :form-param-encoding "ISO-8859-1"})
;; Send form params as a Transit encoded JSON body (POST or PUT) with options
-(client/post "http://site.com" {:form-params {:foo "bar"}
- :content-type :transit+json
- :transit-opts
- {:encode {:handlers {}}
- :decode {:handlers {}}}})
+(client/post "http://example.com" {:form-params {:foo "bar"}
+ :content-type :transit+json
+ :transit-opts
+ {:encode {:handlers {}}
+ :decode {:handlers {}}}})
;; Send form params as a Transit encoded MessagePack body (POST or PUT) with options
-(client/post "http://site.com" {:form-params {:foo "bar"}
- :content-type :transit+msgpack
- :transit-opts
- {:encode {:handlers {}}
- :decode {:handlers {}}}})
+(client/post "http://example.com" {:form-params {:foo "bar"}
+ :content-type :transit+msgpack
+ :transit-opts
+ {:encode {:handlers {}}
+ :decode {:handlers {}}}})
;; Multipart form uploads/posts
;; takes a vector of maps, to preserve the order of entities, :name
;; will be used as the part name unless :part-name is specified
(client/post "http://example.org" {:multipart [{:name "title" :content "My Awesome Picture"}
- {:name "Content/type" :content "image/jpeg"}
- {:name "foo.txt" :part-name "eggplant" :content "Eggplants"}
- {:name "file" :content (clojure.java.io/file "pic.jpg")}]})
+ {:name "Content/type" :content "image/jpeg"}
+ {:name "foo.txt" :part-name "eggplant" :content "Eggplants"}
+ {:name "file" :content (clojure.java.io/file "pic.jpg")}]
+ ;; You can also optionally pass a :mime-subtype
+ :mime-subtype "foo"})
;; Multipart :content values can be one of the following:
;; String, InputStream, File, a byte-array, or an instance of org.apache.http.entity.mime.content.ContentBody
@@ -293,11 +381,32 @@ content encodings.
(println "Got:" ex)
(if (> try-count 4) false true))})
+;; to handle a file with non-ascii filename, try :multipart-charset "UTF-8" and :multipart-mode BROWSER_COMPATIBLE
+;; see also: https://stackoverflow.com/questions/3393445/international-characters-in-filename-in-mutipart-formdata
+(import (org.apache.http.entity.mime HttpMultipartMode))
+
+(client/post "http://example.org" {:multipart [{:content (clojure.java.io/file "日本語.txt")}]
+ :multipart-mode HttpMultipartMode/BROWSER_COMPATIBLE
+ :multipart-charset "UTF-8"} )
+
#+END_SRC
+A word about flattening nested =:query-params= and =:form-params= maps. There are essentially three
+different ways to handle flattening them:
+
+- =:ignore-nested-query-string= :: Do not handle nested query parameters specially, treat them as
+ the exact text they come in as. Defaults to *false*.
+- =:flatten-nested-form-params= :: Flatten nested (map within a map) =:form-params= before encoding
+ it as the body. Defaults to *false*, meaning form params are encoded only
+ =x-www-form-urlencoded=.
+- =:flatten-nested-keys= :: An advanced way of specifying which keys having nested maps should be
+ flattened. A middleware function checks the previous two options
+ (=:ignore-nested-query-string= and =:flatten-nested-form-params=) and modifies this to be the
+ list that will be flattened.
+
** DELETE
:PROPERTIES:
-:CUSTOM_ID: h:23225b05-4fc1-48f1-995c-8daeaf4c7c90
+:CUSTOM_ID: h-c7165d6b-232a-439d-9390-8c05e6ef1e6f
:END:
#+BEGIN_SRC clojure
@@ -306,100 +415,155 @@ content encodings.
#+END_SRC
+** Async HTTP Request
+:PROPERTIES:
+:CUSTOM_ID: h-0e3eb987-5b2b-4874-97ef-b834394d083d
+:END:
+The new async HTTP request API is a Ring-style async API.
+All options for synchronous requests can be used in asynchronous requests.
+Starting an async request is easy, for example:
+
+#+BEGIN_SRC clojure
+;; :async? in options map need to be true
+(client/get "http://example.com"
+ {:async? true}
+ ;; respond callback
+ (fn [response] (println "response is:" response))
+ ;; raise callback
+ (fn [exception] (println "exception message is: " (.getMessage exception))))
+#+END_SRC
+
+All exceptions thrown during the request will be passed to the raise callback.
+
+*** Cancelling Requests
+:PROPERTIES:
+:CUSTOM_ID: cancelling-requests
+:END:
+
+Calls to the http methods with =:async true= return an Apache [[https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/concurrent/BasicFuture.html][BasicFuture]] that you can call =.get=
+or =.cancel= on. See the Javadocs for =BasicFuture= [[https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/concurrent/BasicFuture.html][here]]. For instance:
+
+#+BEGIN_SRC clojure
+(import '(java.util.concurrent TimeoutException TimeUnit))
+
+(let [future (client/get "http://example.com/slow-url"
+ {:async true :oncancel #(println "request was cancelled")}
+ #(println :got %) #(println :err %))]
+ (try
+ (.get future 1 TimeUnit/SECONDS)
+ (catch TimeoutException e
+ ;; Cancel the request, it's taken too long
+ (.cancel future true))))
+#+END_SRC
+
** Coercions
:PROPERTIES:
-:CUSTOM_ID: h:44b36885-b8ea-4e78-9e04-5141995e6771
+:CUSTOM_ID: h-8902cd95-e01e-4d9b-9dc8-5f5c8f04504b
:END:
+
+clj-http allows coercing the body of the request either before it is sent (input coercion), or after
+it's received (output coercion) from the server.
+
*** Input coercion
:PROPERTIES:
-:CUSTOM_ID: h:b72eb0a0-5546-440b-95ea-ff10aa631fd8
+:CUSTOM_ID: h-bed01743-2209-473d-ae86-bd187f059e0c
:END:
#+BEGIN_SRC clojure
;; body as a byte-array
-(client/post "http://site.com/resources" {:body my-byte-array})
+(client/post "http://example.com/resources" {:body my-byte-array})
;; body as a string
-(client/post "http://site.com/resources" {:body "string"})
+(client/post "http://example.com/resources" {:body "string"})
;; :body-encoding is optional and defaults to "UTF-8"
-(client/post "http://site.com/resources"
+(client/post "http://example.com/resources"
{:body "string" :body-encoding "UTF-8"})
;; body as a file
-(client/post "http://site.com/resources"
+(client/post "http://example.com/resources"
{:body (clojure.java.io/file "/tmp/foo") :body-encoding "UTF-8"})
;; :length is optional for passing in an InputStream; if not
;; supplied it will default to -1 to signal to HttpClient to use
;; chunked encoding
-(client/post "http://site.com/resources"
+(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo")})
-(client/post "http://site.com/resources"
+(client/post "http://example.com/resources"
{:body (clojure.java.io/input-stream "/tmp/foo") :length 1000})
#+END_SRC
*** Output coercion
:PROPERTIES:
-:CUSTOM_ID: h:f617095a-dbda-40a8-8662-db62d0389fd5
+:CUSTOM_ID: h-0c8966a6-f220-4f1e-a79e-a520fb313f9e
:END:
#+BEGIN_SRC clojure
;; The default output is a string body
-(client/get "http://site.com/foo.txt")
+(client/get "http://example.com/foo.txt")
;; Coerce as a byte-array
-(client/get "http://site.com/favicon.ico" {:as :byte-array})
+(client/get "http://example.com/favicon.ico" {:as :byte-array})
;; Coerce as something other than UTF-8 string
-(client/get "http://site.com/string.txt" {:as "UTF-16"})
+(client/get "http://example.com/string.txt" {:as "UTF-16"})
;; Coerce as json
-(client/get "http://site.com/foo.json" {:as :json})
-(client/get "http://site.com/foo.json" {:as :json-strict})
-(client/get "http://site.com/foo.json" {:as :json-string-keys})
-(client/get "http://site.com/foo.json" {:as :json-strict-string-keys})
+(client/get "http://example.com/foo.json" {:as :json})
+(client/get "http://example.com/foo.json" {:as :json-string-keys})
;; Coerce as Transit encoded JSON or MessagePack
-(client/get "http://site.com/foo" {:as :transit+json})
-(client/get "http://site.com/foo" {:as :transit+msgpack})
+(client/get "http://example.com/foo" {:as :transit+json})
+(client/get "http://example.com/foo" {:as :transit+msgpack})
;; Coerce as a clojure datastructure
-(client/get "http://site.com/foo.clj" {:as :clojure})
+(client/get "http://example.com/foo.clj" {:as :clojure})
;; Coerce as x-www-form-urlencoded
-(client/post "http://site.com/foo" {:as :x-www-form-urlencoded})
+(client/post "http://example.com/foo" {:as :x-www-form-urlencoded})
;; Try to automatically coerce the output based on the content-type
;; header (this is currently a BETA feature!). Currently supports
;; text, json and clojure (with automatic charset detection)
;; clojure coercion requires "application/clojure" or
;; "application/edn" in the content-type header
-(client/get "http://site.com/foo.json" {:as :auto})
+(client/get "http://example.com/foo.json" {:as :auto})
;; Return the body as a stream
-(client/get "http://site.com/bigrequest.html" {:as :stream})
+(client/get "http://example.com/bigrequest.html" {:as :stream})
;; Note that the connection to the server will NOT be closed until the
;; stream has been read
+
+;; Return the body as a java.io.BufferedReader
+(client/get "http://example.com/bigrequest.html" {:as :reader})
+;; As above, the connection will remain open until the stream has been
+;; read. The reader will attempt to respect the server-specified charset,
+;; if any, defaulting to UTF-8.
#+END_SRC
-JSON coercion defaults to only an "unexceptional" statuses, meaning status codes
-in the #{200 201 202 203 204 205 206 207 300 301 302 303 307} range. If you
-would like to change this, you can send the =:coerce= option, which can be set
-to:
+Output coercion with =:as :json=, =:as :json-string-keys= or =:as :x-www-form-urlencoded=, will only work with an optional dependency, see [[#optional-dependencies][Optional Dependencies]].
+
+By default, JSON coercion is only applied when the response's status
+is considered "unexceptional". If the =:unexceptional-status= option
+is provided, then its value is a function which specifies what status
+codes are unexceptional. =:unexceptional-status= defaults to
+=clj-http.client/unexceptional-status?=.
+
+If you would like to change under what conditions coercion is applied,
+you can send the =:coerce= option, which can be set to:
#+BEGIN_SRC clojure
:always ;; always json decode the body
-:unexceptional ;; only json decode when not an HTTP error response
-:exceptional ;; only json decode when it IS an HTTP error response
+:unexceptional ;; json decode when an HTTP response is considered unexceptional
+:exceptional ;; json decode when an HTTP response is considered exceptional
#+END_SRC
The =:coerce= setting defaults to =:unexceptional=.
** Headers
:PROPERTIES:
-:CUSTOM_ID: h:5f5e2c8b-e9da-40ea-a5aa-2afc9fa4f2df
+:CUSTOM_ID: h-ef64574f-f9dc-4356-95b7-d55cc6737b44
:END:
clj-http's treatment of headers is a little more permissive than the [[https://github.com/ring-clojure/ring/blob/master/SPEC][ring spec]]
@@ -426,18 +590,19 @@ disabled by using with-middleware to specify different behavior.
** Query-string parameters
:PROPERTIES:
-:CUSTOM_ID: h:dd49992c-a516-4af0-9735-4f4340773361
+:CUSTOM_ID: h-dd49992c-a516-4af0-9735-4f4340773361
:END:
-There are three different ways that query string parameters for array values can
+There are four different ways that query string parameters for array values can
be generated, depending on what the resulting query string should look like,
they are:
- A repeating parameter (default)
- Array style
- Indexed array style
+- Comma separated style
-Here is an example of the input and output for the ~:query_string~ parameter,
+Here is an example of the input and output for the ~:query-params~ parameter,
controlled by the ~:multi-param-style~ option:
#+BEGIN_SRC clojure
@@ -449,11 +614,13 @@ controlled by the ~:multi-param-style~ option:
;; with :multi-param-style :indexed, a repeating param with array suffix and
;; index (Rails-style):
:a [1 2 3] => "a[0]=1&a[1]=2&a[2]=3"
+;; with :multi-param-style :comma-separated, a param with comma-separated values
+:a [1 2 3] => "a=1,2,3"
#+END_SRC
** Meta Tag Headers
:PROPERTIES:
-:CUSTOM_ID: h:1f1a4258-849f-4324-8687-d066c15de09b
+:CUSTOM_ID: h-01663a63-8bc8-45da-8a3d-341402f3f3fa
:END:
HTML 4.01 allows using the tag ~<meta http-equiv="..." />~ and HTML 5 allows
@@ -501,7 +668,7 @@ using:
#+END_SRC
Note that this feature is currently beta and uses [[https://github.com/weavejester/crouton][Crouton]] to parse the body of
-the request. If you do not want to use this feature, you can include Crouton in
+the request. If you want to use this feature, you can include Crouton in
addition to clj-http as a dependency like so:
#+BEGIN_SRC clojure
@@ -517,7 +684,7 @@ clj-http will automatically disable the =:decode-body-headers= option.
** Link Headers
:PROPERTIES:
-:CUSTOM_ID: h:338ed551-06d7-4889-91cd-b21aec21d15f
+:CUSTOM_ID: h-f7464c54-4928-474f-9132-08e6b6f3c19d
:END:
clj-http parses any [[http://tools.ietf.org/html/rfc5988][link headers]] returned in the response, and adds them to the
@@ -532,25 +699,76 @@ APIs:
** Redirects
:PROPERTIES:
-:CUSTOM_ID: h:0176f085-4ddb-4dfd-9007-d27a6e598ebd
+:CUSTOM_ID: h-71c966ae-f764-4bd7-801c-0f3c8413c502
:END:
clj-http conforms its behaviour regarding automatic redirects to the [[https://tools.ietf.org/html/rfc2616#section-10.3][RFC]].
-It means that redirects on status =301=, =302= and =307= are not redirected on
+It means that redirects on status =301=, =302=, =307= and =308= are not redirected on
methods other than =GET= and =HEAD=. If you want a behaviour closer to what most
-browser have, you can set =:force-redirects= to =true= in your request to have
+browser have, you can set =:redirect-strategy= to =:lax= in your request to have
automatic redirection work on all methods by transforming the method of the
request to =GET=.
+Redirect Options:
+
+- =:trace-redirects= :: If true, clj-http will enhance the response object with a
+ list of redirected URLs with key: =:trace-redirects=.
+- =:redirect-strategy= :: Sets the redirect strategy for clj-http. Accepts the following:
+ - =:none= - Perform no redirects
+ - =:default= - See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultRedirectStrategy.html
+ - =:lax= - See https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/LaxRedirectStrategy.html
+ - =:graceful= - Similar to =:default=, but does not throw exceptions when max redirects is reached. This is the redirects behaviour in 2.x
+ - =nil= - When nil, assumes =:default=
+
+You may also pass in an instance of RedirectStrategy (in the =:redirect-strategy= key) if you want a
+behavior that's not implemented.
+
+Additionally, clj-http will attempt to validate that a redirect host is not invalid, you can disable
+this by setting =:validate-redirects false= in the request (the default is true)
+
+NOTE: The options =:force-redirects= and =:follow-redirects= (present in clj-http 2.x are no longer
+used). You can use =:graceful= to mostly emulate the old redirect behaviour.
+
+*** How to create a custom RedirectStrategy
+:PROPERTIES:
+:CUSTOM_ID: h:a3b8b124-411f-4c4c-ac4b-777624e76bf1
+:END:
+As mentioned earlier, it's possible to pass a custom instance of RedirectStrategy. The snippet below shows how to create a custom =RedirectStrategy= by wrapping the default strategy.
+
+#+begin_src clojure
+ (def default-strategy org.apache.http.impl.client.DefaultRedirectStrategy/INSTANCE)
+
+ (def logging-redirect-strategy
+ (reify org.apache.http.client.RedirectStrategy
+ (getRedirect [this request response context]
+ (println "attempting redirect...")
+ (.getRedirect default-strategy request response context))
+ (isRedirected [this request response context]
+ (println "checking isRedirected")
+ (.isRedirected default-strategy request response context))))
+
+ (client/get "https://httpbin.org/absolute-redirect/3" {:redirect-strategy logging-redirect-strategy})
+ ;; this will output the following:
+ ;;
+ ;; checking isRedirected
+ ;; attempting redirect...
+ ;; checking isRedirected
+ ;; attempting redirect...
+ ;; checking isRedirected
+ ;; attempting redirect...
+ ;; checking isRedirected
+#+end_src
+
+
** Cookies
:PROPERTIES:
-:CUSTOM_ID: h:82472506-4fbe-4c1d-9c09-b6f764647c24
+:CUSTOM_ID: h-3bb89b16-4be3-455e-98ec-c5ca5830ddb9
:END:
*** Cookiestores
:PROPERTIES:
-:CUSTOM_ID: h:d9887431-486f-456f-b698-3f708e46367f
+:CUSTOM_ID: h-1d86fe30-f690-4c2a-9a1c-231669f4591a
:END:
clj-http can simplify the maintenance of cookies across requests if it is
@@ -558,9 +776,9 @@ provided with a _cookie store_.
#+BEGIN_SRC clojure
(binding [clj-http.core/*cookie-store* (clj-http.cookies/cookie-store)]
- (client/post "http://site.com/login" {:form-params {:username "..."
- :password "..."}})
- (client/get "http://site.com/secured-page")
+ (client/post "http://example.com/login" {:form-params {:username "..."
+ :password "..."}})
+ (client/get "http://example.com/secured-page")
...)
#+END_SRC
@@ -575,14 +793,14 @@ each request, specify the cookie-store with the =:cookie-store= option:
#+BEGIN_SRC clojure
(let [my-cs (clj-http.cookies/cookie-store)]
- (client/post "http://site.com/login" {:form-params {:username "..."
- :password "..."}
- :cookie-store my-cs})
- (client/post "http://site.com/update" {:body my-data
- :cookie-store my-cs}))
+ (client/post "http://example.com/login" {:form-params {:username "..."
+ :password "..."}
+ :cookie-store my-cs})
+ (client/post "http://example.com/update" {:body my-data
+ :cookie-store my-cs}))
#+END_SRC
-You can also us the =get-cookies= function to retrieve the cookies
+You can also use the =get-cookies= function to retrieve the cookies
from a cookie store:
#+BEGIN_SRC clojure
@@ -609,7 +827,7 @@ from a cookie store:
*** Keystores, Trust-stores
:PROPERTIES:
-:CUSTOM_ID: h:1546e3f1-3f9f-459a-9015-628afa22f59e
+:CUSTOM_ID: h-7968467a-1441-4a73-9307-9a7a5fd8e733
:END:
You can also specify your own keystore/trust-store to be used:
@@ -628,19 +846,19 @@ files or =KeyStore= instances.
** Exceptions
:PROPERTIES:
-:CUSTOM_ID: h:dfb56fc9-a958-41ad-8de2-af15a4cd4902
+:CUSTOM_ID: h-ed9e04f1-1c7b-4c2e-9259-94d2a3e65a89
:END:
The client will throw exceptions on, well, exceptional status codes, meaning all
-HTTP responses other than =#{200 201 202 203 204 205 206 207 300 301 302 303
+HTTP responses other than =#{200 201 202 203 204 205 206 207 300 301 302 303 304
307}=. clj-http will throw a [[http://github.com/scgilardi/slingshot][Slingshot]] Stone that can be caught by a regular
=(catch Exception e ...)= or in Slingshot's =try+= block:
#+BEGIN_SRC clojure
-(client/get "http://site.com/broken")
+(client/get "http://example.com/broken")
=> ExceptionInfo clj-http: status 404 clj-http.client/wrap-exceptions/fn--583 (client.clj:41)
;; Or, if you would like the Exception message to contain the entire response:
-(client/get "http://site.com/broken" {:throw-entire-message? true})
+(client/get "http://example.com/broken" {:throw-entire-message? true})
=> ExceptionInfo clj-http: status 404 {:status 404,
:headers {"server" "nginx/1.0.4",
"x-runtime" "12ms",
@@ -655,21 +873,23 @@ HTTP responses other than =#{200 201 202 203 204 205 206 207 300 301 302 303
clj-http.client/wrap-exceptions/fn--584 (client.clj:42
;; You can also ignore HTTP-status-code exceptions and handle them yourself:
-(client/get "http://site.com/broken" {:throw-exceptions false})
+(client/get "http://example.com/broken" {:throw-exceptions false})
;; Or ignore an unknown host (methods return 'nil' if this is set to
;; true and the host does not exist:
-(client/get "http://aoeuntahuf89o.com" {:ignore-unknown-host? true})
+(client/get "http://example.invalid" {:ignore-unknown-host? true})
+;; Or customize the http statuses that will not throw:
+(client/get "http://example.com/broken" {:unexceptional-status #(<= 200 % 299)})
#+END_SRC
(spacing added by me to be human readable)
How to use with Slingshot:
-#+BEGIN_SRC
+#+BEGIN_SRC clojure
; Response map is thrown as exception obj.
; We filter out by status codes
(try+
- (client/get "http://some-site.com/broken")
+ (client/get "http://example.com/broken")
(catch [:status 403] {:keys [request-time headers body]}
(log/warn "403" request-time headers))
(catch [:status 404] {:keys [request-time headers body]}
@@ -681,7 +901,7 @@ How to use with Slingshot:
** Decompression
:PROPERTIES:
-:CUSTOM_ID: h:1ff48808-dc42-46de-8cef-12983c446d80
+:CUSTOM_ID: h-f780c96c-90be-4d83-9b53-227a9e5942ab
:END:
By default, clj-http will add the ={"Accept-Encoding" "gzip, deflate"}= header
@@ -710,7 +930,7 @@ is enabled.
** Debugging
:PROPERTIES:
-:CUSTOM_ID: h:f86f4daa-356e-40ca-b87e-bf347ec1f38e
+:CUSTOM_ID: debugging
:END:
There are four debugging methods you can use:
@@ -737,53 +957,187 @@ There are four debugging methods you can use:
(client/get "http://example.org" {:response-interceptor (fn [resp ctx] (println ctx))})
#+END_SRC
+*** Logging
+:PROPERTIES:
+:CUSTOM_ID: h-0d505652-d453-48a2-a868-46aef2b8af66
+:END:
+
+Finally, if you want to access the logging that the Apache client does internally, you can set up
+your dependencies to add the [[https://logging.apache.org/log4j/2.x/][log4j2]] libraries and configure the logging for clj-http. In order to do
+this, you'll need to add
+
+#+BEGIN_SRC clojure
+[org.apache.logging.log4j/log4j-api "2.11.0"]
+[org.apache.logging.log4j/log4j-core "2.11.0"]
+[org.apache.logging.log4j/log4j-1.2-api "2.11.0"]
+#+END_SRC
+
+To your =project.clj= and have a usable log4j2.properties. I have provided one in
+=resources/log4j2.properties=. Make sure to set:
+
+#+BEGIN_SRC fundamental
+rootLogger.level = debug
+#+END_SRC
+
+If you want to see debug information (or "trace" for trace logging). When you perform a request you
+should see something akin to this in the logs:
+
+#+BEGIN_SRC fundamental
+[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAddCookies] CookieSpec selected: default
+[2018-03-20T20:36:34,635][DEBUG][o.a.h.c.p.RequestAuthCache] Auth cache not set in the context
+[2018-03-20T20:36:34,635][DEBUG][o.a.h.i.c.BasicHttpClientConnectionManager] Get connection for route {s}->https://example.com:443
+[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] http-outgoing-1: set socket timeout to 0
+[2018-03-20T20:36:34,636][DEBUG][o.a.h.i.e.MainClientExec ] Opening connection {s}->https://example.com:443
+[2018-03-20T20:36:34,644][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connecting to example.com/10.0.0.1:443
+[2018-03-20T20:36:34,644][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Connecting socket to example.com/10.0.0.1:443 with timeout 0
+[2018-03-20T20:36:34,692][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled protocols: [TLSv1, TLSv1.1, TLSv1.2]
+[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Enabled cipher suites:[TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ... etc ...]
+[2018-03-20T20:36:34,693][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Starting handshake
+[2018-03-20T20:36:34,841][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] Secure session established
+[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated protocol: TLSv1.2
+[2018-03-20T20:36:34,842][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] negotiated cipher suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
+[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer principal: CN=example.com
+[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] peer alternative names: [example.com, www.example.com]
+[2018-03-20T20:36:34,843][DEBUG][o.a.h.c.s.SSLConnectionSocketFactory] issuer principal: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
+[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] Connection established 192.168.0.29:36792<->10.0.0.1:443
+[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Executing request POST /post HTTP/1.1
+[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Target auth state: UNCHALLENGED
+[2018-03-20T20:36:34,844][DEBUG][o.a.h.i.e.MainClientExec ] Proxy auth state: UNCHALLENGED
+[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> POST /post HTTP/1.1
+[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Connection: close
+[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> accept-encoding: gzip, deflate
+[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Length: 14
+[2018-03-20T20:36:34,845][DEBUG][o.a.h.headers ] http-outgoing-1 >> Content-Type: text/plain; charset=UTF-8
+[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> Host: example.com
+[2018-03-20T20:36:34,846][DEBUG][o.a.h.headers ] http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.5 (Java/9.0.1)
+[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "POST /post HTTP/1.1[\r][\n]"
+[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Connection: close[\r][\n]"
+[2018-03-20T20:36:34,846][DEBUG][o.a.h.wire ] http-outgoing-1 >> "accept-encoding: gzip, deflate[\r][\n]"
+[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Length: 14[\r][\n]"
+[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
+[2018-03-20T20:36:34,847][DEBUG][o.a.h.wire ] http-outgoing-1 >> "Host: example.com[\r][\n]"
+etc etc it will go on forever and be very verbose
+#+END_SRC
+
+This provides both the data sent and received on the wire for debugging purposes.
+
+I've also provided an example for changing the log level from clojure in
+=examples/logging-apache-requests.clj=.
+
+* Caching
+:PROPERTIES:
+:CUSTOM_ID: h-2c4ee611-ca22-432e-9c33-18040566661e
+:END:
+
+clj-http supports Apache's caching client, essentially it "provides an HTTP/1.1-compliant caching
+layer to be used with HttpClient--the Java equivalent of a browser cache." (see [[https://hc.apache.org/httpcomponents-client-ga/tutorial/html/caching.html][the explanation in
+the apache docs]]). In order to use the cache, a reusable connection manager *and* http-client must be
+used.
+
+An example of basic usage with the default options:
+
+#+BEGIN_SRC clojure
+(let [cm (conn/make-reusable-conn-manager {})
+ client (:http-client (http/get "http://example.com"
+ {:connection-manager cm :cache true}))]
+ (http/get "http://example.com"
+ {:connection-manager cm :http-client client :cache true})
+ (http/get "http://example.com"
+ {:connection-manager cm :http-client client :cache true})
+ (http/get "http://example.com"
+ {:connection-manager cm :http-client client :cache true}))
+#+END_SRC
+
+You can build your own cache config by providing either a map of caching configuration options, or
+by providing a =CacheConfig= object, as seen below:
+
+#+BEGIN_SRC clojure
+(let [cm (conn/make-reusable-conn-manager {})
+ cache-config (core/build-cache-config
+ {:cache-config {:max-object-size 4096}})
+ client (:http-client (http/get "http://example.com"
+ {:connection-manager cm :cache true}))]
+ (http/get "http://example.com"
+ ;; Use the default cache config settings
+ {:connection-manager cm :http-client client :cache true})
+ (http/get "http://example.com"
+ {:connection-manager cm :http-client client :cache true
+ ;; Provide cache configuration options as a map
+ :cache-config {:max-object-size 9152
+ :max-cache-entries 100}})
+ (http/get "http://example.com"
+ {:connection-manager cm :http-client client :cache true
+ ;; Provide the cache configuration as a CacheConfig object
+ :cache-config cache-config}))
+#+END_SRC
+
+In the response, clj-http provides the =:cached= key to indicate whether the response was cached,
+missed, etc:
+
+- nil :: Caching was not used for this request
+- =:CACHE_HIT= :: A response was generated from the cache with no requests sent upstream.
+- =:CACHE_MISS= :: The response came from an upstream server.
+- =:CACHE_MODULE_RESPONSE= :: The response was generated directly by the caching module.
+- =:VALIDATED= :: The response was generated from the cache after validating the entry with the origin server.
+
* Authentication
:PROPERTIES:
-:CUSTOM_ID: h:3c375d7a-7acc-45cb-a7a4-6f2bdf4cad95
+:CUSTOM_ID: h-87f38469-36b4-44c6-ae74-0d8f5e80c2ed
:END:
** Basic Auth
:PROPERTIES:
-:CUSTOM_ID: h:8ae77bcc-68c6-4560-affb-4bbe02c6b7a9
+:CUSTOM_ID: h-d3ea348f-88ed-4193-bb16-d8d5accdc2aa
:END:
-#+BEGIN_SRC
+#+BEGIN_SRC clojure
-(client/get "http://site.com/protected" {:basic-auth ["user" "pass"]})
-(client/get "http://site.com/protected" {:basic-auth "user:pass"})
+(client/get "http://example.com/protected" {:basic-auth ["user" "pass"]})
+(client/get "http://example.com/protected" {:basic-auth "user:pass"})
#+END_SRC
** Digest Auth
:PROPERTIES:
-:CUSTOM_ID: h:47c07a03-677f-4a4f-967f-242329a8ab07
+:CUSTOM_ID: h-d1904589-e71e-43db-8b93-0f94ccecaabe
:END:
-#+BEGIN_SRC
+#+BEGIN_SRC clojure
-(client/get "http://site.com/protected" {:digest-auth ["user" "pass"]})
+(client/get "http://example.com/protected" {:digest-auth ["user" "pass"]})
+
+#+END_SRC
+
+** NTLM Auth
+:PROPERTIES:
+:CUSTOM_ID: h-AE80FFDC-2016-4883-9512-2BE16640339D
+:END:
+
+#+BEGIN_SRC clojure
+
+(client/get "http://example.com/protected" {:ntlm-auth ["user" "pass" "host" "domain"]})
#+END_SRC
** oAuth2
:PROPERTIES:
-:CUSTOM_ID: h:e34482c0-15d2-483a-a183-d5e3f1f662a6
+:CUSTOM_ID: h-dd077440-a1de-437e-b34e-5d6d0d1da4bd
:END:
-#+BEGIN_SRC
+#+BEGIN_SRC clojure
-(client/get "http://site.com/protected" {:oauth-token "secret-token"})
+(client/get "http://example.com/protected" {:oauth-token "secret-token"})
#+END_SRC
* Advanced Usage
:PROPERTIES:
-:CUSTOM_ID: h:e6aed224-7ed5-4340-bc4e-7874eefadd87
+:CUSTOM_ID: h-d52ca837-a575-402f-81fe-53241d85f2db
:END:
** Raw Request
:PROPERTIES:
-:CUSTOM_ID: h:71bf84d3-2ff0-44a8-99aa-214d339cf7d2
+:CUSTOM_ID: h-0d2eadbf-c1ad-4514-a932-9d173582a790
:END:
A more general =request= function is also available, which is useful as a
@@ -792,12 +1146,12 @@ primitive for building higher-level interfaces:
#+BEGIN_SRC clojure
(defn api-action [method path & [opts]]
(client/request
- (merge {:method method :url (str "http://site.com/" path)} opts)))
+ (merge {:method method :url (str "http://example.com/" path)} opts)))
#+END_SRC
*** Boolean options
:PROPERTIES:
-:CUSTOM_ID: h:4a5870a5-693f-441d-a69c-da96eebbbb6e
+:CUSTOM_ID: h-a37c718c-43bb-43ce-936a-21ef65147295
:END:
Since 0.9.0, all boolean options can be expressed as either ={:debug true}= or
@@ -805,7 +1159,7 @@ Since 0.9.0, all boolean options can be expressed as either ={:debug true}= or
** Persistent Connections
:PROPERTIES:
-:CUSTOM_ID: h:5f755de8-c106-4b89-aa0a-3074ef96efc4
+:CUSTOM_ID: h-4e9f116d-c293-4a0c-8e11-435c440bfe97
:END:
clj-http can use persistent connections to speed up connections if multiple
@@ -813,27 +1167,69 @@ connections are being used:
#+BEGIN_SRC clojure
(with-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
- (get "http://aoeu.com/1")
- (post "http://aoeu.com/2")
- (get "http://aoeu.com/3")
+ (get "http://example.org/1")
+ (post "http://example.org/2")
+ (get "http://example.org/3")
...
- (get "http://aoeu.com/999"))
+ (get "http://example.org/999"))
+#+END_SRC
+
+For async request, you can use =with-async-connection-pool=
+
+#+BEGIN_SRC clojure
+(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
+ (get "http://example.org/1" {:async? true} resp1 exce1)
+ (post "http://example.org/2" {:async? true} resp2 exce2)
+ (get "http://example.org/3" {:async? true} resp3 exce3)
+ ...
+ (get "http://example.org/999" {:async? true} resp999 exce999))
#+END_SRC
This is MUCH faster than sequentially performing all requests, because a
persistent connection can be used instead creating a new connection for each
request.
+If you want to start an async request in the =respond= callback of an async request and
+reuse the pool context, just use =reuse-pool=.
+
+#+BEGIN_SRC clojure
+(with-async-connection-pool {:timeout 5 :threads 4 :insecure? false :default-per-route 10}
+ (get "http://example.org/1" {:async? true} resp1 exce1)
+ (post "http://example.org/2"
+ {:async? true}
+ (fn [resp] (get "http://example.org/3"
+ (reuse-pool {:async? true} resp)
+ resp3 exce3))
+ exce2))
+#+END_SRC
+
+There are many advanced options available when creating asynchronous connection pools that can be
+configured by passing an =:io-config= map in the connection manager parameters. It supports:
+
+- =:connect-timeout=
+- =:interest-op-queued=
+- =:io-thread-count=
+- =:rcv-buf-size=
+- =:select-interval=
+- =:shutdown-grace-period=
+- =:snd-buf-size=
+- =:so-keep-alive=
+- =:so-linger=
+- =:so-timeout=
+- =:tcp-no-delay=
+
+See the docstring on =with-async-connection-pool= for more information about these options.
+
If you would prefer to handle managing the connection manager yourself, you can
-create a connection manager yourself and specify it for each request:
+create a connection manager and specify it for each request:
#+BEGIN_SRC clojure
(def cm (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 2 :threads 3}))
(def cm2 (clj-http.conn-mgr/make-reusable-conn-manager {:timeout 10 :threads 1}))
-(get "http://aoeu.com/1" {:connection-manager cm2})
-(post "http://aoeu.com/2" {:connection-manager cm})
-(get "http://aoeu.com/3" {:connection-manager cm2})
+(get "http://example.org/1" {:connection-manager cm2})
+(post "http://example.org/2" {:connection-manager cm})
+(get "http://example.org/3" {:connection-manager cm2})
;; Don't forget to shut it down when you're done!
(clj-http.conn-mgr/shutdown-manager cm)
@@ -843,9 +1239,58 @@ create a connection manager yourself and specify it for each request:
See the docstring on =make-reusable-conn-manager= for options and default
values.
+In the current version, pooled async request CANNOT specify connection manager.
+
+** Re-using =HttpClient= between requests
+:PROPERTIES:
+:CUSTOM_ID: h-b79b07fb-d024-49a2-a7f7-53863d1b8d6d
+:END:
+
+In some cases, you may want to re-use the same =HttpClient= object between requests, either so you
+don't have to build it every time, or because you make some configuration change to the request.
+clj-http will return the built HTTP client in =:http-client= which you can then specify in
+subsequent requests (with =:http-client=). Note that in order to reuse the client a connection
+manager must be used.
+
+#+BEGIN_SRC clojure
+;; Re-use the HttpClient clj-http builds for you:
+(let [cm (conn/make-reusable-conn-manager {})
+ resp (client/get "http://example.com" {:connection-manager cm})
+ hclient (:http-client resp)]
+ (client/get "http://example.com/1"
+ {:connection-manager cm :http-client hclient})
+ (client/get "http://example.com/2"
+ {:connection-manager cm :http-client hclient})
+ (client/get "http://example.com/3"
+ {:connection-manager cm :http-client hclient}))
+
+;; You can also build your own, using clj-http's helper or manually building it:
+(let [cm (conn/make-reusable-conn-manager {})
+ hclient (core/build-http-client {} false cm)]
+ (client/get "http://example.com/1"
+ {:connection-manager cm :http-client hclient})
+ (client/get "http://example.com/2"
+ {:connection-manager cm :http-client hclient})
+ (client/get "http://example.com/3"
+ {:connection-manager cm :http-client hclient}))
+
+;; Async http clients may also be created and re-used:
+(let [acm (conn/make-reuseable-async-conn-manager {})
+ ahclient (core/build-async-http-client {} acm)]
+ (client/get "http://example.com/1"
+ {:connection-manager cm :http-client ahclient}
+ handle-response handle-failure)
+ (client/get "http://example.com/2"
+ {:connection-manager cm :http-client ahclient}
+ handle-response handle-failure)
+ (client/get "http://example.com/3"
+ {:connection-manager cm :http-client ahclient}
+ handle-response handle-failure))
+#+END_SRC
+
** Proxies
:PROPERTIES:
-:CUSTOM_ID: h:b5007a6f-f0bf-4d98-9ab9-b23fcebfa49a
+:CUSTOM_ID: h-49f9ca81-0bad-4cd8-87ac-c09a85fa5500
:END:
A proxy can be specified by setting the Java properties: =<scheme>.proxyHost=
@@ -856,7 +1301,13 @@ Additionally, per-request proxies can be specified with the =proxy-host= and
=proxy-port= options (this overrides =http.nonProxyHosts= too):
#+BEGIN_SRC clojure
-(client/get "http://foo.com" {:proxy-host "127.0.0.1" :proxy-port 8118})
+(client/get "http://example.com" {:proxy-host "127.0.0.1" :proxy-port 8118})
+#+END_SRC
+
+Proxy credentials can also be explicitly set as
+
+#+BEGIN_SRC clojure
+(client/get "http://example.com" {:proxy-host "127.0.0.1" :proxy-port 8118 :proxy-user "proxy-user" :proxy-pass "superSecurePassword"})
#+END_SRC
You can also specify the =proxy-ignore-hosts= parameter with a list of
@@ -880,11 +1331,29 @@ would:
(conn-mgr/make-socks-proxied-conn-manager "localhost" 8081)})
#+END_SRC
+If your SOCKS connection requires a keystore / trust-store, you can specify that too:
+
+#+BEGIN_SRC clojure
+(ns foo.bar
+ (:require [clj-http.client :as client]
+ [clj-http.conn-mgr :as conn-mgr]))
+
+(client/get "https://google.com"
+ {:connection-manager
+ (conn-mgr/make-socks-proxied-conn-manager "localhost" 8081
+ {:keystore "/path/to/keystore.ks"
+ :keystore-type "jks" ; default: jks
+ :keystore-pass "secretpass"
+ :trust-store "/path/to/trust-store.ks"
+ :trust-store-type "jks" ; default jks
+ :trust-store-pass "trustpass"})})
+#+END_SRC
+
You can also store the proxied connection manager and reuse it later.
** Custom Middleware
:PROPERTIES:
-:CUSTOM_ID: h:afec8fd4-580a-4a82-9521-628f8fa4fbd8
+:CUSTOM_ID: h-c51cba6c-5c1b-4941-93c3-f769bb533562
:END:
Sometime it is desirable to run a request with some middleware enabled and some
@@ -903,16 +1372,117 @@ which is a vector of the default middleware that clj-http uses.
=clj-http.client/*current-middleware*= is bound to the current list of
middleware during request time.
+** Modifying Apache-specific features of the =HttpClientBuilder= and =HttpAsyncClientBuilder=
+:PROPERTIES:
+:CUSTOM_ID: h:844f078c-531e-445e-b7ce-76092bcc9928
+:END:
+
+While clj-http tries to provide the features needed, there are times when it does not provide access
+to a parameter that you need. In these cases, you can use a couple of advanced parameters to provide
+arbitrary configuration functions to be run on the =HttpClientBuilder= by specifying
+=:http-builder-fns= and =:async-http-builder-fns=.
+
+Each of these variables is a sequence of functions of two arguments, the http builder
+(=HttpClientBuilder= for =:http-builder-fns= and =HttpAsyncClientBuilder= for
+=:async-http-builder-fns=) and the request map.
+
+#+BEGIN_SRC clojure
+;; A function that takes a builder and disables Apache's cookie management
+(defun my-cookie-disabler [^HttpClientBuilder builder
+ request]
+ (when (:disable-cookies request)
+ (.disableCookieManagement builder)))
+
+;; The functions to modify the builder are passed in
+(http/post "http://www.example.org" {:http-builder-fns [my-cookie-disabler]
+ :disable-cookies true})
+#+END_SRC
+
+The functions are run in the order they are passed in (inside a =doseq=).
+
+By specifying =:http-client-builder=, your own instance of
+=HttpClientBuilder= will be used. A supplied =HttpClientBuilder= which
+sets the connection manager, redirect strategy, retry handler, route
+planner, cache, or cookie spec registry may find these overridden by
+clj-http's =:connection-manager=, =:redirect-strategy=,
+=:retry-handler=, =:cache=, or =:cookie-policy-registry= or
+=:cookie-spec=, respectively.
+
+** Incrementally JSON Parsing
+:PROPERTIES:
+:CUSTOM_ID: h:b01c16e8-7179-468e-8890-316939ec0e38
+:END:
+[[https://github.com/dakrone/cheshire][cheshire]] supports incrementally parsing JSON using lazy sequences. This approach can useful for
+processing large top-level JSON arrays because it doesn't require upfront work consuming the entire stream.
+
+#+begin_src clojure
+ (require '[cheshire.core :as json])
+
+ (defn print-all-pokemon-names [pokemons]
+ (for [pokemon pokemons]
+ (println (get-in pokemon [:name :english]))))
+
+ (let [url "https://raw.githubusercontent.com/fanzeyi/pokemon.json/master/pokedex.json"
+ response (get url {:as :reader})]
+ (with-open [reader (:body response)] ; closes the underlying connection when we're done
+ (let [pokemons (json/parse-stream reader true)]
+ ; You must perform all reads from the stream inside `with-open`,
+ ; any , any lazy
+ (doall (print-all-pokemon-names pokemons)))))
+#+end_src
+
+Keep in mind that the =reader= object wraps a HTTP connection. The user needs to be aware of two
+things:
+
+1. The user should close the reader after processing the stream, otherwise the underlying HTTP
+ Connection may leak and create subtle bugs. Clojure's [[https://clojuredocs.org/clojure.core/with-open][with-open]] is useful here.
+
+2. You should realize any lazy sequences before closing the connection. Use [[https://clojuredocs.org/clojure.core/doall][doall]] or [[https://clojure.org/reference/transducers][transducers]] to
+ prevent bugs from lazy IO. See [[https://stuartsierra.com/2015/08/25/clojure-donts-lazy-effects][Clojure Don'ts: Lazy Effects]].
+
+In previous versions of =clj-http= (<= 3.10.0), =clj-http= defaulted to lazily parsing JSON, but this
+was slow and also confused users who didn't expect laziness.
+
+** DNS Resolution
+
+Users may add their own DNS resolver function to override the default DNS Resolver. This is useful in situations where you are unable to change the name to IP Address mapping. It is analogous to the =--resolve= flag present in =curl=. This example uses =org.apache.http.impl.conn.InMemoryDnsResolver= to resolve =example.com= to IP Address =127.0.0.1=.
+
+#+BEGIN_SRC clojure
+(client/get "https://example.com" {:dns-resolver (doto (InMemoryDnsResolver.)
+ (.add "example.com" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))})
+#+END_SRC
+
+This option is supported for all of the connection managers.
+
+The =dns-resolver= can be any instance of =DnsResolver=. Here is an example of a custom implementation that attempts to look up the hostname in the supplied map and falls back to the default SystemDnsResolver if not found. Note how IPV6 addresses are specified.
+
+#+BEGIN_SRC clojure
+(defn custom-dns-resolver
+ [host-map]
+ (let [system-dns-resolver (org.apache.http.impl.conn.SystemDefaultDnsResolver.)]
+ (reify
+ org.apache.http.conn.DnsResolver
+ (^"[Ljava.net.InetAddress;" resolve [this ^String host]
+ (if-let [address (get host-map host)]
+ (into-array [(java.net.InetAddress/getByAddress host (byte-array address))])
+ (.resolve system-dns-resolver host))))))
+
+(client/get "https://example.com" {:dns-resolver (custom-dns-resolver {"example.com" [127 0 0 1]
+ "www.google.com" [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]})})
+#+END_SRC
+
+
* Development
:PROPERTIES:
-:CUSTOM_ID: h:0fb11882-d060-4e2e-85f7-3bf18dc9051b
+:CUSTOM_ID: h-65bbf017-2e8b-4c43-824b-24b89cc27a70
:END:
-Please send a pull request or open an issue if you have any problems.
+Please send a pull request or open an issue if you have any problems. See =CONTRIBUTING.md= for more
+information.
** Faking Responses
:PROPERTIES:
-:CUSTOM_ID: h:8bacf773-9af1-4098-907e-f96a780d3fca
+:CUSTOM_ID: h-c3d9c7e0-cc3f-47bf-91e3-b12567b08eb6
:END:
If you need to fake clj-http responses (for things like testing and such), check
@@ -920,7 +1490,7 @@ out the [[https://github.com/myfreeweb/clj-http-fake][clj-http-fake]] library.
** Optional Dependencies
:PROPERTIES:
-:CUSTOM_ID: h:a847429d-741f-4a5a-8d27-916ea1017461
+:CUSTOM_ID: h-f1fbdad3-cf40-41e0-8ae0-8716419be228
:END:
In 2.0.0+ clj-http's optional dependencies at excluded by default, in order to
@@ -943,20 +1513,20 @@ without them.
** clj-http-lite
:PROPERTIES:
-:CUSTOM_ID: h:472e4fba-3c50-4eb6-95fb-95d0d9afdbad
+:CUSTOM_ID: h-ba6b263b-74a5-40f3-afc1-b0d785554c2b
:END:
-Like clj-http but need something more lightweight without as many external
-dependencies? Check out [[https://github.com/hiredman/clj-http-lite][clj-http-lite]] for a project that can be used as a
+Like clj-http but need something more lightweight without as any external
+dependencies? Check out [[https://github.com/clj-commons/clj-http-lite][clj-http-lite]] for a project that can be used as a
drop-in replacement for clj-http.
** Troubleshooting
:PROPERTIES:
-:CUSTOM_ID: h:e97ba275-3324-4102-9beb-9bcbc483ad15
+:CUSTOM_ID: h-c543201e-a0e5-4e84-8eb2-6bf3e21a3140
:END:
*** VerifyError class org.codehaus.jackson.smile.SmileParser overrides final method getBinaryValue...
:PROPERTIES:
-:CUSTOM_ID: h:048d8994-647a-4325-ab5d-f96fa12d5798
+:CUSTOM_ID: h-c3a8ebc3-a247-4327-8b71-0097d1380873
:END:
This is actually caused by your project attempting to use [[https://github.com/mmcgrana/clj-json/][clj-json]] and [[https://github.com/dakrone/cheshire][cheshire]]
@@ -979,7 +1549,7 @@ and clj-json can now live together without causing problems.
*** NoHttpResponseException ... due to stale connections**
:PROPERTIES:
-:CUSTOM_ID: h:c2a2ea17-d402-43ba-b57b-2a5bc75a6750
+:CUSTOM_ID: h-9d7cf050-ed5b-4d23-8b02-97a9b9c94737
:END:
Persistent connections kept alive by the connection manager become stale: the
@@ -992,7 +1562,7 @@ This can be solved by using (with-connection-pool) as described in the
* Tests
:PROPERTIES:
-:CUSTOM_ID: h:34bd658a-26a2-4731-9b7d-dd93bce8c35a
+:CUSTOM_ID: h-a52feb3d-d966-4287-a07e-ad7aa7918fd5
:END:
To run the tests:
@@ -1004,35 +1574,47 @@ $ lein test
Run all tests (including integration):
$ lein test :all
-Run tests against 1.2.1, 1.3 and 1.4
+Run tests against all clojure versions
$ lein all test
$ lein all test :all
#+END_SRC
* Testimonials
:PROPERTIES:
-:CUSTOM_ID: h:3e19427d-7d2b-465e-8fa4-1d59f9555924
+:CUSTOM_ID: h-3044d1f7-6772-43c2-9ded-8c71c7f9ada2
:END:
-With close to a [million](https://clojars.org/clj-http) downloads, clj-http is a
-widely used, battle-tested clojure library. It is also included in other
-libraries (like database clients) as a low-level http wrapper.
+With over [[https://clojars.org/clj-http][three million]] downloads, clj-http is a widely used, battle-tested clojure library. It is
+also included in other libraries (like database clients) as a low-level http wrapper.
Libraries using clj-http:
-- [[https://github.com/mattrepl/clj-oauth] [clj-oauth]]
-- [[[[https://github.com/clojurewerkz/elastisch]]] [elasticsearch]]
-- [[https://github.com/olauzon/capacitor] [influxdb]]
+- [[https://github.com/mattrepl/clj-oauth][clj-oauth]]
+- [[https://github.com/clojurewerkz/elastisch][elasticsearch]]
+- [[https://github.com/olauzon/capacitor][influxdb]]
Libraries inspired by clj-http:
-- [[https://github.com/mpenet/jet] [jet]]
-- [[https://github.com/hiredman/clj-http-lite] [clj-http-lite]]
+- [[https://github.com/mpenet/jet][jet]]
+- [[https://github.com/clj-commons/clj-http-lite][clj-http-lite]]
+
+* Other Libraries Providing Middleware
+:PROPERTIES:
+:CUSTOM_ID: other-middleware
+:END:
+
+- [[https://github.com/sharetribe/aws-sig4][aws-sig4]] :: a pure clojure implementation of AWS v4 signature request signing as middleware
+
+(feel free to open a PR or issue if you'd like to add middleware here)
* License
:PROPERTIES:
-:CUSTOM_ID: h:9968d81c-ff40-40f5-be27-60bab27c64c9
+:CUSTOM_ID: h-2de3db75-7a1b-42b8-ad3b-6ef27fc2a5ea
:END:
Released under the MIT License:
<http://www.opensource.org/licenses/mit-license.php>
+
+# Local Variables:
+# fill-column: 100
+# End:
diff --git a/changelog.org b/changelog.org
index 38b19b4..04e5cfd 100644
--- a/changelog.org
+++ b/changelog.org
@@ -9,6 +9,126 @@
* Changelog
List of user-visible changes that have gone into each release
+** 3.12.4 (unreleased)
+** 3.12.3
+- Allow http-client re-use in async situation (#599)
+ https://github.com/dakrone/clj-http/pull/599
+** 3.12.2
+- Upgrade Dependencies (#598)
+ https://github.com/dakrone/clj-http/pull/598
+** 3.12.1
+- Bugfix for :normalize-uri (#584)
+ https://github.com/dakrone/clj-http/pull/584
+** 3.12.0
+- Create SSLContext consistently for all connection managers (#575)
+ https://github.com/dakrone/clj-http/pull/575
+- Adds RequestConfig Option :normalize-uri (#583)
+ https://github.com/dakrone/clj-http/pull/583
+** 3.11.0
+- Adds workaround for Async Multipart uploads greater than 25 kb (#574)
+ https://github.com/dakrone/clj-http/pull/574
+- Adds an additional style for multi-param-style added (#562)
+ https://github.com/dakrone/clj-http/pull/562
+- Close transit input stream after reading response (#565)
+ https://github.com/dakrone/clj-http/pull/565
+- Bump patch versions of apache httpcomponents to latest. (#569)
+ https://github.com/dakrone/clj-http/pull/569
+- Fixed decode-json-body (#568)
+ https://github.com/dakrone/clj-http/pull/568
+- Handle quoted parameter values in content type (#573)
+ https://github.com/dakrone/clj-http/pull/573
+** 3.10.3
+- Improve error message when using incompatible version of cheshire
+ https://github.com/dakrone/clj-http/pull/558
+- Properly handle "308 Permanent Redirect" status code
+ https://github.com/dakrone/clj-http/pull/554
+** 3.10.2
+- Fix performance regressions from #528
+ https://github.com/dakrone/clj-http/pull/546
+- Adds support for custom DNS Resolvers
+ https://github.com/dakrone/clj-http/pull/545
+- Buffer :debug output to improve readability
+ https://github.com/dakrone/clj-http/pull/544
+- Improve compatbility with GraalVM
+ https://github.com/dakrone/clj-http/pull/543
+- Bug fix: Check first byte before wrapping response stream with gunzip
+ https://github.com/dakrone/clj-http/pull/549
+** 3.10.1
+- JSON parsing is always strict. See [[file:README.org::*Incrementally%20JSON%20Parsing][README#Incrementally JSON Parsing]]. This is
+ a *breaking change* and users *must* upgrade to cheshire >= 5.9.0.
+ https://github.com/dakrone/clj-http/pull/507
+** 3.10.0
+- Add trust-manager and key-managers support to the client
+ https://github.com/dakrone/clj-http/pull/469
+- Improving consistency of connection option names
+ https://github.com/dakrone/clj-http/pull/483
+ https://github.com/dakrone/clj-http/issues/477
+- Ensure Socket Timeout is set for BasicHttpClientConnectionManager
+ https://github.com/dakrone/clj-http/pull/463
+- Reduce body allocation and copying
+ https://github.com/dakrone/clj-http/pull/475
+** 3.9.1
+- Fix body parsing when first byte value is 255
+ https://github.com/dakrone/clj-http/pull/449
+- Add custom =:unexceptional-status= option
+ https://github.com/dakrone/clj-http/pull/451
+** 3.9.0
+- Add support for reusable http clients, returning the client in =:http-client= and allowing one to
+ be specified (with the same setting) - https://github.com/dakrone/clj-http/issues/441
+- Cancelling the =Future= returned from an async http request now also aborts the HttpRequest object
+- Async connection managers no longer put the connection manager in an illegal ACTIVE state [[https://github.com/dakrone/clj-http/issues/443][#443]]
+- Added the =:cookie-spec= and =:cookie-policy-registry= options for specifying a custom cookie spec
+ for parsing cookies. Since clj-http doesn't rely on Apache's cookies handling, this is for
+ advanced users who wish to add their own cookie validation, or use Apache's handling instead of
+ clj-http's. It also allows a user who wants to registry a custom spec to reuse the spec without
+ creating it for every request. Semi-related to https://github.com/dakrone/clj-http/issues/444
+- Added support for caching HTTP responses from a server. This can dramatically speed up requests to
+ the same URL. Filling and invalidating the cache is handled by Apache's httpclient-cache project,
+ with configuration exposed under the =:cache= and =:cache-config= parameters in the option map.
+ https://github.com/dakrone/clj-http/issues/445
+
+** 3.8.0
+- Reintroduce the =:save-request= and =:debug-body= options
+- +Wrap nested querystring params before form params, fixing
+ https://github.com/dakrone/clj-http/issues/427+ Reverted, see further below
+- Merged https://github.com/dakrone/clj-http/pull/426 to allow an empty SSLGenericSocketFactory
+ context
+- Merged https://github.com/dakrone/clj-http/pull/424 to add :mime-subtype request parameter to
+ override mime subtype
+- create-multipart-entity with three arguments arity lets the selection of =HttpMultipartMode=
+- new request key :http-multipart-mode which is HttpMultipartMode/STRICT by default
+- Added =:ignore-nested-query-string=, =:flatten-nested-form-params=, and =:flatten-nested-keys=
+ options for finer-grained control over which nested parts of the request are flattened. Fixes
+ https://github.com/dakrone/clj-http/issues/427
+- Added =:http-builder-fns= and =:async-http-builder-fns= to support arbitrary customizations to the
+ =HttpClientBuilder= and =HttpAsyncClientBuilder=
+- Fixed an issue where redirects to a bad location could cause the async client to hang -
+ https://github.com/dakrone/clj-http/pull/435
+- =client/parse-url= now includes the original URL in the =:url= key
+- =core/get-cookie-policy= is now a multimethod. This allows users to customize the return of their
+ own cookie validation method.
+- Empty responses with coercion no longer throw exceptions when processing empty gzipped response
+ streams. Fixes https://github.com/dakrone/clj-http/issues/257
+
+** 3.7.0
+This list contains all the changes since 3.0.0.
+
+Added:
+- HttpRequestInterceptor support 155bd23
+- protocol-version and reason-phrase f430517
+- support for async HTTP requests (like Ring) 44d10ec
+- support for different multi-param encoding (:repeating, :array, :indexed) cddeb3e
+- Add unparse function aec7dd1
+- Added :redirect-strategy :graceful
+- Allow RequestConfig and HttpClientContext to be injected feb3c48
+
+Removed:
+- :save-request
+
+Changed:
+- re-written middleware using apache http client 4.5
+- Fix retry-handler to be added in correct place a2c31f5
+- POST Mutipart: Use charset "UTF-8" instead of "ASCII" as default charset to support internationalization 983508f
** 2.0.0
- merged https://github.com/dakrone/clj-http/pull/274 to update Potemkin so it
@@ -106,6 +226,9 @@ List of user-visible changes that have gone into each release
and clean up whitespace for new clojure-mode
- Merged https://github.com/dakrone/clj-http/pull/171 to support SOCKS proxies
* Work log
+** 2015-07-24
+- branched master to create 2.x
+- start major rewrite on master branch for non-deprecated Apache usage
** Released 2.0.0
** 2015-07-18
- merged https://github.com/dakrone/clj-http/pull/274 to update Potemkin so it
diff --git a/debian/changelog b/debian/changelog
index 31650c1..58c1ed6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,9 +1,13 @@
-clj-http-clojure (2.3.0-2) UNRELEASED; urgency=normal
+clj-http-clojure (3.12.3+git20221108.0.1cc8be6-1) UNRELEASED; urgency=normal
[ Louis-Philippe Véronneau ]
* d/control: Migrate to the Clojure Team.
- -- Louis-Philippe Véronneau <pollo@debian.org> Sat, 08 Jan 2022 17:54:11 -0500
+ [ Debian Janitor ]
+ * New upstream snapshot.
+ * New upstream snapshot.
+
+ -- Louis-Philippe Véronneau <pollo@debian.org> Wed, 18 Jan 2023 23:44:38 -0000
clj-http-clojure (2.3.0-1) unstable; urgency=medium
diff --git a/examples/body_coercion.clj b/examples/body_coercion.clj
new file mode 100644
index 0000000..040a397
--- /dev/null
+++ b/examples/body_coercion.clj
@@ -0,0 +1,23 @@
+(ns clj-http.examples.body-coercion
+ (:require [clj-http.client :as http]
+ [camel-snake-kebab.core :refer [->kebab-case-keyword]]))
+
+;; register your own body coercers by participating in the coerce-response-body multimethod
+;; dispatch to it by using {:as :json-kebab-keys} as an argument to http client calls
+
+;; this example uses camel-snake-kebab to turn a camel-cased JSON API into
+;; idiomatic kebab-cased keywords in clojure data structures and is much
+;; faster than applying via postwalk or similar
+
+(defmethod http/coerce-response-body :json-kebab-keys [req resp]
+ (http/coerce-json-body req resp (memoize ->kebab-case-keyword) false))
+
+;; example of use; note that in the response, the first field is called userId
+;;
+;; (:body (http/get "http://jsonplaceholder.typicode.com/posts/1" {:as :json-kebab-keys}))
+;; =>
+;; {:user-id 1,
+;; :id 1,
+;; :title "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
+;; :body "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
+;; }
diff --git a/examples/caching_middleware.clj b/examples/caching_middleware.clj
new file mode 100644
index 0000000..7e71b23
--- /dev/null
+++ b/examples/caching_middleware.clj
@@ -0,0 +1,74 @@
+(ns clj-http.examples.caching-middleware
+ "Example middleware that caches successful GET requests using core.cache."
+ (:require
+ [clj-http.client :as http]
+ [clojure.core.cache :as cache])
+ (:import
+ (java.nio.charset StandardCharsets)))
+
+(def http-cache (atom (cache/ttl-cache-factory {} :ttl (* 60 60 1000))))
+
+(defn slurp-bytes
+ "Read all bytes from the stream.
+ Use for example when the bytes will be in demand after stream has been closed."
+ [stream]
+ (.getBytes (slurp stream) StandardCharsets/UTF_8))
+
+(defn- cached-response
+ "Look up the response in the cache using URL as the cache key.
+ If the cache has the response, return the cached value.
+ If the cache does not have the response, invoke the remaining middleware functions
+ to perform the request and receive the response.
+ If the response is successful (2xx) and is a GET, store the response in the cache.
+ Return the response."
+ ([client req]
+ (let [cache-key (str (:server-name req) (:uri req) "?" (:query-string req))]
+ (if (cache/has? @http-cache cache-key)
+ (do
+ (println "CACHE HIT")
+ (reset! http-cache (cache/hit @http-cache cache-key)) ; update cache stats
+ (cache/lookup @http-cache cache-key)) ; return cached value
+ ; do not invoke further middleware
+ (do
+ (println "CACHE MISS")
+ (let [resp (update (client req) :body slurp-bytes)] ; middleware chain invoked
+ (when (and (http/success? resp) (= (:request-method req) :get))
+ (reset! http-cache (cache/miss @http-cache cache-key resp)) ; update cache value
+ resp)))))))
+
+(defn wrap-caching-middleware
+ "Middleware are functions that add functionality to handlers.
+ The argument client is a handler.
+ This wrapper function adds response caching to the client."
+ [client]
+ (fn
+ ([req]
+ (cached-response client req))))
+
+(defn example
+ "Add the caching middleware and perform a GET request using the URI argument.
+ Subsequent invocations of this function using an identical URI argument
+ before the Time To Live expires can be expected to hit the cache."
+ [& uri]
+ (-> (time (http/with-additional-middleware [#'wrap-caching-middleware]
+ (http/get (or uri "https://api.github.com")
+ {
+ ;; :debug true
+ ;; :debug-body true
+ ;; :throw-entire-message? true
+ })))
+ (select-keys ,,, [:status :reason-phrase :headers])))
+
+;; Try this out:
+;;
+;; user> (use '[clj-http.examples.caching-middleware :as mw])
+;; nil
+;; user> (mw/example)
+;; CACHE MISS
+;; "Elapsed time: 1910.027361 msecs"
+;; {:status 200, :reason-phrase "OK"}
+;; user> (mw/example)
+;; CACHE HIT
+;; "Elapsed time: 0.83484 msecs"
+;; {:status 200, :reason-phrase "OK"}
+;; user>
diff --git a/examples/kubernetes_pod.clj b/examples/kubernetes_pod.clj
new file mode 100644
index 0000000..4add03b
--- /dev/null
+++ b/examples/kubernetes_pod.clj
@@ -0,0 +1,20 @@
+(:ns clj-http.examples.kubernetes-pod
+ "This is an example of calling the Kubernetes API from inside a pod. K8s uses a
+ custom CA so that you can authenticate the API server, and provides a token per pod
+ so that each pod can authenticate itself with the APi server.
+
+ If you are still having 401/403 errors, look carefully at the message, if it includes
+ a ServiceAccount name, this part worked, and your problem is likely at the Role/RoleBinding level."
+ (:require [clj-http.client :as http]
+ [less.awful.ssl :refer [trust-store]]))
+
+;; Note that this is not a working example, you'll need to figure out your K8s API path.
+(let [k8s-trust-store (trust-store (clojure.java.io/file "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"))
+ bearer-token (format "Bearer %s" (slurp "/var/run/secrets/kubernetes.io/serviceaccount/token"))
+ kube-api-host (System/getenv "KUBERNETES_SERVICE_HOST")
+ kube-api-port (System/getenv "KUBERNETES_SERVICE_PORT")]
+ (http/get
+ (format "https://%s:%s/apis/<something-protected>" kube-api-host kube-api-port)
+ {:trust-store k8s-trust-store
+ :headers {:authorization bearer-token}}))
+
diff --git a/examples/logging-apache-requests.clj b/examples/logging-apache-requests.clj
new file mode 100644
index 0000000..8c1d573
--- /dev/null
+++ b/examples/logging-apache-requests.clj
@@ -0,0 +1,23 @@
+(ns clj-http.examples.logging-apache-requests
+ "This is an example of configuring Apache's log4j2 logging from Clojure, so
+ that the http client logging can be seen"
+ (:require [clj-http.client :as http])
+ (:import (org.apache.logging.log4j Level
+ LogManager)))
+
+;; This is a helper function to change the log level for log4j2. If you use a
+;; different logging framework (and subsequently a different bridge for log4j
+;; then you'll need to substitute your own logging configuration
+(defn change-log-level! [logger-name level]
+ (let [ctx (LogManager/getContext false)
+ config (.getConfiguration ctx)
+ logger-config (.getLoggerConfig config logger-name)]
+ (.setLevel logger-config level)
+ (.updateLoggers ctx)))
+
+;; Here is an example of using it to change the root logger to "DEBUG" and the
+;; back to "INFO" after a request has been completed
+(defn post-page-with-debug []
+ (change-log-level! LogManager/ROOT_LOGGER_NAME Level/DEBUG)
+ (http/post "https://httpbin.org/post" {:body "this is a test"})
+ (change-log-level! LogManager/ROOT_LOGGER_NAME Level/INFO))
diff --git a/examples/progress_download.clj b/examples/progress_download.clj
index 62c86ab..1c2a4a1 100644
--- a/examples/progress_download.clj
+++ b/examples/progress_download.clj
@@ -27,7 +27,7 @@
"Addes value into a vector at an specific index."
(-> (subvec v 0 idx)
(conj val)
- (concat (subvec v idx))))
+ (into (subvec v idx))))
(defn insert-after [v needle val]
"Finds an item into a vector and adds val just after it.
diff --git a/project.clj b/project.clj
index b2991d6..52f6afe 100644
--- a/project.clj
+++ b/project.clj
@@ -1,4 +1,4 @@
-(defproject clj-http "2.3.0"
+(defproject clj-http "3.12.4-SNAPSHOT"
:description "A Clojure HTTP library wrapping the Apache HttpComponents client."
:url "https://github.com/dakrone/clj-http/"
:license {:name "The MIT License"
@@ -7,28 +7,44 @@
:global-vars {*warn-on-reflection* false}
:min-lein-version "2.0.0"
:exclusions [org.clojure/clojure]
- :dependencies [[org.apache.httpcomponents/httpcore "4.4.5"]
- [org.apache.httpcomponents/httpclient "4.5.2"]
- [org.apache.httpcomponents/httpmime "4.5.2"]
- [commons-codec "1.10"]
- [commons-io "2.5"]
+ :dependencies [[org.apache.httpcomponents/httpcore "4.4.14"]
+ [org.apache.httpcomponents/httpclient "4.5.13"]
+ [org.apache.httpcomponents/httpclient-cache "4.5.13"]
+ [org.apache.httpcomponents/httpasyncclient "4.1.4"]
+ [org.apache.httpcomponents/httpmime "4.5.13"]
+ [commons-codec "1.15"]
+ [commons-io "2.8.0"]
[slingshot "0.12.2"]
- [potemkin "0.4.3"]]
+ [potemkin "0.4.5"]]
+ :resource-paths ["resources"]
:profiles {:dev {:dependencies [;; optional deps
- [cheshire "5.6.3"]
- [crouton "0.1.2"]
- [org.clojure/tools.reader "0.10.0"]
- [com.cognitect/transit-clj "0.8.288"]
- [ring/ring-codec "1.0.1"]
+ [cheshire "5.10.0"]
+ [crouton "0.1.2" :exclusions [[org.jsoup/jsoup]]]
+ [org.jsoup/jsoup "1.13.1"]
+ [org.clojure/tools.reader "1.3.5"]
+ [com.cognitect/transit-clj "1.0.324"]
+ [ring/ring-codec "1.1.3"]
;; other (testing) deps
- [org.clojure/clojure "1.8.0"]
- [org.clojure/tools.logging "0.3.1"]
- [log4j "1.2.17"]
- [ring/ring-jetty-adapter "1.5.0"]
- [ring/ring-devel "1.5.0"]]}
+ [org.clojure/clojure "1.10.3"]
+ [org.clojure/tools.logging "1.1.0"]
+ [ring/ring-jetty-adapter "1.9.3"]
+ [ring/ring-devel "1.9.3"]
+ ;; caching example deps
+ [org.clojure/core.cache "1.0.207"]
+ ;; logging
+ [org.apache.logging.log4j/log4j-api "2.17.1"]
+ [org.apache.logging.log4j/log4j-core "2.17.1"]
+ [org.apache.logging.log4j/log4j-1.2-api "2.17.1"]]
+ :plugins [[lein-ancient "0.7.0"]
+ [jonase/eastwood "0.2.5"]
+ [lein-kibit "0.1.5"]
+ [lein-nvd "0.5.2"]]}
:1.6 {:dependencies [[org.clojure/clojure "1.6.0"]]}
- :1.7 {:dependencies [[org.clojure/clojure "1.7.0"]]}}
- :aliases {"all" ["with-profile" "dev,1.6:dev,1.7:dev"]}
+ :1.7 {:dependencies [[org.clojure/clojure "1.7.0"]]}
+ :1.8 {:dependencies [[org.clojure/clojure "1.8.0"]]}
+ :1.9 {:dependencies [[org.clojure/clojure "1.9.0"]]}
+ :1.10 {:dependencies [[org.clojure/clojure "1.10.1"]]}}
+ :aliases {"all" ["with-profile" "dev,1.6:dev,1.7:dev,1.8:dev,1.9:dev,1.10:dev"]}
:plugins [[codox "0.6.4"]]
:test-selectors {:default #(not (:integration %))
:integration :integration
diff --git a/resources/example-log4j2.properties b/resources/example-log4j2.properties
new file mode 100644
index 0000000..faa1bd1
--- /dev/null
+++ b/resources/example-log4j2.properties
@@ -0,0 +1,38 @@
+###
+# While no means required, this is an example log4j2.properties that you can use
+# for debugging clj-http (mostly the apache http client side). See the readme or
+# examples directory for how to use it.
+
+# Change this to "debug" to get debugging information
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = fileLogger
+
+# Give directory path where log files should get stored
+property.basePath = ./log/
+status = error
+
+# ConsoleAppender will print logs on console
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+# Specify the pattern of the logs
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
+
+# RollingFileAppender will print logs in file which can be rotated based on time
+# or size
+appender.rolling.type = RollingFile
+appender.rolling.name = fileLogger
+appender.rolling.fileName=${basePath}/clj-http.log
+appender.rolling.filePattern=${basePath}clj-http_%d{yyyyMMdd}.log.gz
+appender.rolling.layout.type = PatternLayout
+appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
+appender.rolling.policies.type = Policies
+
+# Mention package name here in place of example. Classes in this package or
+# subpackages will use ConsoleAppender and RollingFileAppender for logging
+logger.example.name = example
+logger.example.level = debug
+logger.example.additivity = false
+logger.example.appenderRef.rolling.ref = fileLogger
+logger.example.appenderRef.console.ref = console
diff --git a/src/clj_http/client.clj b/src/clj_http/client.clj
index ae6815c..8c38cf0 100644
--- a/src/clj_http/client.clj
+++ b/src/clj_http/client.clj
@@ -1,21 +1,22 @@
(ns clj-http.client
"Batteries-included HTTP client."
+ (:refer-clojure :exclude [get update])
(:require [clj-http.conn-mgr :as conn]
[clj-http.cookies :refer [wrap-cookies]]
[clj-http.core :as core]
[clj-http.headers :refer [wrap-header-map]]
[clj-http.links :refer [wrap-links]]
- [clj-http.util :refer [opt] :as util]
+ [clj-http.util :as util :refer [opt]]
+ [clojure.java.io :as io]
[clojure.stacktrace :refer [root-cause]]
[clojure.string :as str]
[clojure.walk :refer [keywordize-keys prewalk]]
[slingshot.slingshot :refer [throw+]])
- (:import (java.io InputStream File ByteArrayOutputStream ByteArrayInputStream)
- (java.net URL UnknownHostException)
- (org.apache.http.entity BufferedHttpEntity ByteArrayEntity
- InputStreamEntity FileEntity StringEntity)
- (org.apache.http.impl.conn PoolingClientConnectionManager))
- (:refer-clojure :exclude [get update]))
+ (:import [java.io BufferedReader ByteArrayInputStream ByteArrayOutputStream EOFException File InputStream]
+ [java.net UnknownHostException URL]
+ [org.apache.http.entity BufferedHttpEntity ByteArrayEntity FileEntity InputStreamEntity StringEntity]
+ org.apache.http.impl.conn.PoolingHttpClientConnectionManager
+ org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager))
;; Cheshire is an optional dependency, so we check for it at compile time.
(def json-enabled?
@@ -98,11 +99,17 @@
(defn ^:dynamic parse-transit
"Resolve and apply Transit's JSON/MessagePack decoding."
- [in type & [opts]]
+ [^InputStream in type & [opts]]
{:pre [transit-enabled?]}
- (let [reader (ns-resolve 'cognitect.transit 'reader)
- read (ns-resolve 'cognitect.transit 'read)]
- (read (reader in type (transit-read-opts opts)))))
+ (try
+ (let [reader (ns-resolve 'cognitect.transit 'reader)
+ read (ns-resolve 'cognitect.transit 'read)]
+ (read (reader in type (transit-read-opts opts))))
+ (catch RuntimeException e
+ ;; Ignore exceptions from trying to read an empty stream.
+ (if (instance? EOFException (.getCause e))
+ nil
+ (throw e)))))
(defn ^:dynamic transit-encode
"Resolve and apply Transit's JSON/MessagePack encoding."
@@ -124,14 +131,10 @@
"Resolve and apply cheshire's json decoding dynamically."
[& args]
{:pre [json-enabled?]}
- (apply (ns-resolve (symbol "cheshire.core") (symbol "decode")) args))
-
-(defn ^:dynamic json-decode-strict
- "Resolve and apply cheshire's json decoding dynamically (with lazy parsing
- disabled)."
- [& args]
- {:pre [json-enabled?]}
- (apply (ns-resolve (symbol "cheshire.core") (symbol "decode-strict")) args))
+ (if-let [json-decode-fn (ns-resolve (symbol "cheshire.core") (symbol "parse-stream-strict"))]
+ (apply json-decode-fn args)
+ (throw
+ (IllegalStateException. "Missing #'cheshire.core/parse-stream-strict. Ensure the version of `cheshire` is >= 5.9.0"))))
(defn ^:dynamic form-decode
"Resolve and apply ring-codec's form decoding dynamically."
@@ -176,14 +179,35 @@
{:scheme (keyword (.getProtocol url-parsed))
:server-name (.getHost url-parsed)
:server-port (when-pos (.getPort url-parsed))
+ :url url
:uri (url-encode-illegal-characters (.getPath url-parsed))
:user-info (if-let [user-info (.getUserInfo url-parsed)]
(util/url-decode user-info))
:query-string (url-encode-illegal-characters (.getQuery url-parsed))}))
+(defn unparse-url
+ "Takes a map of url-parts and generates a string representation.
+ WARNING: does not do any sort of encoding! Don't use this for strict RFC
+ following!"
+ [{:keys [scheme server-name server-port uri user-info query-string]}]
+ (str (name scheme) "://"
+ (if (seq user-info)
+ (str user-info "@" server-name)
+ server-name)
+ (when server-port
+ (str ":" server-port))
+ uri
+ (when (seq query-string)
+ (str "?" query-string))))
+
;; Statuses for which clj-http will not throw an exception
(def unexceptional-status?
- #{200 201 202 203 204 205 206 207 300 301 302 303 304 307})
+ #{200 201 202 203 204 205 206 207 300 301 302 303 304 307 308})
+
+(defn unexceptional-status-for-request?
+ [req status]
+ ((or (:unexceptional-status req) unexceptional-status?)
+ status))
;; helper methods to determine realm of a response
(defn success?
@@ -210,43 +234,122 @@
[{:keys [status]}]
(<= 500 status 599))
+(defn- exceptions-response
+ [req {:keys [status] :as resp}]
+ (if (unexceptional-status-for-request? req status)
+ resp
+ (if (false? (opt req :throw-exceptions))
+ resp
+ (let [data (assoc resp :type ::unexceptional-status)]
+ (if (opt req :throw-entire-message)
+ (throw+ data "clj-http: status %d %s" (:status %) resp)
+ (throw+ data "clj-http: status %s" (:status %)))))))
+
(defn wrap-exceptions
"Middleware that throws a slingshot exception if the response is not a
regular response. If :throw-entire-message? is set to true, the entire
response is used as the message, instead of just the status number."
[client]
- (fn [req]
- (let [{:keys [status] :as resp} (client req)]
- (if (unexceptional-status? status)
- resp
- (if (false? (opt req :throw-exceptions))
- resp
- (if (opt req :throw-entire-message)
- (throw+ resp "clj-http: status %d %s" (:status %) resp)
- (throw+ resp "clj-http: status %s" (:status %))))))))
+ (fn
+ ([req]
+ (exceptions-response req (client req)))
+ ([req response raise]
+ (client req
+ (fn [resp]
+ (response (exceptions-response req resp)))
+ raise))))
(declare wrap-redirects)
+(declare reuse-pool)
+
+(defn- follow-redirect-request
+ [req redirect trace-redirects resp]
+ (-> req
+ (merge (parse-url redirect))
+ (dissoc :query-params)
+ (assoc :url redirect)
+ (assoc :trace-redirects trace-redirects)
+ (reuse-pool resp)))
(defn follow-redirect
"Attempts to follow the redirects from the \"location\" header, if no such
header exists (bad server!), returns the response without following the
request."
- [client {:keys [uri url scheme server-name server-port] :as req}
+ [client {:keys [uri url scheme server-name server-port async? respond raise]
+ :as req}
{:keys [trace-redirects ^InputStream body] :as resp}]
(let [url (or url (str (name scheme) "://" server-name
(when server-port (str ":" server-port)) uri))]
(if-let [raw-redirect (get-in resp [:headers "location"])]
(let [redirect (str (URL. (URL. url) raw-redirect))]
(try (.close body) (catch Exception _))
- ((wrap-redirects client) (-> req
- (merge (parse-url redirect))
- (dissoc :query-params)
- (assoc :url redirect)
- (assoc :trace-redirects trace-redirects))))
+ (if-not async?
+ ((wrap-redirects client)
+ (follow-redirect-request req redirect trace-redirects resp))
+ (if (some nil? [respond raise])
+ (raise
+ (IllegalArgumentException.
+ "If :async? is true, you must set :respond and :raise"))
+ ((wrap-redirects client)
+ (follow-redirect-request req redirect trace-redirects resp)
+ respond raise))))
;; Oh well, we tried, but if no location is set, return the response
- resp)))
+ (if-not async?
+ resp
+ (respond resp)))))
-(defn wrap-redirects
+(defn- respond*
+ [resp req]
+ (if (opt req :async)
+ ((:respond req) resp)
+ resp))
+
+(defn- redirects-response
+ [client
+ {:keys [request-method max-redirects redirects-count trace-redirects url]
+ :or {redirects-count 1 trace-redirects []
+ ;; max-redirects default taken from Firefox
+ max-redirects 20}
+ :as req} {:keys [status] :as resp}]
+ (let [resp-r (assoc resp :trace-redirects
+ (if url
+ (conj trace-redirects url)
+ trace-redirects))]
+ (cond
+ (false? (opt req :follow-redirects))
+ (respond* resp req)
+ (not (redirect? resp-r))
+ (respond* resp-r req)
+ (and max-redirects (> redirects-count max-redirects))
+ (if (opt req :throw-exceptions)
+ (throw+ resp-r "Too many redirects: %s" redirects-count)
+ (respond* resp-r req))
+ (= 303 status)
+ (follow-redirect client (assoc req :request-method :get
+ :redirects-count (inc redirects-count))
+ resp-r)
+ (#{301 302} status)
+ (cond
+ (#{:get :head} request-method)
+ (follow-redirect client (assoc req :redirects-count
+ (inc redirects-count)) resp-r)
+ (opt req :force-redirects)
+ (follow-redirect client (assoc req
+ :request-method :get
+ :redirects-count (inc redirects-count))
+ resp-r)
+ :else
+ (respond* resp-r req))
+ (#{307 308} status)
+ (if (or (#{:get :head} request-method)
+ (opt req :force-redirects))
+ (follow-redirect client (assoc req :redirects-count
+ (inc redirects-count)) resp-r)
+ (respond* resp-r req))
+ :else
+ (respond* resp-r req))))
+
+(defn ^:deprecated wrap-redirects
"Middleware that follows redirects in the response. A slingshot exception is
thrown if too many redirects occur. Options
@@ -259,49 +362,17 @@
:redirects-count - number of redirects
:trace-redirects - vector of sites the request was redirected from"
[client]
- (fn [{:keys [request-method max-redirects redirects-count trace-redirects url]
- :or {redirects-count 1 trace-redirects []
- ;; max-redirects default taken from Firefox
- max-redirects 20}
- :as req}]
- (let [{:keys [status] :as resp} (client req)
- resp-r (assoc resp :trace-redirects
- (if url
- (conj trace-redirects url)
- trace-redirects))]
- (cond
- (false? (opt req :follow-redirects))
- resp
- (not (redirect? resp-r))
- resp-r
- (and max-redirects (> redirects-count max-redirects))
- (if (opt req :throw-exceptions)
- (throw+ resp-r "Too many redirects: %s" redirects-count)
- resp-r)
- (= 303 status)
- (follow-redirect client (assoc req :request-method :get
- :redirects-count (inc redirects-count))
- resp-r)
- (#{301 302} status)
- (cond
- (#{:get :head} request-method)
- (follow-redirect client (assoc req :redirects-count
- (inc redirects-count)) resp-r)
- (opt req :force-redirects)
- (follow-redirect client (assoc req
- :request-method :get
- :redirects-count (inc redirects-count))
- resp-r)
- :else
- resp-r)
- (= 307 status)
- (if (or (#{:get :head} request-method)
- (opt req :force-redirects))
- (follow-redirect client (assoc req :redirects-count
- (inc redirects-count)) resp-r)
- resp-r)
- :else
- resp-r))))
+ (fn
+ ([req]
+ (redirects-response client req (client req)))
+ ([req respond raise]
+ (client req
+ #(redirects-response client
+ (assoc req :async? true
+ :respond respond
+ :raise raise)
+ %)
+ raise))))
;; Multimethods for Content-Encoding dispatch automatically
;; decompressing response bodies
@@ -327,75 +398,107 @@
:orig-content-encoding
(get-in resp [:headers "content-encoding"])))
+(defn- decompression-request
+ [req]
+ (if (false? (opt req :decompress-body))
+ req
+ (update-in req [:headers "accept-encoding"]
+ #(str/join ", " (remove nil? [% "gzip, deflate"])))))
+
+(defn- decompression-response
+ [req resp]
+ (if (false? (opt req :decompress-body))
+ resp
+ (decompress-body resp)))
+
(defn wrap-decompression
"Middleware handling automatic decompression of responses from web servers. If
:decompress-body is set to false, does not automatically set `Accept-Encoding`
header or decompress body."
[client]
- (fn [req]
- (if (false? (opt req :decompress-body))
- (client req)
- (let [req-c (update req :headers assoc "accept-encoding" "gzip, deflate")
- resp-c (client req-c)]
- (decompress-body resp-c)))))
+ (fn
+ ([req]
+ (decompression-response req (client (decompression-request req))))
+ ([req respond raise]
+ (client (decompression-request req)
+ #(respond (decompression-response req %))
+ raise))))
;; Multimethods for coercing body type to the :as key
(defmulti coerce-response-body (fn [req _] (:as req)))
(defmethod coerce-response-body :byte-array [_ resp]
- (assoc resp :body (util/force-byte-array (:body resp))))
+ (update resp :body util/force-byte-array))
(defmethod coerce-response-body :stream [_ resp]
- (let [body (:body resp)]
- (cond (instance? InputStream body) resp
- ;; This shouldn't happen, but we plan for it anyway
- (instance? (Class/forName "[B") body)
- (assoc resp :body (ByteArrayInputStream. body)))))
+ (update resp :body util/force-stream))
+
+(defn- response-charset [response]
+ (or (-> response :content-type-params :charset)
+ "UTF-8"))
+
+(defmethod coerce-response-body :reader
+ [_ {:keys [body] :as resp}]
+ (let [header (get-in resp [:headers "content-type"])
+ parsed-values (util/parse-content-type header)
+ charset (response-charset parsed-values)]
+ (assoc resp :body (io/reader body :encoding charset))))
+
+(defn- can-parse-body? [{:keys [coerce] :as request} {:keys [status] :as _response}]
+ (or (= coerce :always)
+ (and (unexceptional-status-for-request? request status)
+ (or (nil? coerce)
+ (= coerce :unexceptional)))
+ (and (not (unexceptional-status-for-request? request status))
+ (= coerce :exceptional))))
+
+(defn- decode-json-body [body keyword? charset]
+ (let [^BufferedReader br (io/reader (util/force-stream body) :encoding charset)]
+ (try
+ (.mark br 1)
+ (let [first-char (int (try (.read br) (catch EOFException _ -1)))]
+ (case first-char
+ -1 nil
+ (do (.reset br)
+ (json-decode br keyword?))))
+ (finally (.close br)))))
(defn coerce-json-body
- [{:keys [coerce]} {:keys [body status] :as resp} keyword? strict? & [charset]]
- (let [^String charset (or charset (-> resp :content-type-params :charset)
- "UTF-8")
- body (util/force-byte-array body)
- decode-func (if strict? json-decode-strict json-decode)]
- (if json-enabled?
- (cond
- (= coerce :always)
- (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
-
- (and (unexceptional-status? status)
- (or (nil? coerce) (= coerce :unexceptional)))
- (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
-
- (and (not (unexceptional-status? status)) (= coerce :exceptional))
- (assoc resp :body (decode-func (String. ^"[B" body charset) keyword?))
-
- :else (assoc resp :body (String. ^"[B" body charset)))
- (assoc resp :body (String. ^"[B" body charset)))))
+ [request {:keys [body] :as resp} keyword? & [charset]]
+ {:pre [json-enabled?]}
+ (let [charset (or charset (response-charset resp))
+ body (if (can-parse-body? request resp)
+ (decode-json-body body keyword? charset)
+ (util/force-string body charset))]
+ (assoc resp :body body)))
(defn coerce-clojure-body
- [request {:keys [body] :as resp}]
- (let [^String charset (or (-> resp :content-type-params :charset) "UTF-8")
- body (util/force-byte-array body)]
- (if edn-enabled?
- (assoc resp :body (parse-edn (String. ^"[B" body charset)))
- (binding [*read-eval* false]
- (assoc resp :body (read-string (String. ^"[B" body charset)))))))
+ [_request {:keys [body] :as resp}]
+ (let [charset (response-charset resp)
+ body (util/force-string body charset)]
+ (assoc resp :body (cond
+ (empty? body) nil
+ edn-enabled? (parse-edn body)
+ :else (binding [*read-eval* false]
+ (read-string body))))))
(defn coerce-transit-body
- [{:keys [transit-opts] :as request} {:keys [body] :as resp} type]
- (if transit-enabled?
- (assoc resp :body (parse-transit body type transit-opts))
- resp))
+ [{:keys [transit-opts] :as request}
+ {:keys [body] :as resp} type & [charset]]
+ {:pre [transit-enabled?]}
+ (let [charset (or charset (response-charset resp))
+ body (if (can-parse-body? request resp)
+ (with-open [in (util/force-stream body)]
+ (parse-transit in type transit-opts))
+ (util/force-string body charset))]
+ (assoc resp :body body)))
(defn coerce-form-urlencoded-body
- [request {:keys [body] :as resp}]
- (let [^String charset (or (-> resp :content-type-params :charset) "UTF-8")
- body-bytes (util/force-byte-array body)]
- (if ring-codec-enabled?
- (assoc resp :body (-> (String. ^"[B" body-bytes charset)
- form-decode keywordize-keys))
- (assoc resp :body (String. ^"[B" body-bytes charset)))))
+ [_request {:keys [body] :as resp}]
+ {:pre [ring-codec-enabled?]}
+ (let [charset (response-charset resp)
+ body (util/force-string body charset)]
+ (assoc resp :body (-> body form-decode keywordize-keys))))
(defmulti coerce-content-type (fn [req resp] (:content-type resp)))
@@ -428,16 +531,20 @@
(coerce-content-type request))))
(defmethod coerce-response-body :json [req resp]
- (coerce-json-body req resp true false))
+ (coerce-json-body req resp true))
+(defmethod coerce-response-body :json-string-keys [req resp]
+ (coerce-json-body req resp false))
+
+;; There is no longer any distinction between strict and non-strict JSON parsing
+;; options.
+;;
+;; `:json-strict` and `:json-strict-string-keys` will be removed in a future version
(defmethod coerce-response-body :json-strict [req resp]
- (coerce-json-body req resp true true))
+ (coerce-json-body req resp true))
(defmethod coerce-response-body :json-strict-string-keys [req resp]
- (coerce-json-body req resp false true))
-
-(defmethod coerce-response-body :json-string-keys [req resp]
- (coerce-json-body req resp false false))
+ (coerce-json-body req resp false))
(defmethod coerce-response-body :clojure [req resp]
(coerce-clojure-body req resp))
@@ -453,10 +560,13 @@
(defmethod coerce-response-body :default
[{:keys [as]} {:keys [body] :as resp}]
- (let [body-bytes (util/force-byte-array body)]
- (cond
- (string? as) (assoc resp :body (String. ^"[B" body-bytes ^String as))
- :else (assoc resp :body (String. ^"[B" body-bytes "UTF-8")))))
+ (assoc resp :body (util/force-string body (if (string? as) as "UTF-8"))))
+
+(defn- output-coercion-response
+ [req {:keys [body] :as resp}]
+ (if body
+ (coerce-response-body req resp)
+ resp))
(defn wrap-output-coercion
"Middleware converting a response body from a byte-array to a different
@@ -464,11 +574,13 @@
`coerce-response-body` multimethod may be extended to add
additional coercions."
[client]
- (fn [req]
- (let [{:keys [body] :as resp} (client req)]
- (if body
- (coerce-response-body req resp)
- resp))))
+ (fn
+ ([req]
+ (output-coercion-response req (client req)))
+ ([req respond raise]
+ (client req
+ #(respond (output-coercion-response req %))
+ raise))))
(defn maybe-wrap-entity
"Wrap an HttpEntity in a BufferedHttpEntity if warranted."
@@ -477,45 +589,52 @@
(BufferedHttpEntity. entity)
entity))
+(defn- input-coercion-request
+ [{:keys [body body-encoding length]
+ :or {^String body-encoding "UTF-8"} :as req}]
+ (if body
+ (cond
+ (string? body)
+ (-> req (assoc :body (maybe-wrap-entity
+ req (StringEntity. ^String body
+ ^String body-encoding))
+ :character-encoding (or body-encoding
+ "UTF-8")))
+ (instance? File body)
+ (-> req (assoc :body
+ (maybe-wrap-entity
+ req (FileEntity. ^File body
+ ^String body-encoding))))
+
+ ;; A length of -1 instructs HttpClient to use chunked encoding.
+ (instance? InputStream body)
+ (-> req
+ (assoc :body
+ (if length
+ (InputStreamEntity.
+ ^InputStream body (long length))
+ (maybe-wrap-entity
+ req
+ (InputStreamEntity. ^InputStream body -1)))))
+
+ (instance? (Class/forName "[B") body)
+ (-> req (assoc :body (maybe-wrap-entity
+ req (ByteArrayEntity. body))))
+
+ :else
+ req)
+ req))
+
(defn wrap-input-coercion
"Middleware coercing the :body of a request from a number of formats into an
Apache Entity. Currently supports Strings, Files, InputStreams
and byte-arrays."
[client]
- (fn [{:keys [body body-encoding length]
- :or {^String body-encoding "UTF-8"} :as req}]
- (if body
- (cond
- (string? body)
- (client (-> req (assoc :body (maybe-wrap-entity
- req (StringEntity. ^String body
- ^String body-encoding))
- :character-encoding (or body-encoding
- "UTF-8"))))
- (instance? File body)
- (client (-> req (assoc :body
- (maybe-wrap-entity
- req (FileEntity. ^File body
- ^String body-encoding)))))
-
- ;; A length of -1 instructs HttpClient to use chunked encoding.
- (instance? InputStream body)
- (client (-> req
- (assoc :body
- (if length
- (InputStreamEntity.
- ^InputStream body (long length))
- (maybe-wrap-entity
- req
- (InputStreamEntity. ^InputStream body -1))))))
-
- (instance? (Class/forName "[B") body)
- (client (-> req (assoc :body (maybe-wrap-entity
- req (ByteArrayEntity. body)))))
-
- :else
- (client req))
- (client req))))
+ (fn
+ ([req]
+ (client (input-coercion-request req)))
+ ([req respond raise]
+ (client (input-coercion-request req) respond raise))))
(defn get-headers-from-body
"Given a map of body content, return a map of header-name to header-value."
@@ -541,6 +660,24 @@
{"content-type" (str "text/html; charset=" cs)}))]
headers))
+(defn- additional-header-parsing-response
+ [req resp]
+ (if (and (opt req :decode-body-headers)
+ crouton-enabled?
+ (:body resp)
+ (let [^String content-type (get-in resp [:headers "content-type"])]
+ (or (str/blank? content-type)
+ (.startsWith content-type "text"))))
+ (let [body-bytes (util/force-byte-array (:body resp))
+ body-stream1 (java.io.ByteArrayInputStream. body-bytes)
+ body-map (parse-html body-stream1)
+ additional-headers (get-headers-from-body body-map)
+ body-stream2 (java.io.ByteArrayInputStream. body-bytes)]
+ (assoc resp
+ :headers (merge (:headers resp) additional-headers)
+ :body body-stream2))
+ resp))
+
(defn wrap-additional-header-parsing
"Middleware that parses additional http headers from the body of a web page,
adding them into the headers map of the response if any are found. Only looks
@@ -548,65 +685,76 @@
be silently disabled if crouton is excluded from clj-http's dependencies. Will
do nothing if no body is returned, e.g. HEAD requests"
[client]
- (fn [req]
- (let [resp (client req)]
- (if (and (opt req :decode-body-headers)
- crouton-enabled?
- (:body resp)
- (let [content-type (get-in resp [:headers "content-type"])]
- (or (str/blank? content-type)
- (.startsWith content-type "text"))))
- (let [body-bytes (util/force-byte-array (:body resp))
- body-stream1 (java.io.ByteArrayInputStream. body-bytes)
- body-map (parse-html body-stream1)
- additional-headers (get-headers-from-body body-map)
- body-stream2 (java.io.ByteArrayInputStream. body-bytes)]
- (assoc resp
- :headers (merge (:headers resp) additional-headers)
- :body body-stream2))
- resp))))
+ (fn
+ ([req]
+ (additional-header-parsing-response req (client req)))
+ ([req respond raise]
+ (client req
+ #(respond (additional-header-parsing-response req %)) raise))))
(defn content-type-value [type]
(if (keyword? type)
(str "application/" (name type))
type))
+(defn- content-type-request
+ [{:keys [content-type character-encoding] :as req}]
+ (if content-type
+ (let [ctv (content-type-value content-type)
+ ct (if character-encoding
+ (str ctv "; charset=" character-encoding)
+ ctv)]
+ (update-in req [:headers] assoc "content-type" ct))
+ req))
+
(defn wrap-content-type
"Middleware converting a `:content-type <keyword>` option to the formal
application/<name> format and adding it as a header."
[client]
- (fn [{:keys [content-type character-encoding] :as req}]
- (if content-type
- (let [ctv (content-type-value content-type)
- ct (if character-encoding
- (str ctv "; charset=" character-encoding)
- ctv)]
- (client (update-in req [:headers] assoc "content-type" ct)))
- (client req))))
+ (fn
+ ([req]
+ (client (content-type-request req)))
+ ([req respond raise]
+ (client (content-type-request req) respond raise))))
+
+(defn- accept-request
+ [{:keys [accept] :as req}]
+ (if accept
+ (-> req (dissoc :accept)
+ (assoc-in [:headers "accept"]
+ (content-type-value accept)))
+ req))
(defn wrap-accept
"Middleware converting the :accept key in a request to application/<type>"
[client]
- (fn [{:keys [accept] :as req}]
- (if accept
- (client (-> req (dissoc :accept)
- (assoc-in [:headers "accept"]
- (content-type-value accept))))
- (client req))))
+ (fn
+ ([req]
+ (client (accept-request req)))
+ ([req respond raise]
+ (client (accept-request req) respond raise))))
(defn accept-encoding-value [accept-encoding]
(str/join ", " (map name accept-encoding)))
+(defn- accept-encoding-request
+ [{:keys [accept-encoding] :as req}]
+ (if accept-encoding
+ (-> req
+ (dissoc :accept-encoding)
+ (assoc-in [:headers "accept-encoding"]
+ (accept-encoding-value accept-encoding)))
+ req))
+
(defn wrap-accept-encoding
"Middleware converting the :accept-encoding option to an acceptable
Accept-Encoding header in the request."
[client]
- (fn [{:keys [accept-encoding] :as req}]
- (if accept-encoding
- (client (-> req (dissoc :accept-encoding)
- (assoc-in [:headers "accept-encoding"]
- (accept-encoding-value accept-encoding))))
- (client req))))
+ (fn
+ ([req]
+ (client (accept-encoding-request req)))
+ ([req respond raise]
+ (client (accept-encoding-request req) respond raise))))
(defn detect-charset
"Given a charset header, detect the charset, returns UTF-8 if not found."
@@ -617,20 +765,28 @@
(second found))
"UTF-8"))
-(defn- multi-param-suffix [index multi-param-style]
- (case multi-param-style
- :indexed (str "[" index "]")
- :array "[]"
- ""))
+(defn- multi-param-entries [key values multi-param-style encoding]
+ (let [key (util/url-encode (name key) encoding)
+ values (map #(util/url-encode (str %) encoding) values)]
+ (case multi-param-style
+ :indexed
+ (map-indexed #(vector (str key \[ %1 \]) %2) values)
+
+ :array
+ (map #(vector (str key "[]") %) values)
+
+ :comma-separated
+ ;; See sub-delims in https://tools.ietf.org/html/rfc3986#section-2.2
+ [[key (str/join "," values)]]
+
+ ;; default: repeat the key multiple times
+ (map #(vector key %) values))))
(defn generate-query-string-with-encoding [params encoding multi-param-style]
(str/join "&"
(mapcat (fn [[k v]]
(if (sequential? v)
- (map-indexed #(str (util/url-encode (name k) encoding)
- (multi-param-suffix %1 multi-param-style)
- "="
- (util/url-encode (str %2) encoding)) v)
+ (map #(str/join "=" %) (multi-param-entries k v multi-param-style encoding))
[(str (util/url-encode (name k) encoding)
"="
(util/url-encode (str v) encoding))]))
@@ -640,25 +796,32 @@
(let [encoding (detect-charset content-type)]
(generate-query-string-with-encoding params encoding multi-param-style)))
+(defn- query-params-request
+ [{:keys [query-params content-type multi-param-style]
+ :or {content-type :x-www-form-urlencoded}
+ :as req}]
+ (if query-params
+ (-> req (dissoc :query-params)
+ (update-in [:query-string]
+ (fn [old-query-string new-query-string]
+ (if-not (empty? old-query-string)
+ (str old-query-string "&" new-query-string)
+ new-query-string))
+ (generate-query-string
+ query-params
+ (content-type-value content-type)
+ multi-param-style)))
+ req))
+
(defn wrap-query-params
"Middleware converting the :query-params option to a querystring on
the request."
[client]
- (fn [{:keys [query-params content-type multi-param-style]
- :or {content-type :x-www-form-urlencoded}
- :as req}]
- (if query-params
- (client (-> req (dissoc :query-params)
- (update-in [:query-string]
- (fn [old-query-string new-query-string]
- (if-not (empty? old-query-string)
- (str old-query-string "&" new-query-string)
- new-query-string))
- (generate-query-string
- query-params
- (content-type-value content-type)
- multi-param-style))))
- (client req))))
+ (fn
+ ([req]
+ (client (query-params-request req)))
+ ([req respond raise]
+ (client (query-params-request req) respond raise))))
(defn basic-auth-value [basic-auth]
(let [basic-auth (if (string? basic-auth)
@@ -666,47 +829,75 @@
(str (first basic-auth) ":" (second basic-auth)))]
(str "Basic " (util/base64-encode (util/utf8-bytes basic-auth)))))
+(defn- basic-auth-request
+ [req]
+ (if-let [basic-auth (:basic-auth req)]
+ (-> req (dissoc :basic-auth)
+ (assoc-in [:headers "authorization"]
+ (basic-auth-value basic-auth)))
+ req))
+
(defn wrap-basic-auth
"Middleware converting the :basic-auth option into an Authorization header."
[client]
- (fn [req]
- (if-let [basic-auth (:basic-auth req)]
- (client (-> req (dissoc :basic-auth)
- (assoc-in [:headers "authorization"]
- (basic-auth-value basic-auth))))
- (client req))))
+ (fn
+ ([req]
+ (client (basic-auth-request req)))
+ ([req respond raise]
+ (client (basic-auth-request req) respond raise))))
+
+(defn- oauth-request
+ [req]
+ (if-let [oauth-token (:oauth-token req)]
+ (-> req (dissoc :oauth-token)
+ (assoc-in [:headers "authorization"]
+ (str "Bearer " oauth-token)))
+ req))
(defn wrap-oauth
"Middleware converting the :oauth-token option into an Authorization header."
[client]
- (fn [req]
- (if-let [oauth-token (:oauth-token req)]
- (client (-> req (dissoc :oauth-token)
- (assoc-in [:headers "authorization"]
- (str "Bearer " oauth-token))))
- (client req))))
+ (fn
+ ([req]
+ (client (oauth-request req)))
+ ([req respond raise]
+ (client (oauth-request req) respond raise))))
(defn parse-user-info [user-info]
(when user-info
(str/split user-info #":")))
+(defn- user-info-request
+ [req]
+ (if-let [[user password] (parse-user-info (:user-info req))]
+ (assoc req :basic-auth [user password])
+ req))
+
(defn wrap-user-info
"Middleware converting the :user-info option into a :basic-auth option"
[client]
- (fn [req]
- (if-let [[user password] (parse-user-info (:user-info req))]
- (client (assoc req :basic-auth [user password]))
- (client req))))
+ (fn
+ ([req]
+ (client (user-info-request req)))
+ ([req respond raise]
+ (client (user-info-request req) respond raise))))
+
+(defn- method-request
+ [req]
+ (if-let [m (:method req)]
+ (-> req (dissoc :method)
+ (assoc :request-method m))
+ req))
(defn wrap-method
"Middleware converting the :method option into the :request-method option"
[client]
- (fn [req]
- (if-let [m (:method req)]
- (client (-> req (dissoc :method)
- (assoc :request-method m)))
- (client req))))
+ (fn
+ ([req]
+ (client (method-request req)))
+ ([req respond raise]
+ (client (method-request req) respond raise))))
(defmulti coerce-form-params
(fn [req] (keyword (content-type-value (:content-type req)))))
@@ -748,21 +939,31 @@
form-params
form-param-encoding]}]
(if form-param-encoding
- (generate-query-string-with-encoding form-params form-param-encoding multi-param-style)
- (generate-query-string form-params (content-type-value content-type) multi-param-style)))
+ (generate-query-string-with-encoding form-params
+ form-param-encoding multi-param-style)
+ (generate-query-string form-params
+ (content-type-value content-type)
+ multi-param-style)))
+
+(defn- form-params-request
+ [{:keys [form-params content-type request-method]
+ :or {content-type :x-www-form-urlencoded}
+ :as req}]
+ (if (and form-params (#{:post :put :patch :delete} request-method))
+ (-> req
+ (dissoc :form-params)
+ (assoc :content-type (content-type-value content-type)
+ :body (coerce-form-params req)))
+ req))
(defn wrap-form-params
"Middleware wrapping the submission or form parameters."
[client]
- (fn [{:keys [form-params content-type request-method]
- :or {content-type :x-www-form-urlencoded}
- :as req}]
- (if (and form-params (#{:post :put :patch} request-method))
- (client (-> req
- (dissoc :form-params)
- (assoc :content-type (content-type-value content-type)
- :body (coerce-form-params req))))
- (client req))))
+ (fn
+ ([req]
+ (client (form-params-request req)))
+ ([req respnd raise]
+ (client (form-params-request req) respnd raise))))
(defn- nest-params
[request param-key]
@@ -780,39 +981,81 @@
params))
request))
+(defn- nest-params-request
+ [{:keys [flatten-nested-keys] :as req}]
+ (if (seq flatten-nested-keys)
+ (reduce
+ nest-params
+ req
+ flatten-nested-keys)
+ req))
+
(defn wrap-nested-params
"Middleware wrapping nested parameters for query strings."
[client]
- (fn [{:keys [content-type]
- :as req}]
- (if (or (nil? content-type)
- (= content-type :x-www-form-urlencoded))
- (client (reduce
- nest-params
- req
- [:query-params :form-params]))
- (client req))))
+ (fn
+ ([req]
+ (client (nest-params-request req)))
+ ([req respond raise]
+ (client (nest-params-request req) respond raise))))
+
+(defn- nested-keys-to-flatten
+ [{:keys [flatten-nested-keys] :as req}]
+ (when (and (or (not (nil? (opt req :ignore-nested-query-string)))
+ (not (nil? (opt req :flatten-nested-form-params))))
+ flatten-nested-keys)
+ (throw (IllegalArgumentException.
+ (str "only :flatten-nested-keys or :ignore-nested-query-string/"
+ ":flatten-nested-keys may be specified, not both"))))
+ (let [iqs-key (when-not (opt req :ignore-nested-query-string) :query-params)
+ ifp-key (when (opt req :flatten-nested-form-params) :form-params)]
+ (or flatten-nested-keys
+ (remove nil? (list iqs-key ifp-key)))))
+
+(defn wrap-flatten-nested-params
+ "Middleware wrapping options for whether or not to flatten `:query-params` and
+ `:form-params`. Modifies the request by adding a `:flatten-nested-keys`
+ sequence of the nested keys that will be flattened."
+ [client]
+ (fn
+ ([req]
+ (client
+ (assoc req :flatten-nested-keys (nested-keys-to-flatten req))))
+ ([req respond raise]
+ (client
+ (assoc req :flatten-nested-keys (nested-keys-to-flatten req))
+ respond raise))))
+
+(defn- url-request
+ [req]
+ (if-let [url (:url req)]
+ (-> req (dissoc :url) (merge (parse-url url)))
+ req))
(defn wrap-url
"Middleware wrapping request URL parsing."
[client]
- (fn [req]
- (if-let [url (:url req)]
- (client (-> req (dissoc :url) (merge (parse-url url))))
- (client req))))
+ (fn
+ ([req]
+ (client (url-request req)))
+ ([req respond raise]
+ (client (url-request req) respond raise))))
(defn wrap-unknown-host
"Middleware ignoring unknown hosts when the :ignore-unknown-host? option
is set."
[client]
- (fn [req]
- (try
- (client req)
- (catch Exception e
- (if (= (type (root-cause e)) UnknownHostException)
- (when-not (opt req :ignore-unknown-host)
- (throw (root-cause e)))
- (throw (root-cause e)))))))
+ (fn
+ ([req]
+ (try
+ (client req)
+ (catch Exception e
+ (if (= (type (root-cause e)) UnknownHostException)
+ (when-not (opt req :ignore-unknown-host)
+ (throw (root-cause e)))
+ (throw (root-cause e))))))
+ ([req respond raise]
+ (client (assoc req :unknown-host-respond respond) respond raise))))
(defn wrap-lower-case-headers
"Middleware lowercasing all headers, as per RFC (case-insensitive) and
@@ -822,18 +1065,33 @@
#(if-let [headers (:headers %1)]
(assoc %1 :headers (util/lower-case-keys headers))
%1)]
- (fn [req]
- (-> (client (lower-case-headers req))
- (lower-case-headers)))))
+ (fn
+ ([req]
+ (-> (client (lower-case-headers req))
+ (lower-case-headers)))
+ ([req respond raise]
+ (client (lower-case-headers req)
+ #(respond (lower-case-headers %))
+ raise)))))
+
+(defn- request-timing-response
+ [resp start]
+ (assoc resp :request-time (- (System/currentTimeMillis) start)))
(defn wrap-request-timing
"Middleware that times the request, putting the total time (in milliseconds)
of the request into the :request-time key in the response."
[client]
- (fn [req]
- (let [start (System/currentTimeMillis)
- resp (client req)]
- (assoc resp :request-time (- (System/currentTimeMillis) start)))))
+ (fn
+ ([req]
+ (let [start (System/currentTimeMillis)
+ resp (client req)]
+ (request-timing-response resp start)))
+ ([req respond raise]
+ (let [start (System/currentTimeMillis)]
+ (client req
+ #(respond (request-timing-response % start))
+ raise)))))
(def default-middleware
"The default list of middleware clj-http uses for wrapping requests."
@@ -844,7 +1102,6 @@
wrap-oauth
wrap-user-info
wrap-url
- wrap-redirects
wrap-decompression
wrap-input-coercion
;; put this before output-coercion, so additional charset
@@ -857,6 +1114,7 @@
wrap-content-type
wrap-form-params
wrap-nested-params
+ wrap-flatten-nested-params
wrap-method
wrap-cookies
wrap-links
@@ -893,7 +1151,12 @@
* :accept-encoding
* :as
- The following additional behaviors over also automatically enabled:
+ The following keys make an async HTTP request, like ring's CPS handler.
+ * :async?
+ * :respond
+ * :raise
+
+ The following additional behaviors are also automatically enabled:
* Exceptions are thrown for status codes other than 200-207, 300-303, or 307
* Gzip and deflate responses are accepted and decompressed
* Input and output bodies are coerced as required and indicated by the :as
@@ -906,59 +1169,68 @@
`(when (nil? ~url)
(throw (IllegalArgumentException. "Host URL cannot be nil"))))
+(defn- request*
+ [req [respond raise]]
+ (if (opt req :async)
+ (if (some nil? [respond raise])
+ (throw (IllegalArgumentException.
+ "If :async? is true, you must pass respond and raise"))
+ (request (dissoc req :respond :raise) respond raise))
+ (request req)))
+
(defn get
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :get :url url})))
+ (request* (merge req {:method :get :url url}) r))
(defn head
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :head :url url})))
+ (request* (merge req {:method :head :url url}) r))
(defn post
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :post :url url})))
+ (request* (merge req {:method :post :url url}) r))
(defn put
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :put :url url})))
+ (request* (merge req {:method :put :url url}) r))
(defn delete
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :delete :url url})))
+ (request* (merge req {:method :delete :url url}) r))
(defn options
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :options :url url})))
+ (request* (merge req {:method :options :url url}) r))
(defn copy
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :copy :url url})))
+ (request* (merge req {:method :copy :url url}) r))
(defn move
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :move :url url})))
+ (request* (merge req {:method :move :url url}) r))
(defn patch
"Like #'request, but sets the :method and :url as appropriate."
- [url & [req]]
+ [url & [req & r]]
(check-url! url)
- (request (merge req {:method :patch :url url})))
+ (request* (merge req {:method :patch :url url}) r))
(defmacro with-middleware
"Perform the body of the macro with a custom middleware list.
@@ -987,9 +1259,9 @@
(defmacro with-connection-pool
"Macro to execute the body using a connection manager. Creates a
- PoolingClientConnectionManager to use for all requests within the body of
- the expression. An option map is allowed to set options for the connection
- manager.
+ PoolingHttpClientConnectionManager to use for all requests within the
+ body of the expression. An option map is allowed to set options for the
+ connection manager.
The following options are supported:
@@ -1023,5 +1295,59 @@
~@body
(finally
(.shutdown
- ^PoolingClientConnectionManager
+ ^PoolingHttpClientConnectionManager
conn/*connection-manager*))))))
+
+(defn reuse-pool
+ "A helper function takes a request options map and a response map respond
+ from a pooled async request, the returned options map will be set to reuse
+ the connection pool which used by the former request"
+ [options response]
+ (if-let [info (:pooling-info response)]
+ (assoc options :pooling-info info)
+ options))
+
+(defmacro with-async-connection-pool
+ "Macro to execute the body using a connection manager. Creates a
+ PoolingNHttpClientConnectionManager to use for all requests within the body of
+ the expression. An option map is allowed to set options for the connection
+ manager.
+
+ Handles the same options as `with-connection-pool` plus:
+ :io-config which should be a map containing some of the following keys:
+
+ :connect-timeout - int the default connect timeout value for connection
+ requests (default 0, meaning no timeout)
+ :interest-op-queued - boolean, whether or not I/O interest operations are to
+ be queued and executed asynchronously or to be applied to the underlying
+ SelectionKey immediately (default false)
+ :io-thread-count - int, the number of I/O dispatch threads to be used
+ (default is the number of available processors)
+ :rcv-buf-size - int the default value of the SO_RCVBUF parameter for
+ newly created sockets (default is 0, meaning the system default)
+ :select-interval - long, time interval in milliseconds at which to check for
+ timed out sessions and session requests (default 1000)
+ :shutdown-grace-period - long, grace period in milliseconds to wait for
+ individual worker threads to terminate cleanly (default 500)
+ :snd-buf-size - int, the default value of the SO_SNDBUF parameter for
+ newly created sockets (default is 0, meaning the system default)
+ :so-keep-alive - boolean, the default value of the SO_KEEPALIVE parameter for
+ newly created sockets (default false)
+ :so-linger - int, the default value of the SO_LINGER parameter for
+ newly created sockets (default -1)
+ :so-timeout - int, the default socket timeout value for I/O operations
+ (default 0, meaning no timeout)
+ :tcp-no-delay - boolean, the default value of the TCP_NODELAY parameter for
+ newly created sockets (default true)
+
+ If the value 'nil' is specified or the value is not set, the default value
+ will be used."
+ [opts & body]
+ `(let [cm# (conn/make-reuseable-async-conn-manager ~opts)]
+ (binding [conn/*async-connection-manager* cm#]
+ (try
+ ~@body
+ (finally
+ (.shutdown
+ ^PoolingNHttpClientConnectionManager
+ cm#))))))
diff --git a/src/clj_http/conn_mgr.clj b/src/clj_http/conn_mgr.clj
index f409aa5..111c64f 100644
--- a/src/clj_http/conn_mgr.clj
+++ b/src/clj_http/conn_mgr.clj
@@ -2,68 +2,56 @@
"Utility methods for Scheme registries and HTTP connection managers"
(:require [clj-http.util :refer [opt]]
[clojure.java.io :as io])
- (:import (java.net Socket Proxy Proxy$Type InetSocketAddress)
- (java.security KeyStore)
- (java.security.cert X509Certificate)
- (javax.net.ssl SSLSession SSLSocket)
- (org.apache.http.conn ClientConnectionManager)
- (org.apache.http.conn.params ConnPerRouteBean)
- (org.apache.http.conn.ssl SSLSocketFactory TrustStrategy
- X509HostnameVerifier SSLContexts)
- (org.apache.http.conn.scheme PlainSocketFactory
- SchemeRegistry Scheme)
- (org.apache.http.impl.conn BasicClientConnectionManager
- PoolingClientConnectionManager
- SchemeRegistryFactory
- SingleClientConnManager)))
-
-(def ^SSLSocketFactory insecure-socket-factory
- (SSLSocketFactory. (reify TrustStrategy
- (isTrusted [_ _ _] true))
- (reify X509HostnameVerifier
- (^void verify [this ^String host ^SSLSocket sock]
- ;; for some strange reason, only TLSv1 really
- ;; works here, if you know why, tell me.
- (.setEnabledProtocols
- sock (into-array String ["TLSv1"]))
- (.setWantClientAuth sock false)
- (let [session (.getSession sock)]
- (when-not session
- (.startHandshake sock))
- (aget (.getPeerCertificates session) 0)
- ;; normally you'd want to verify the cert
- ;; here, but since this is an insecure
- ;; socketfactory, we don't
- nil))
- (^void verify [_ ^String _ ^X509Certificate _]
- nil)
- (^void verify [_ ^String _ ^"[Ljava.lang.String;" _
- ^"[Ljava.lang.String;" _]
- nil)
- (^boolean verify [_ ^String _ ^SSLSession _]
- true))))
-
-(def ^SSLSocketFactory secure-ssl-socket-factory
- (doto (SSLSocketFactory/getSocketFactory)
- (.setHostnameVerifier SSLSocketFactory/STRICT_HOSTNAME_VERIFIER)))
-
-;; New Generic Socket Factories that can support socks proxy
-(defn ^SSLSocketFactory SSLGenericSocketFactory
- "Given a function that returns a new socket, create an SSLSocketFactory that
- will use that socket."
- [socket-factory]
- (proxy [SSLSocketFactory] [(SSLContexts/createDefault)]
- (connectSocket [socket remoteAddress localAddress params]
- (let [^SSLSocketFactory this this] ;; avoid reflection
- (proxy-super connectSocket (socket-factory)
- remoteAddress localAddress params)))))
-
-(defn ^PlainSocketFactory PlainGenericSocketFactory
- "Given a Function that returns a new socket, create a PlainSocketFactory that
- will use that socket."
+ (:import [java.net InetSocketAddress Proxy Proxy$Type Socket]
+ java.security.KeyStore
+ [javax.net.ssl HostnameVerifier KeyManager SSLContext TrustManager]
+ [org.apache.http.config ConnectionConfig Registry RegistryBuilder SocketConfig]
+ org.apache.http.conn.HttpClientConnectionManager
+ org.apache.http.conn.socket.PlainConnectionSocketFactory
+ [org.apache.http.conn.ssl DefaultHostnameVerifier NoopHostnameVerifier SSLConnectionSocketFactory SSLContexts TrustStrategy]
+ [org.apache.http.impl.conn BasicHttpClientConnectionManager PoolingHttpClientConnectionManager]
+ org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager
+ org.apache.http.impl.nio.DefaultHttpClientIODispatch
+ [org.apache.http.impl.nio.reactor DefaultConnectingIOReactor IOReactorConfig]
+ [org.apache.http.nio.conn NHttpClientConnectionManager NoopIOSessionStrategy]
+ org.apache.http.nio.conn.ssl.SSLIOSessionStrategy
+ org.apache.http.nio.protocol.HttpAsyncRequestExecutor))
+
+;; -- Interop Helpers ---------------------------------------------------------
+(defn ^Registry into-registry [registry]
+ (cond
+ (instance? Registry registry)
+ registry
+
+ (map? registry)
+ (let [registry-builder (RegistryBuilder/create)]
+ (doseq [[k v] registry]
+ (.register registry-builder k v))
+ (.build registry-builder))
+
+ :else
+ (throw (IllegalArgumentException. "Cannot coerce into a Registry"))))
+
+;; -- SocketFactory -----------------------------------------------------------
+(defn ^SSLConnectionSocketFactory SSLGenericSocketFactory
+ "Given a function that returns a new socket, create an
+ SSLConnectionSocketFactory that will use that socket."
+ ([socket-factory]
+ (SSLGenericSocketFactory socket-factory nil))
+ ([socket-factory ^SSLContext ssl-context]
+ (let [^SSLContext ssl-context' (or ssl-context (SSLContexts/createDefault))]
+ (proxy [SSLConnectionSocketFactory] [ssl-context']
+ (connectSocket [timeout socket host remoteAddress localAddress context]
+ (let [^SSLConnectionSocketFactory this this] ;; avoid reflection
+ (proxy-super connectSocket timeout (socket-factory) host remoteAddress
+ localAddress context)))))))
+
+(defn ^PlainConnectionSocketFactory PlainGenericSocketFactory
+ "Given a Function that returns a new socket, create a
+ PlainConnectionSocketFactory that will use that socket."
[socket-factory]
- (proxy [PlainSocketFactory] []
- (createSocket [params]
+ (proxy [PlainConnectionSocketFactory] []
+ (createSocket [context]
(socket-factory))))
(defn socks-proxied-socket
@@ -71,28 +59,7 @@
[^String hostname ^Integer port]
(Socket. (Proxy. Proxy$Type/SOCKS (InetSocketAddress. hostname port))))
-(defn make-socks-proxied-conn-manager
- "Given an optional hostname and a port, create a connection manager that's
- proxied using a SOCKS proxy."
- [^String hostname ^Integer port]
- (let [socket-factory #(socks-proxied-socket hostname port)
- reg (doto (SchemeRegistry.)
- (.register
- (Scheme. "https" 443 (SSLGenericSocketFactory socket-factory)))
- (.register
- (Scheme. "http" 80 (PlainGenericSocketFactory socket-factory))))]
- (PoolingClientConnectionManager. reg)))
-
-(def insecure-scheme-registry
- (doto (SchemeRegistry.)
- (.register (Scheme. "http" 80 (PlainSocketFactory/getSocketFactory)))
- (.register (Scheme. "https" 443 insecure-socket-factory))))
-
-(def regular-scheme-registry
- (doto (SchemeRegistry.)
- (.register (Scheme. "http" 80 (PlainSocketFactory/getSocketFactory)))
- (.register (Scheme. "https" 443 secure-ssl-socket-factory))))
-
+;; -- SSL Contexts ------------------------------------------------------------
(defn ^KeyStore get-keystore*
[keystore-file keystore-type ^String keystore-pass]
(when keystore-file
@@ -107,54 +74,164 @@
keystore
(apply get-keystore* keystore args)))
-(defn ^SchemeRegistry get-keystore-scheme-registry
- [{:keys [keystore keystore-type keystore-pass keystore-instance
- trust-store trust-store-type trust-store-pass]
- :as req}]
+(defn- ssl-context-for-keystore
+ ;; TODO: use something else for passwords
+ ;; Note: JVM strings aren't ideal for passwords - see
+ ;; https://tinyurl.com/azm3ab9
+ [{:keys [keystore keystore-type ^String keystore-pass
+ trust-store trust-store-type trust-store-pass]}]
(let [ks (get-keystore keystore keystore-type keystore-pass)
- ts (get-keystore trust-store trust-store-type trust-store-pass)
- factory (SSLSocketFactory. ks keystore-pass ts)]
- (if (opt req :insecure)
- (.setHostnameVerifier factory
- SSLSocketFactory/ALLOW_ALL_HOSTNAME_VERIFIER))
- (doto (SchemeRegistryFactory/createDefault)
- (.register (Scheme. "https" 443 factory)))))
-
-(defn ^BasicClientConnectionManager make-regular-conn-manager
- [{:keys [keystore trust-store] :as req}]
- (cond
- (or keystore trust-store)
- (BasicClientConnectionManager. (get-keystore-scheme-registry req))
+ ts (get-keystore trust-store trust-store-type trust-store-pass)]
+ (-> (SSLContexts/custom)
+ (.loadKeyMaterial
+ ks (when keystore-pass
+ (.toCharArray keystore-pass)))
+ (.loadTrustMaterial
+ ts nil)
+ (.build))))
+
+(defn- ssl-context-for-trust-or-key-manager
+ "Given an instance or seqable data structure of TrustManager or KeyManager
+ will create and return an SSLContexts object including the resulting managers"
+ [{:keys [trust-managers key-managers]}]
+ (let [x-or-xs->x-array (fn [type x-or-xs]
+ (cond
+ (or (-> x-or-xs class .isArray)
+ (sequential? x-or-xs))
+ (into-array type (seq x-or-xs))
+
+ :else
+ (into-array type [x-or-xs])))
+ trust-managers (when trust-managers
+ (x-or-xs->x-array TrustManager trust-managers))
+ key-managers (when key-managers
+ (x-or-xs->x-array KeyManager key-managers))]
+ (doto (.build (SSLContexts/custom))
+ (.init key-managers trust-managers nil))))
+
+(defn- ssl-context-insecure
+ "Creates a SSL Context that trusts all material."
+ []
+ (-> (SSLContexts/custom)
+ (.loadTrustMaterial nil (reify TrustStrategy
+ (isTrusted [_ chain auth-type] true)))
+ (.build)))
+
+(defn ^SSLContext get-ssl-context
+ "Gets the SSL Context from a request or connection pool settings"
+ [{:keys [keystore trust-store key-managers trust-managers] :as config}]
+ (cond (or keystore trust-store)
+ (ssl-context-for-keystore config)
+
+ (or key-managers trust-managers)
+ (ssl-context-for-trust-or-key-manager config)
+
+ (opt config :insecure)
+ (ssl-context-insecure)
+
+ :else
+ (SSLContexts/createDefault)))
+
+(defn ^HostnameVerifier get-hostname-verifier [config]
+ (if (opt config :insecure)
+ NoopHostnameVerifier/INSTANCE
+ (DefaultHostnameVerifier.)))
+
+;; -- Connection Managers -----------------------------------------------------
+(defn make-socks-proxied-conn-manager
+ "Given an optional hostname and a port, create a connection manager that's
+ proxied using a SOCKS proxy."
+ ([^String hostname ^Integer port]
+ (make-socks-proxied-conn-manager hostname port {}))
+ ([^String hostname ^Integer port
+ {:keys [keystore keystore-type keystore-pass
+ trust-store trust-store-type trust-store-pass
+ trust-managers key-managers] :as config}]
+ (let [socket-factory #(socks-proxied-socket hostname port)
+ registry (into-registry
+ {"http" (PlainGenericSocketFactory socket-factory)
+ "https" (SSLGenericSocketFactory socket-factory (get-ssl-context config))})]
+ (PoolingHttpClientConnectionManager. registry))))
- (opt req :insecure) (BasicClientConnectionManager. insecure-scheme-registry)
+(defn ^BasicHttpClientConnectionManager make-regular-conn-manager
+ [{:keys [dns-resolver
+ keystore trust-store
+ key-managers trust-managers
+ socket-timeout] :as config}]
- :else (BasicClientConnectionManager. regular-scheme-registry)))
+ (let [registry (into-registry
+ {"http" (PlainConnectionSocketFactory/getSocketFactory)
+ "https" (SSLConnectionSocketFactory.
+ (get-ssl-context config)
+ (get-hostname-verifier config))})
+ conn-manager (BasicHttpClientConnectionManager. registry
+ nil nil
+ dns-resolver)]
+ (when socket-timeout
+ (.setSocketConfig conn-manager
+ (-> (.getSocketConfig conn-manager)
+ (SocketConfig/copy)
+ (.setSoTimeout socket-timeout) ;modify only the socket-timeout
+ (.build))))
+ conn-manager))
+
+(defn- ^DefaultConnectingIOReactor make-ioreactor
+ [{:keys [connect-timeout interest-op-queued io-thread-count rcv-buf-size
+ select-interval shutdown-grace-period snd-buf-size
+ so-keep-alive so-linger so-timeout tcp-no-delay]}]
+ (as-> (IOReactorConfig/custom) c
+ (if-some [v connect-timeout] (.setConnectTimeout c v) c)
+ (if-some [v interest-op-queued] (.setInterestOpQueued c v) c)
+ (if-some [v io-thread-count] (.setIoThreadCount c v) c)
+ (if-some [v rcv-buf-size] (.setRcvBufSize c v) c)
+ (if-some [v select-interval] (.setSelectInterval c v) c)
+ (if-some [v shutdown-grace-period] (.setShutdownGracePeriod c v) c)
+ (if-some [v snd-buf-size] (.setSndBufSize c v) c)
+ (if-some [v so-keep-alive] (.setSoKeepAlive c v) c)
+ (if-some [v so-linger] (.setSoLinger c v) c)
+ (if-some [v so-timeout] (.setSoTimeout c v) c)
+ (if-some [v tcp-no-delay] (.setTcpNoDelay c v) c)
+ (DefaultConnectingIOReactor. (.build c))))
+
+(defn ^PoolingNHttpClientConnectionManager
+ make-regular-async-conn-manager
+ [{:keys [keystore trust-store
+ key-managers trust-managers] :as config}]
+ (let [^Registry registry (into-registry
+ {"http" (NoopIOSessionStrategy/INSTANCE)
+ "https" (SSLIOSessionStrategy.
+ (get-ssl-context config)
+ (get-hostname-verifier config))})
+ io-reactor (make-ioreactor {:shutdown-grace-period 1})]
+ (doto (PoolingNHttpClientConnectionManager. io-reactor registry)
+ (.setMaxTotal 1))))
+
+(definterface ReuseableAsyncConnectionManager)
;; need the fully qualified class name because this fn is later used in a
;; macro from a different ns
-(defn ^org.apache.http.impl.conn.PoolingClientConnectionManager
+(defn ^org.apache.http.impl.conn.PoolingHttpClientConnectionManager
make-reusable-conn-manager*
"Given an timeout and optional insecure? flag, create a
- PoolingClientConnectionManager with <timeout> seconds set as the
+ PoolingHttpClientConnectionManager with <timeout> seconds set as the
timeout value."
- [{:keys [timeout keystore trust-store] :as config}]
- (let [registry (cond
- (opt config :insecure) insecure-scheme-registry
-
- (or keystore trust-store)
- (get-keystore-scheme-registry config)
+ [{:keys [dns-resolver
+ timeout
+ keystore trust-store
+ key-managers trust-managers] :as config}]
+ (let [registry (into-registry
+ {"http" (PlainConnectionSocketFactory/getSocketFactory)
+ "https" (SSLConnectionSocketFactory.
+ (get-ssl-context config)
+ (get-hostname-verifier config))})]
+ (PoolingHttpClientConnectionManager.
+ registry nil nil dns-resolver timeout java.util.concurrent.TimeUnit/SECONDS)))
- :else regular-scheme-registry)]
- (PoolingClientConnectionManager.
- registry timeout java.util.concurrent.TimeUnit/SECONDS)))
+(defn reusable? [conn-mgr]
+ (or (instance? PoolingHttpClientConnectionManager conn-mgr)
+ (instance? ReuseableAsyncConnectionManager conn-mgr)))
-(def dmcpr ConnPerRouteBean/DEFAULT_MAX_CONNECTIONS_PER_ROUTE)
-
-(defn reusable? [^ClientConnectionManager conn-mgr]
- (not (or (instance? SingleClientConnManager conn-mgr)
- (instance? BasicClientConnectionManager conn-mgr))))
-
-(defn ^PoolingClientConnectionManager make-reusable-conn-manager
+(defn ^PoolingHttpClientConnectionManager make-reusable-conn-manager
"Creates a default pooling connection manager with the specified options.
The following options are supported:
@@ -173,27 +250,113 @@
:trust-store - trust store file to be used for connection manager
:trust-store-pass - trust store password
- Note that :insecure? and :keystore/:trust-store options are mutually exclusive
+ :key-managers - KeyManager objects to be used for connection manager
+ :trust-managers - TrustManager objects to be used for connection manager
+
+ :dns-resolver - Use a custom DNS resolver instead of the default DNS resolver.
+
+ Note that :insecure? and :keystore/:trust-store/:key-managers/:trust-managers options are mutually exclusive
+
+ Note that :key-managers/:trust-managers have precedence over :keystore/:trust-store options
+
If the value 'nil' is specified or the value is not set, the default value
will be used."
[opts]
(let [timeout (or (:timeout opts) 5)
threads (or (:threads opts) 4)
- default-per-route (or (:default-per-route opts) dmcpr)
+ default-per-route (:default-per-route opts)
insecure? (opt opts :insecure)
- leftovers (dissoc opts :timeout :threads :insecure? :insecure)]
- (doto (make-reusable-conn-manager* (merge {:timeout timeout
- :insecure? insecure?}
- leftovers))
- (.setMaxTotal threads)
- (.setDefaultMaxPerRoute default-per-route))))
-
-(defn shutdown-manager
+ leftovers (dissoc opts :timeout :threads :insecure? :insecure)
+ conn-man (make-reusable-conn-manager* (merge {:timeout timeout
+ :insecure? insecure?}
+ leftovers))]
+ (.setMaxTotal conn-man threads)
+ (when default-per-route
+ (.setDefaultMaxPerRoute conn-man default-per-route))
+ conn-man))
+
+(defn- ^PoolingNHttpClientConnectionManager make-reusable-async-conn-manager*
+ [{:keys [dns-resolver
+ timeout keystore trust-store io-config
+ key-managers trust-managers] :as config}]
+ (let [registry (into-registry
+ {"http" (NoopIOSessionStrategy/INSTANCE)
+ "https" (SSLIOSessionStrategy.
+ (get-ssl-context config)
+ (get-hostname-verifier config))})
+ io-reactor (make-ioreactor io-config)
+ protocol-handler (HttpAsyncRequestExecutor.)
+ io-event-dispatch (DefaultHttpClientIODispatch. protocol-handler
+ ConnectionConfig/DEFAULT)]
+ (future (.execute io-reactor io-event-dispatch))
+ (proxy [PoolingNHttpClientConnectionManager ReuseableAsyncConnectionManager]
+ [io-reactor nil registry nil dns-resolver timeout
+ java.util.concurrent.TimeUnit/SECONDS])))
+
+(defn ^PoolingNHttpClientConnectionManager make-reusable-async-conn-manager
+ "Creates a default pooling async connection manager with the specified
+ options. Handles the same options as make-reusable-conn-manager plus
+ :io-config which should be a map containing some of the following keys:
+
+ :connect-timeout - int the default connect timeout value for connection
+ requests (default 0, meaning no timeout)
+ :interest-op-queued - boolean, whether or not I/O interest operations are to
+ be queued and executed asynchronously or to be applied to the underlying
+ SelectionKey immediately (default false)
+ :io-thread-count - int, the number of I/O dispatch threads to be used
+ (default is the number of available processors)
+ :rcv-buf-size - int the default value of the SO_RCVBUF parameter for
+ newly created sockets (default is 0, meaning the system default)
+ :select-interval - long, time interval in milliseconds at which to check for
+ timed out sessions and session requests (default 1000)
+ :shutdown-grace-period - long, grace period in milliseconds to wait for
+ individual worker threads to terminate cleanly (default 500)
+ :snd-buf-size - int, the default value of the SO_SNDBUF parameter for
+ newly created sockets (default is 0, meaning the system default)
+ :so-keep-alive - boolean, the default value of the SO_KEEPALIVE parameter for
+ newly created sockets (default false)
+ :so-linger - int, the default value of the SO_LINGER parameter for
+ newly created sockets (default -1)
+ :so-timeout - int, the default socket timeout value for I/O operations
+ (default 0, meaning no timeout)
+ :tcp-no-delay - boolean, the default value of the TCP_NODELAY parameter for
+ newly created sockets (default true)
+
+ If the value 'nil' is specified or the value is not set, the default value
+ will be used."
+ [opts]
+ (let [timeout (or (:timeout opts) 5)
+ threads (or (:threads opts) 4)
+ default-per-route (:default-per-route opts)
+ insecure? (opt opts :insecure)
+ leftovers (dissoc opts :timeout :threads :insecure? :insecure)
+ conn-man (make-reusable-async-conn-manager*
+ (merge {:timeout timeout :insecure? insecure?} leftovers))]
+ (.setMaxTotal conn-man threads)
+ (when default-per-route
+ (.setDefaultMaxPerRoute conn-man default-per-route))
+ conn-man))
+
+(defn ^PoolingNHttpClientConnectionManager make-reuseable-async-conn-manager
+ "Wraps correctly-spelled version - keeping for backwards compatibility."
+ [opts]
+ (make-reusable-async-conn-manager opts))
+
+(defmulti shutdown-manager
"Shut down the given connection manager, if it is not nil"
- [^ClientConnectionManager manager]
- (and manager (.shutdown manager)))
+ class)
+(defmethod shutdown-manager nil [conn-mgr] nil)
+(defmethod shutdown-manager org.apache.http.conn.HttpClientConnectionManager
+ [^HttpClientConnectionManager conn-mgr] (.shutdown conn-mgr))
+(defmethod shutdown-manager
+ org.apache.http.nio.conn.NHttpClientConnectionManager
+ [^NHttpClientConnectionManager conn-mgr] (.shutdown conn-mgr))
(def ^:dynamic *connection-manager*
"connection manager to be rebound during request execution"
nil)
+
+(def ^:dynamic *async-connection-manager*
+ "connection manager to be rebound during async request execution"
+ nil)
diff --git a/src/clj_http/cookies.clj b/src/clj_http/cookies.clj
index c1a74d7..c307235 100644
--- a/src/clj_http/cookies.clj
+++ b/src/clj_http/cookies.clj
@@ -2,18 +2,15 @@
"Namespace dealing with HTTP cookies"
(:require [clj-http.util :refer [opt]]
[clojure.string :refer [blank? join lower-case]])
- (:import (org.apache.http.client.params ClientPNames CookiePolicy)
- (org.apache.http.cookie ClientCookie CookieOrigin CookieSpec)
- (org.apache.http.params BasicHttpParams)
- (org.apache.http.impl.cookie BasicClientCookie2)
- (org.apache.http.impl.cookie BrowserCompatSpecFactory)
- (org.apache.http.message BasicHeader)
- org.apache.http.client.CookieStore
- (org.apache.http.impl.client BasicCookieStore)
- (org.apache.http Header)
- (org.apache.http.protocol BasicHttpContext)))
-
-(defn cookie-spec ^CookieSpec []
+ (:import org.apache.http.client.CookieStore
+ [org.apache.http.cookie ClientCookie CookieOrigin CookieSpec]
+ org.apache.http.Header
+ org.apache.http.impl.client.BasicCookieStore
+ [org.apache.http.impl.cookie BasicClientCookie2 BrowserCompatSpecFactory]
+ org.apache.http.message.BasicHeader
+ org.apache.http.protocol.BasicHttpContext))
+
+(defn cookie-spec ^org.apache.http.cookie.CookieSpec []
(.create
(BrowserCompatSpecFactory.)
(BasicHttpContext.)))
@@ -25,7 +22,7 @@
(if (not (nil? (get m k)))
(assoc newm k (get m k))
newm))
- (sorted-map) (sort (keys m))))
+ {} (keys m)))
(defn to-cookie
"Converts a ClientCookie object into a tuple where the first item is
@@ -115,16 +112,24 @@
(dissoc :cookies))
request))
+(defn- cookies-response
+ [request response]
+ (if (= false (opt request :decode-cookies))
+ response
+ (decode-cookie-header response)))
+
(defn wrap-cookies
"Middleware wrapping cookie handling. Handles converting
the :cookies request parameter into the 'Cookies' header for an HTTP
request."
[client]
- (fn [request]
- (let [response (client (encode-cookie-header request))]
- (if (= false (opt request :decode-cookies))
- response
- (decode-cookie-header response)))))
+ (fn
+ ([request]
+ (cookies-response request (client (encode-cookie-header request))))
+ ([request respond raise]
+ (client (encode-cookie-header request)
+ #(respond (cookies-response request %))
+ raise))))
(defn cookie-store
"Returns a new, empty instance of the default implementation of the
@@ -137,3 +142,13 @@
[^CookieStore cookie-store]
(when cookie-store
(into {} (map to-cookie (.getCookies cookie-store)))))
+
+(defn add-cookie
+ "Add a ClientCookie to a cookie-store"
+ [^CookieStore cookie-store ^ClientCookie cookie]
+ (.addCookie cookie-store cookie))
+
+(defn clear-cookies
+ "Clears all cookies from cookie-store"
+ [^CookieStore cookie-store]
+ (.clear cookie-store))
diff --git a/src/clj_http/core.clj b/src/clj_http/core.clj
index 11d8325..2ed4ea1 100644
--- a/src/clj_http/core.clj
+++ b/src/clj_http/core.clj
@@ -1,37 +1,31 @@
(ns clj-http.core
- "Core HTTP request/response implementation."
+ "Core HTTP request/response implementation. Rewrite for Apache 4.3"
(:require [clj-http.conn-mgr :as conn]
[clj-http.headers :as headers]
[clj-http.multipart :as mp]
[clj-http.util :refer [opt]]
- [clojure.pprint])
- (:import (java.io ByteArrayOutputStream FilterInputStream InputStream)
-
- (org.apache.http HeaderIterator HttpEntity
- HttpEntityEnclosingRequest
- HttpResponse Header HttpHost
- HttpRequestInterceptor HttpResponseInterceptor)
- (org.apache.http.auth UsernamePasswordCredentials AuthScope
- NTCredentials)
- (org.apache.http.params CoreConnectionPNames)
- (org.apache.http.client HttpClient HttpRequestRetryHandler)
- (org.apache.http.client.methods HttpDelete
- HttpEntityEnclosingRequestBase
- HttpGet HttpHead HttpOptions
- HttpPatch HttpPost HttpPut
- HttpUriRequest)
- (org.apache.http.client.params CookiePolicy ClientPNames)
- (org.apache.http.conn ClientConnectionManager)
- (org.apache.http.conn.routing HttpRoute)
- (org.apache.http.conn.params ConnRoutePNames)
- (org.apache.http.cookie CookieSpecFactory)
- (org.apache.http.cookie.params CookieSpecPNames)
- (org.apache.http.entity ByteArrayEntity StringEntity)
-
- (org.apache.http.impl.client DefaultHttpClient)
- (org.apache.http.impl.conn ProxySelectorRoutePlanner)
- (org.apache.http.impl.cookie BrowserCompatSpec)
- (java.net URI)))
+ clojure.pprint)
+ (:import [java.io ByteArrayOutputStream FilterInputStream InputStream]
+ [java.net InetAddress ProxySelector URI URL]
+ java.util.Locale
+ [org.apache.http HeaderIterator HttpEntity HttpEntityEnclosingRequest HttpHost HttpRequestInterceptor HttpResponse HttpResponseInterceptor ProtocolException]
+ [org.apache.http.auth AuthScope NTCredentials UsernamePasswordCredentials]
+ [org.apache.http.client CredentialsProvider HttpRequestRetryHandler RedirectStrategy]
+ org.apache.http.client.cache.HttpCacheContext
+ [org.apache.http.client.config CookieSpecs RequestConfig]
+ [org.apache.http.client.methods CloseableHttpResponse HttpDelete HttpEntityEnclosingRequestBase HttpGet HttpHead HttpOptions HttpPatch HttpPost HttpPut HttpRequestBase HttpUriRequest]
+ org.apache.http.client.protocol.HttpClientContext
+ org.apache.http.client.utils.URIUtils
+ org.apache.http.config.RegistryBuilder
+ org.apache.http.conn.routing.HttpRoutePlanner
+ org.apache.http.cookie.CookieSpecProvider
+ [org.apache.http.entity ByteArrayEntity StringEntity]
+ [org.apache.http.impl.client BasicCredentialsProvider CloseableHttpClient DefaultRedirectStrategy HttpClientBuilder HttpClients LaxRedirectStrategy]
+ [org.apache.http.impl.client.cache CacheConfig CachingHttpClientBuilder]
+ [org.apache.http.impl.conn DefaultProxyRoutePlanner SystemDefaultRoutePlanner]
+ [org.apache.http.impl.nio.client CloseableHttpAsyncClient HttpAsyncClientBuilder HttpAsyncClients]))
+
+(def CUSTOM_COOKIE_POLICY "_custom")
(defn parse-headers
"Takes a HeaderIterator and returns a map of names to values.
@@ -52,19 +46,352 @@
(headers/assoc-join hs k v))
(headers/header-map)))))
-(defn set-client-param [^HttpClient client key val]
- (when-not (nil? val)
- (-> client
- (.getParams)
- (.setParameter key val))))
+(defn graceful-redirect-strategy
+ "Similar to the default redirect strategy, however, does not throw an error
+ when the maximum number of redirects has been reached. Still supports
+ validating that the new redirect host is not empty."
+ [req]
+ (let [validate? (opt req :validate-redirects)]
+ (reify RedirectStrategy
+ (getRedirect [this request response context]
+ (let [new-request (.getRedirect DefaultRedirectStrategy/INSTANCE
+ request response context)]
+ (when (or validate? (nil? validate?))
+ (let [uri (.getURI new-request)
+ new-host (URIUtils/extractHost uri)]
+ (when (nil? new-host)
+ (throw
+ (ProtocolException.
+ (str "Redirect URI does not specify a valid host name: "
+ uri))))))
+ new-request))
+
+ (isRedirected [this request response context]
+ (let [^HttpClientContext typed-context context
+ max-redirects (-> (.getRequestConfig typed-context)
+ .getMaxRedirects)
+ num-redirects (count (.getRedirectLocations typed-context))]
+ (if (<= max-redirects num-redirects)
+ false
+ (.isRedirected DefaultRedirectStrategy/INSTANCE
+ request response typed-context)))))))
+
+(defn default-redirect-strategy
+ "Proxies calls to whatever original redirect strategy is passed in, however,
+ if :validate-redirects is set in the request, checks that the redirected host
+ is not empty."
+ [^RedirectStrategy original req]
+ (let [validate? (opt req :validate-redirects)]
+ (reify RedirectStrategy
+ (getRedirect [this request response context]
+ (let [new-request (.getRedirect original request response context)]
+ (when (or validate? (nil? validate?))
+ (let [uri (.getURI new-request)
+ new-host (URIUtils/extractHost uri)]
+ (when (nil? new-host)
+ (throw
+ (ProtocolException.
+ (str "Redirect URI does not specify a valid host name: "
+ uri))))))
+ new-request))
+
+ (isRedirected [this request response context]
+ (.isRedirected original request response context)))))
+
+(defn get-redirect-strategy [{:keys [redirect-strategy] :as req}]
+ (case redirect-strategy
+ :none (reify RedirectStrategy
+ (getRedirect [this request response context] nil)
+ (isRedirected [this request response context] false))
+
+ ;; Like default, but does not throw exceptions when max redirects is
+ ;; reached.
+ :graceful (graceful-redirect-strategy req)
+
+ :default (default-redirect-strategy DefaultRedirectStrategy/INSTANCE req)
+ :lax (default-redirect-strategy (LaxRedirectStrategy.) req)
+ nil (default-redirect-strategy DefaultRedirectStrategy/INSTANCE req)
+
+ ;; use directly as reifed RedirectStrategy
+ redirect-strategy))
+
+(defn ^HttpClientBuilder add-retry-handler [^HttpClientBuilder builder handler]
+ (when handler
+ (.setRetryHandler
+ builder
+ (proxy [HttpRequestRetryHandler] []
+ (retryRequest [e cnt context]
+ (handler e cnt context)))))
+ builder)
+
+(defn create-custom-cookie-policy-registry
+ "Given a function that will take an HttpContext and return a CookieSpec,
+ create a new Registry for the cookie policy under the CUSTOM_COOKIE_POLICY
+ string."
+ [cookie-spec-fn]
+ (-> (RegistryBuilder/create)
+ (.register CUSTOM_COOKIE_POLICY
+ (proxy [CookieSpecProvider] []
+ (create [context]
+ (cookie-spec-fn context))))
+ (.build)))
+
+(defmulti get-cookie-policy
+ "Method to retrieve the cookie policy that should be used for the request.
+ This is a multimethod that may be extended to return your own cookie policy.
+ Dispatches based on the `:cookie-policy` key in the request map."
+ (fn get-cookie-dispatch [request] (:cookie-policy request)))
+
+(defmethod get-cookie-policy :none none-cookie-policy
+ [_] CookieSpecs/IGNORE_COOKIES)
+(defmethod get-cookie-policy :default default-cookie-policy
+ [_] CookieSpecs/DEFAULT)
+(defmethod get-cookie-policy nil nil-cookie-policy
+ [_] CookieSpecs/DEFAULT)
+(defmethod get-cookie-policy :netscape netscape-cookie-policy
+ [_] CookieSpecs/NETSCAPE)
+(defmethod get-cookie-policy :standard standard-cookie-policy
+ [_] CookieSpecs/STANDARD)
+(defmethod get-cookie-policy :stardard-strict standard-strict-cookie-policy
+ [_] CookieSpecs/STANDARD_STRICT)
+
+(defn request-config [{:keys [connection-timeout
+ connection-request-timeout
+ socket-timeout
+ max-redirects
+ cookie-spec
+ normalize-uri
+ ; deprecated
+ conn-request-timeout
+ conn-timeout]
+ :as req}]
+ (let [config (-> (RequestConfig/custom)
+ (.setConnectTimeout (or connection-timeout conn-timeout -1))
+ (.setSocketTimeout (or socket-timeout -1))
+ (.setConnectionRequestTimeout
+ (or connection-request-timeout conn-request-timeout -1))
+ (.setRedirectsEnabled true)
+ (.setCircularRedirectsAllowed
+ (boolean (opt req :allow-circular-redirects)))
+ (.setRelativeRedirectsAllowed
+ ((complement false?)
+ (opt req :allow-relative-redirects))))]
+ (if cookie-spec
+ (.setCookieSpec config CUSTOM_COOKIE_POLICY)
+ (.setCookieSpec config (get-cookie-policy req)))
+ (when max-redirects (.setMaxRedirects config max-redirects))
+ (when-not (nil? normalize-uri) (.setNormalizeUri config normalize-uri))
+ (.build config)))
+
+(defmulti ^:private construct-http-host (fn [proxy-host proxy-port]
+ (class proxy-host)))
+(defmethod construct-http-host String
+ [^String proxy-host ^Long proxy-port]
+ (if proxy-port
+ (HttpHost. proxy-host proxy-port)
+ (HttpHost. proxy-host)))
+(defmethod construct-http-host java.net.InetAddress
+ [^InetAddress proxy-host ^Long proxy-port]
+ (if proxy-port
+ (HttpHost. proxy-host proxy-port)
+ (HttpHost. proxy-host)))
+
+(defn ^HttpRoutePlanner get-route-planner
+ "Return an HttpRoutePlanner that either use the supplied proxy settings
+ if any, or the JVM/system proxy settings otherwise"
+ [^String proxy-host ^Long proxy-port proxy-ignore-hosts http-url]
+ (let [ignore-proxy? (and http-url
+ (contains? (set proxy-ignore-hosts)
+ (.getHost (URL. http-url))))]
+ (if (and proxy-host (not ignore-proxy?))
+ (DefaultProxyRoutePlanner. (construct-http-host proxy-host proxy-port))
+ (SystemDefaultRoutePlanner. (ProxySelector/getDefault)))))
+
+(defn build-cache-config
+ "Given a request with :cache-config as a map or a CacheConfig object, return a
+ CacheConfig object, or nil if no cache config is found. If :cache-config is a
+ map, it checks for the following options:
+ - :allow-303-caching
+ - :asynchronous-worker-idle-lifetime-secs
+ - :asynchronous-workers-core
+ - :asynchronous-workers-max
+ - :heuristic-caching-enabled
+ - :heuristic-coefficient
+ - :heuristic-default-lifetime
+ - :max-cache-entries
+ - :max-object-size
+ - :max-update-retries
+ - :never-cache-http10-responses-with-query-string
+ - :revalidation-queue-size
+ - :shared-cache
+ - :weak-etag-on-put-delete-allowed"
+ [request]
+ (when-let [cc (:cache-config request)]
+ (if (instance? CacheConfig cc)
+ cc
+ (let [config (CacheConfig/custom)
+ {:keys [allow-303-caching
+ asynchronous-worker-idle-lifetime-secs
+ asynchronous-workers-core
+ asynchronous-workers-max
+ heuristic-caching-enabled
+ heuristic-coefficient
+ heuristic-default-lifetime
+ max-cache-entries
+ max-object-size
+ max-update-retries
+ never-cache-http10-responses-with-query-string
+ revalidation-queue-size
+ shared-cache
+ weak-etag-on-put-delete-allowed]} cc]
+ (when (instance? Boolean allow-303-caching)
+ (.setAllow303Caching config allow-303-caching))
+ (when asynchronous-worker-idle-lifetime-secs
+ (.setAsynchronousWorkerIdleLifetimeSecs
+ config asynchronous-worker-idle-lifetime-secs))
+ (when asynchronous-workers-core
+ (.setAsynchronousWorkersCore config asynchronous-workers-core))
+ (when asynchronous-workers-max
+ (.setAsynchronousWorkersMax config asynchronous-workers-max))
+ (when (instance? Boolean heuristic-caching-enabled)
+ (.setHeuristicCachingEnabled config heuristic-caching-enabled))
+ (when heuristic-coefficient
+ (.setHeuristicCoefficient config heuristic-coefficient))
+ (when heuristic-default-lifetime
+ (.setHeuristicDefaultLifetime config heuristic-default-lifetime))
+ (when max-cache-entries
+ (.setMaxCacheEntries config max-cache-entries))
+ (when max-object-size
+ (.setMaxObjectSize config max-object-size))
+ (when max-update-retries
+ (.setMaxUpdateRetries config max-update-retries))
+ ;; I would add this option, but there is a bug in 4.x CacheConfig that
+ ;; it does not actually correctly use the object from the builder.
+ ;; It's fixed in 5.0 however
+ ;; (when (boolean? never-cache-http10-responses-with-query-string)
+ ;; (.setNeverCacheHTTP10ResponsesWithQueryString
+ ;; config never-cache-http10-responses-with-query-string))
+ (when revalidation-queue-size
+ (.setRevalidationQueueSize config revalidation-queue-size))
+ (when (instance? Boolean shared-cache)
+ (.setSharedCache config shared-cache))
+ (when (instance? Boolean weak-etag-on-put-delete-allowed)
+ (.setWeakETagOnPutDeleteAllowed config weak-etag-on-put-delete-allowed))
+ (.build config)))))
+
+(defn build-http-client
+ "Builds an Apache `HttpClient` from a clj-http request map. Optional arguments
+ `http-url` and `proxy-ignore-hosts` are used to specify the host and a list of
+ hostnames to ignore for any proxy settings. They can be safely ignored if not
+ using proxies."
+ [{:keys [retry-handler request-interceptor
+ response-interceptor proxy-host proxy-port
+ http-builder-fns cookie-spec
+ cookie-policy-registry
+ ^HttpClientBuilder http-client-builder]
+ :as req}
+ caching?
+ conn-mgr
+ & [http-url proxy-ignore-hosts]]
+ ;; have to let first, otherwise we get a reflection warning on (.build)
+ (let [cache? (opt req :cache)
+ builder (-> (cond
+ http-client-builder http-client-builder
+ caching?
+ ^HttpClientBuilder (CachingHttpClientBuilder/create)
+ :else
+ ^HttpClientBuilder (HttpClients/custom))
+ (.setConnectionManager conn-mgr)
+ (.setRedirectStrategy
+ (get-redirect-strategy req))
+ (add-retry-handler retry-handler)
+
+ ;; prefer using clj-http.client/wrap-decompression
+ ;; for consistency between sync/async client options
+ (.disableContentCompression)
+
+ ;; By default, get the proxy settings
+ ;; from the jvm or system properties
+ (.setRoutePlanner
+ (get-route-planner
+ proxy-host proxy-port
+ proxy-ignore-hosts http-url)))]
+ (when cache?
+ (.setCacheConfig ^CachingHttpClientBuilder builder (build-cache-config req)))
+ (when (or cookie-policy-registry cookie-spec)
+ (if cookie-policy-registry
+ ;; They have a custom registry they'd like to re-use, so use that
+ (.setDefaultCookieSpecRegistry builder cookie-policy-registry)
+ ;; They have only a one-time function for cookie spec, so use that
+ (.setDefaultCookieSpecRegistry
+ builder (create-custom-cookie-policy-registry cookie-spec))))
+ (when request-interceptor
+ (.addInterceptorLast
+ builder (proxy [HttpRequestInterceptor] []
+ (process [req ctx]
+ (request-interceptor req ctx)))))
+
+ (when response-interceptor
+ (.addInterceptorLast
+ builder (proxy [HttpResponseInterceptor] []
+ (process [resp ctx]
+ (response-interceptor
+ resp ctx)))))
+ (doseq [http-builder-fn http-builder-fns]
+ (http-builder-fn builder req))
+ (.build builder)))
+
+(defn build-async-http-client
+ "Builds an Apache `HttpAsyncClient` from a clj-http request map. Optional
+ arguments `http-url` and `proxy-ignore-hosts` are used to specify the host
+ and a list of hostnames to ignore for any proxy settings. They can be safely
+ ignored if not using proxies."
+ [{:keys [request-interceptor response-interceptor
+ proxy-host proxy-port async-http-builder-fns]
+ :as req}
+ conn-mgr & [http-url proxy-ignore-hosts]]
+ ;; have to let first, otherwise we get a reflection warning on (.build)
+ (let [^HttpAsyncClientBuilder builder (-> (HttpAsyncClients/custom)
+ (.setConnectionManager conn-mgr)
+ (.setRedirectStrategy
+ (get-redirect-strategy req))
+ ;; By default, get the proxy
+ ;; settings from the jvm or system
+ ;; properties
+ (.setRoutePlanner
+ (get-route-planner
+ proxy-host proxy-port
+ proxy-ignore-hosts http-url)))]
+ (when (conn/reusable? conn-mgr)
+ (.setConnectionManagerShared builder true))
+
+ (when request-interceptor
+ (.addInterceptorLast
+ builder (proxy [HttpRequestInterceptor] []
+ (process [req ctx]
+ (request-interceptor req ctx)))))
+
+ (when response-interceptor
+ (.addInterceptorLast
+ builder (proxy [HttpResponseInterceptor] []
+ (process [resp ctx]
+ (response-interceptor
+ resp ctx)))))
+ (doseq [async-http-builder-fn async-http-builder-fns]
+ (async-http-builder-fn builder req))
+ (.build builder)))
+
+(defn http-get []
+ (HttpGet. "https://www.google.com"))
(defn make-proxy-method-with-body
[method]
- (fn [^String url]
+ (fn [url]
(doto (proxy [HttpEntityEnclosingRequestBase] []
- (getMethod [] (.toUpperCase (name method))))
+ (getMethod [] (.toUpperCase (name method) Locale/ROOT)))
(.setURI (URI. url)))))
+(def proxy-head-with-body (make-proxy-method-with-body :head))
(def proxy-delete-with-body (make-proxy-method-with-body :delete))
(def proxy-get-with-body (make-proxy-method-with-body :get))
(def proxy-copy-with-body (make-proxy-method-with-body :copy))
@@ -73,77 +400,53 @@
(def ^:dynamic *cookie-store* nil)
-(defn- set-routing
- "Use ProxySelectorRoutePlanner to choose proxy sensible based on
- http.nonProxyHosts"
- [^DefaultHttpClient client]
- (.setRoutePlanner client
- (ProxySelectorRoutePlanner.
- (.. client getConnectionManager getSchemeRegistry) nil))
- client)
-
-(defn maybe-force-proxy [^DefaultHttpClient client
- ^HttpEntityEnclosingRequestBase request
- proxy-host proxy-port proxy-ignore-hosts]
- (let [uri (.getURI request)]
- (when (and (nil? ((set proxy-ignore-hosts) (.getHost uri))) proxy-host)
- (let [target (HttpHost. (.getHost uri) (.getPort uri) (.getScheme uri))
- route (HttpRoute. target nil (HttpHost. ^String proxy-host
- (int proxy-port))
- (.. client getConnectionManager getSchemeRegistry
- (getScheme target) isLayered))]
- (set-client-param client ConnRoutePNames/FORCED_ROUTE route)))
- request))
-
-(defn cookie-spec
- "Create an instance of a
- org.apache.http.impl.cookie.BrowserCompatSpec with a validate
- function that you pass in. This function takes two parameters, a
- cookie and an origin."
- [f]
- (proxy [BrowserCompatSpec] []
- (validate [cookie origin] (f cookie origin))))
-
-(defn cookie-spec-factory
- "Create an instance of a org.apache.http.cookie.CookieSpecFactory
- with a newInstance implementation that returns a cookie
- specification with a validate function that you pass in. The
- function takes two parameters: cookie and origin."
- [f]
- (proxy
- [CookieSpecFactory] []
- (newInstance [params] (cookie-spec f))))
-
-(defn add-client-params!
- "Add various client params to the http-client object, if needed."
- [^DefaultHttpClient http-client kvs]
- (let [cookie-policy (:cookie-policy kvs)
- cookie-policy-name (str (type cookie-policy))
- kvs (dissoc kvs :cookie-policy)]
- (when cookie-policy
- (-> http-client
- .getCookieSpecs
- (.register cookie-policy-name (cookie-spec-factory cookie-policy))))
- (doto http-client
- (set-client-param ClientPNames/COOKIE_POLICY
- (if cookie-policy
- cookie-policy-name
- CookiePolicy/BROWSER_COMPATIBILITY))
- (set-client-param CookieSpecPNames/SINGLE_COOKIE_HEADER true)
- (set-client-param ClientPNames/HANDLE_REDIRECTS false))
-
- (doseq [[k v] kvs]
- (set-client-param http-client
- k (cond
- (and (not= ClientPNames/CONN_MANAGER_TIMEOUT k)
- (instance? Long v))
- (Integer. ^Long v)
- true v)))))
+(defn make-proxy-method [method url]
+ (doto (proxy [HttpRequestBase] []
+ (getMethod
+ []
+ (str method)))
+ (.setURI (URI/create url))))
+
+(defn http-request-for
+ "Provides the HttpRequest object for a particular request-method and url"
+ [request-method ^String http-url body]
+ (case request-method
+ :get (if body
+ (proxy-get-with-body http-url)
+ (HttpGet. http-url))
+ :head (if body
+ (proxy-head-with-body http-url)
+ (HttpHead. http-url))
+ :put (HttpPut. http-url)
+ :post (HttpPost. http-url)
+ :options (HttpOptions. http-url)
+ :delete (if body
+ (proxy-delete-with-body http-url)
+ (HttpDelete. http-url))
+ :copy (proxy-copy-with-body http-url)
+ :move (proxy-move-with-body http-url)
+ :patch (if body
+ (proxy-patch-with-body http-url)
+ (HttpPatch. http-url))
+ (if body
+ ((make-proxy-method-with-body request-method) http-url)
+ (make-proxy-method request-method http-url))))
+
+(defn ^HttpClientContext http-context [caching? request-config http-client-context]
+ (let [^HttpClientContext typed-context (or http-client-context
+ (if caching?
+ (HttpCacheContext/create)
+ (HttpClientContext/create)))]
+ (doto typed-context
+ (.setRequestConfig request-config))))
+
+(defn ^CredentialsProvider credentials-provider []
+ (BasicCredentialsProvider.))
(defn- coerce-body-entity
- "Coerce the http-entity from an HttpResponse to either a byte-array, or a
- stream that closes itself and the connection manager when closed."
- [{:keys [as]} ^HttpEntity http-entity ^ClientConnectionManager conn-mgr]
+ "Coerce the http-entity from an HttpResponse to a stream that closes itself
+ and the connection manager when closed."
+ [^HttpEntity http-entity conn-mgr ^CloseableHttpResponse response]
(if http-entity
(proxy [FilterInputStream]
[^InputStream (.getContent http-entity)]
@@ -153,6 +456,8 @@
(let [^InputStream this this]
(proxy-super close))
(finally
+ (when (instance? CloseableHttpResponse response)
+ (.close response))
(when-not (conn/reusable? conn-mgr)
(conn/shutdown-manager conn-mgr))))))
(when-not (conn/reusable? conn-mgr)
@@ -161,175 +466,200 @@
(defn- print-debug!
"Print out debugging information to *out* for a given request."
[{:keys [debug-body body] :as req} http-req]
- (println "Request:" (type body))
- (clojure.pprint/pprint
- (assoc req
- :body (if (opt req :debug-body)
- (cond
- (isa? (type body) String)
- body
-
- (isa? (type body) HttpEntity)
- (let [baos (ByteArrayOutputStream.)]
- (.writeTo ^HttpEntity body baos)
- (.toString baos "UTF-8"))
-
- :else nil)
- (if (isa? (type body) String)
- (format "... %s bytes ..."
- (count body))
- (and body (bean body))))
- :body-type (type body)))
- (println "HttpRequest:")
- (clojure.pprint/pprint (bean http-req)))
+ (println
+ (with-out-str
+ (println "Request:" (type body))
+ (clojure.pprint/pprint
+ (assoc req
+ :body (if (opt req :debug-body)
+ (cond
+ (isa? (type body) String)
+ body
-(defn http-request-for
- "Provides the HttpRequest object for a particular request-method and url"
- [request-method ^String http-url body]
- (case request-method
- :get (if body
- (proxy-get-with-body http-url)
- (HttpGet. http-url))
- :head (HttpHead. http-url)
- :put (HttpPut. http-url)
- :post (HttpPost. http-url)
- :options (HttpOptions. http-url)
- :delete (if body
- (proxy-delete-with-body http-url)
- (HttpDelete. http-url))
- :copy (proxy-copy-with-body http-url)
- :move (proxy-move-with-body http-url)
- :patch (if body
- (proxy-patch-with-body http-url)
- (HttpPatch. http-url))
- (throw (IllegalArgumentException.
- (str "Invalid request method " request-method)))))
+ (isa? (type body) HttpEntity)
+ (let [baos (ByteArrayOutputStream.)]
+ (.writeTo ^HttpEntity body baos)
+ (.toString baos "UTF-8"))
+
+ :else nil)
+ (if (isa? (type body) String)
+ (format "... %s bytes ..."
+ (count body))
+ (and body (bean body))))
+ :body-type (type body)))
+ (println "HttpRequest:")
+ (clojure.pprint/pprint (bean http-req)))))
+
+(defn- build-response-map
+ [^HttpResponse response req ^HttpUriRequest http-req http-url
+ conn-mgr ^HttpClientContext context ^CloseableHttpClient client]
+ (let [^HttpEntity entity (.getEntity response)
+ status (.getStatusLine response)
+ protocol-version (.getProtocolVersion status)
+ body (:body req)
+ response
+ {:body (coerce-body-entity entity conn-mgr response)
+ :http-client client
+ :headers (parse-headers
+ (.headerIterator response)
+ (opt req :use-header-maps-in-response))
+ :length (if (nil? entity) 0 (.getContentLength entity))
+ :chunked? (if (nil? entity) false (.isChunked entity))
+ :repeatable? (if (nil? entity) false (.isRepeatable entity))
+ :streaming? (if (nil? entity) false (.isStreaming entity))
+ :status (.getStatusCode status)
+ :protocol-version {:name (.getProtocol protocol-version)
+ :major (.getMajor protocol-version)
+ :minor (.getMinor protocol-version)}
+ :reason-phrase (.getReasonPhrase status)
+ :trace-redirects (mapv str (.getRedirectLocations context))
+ :cached (when (instance? HttpCacheContext context)
+ (when-let [cache-resp (.getCacheResponseStatus ^HttpCacheContext context)]
+ (-> cache-resp str keyword)))}]
+ (if (opt req :save-request)
+ (-> response
+ (assoc :request req)
+ (assoc-in [:request :body-type] (type body))
+ (assoc-in [:request :http-url] http-url)
+ (update-in [:request]
+ #(if (opt req :debug-body)
+ (assoc % :body-content
+ (cond
+ (isa? (type (:body %)) String)
+ (:body %)
+
+ (isa? (type (:body %)) HttpEntity)
+ (let [baos (ByteArrayOutputStream.)]
+ (.writeTo ^HttpEntity (:body %) baos)
+ (.toString baos "UTF-8"))
+
+ :else nil))
+ %))
+ (assoc-in [:request :http-req] http-req))
+ response)))
+
+(defn- get-conn-mgr
+ [async? req]
+ (if async?
+ (or conn/*async-connection-manager*
+ (conn/make-regular-async-conn-manager req))
+ (or conn/*connection-manager*
+ (conn/make-regular-conn-manager req))))
(defn request
- "Executes the HTTP request corresponding to the given Ring request map and
- returns the Ring response map corresponding to the resulting HTTP response.
-
- Note that where Ring uses InputStreams for the request and response bodies,
- the clj-http uses ByteArrays for the bodies."
- [{:keys [request-method scheme server-name server-port uri query-string
- headers body multipart socket-timeout conn-timeout proxy-host
- proxy-ignore-hosts proxy-port proxy-user proxy-pass as cookie-store
- retry-handler request-interceptor response-interceptor
- digest-auth ntlm-auth connection-manager client-params]
- :as req}]
- (let [^ClientConnectionManager conn-mgr
- (or connection-manager
- conn/*connection-manager*
- (conn/make-regular-conn-manager req))
- ^DefaultHttpClient http-client (set-routing
- (DefaultHttpClient. conn-mgr))
- scheme (name scheme)]
- (when-let [cookie-store (or cookie-store *cookie-store*)]
- (.setCookieStore http-client cookie-store))
- (when retry-handler
- (.setHttpRequestRetryHandler
- http-client
- (proxy [HttpRequestRetryHandler] []
- (retryRequest [e cnt context]
- (retry-handler e cnt context)))))
- (add-client-params!
- http-client
- ;; merge in map of specified timeouts, to
- ;; support backward compatiblity.
- (merge {CoreConnectionPNames/SO_TIMEOUT socket-timeout
- CoreConnectionPNames/CONNECTION_TIMEOUT conn-timeout}
- client-params))
-
- (when-let [[user pass] digest-auth]
- (.setCredentials
- (.getCredentialsProvider http-client)
- (AuthScope. nil -1 nil)
- (UsernamePasswordCredentials. user pass)))
- (when-let [[user password host domain] ntlm-auth]
- (.setCredentials
- (.getCredentialsProvider http-client)
- (AuthScope. nil -1 nil)
- (NTCredentials. user password host domain)))
-
- (when (and proxy-user proxy-pass)
- (let [authscope (AuthScope. proxy-host proxy-port)
- creds (UsernamePasswordCredentials. proxy-user proxy-pass)]
- (.setCredentials (.getCredentialsProvider http-client)
- authscope creds)))
- (let [http-url (str scheme "://" server-name
- (when server-port (str ":" server-port))
- uri
- (when query-string (str "?" query-string)))
- req (assoc req :http-url http-url)
- proxy-ignore-hosts (or proxy-ignore-hosts
- #{"localhost" "127.0.0.1"})
- ^HttpUriRequest http-req (maybe-force-proxy
- http-client
- (http-request-for request-method
- http-url body)
- proxy-host
- proxy-port
- proxy-ignore-hosts)]
- (when request-interceptor
- (.addRequestInterceptor
- http-client
- (proxy [HttpRequestInterceptor] []
- (process [req ctx]
- (request-interceptor req ctx)))))
- (when response-interceptor
- (.addResponseInterceptor
- http-client
- (proxy [HttpResponseInterceptor] []
- (process [resp ctx]
- (response-interceptor resp ctx)))))
- (when-not (conn/reusable? conn-mgr)
- (.addHeader http-req "Connection" "close"))
- (doseq [[header-n header-v] headers]
- (if (coll? header-v)
- (doseq [header-vth header-v]
- (.addHeader http-req header-n header-vth))
- (.addHeader http-req header-n (str header-v))))
- (if multipart
- (.setEntity ^HttpEntityEnclosingRequest http-req
- (mp/create-multipart-entity multipart))
- (when (and body (instance? HttpEntityEnclosingRequest http-req))
- (if (instance? HttpEntity body)
- (.setEntity ^HttpEntityEnclosingRequest http-req body)
- (.setEntity ^HttpEntityEnclosingRequest http-req
- (if (string? body)
- (StringEntity. ^String body "UTF-8")
- (ByteArrayEntity. body))))))
- (when (opt req :debug) (print-debug! req http-req))
- (try
- (let [http-resp (.execute http-client http-req)
- http-entity (.getEntity http-resp)
- resp {:status (.getStatusCode (.getStatusLine http-resp))
- :headers (parse-headers
- (.headerIterator http-resp)
- (opt req :use-header-maps-in-response))
- :body (coerce-body-entity req http-entity conn-mgr)}]
- (if (opt req :save-request)
- (-> resp
- (assoc :request req)
- (assoc-in [:request :body-type] (type body))
- (update-in [:request]
- #(if (opt req :debug-body)
- (assoc % :body-content
- (cond
- (isa? (type (:body %)) String)
- (:body %)
-
- (isa? (type (:body %)) HttpEntity)
- (let [baos (ByteArrayOutputStream.)]
- (.writeTo ^HttpEntity (:body %) baos)
- (.toString baos "UTF-8"))
-
- :else nil))
- %))
- (assoc-in [:request :http-req] http-req)
- (dissoc :save-request?))
- resp))
- (catch Throwable e
- (when-not (conn/reusable? conn-mgr)
- (conn/shutdown-manager conn-mgr))
- (throw e))))))
+ ([req] (request req nil nil))
+ ([{:keys [body connection-timeout connection-request-timeout connection-manager
+ cookie-store cookie-policy headers multipart query-string
+ redirect-strategy max-redirects retry-handler
+ request-method scheme server-name server-port socket-timeout
+ uri response-interceptor proxy-host proxy-port
+ http-client-context http-request-config http-client
+ proxy-ignore-hosts proxy-user proxy-pass digest-auth ntlm-auth
+ multipart-mode multipart-charset
+ ; deprecated
+ conn-timeout conn-request-timeout]
+ :as req} respond raise]
+ (let [async? (opt req :async)
+ cache? (opt req :cache)
+ scheme (name scheme)
+ http-url (str scheme "://" server-name
+ (when server-port (str ":" server-port))
+ uri
+ (when query-string (str "?" query-string)))
+ conn-mgr (or connection-manager
+ (get-conn-mgr async? req))
+ proxy-ignore-hosts (or proxy-ignore-hosts
+ #{"localhost" "127.0.0.1"})
+ ^RequestConfig request-config (or http-request-config
+ (request-config req))
+ ^HttpClientContext context
+ (http-context cache? request-config http-client-context)
+ ^HttpUriRequest http-req (http-request-for
+ request-method http-url body)]
+ (when-not (conn/reusable? conn-mgr)
+ (.addHeader http-req "Connection" "close"))
+ (when-let [cookie-jar (or cookie-store *cookie-store*)]
+ (.setCookieStore context cookie-jar))
+ (when-let [[user pass] digest-auth]
+ (.setCredentialsProvider
+ context
+ (doto (credentials-provider)
+ (.setCredentials (AuthScope. nil -1 nil)
+ (UsernamePasswordCredentials. user pass)))))
+ (when-let [[user password host domain] ntlm-auth]
+ (.setCredentialsProvider
+ context
+ (doto (credentials-provider)
+ (.setCredentials (AuthScope. nil -1 nil)
+ (NTCredentials. user password host domain)))))
+ (when (and proxy-user proxy-pass)
+ (let [authscope (AuthScope. proxy-host proxy-port)
+ creds (UsernamePasswordCredentials. proxy-user proxy-pass)]
+ (.setCredentialsProvider
+ context
+ (doto (credentials-provider)
+ (.setCredentials authscope creds)))))
+ (if multipart
+ (.setEntity ^HttpEntityEnclosingRequest http-req
+ (mp/create-multipart-entity multipart req))
+ (when (and body (instance? HttpEntityEnclosingRequest http-req))
+ (if (instance? HttpEntity body)
+ (.setEntity ^HttpEntityEnclosingRequest http-req body)
+ (.setEntity ^HttpEntityEnclosingRequest http-req
+ (if (string? body)
+ (StringEntity. ^String body "UTF-8")
+ (ByteArrayEntity. body))))))
+ (doseq [[header-n header-v] headers]
+ (if (coll? header-v)
+ (doseq [header-vth header-v]
+ (.addHeader http-req header-n header-vth))
+ (.addHeader http-req header-n (str header-v))))
+ (when (opt req :debug) (print-debug! req http-req))
+ (if-not async?
+ (let [^CloseableHttpClient client
+ (or http-client
+ (build-http-client req cache?
+ conn-mgr http-url proxy-ignore-hosts))]
+ (try
+ (build-response-map (.execute client http-req context)
+ req http-req http-url conn-mgr context client)
+ (catch Throwable t
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))
+ (throw t))))
+ (let [^CloseableHttpAsyncClient client
+ (or http-client
+ (build-async-http-client req conn-mgr http-url proxy-ignore-hosts))
+ original-thread-bindings (clojure.lang.Var/getThreadBindingFrame)]
+ (when cache?
+ (throw (IllegalArgumentException.
+ "caching is not yet supported for async clients")))
+ (.start client)
+ (.execute client http-req context
+ (reify org.apache.http.concurrent.FutureCallback
+ (failed [this ex]
+ (clojure.lang.Var/resetThreadBindingFrame original-thread-bindings)
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))
+ (if (opt req :ignore-unknown-host)
+ ((:unknown-host-respond req) nil)
+ (raise ex)))
+ (completed [this resp]
+ (clojure.lang.Var/resetThreadBindingFrame original-thread-bindings)
+ (try
+ (respond (build-response-map
+ resp req http-req http-url
+ conn-mgr context client))
+ (catch Throwable t
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))
+ (raise t))))
+ (cancelled [this]
+ (clojure.lang.Var/resetThreadBindingFrame original-thread-bindings)
+ ;; Run the :oncancel function if available
+ (when-let [oncancel (:oncancel req)]
+ (oncancel))
+ ;; Attempt to abort the execution of the request
+ (.abort http-req)
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))))))))))
diff --git a/src/clj_http/core_old.clj b/src/clj_http/core_old.clj
new file mode 100644
index 0000000..52e1707
--- /dev/null
+++ b/src/clj_http/core_old.clj
@@ -0,0 +1,323 @@
+(ns clj-http.core-old
+ "Core HTTP request/response implementation."
+ (:require [clj-http.conn-mgr :as conn]
+ [clj-http.headers :as headers]
+ [clj-http.multipart :as mp]
+ [clj-http.util :refer [opt]]
+ clojure.pprint)
+ (:import [java.io ByteArrayOutputStream FilterInputStream InputStream]
+ java.net.URI
+ [org.apache.http HeaderIterator HttpEntity HttpEntityEnclosingRequest HttpHost HttpResponseInterceptor]
+ [org.apache.http.auth AuthScope NTCredentials UsernamePasswordCredentials]
+ [org.apache.http.client HttpClient HttpRequestRetryHandler]
+ [org.apache.http.client.methods HttpDelete HttpEntityEnclosingRequestBase HttpGet HttpHead HttpOptions HttpPatch HttpPost HttpPut HttpUriRequest]
+ [org.apache.http.client.params ClientPNames CookiePolicy]
+ org.apache.http.conn.ClientConnectionManager
+ org.apache.http.conn.params.ConnRoutePNames
+ org.apache.http.conn.routing.HttpRoute
+ org.apache.http.cookie.CookieSpecFactory
+ org.apache.http.cookie.params.CookieSpecPNames
+ [org.apache.http.entity ByteArrayEntity StringEntity]
+ org.apache.http.impl.client.DefaultHttpClient
+ org.apache.http.impl.conn.ProxySelectorRoutePlanner
+ org.apache.http.impl.cookie.BrowserCompatSpec
+ org.apache.http.params.CoreConnectionPNames))
+
+(defn parse-headers
+ "Takes a HeaderIterator and returns a map of names to values.
+
+ If a name appears more than once (like `set-cookie`) then the value
+ will be a vector containing the values in the order they appeared
+ in the headers."
+ [^HeaderIterator headers & [use-header-maps-in-response?]]
+ (if-not use-header-maps-in-response?
+ (->> (headers/header-iterator-seq headers)
+ (map (fn [[k v]]
+ [(.toLowerCase ^String k) v]))
+ (reduce (fn [hs [k v]]
+ (headers/assoc-join hs k v))
+ {}))
+ (->> (headers/header-iterator-seq headers)
+ (reduce (fn [hs [k v]]
+ (headers/assoc-join hs k v))
+ (headers/header-map)))))
+
+(defn set-client-param [^HttpClient client key val]
+ (when-not (nil? val)
+ (-> client
+ (.getParams)
+ (.setParameter key val))))
+
+(defn make-proxy-method-with-body
+ [method]
+ (fn [^String url]
+ (doto (proxy [HttpEntityEnclosingRequestBase] []
+ (getMethod [] (.toUpperCase (name method))))
+ (.setURI (URI. url)))))
+
+(def proxy-delete-with-body (make-proxy-method-with-body :delete))
+(def proxy-get-with-body (make-proxy-method-with-body :get))
+(def proxy-copy-with-body (make-proxy-method-with-body :copy))
+(def proxy-move-with-body (make-proxy-method-with-body :move))
+(def proxy-patch-with-body (make-proxy-method-with-body :patch))
+
+(def ^:dynamic *cookie-store* nil)
+
+(defn- set-routing
+ "Use ProxySelectorRoutePlanner to choose proxy sensible based on
+ http.nonProxyHosts"
+ [^DefaultHttpClient client]
+ (.setRoutePlanner client
+ (ProxySelectorRoutePlanner.
+ (.. client getConnectionManager getSchemeRegistry) nil))
+ client)
+
+(defn maybe-force-proxy [^DefaultHttpClient client
+ ^HttpEntityEnclosingRequestBase request
+ proxy-host proxy-port proxy-ignore-hosts]
+ (let [uri (.getURI request)]
+ (when (and (nil? ((set proxy-ignore-hosts) (.getHost uri))) proxy-host)
+ (let [target (HttpHost. (.getHost uri) (.getPort uri) (.getScheme uri))
+ route (HttpRoute. target nil (HttpHost. ^String proxy-host
+ (int proxy-port))
+ (.. client getConnectionManager getSchemeRegistry
+ (getScheme target) isLayered))]
+ (set-client-param client ConnRoutePNames/FORCED_ROUTE route)))
+ request))
+
+(defn cookie-spec
+ "Create an instance of a
+ org.apache.http.impl.cookie.BrowserCompatSpec with a validate
+ function that you pass in. This function takes two parameters, a
+ cookie and an origin."
+ [f]
+ (proxy [BrowserCompatSpec] []
+ (validate [cookie origin] (f cookie origin))))
+
+(defn cookie-spec-factory
+ "Create an instance of a org.apache.http.cookie.CookieSpecFactory
+ with a newInstance implementation that returns a cookie
+ specification with a validate function that you pass in. The
+ function takes two parameters: cookie and origin."
+ [f]
+ (proxy
+ [CookieSpecFactory] []
+ (newInstance [params] (cookie-spec f))))
+
+(defn add-client-params!
+ "Add various client params to the http-client object, if needed."
+ [^DefaultHttpClient http-client kvs]
+ (let [cookie-policy (:cookie-policy kvs)
+ cookie-policy-name (str (type cookie-policy))
+ kvs (dissoc kvs :cookie-policy)]
+ (when cookie-policy
+ (-> http-client
+ .getCookieSpecs
+ (.register cookie-policy-name (cookie-spec-factory cookie-policy))))
+ (doto http-client
+ (set-client-param ClientPNames/COOKIE_POLICY
+ (if cookie-policy
+ cookie-policy-name
+ CookiePolicy/BROWSER_COMPATIBILITY))
+ (set-client-param CookieSpecPNames/SINGLE_COOKIE_HEADER true)
+ (set-client-param ClientPNames/HANDLE_REDIRECTS false))
+
+ (doseq [[k v] kvs]
+ (set-client-param http-client
+ k (cond
+ (and (not= ClientPNames/CONN_MANAGER_TIMEOUT k)
+ (instance? Long v))
+ (Integer. ^Long v)
+ true v)))))
+
+(defn- coerce-body-entity
+ "Coerce the http-entity from an HttpResponse to either a byte-array, or a
+ stream that closes itself and the connection manager when closed."
+ [{:keys [as]} ^HttpEntity http-entity ^ClientConnectionManager conn-mgr]
+ (if http-entity
+ (proxy [FilterInputStream]
+ [^InputStream (.getContent http-entity)]
+ (close []
+ (try
+ ;; Eliminate the reflection warning from proxy-super
+ (let [^InputStream this this]
+ (proxy-super close))
+ (finally
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))))))
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))))
+
+(defn- print-debug!
+ "Print out debugging information to *out* for a given request."
+ [{:keys [debug-body body] :as req} http-req]
+ (println "Request:" (type body))
+ (clojure.pprint/pprint
+ (assoc req
+ :body (if (opt req :debug-body)
+ (cond
+ (isa? (type body) String)
+ body
+
+ (isa? (type body) HttpEntity)
+ (let [baos (ByteArrayOutputStream.)]
+ (.writeTo ^HttpEntity body baos)
+ (.toString baos "UTF-8"))
+
+ :else nil)
+ (if (isa? (type body) String)
+ (format "... %s bytes ..."
+ (count body))
+ (and body (bean body))))
+ :body-type (type body)))
+ (println "HttpRequest:")
+ (clojure.pprint/pprint (bean http-req)))
+
+(defn http-request-for
+ "Provides the HttpRequest object for a particular request-method and url"
+ [request-method ^String http-url body]
+ (case request-method
+ :get (if body
+ (proxy-get-with-body http-url)
+ (HttpGet. http-url))
+ :head (HttpHead. http-url)
+ :put (HttpPut. http-url)
+ :post (HttpPost. http-url)
+ :options (HttpOptions. http-url)
+ :delete (if body
+ (proxy-delete-with-body http-url)
+ (HttpDelete. http-url))
+ :copy (proxy-copy-with-body http-url)
+ :move (proxy-move-with-body http-url)
+ :patch (if body
+ (proxy-patch-with-body http-url)
+ (HttpPatch. http-url))
+ (throw (IllegalArgumentException.
+ (str "Invalid request method " request-method)))))
+
+(defn request
+ "Executes the HTTP request corresponding to the given Ring request map and
+ returns the Ring response map corresponding to the resulting HTTP response.
+
+ Note that where Ring uses InputStreams for the request and response bodies,
+ the clj-http uses ByteArrays for the bodies."
+ [{:keys [request-method scheme server-name server-port uri query-string
+ headers body multipart socket-timeout connection-timeout proxy-host
+ proxy-ignore-hosts proxy-port proxy-user proxy-pass as cookie-store
+ retry-handler response-interceptor digest-auth ntlm-auth
+ connection-manager client-params
+ ; deprecated
+ conn-timeout
+ ]
+ :as req}]
+ (let [^ClientConnectionManager conn-mgr
+ (or connection-manager
+ conn/*connection-manager*
+ (conn/make-regular-conn-manager req))
+ ^DefaultHttpClient http-client (set-routing
+ (DefaultHttpClient. conn-mgr))
+ scheme (name scheme)]
+ (when-let [cookie-store (or cookie-store *cookie-store*)]
+ (.setCookieStore http-client cookie-store))
+ (when retry-handler
+ (.setHttpRequestRetryHandler
+ http-client
+ (proxy [HttpRequestRetryHandler] []
+ (retryRequest [e cnt context]
+ (retry-handler e cnt context)))))
+ (add-client-params!
+ http-client
+ ;; merge in map of specified timeouts, to
+ ;; support backward compatibility.
+ (merge {CoreConnectionPNames/SO_TIMEOUT socket-timeout
+ CoreConnectionPNames/CONNECTION_TIMEOUT (or connection-timeout
+ conn-timeout)}
+ client-params))
+
+ (when-let [[user pass] digest-auth]
+ (.setCredentials
+ (.getCredentialsProvider http-client)
+ (AuthScope. nil -1 nil)
+ (UsernamePasswordCredentials. user pass)))
+ (when-let [[user password host domain] ntlm-auth]
+ (.setCredentials
+ (.getCredentialsProvider http-client)
+ (AuthScope. nil -1 nil)
+ (NTCredentials. user password host domain)))
+
+ (when (and proxy-user proxy-pass)
+ (let [authscope (AuthScope. proxy-host proxy-port)
+ creds (UsernamePasswordCredentials. proxy-user proxy-pass)]
+ (.setCredentials (.getCredentialsProvider http-client)
+ authscope creds)))
+ (let [http-url (str scheme "://" server-name
+ (when server-port (str ":" server-port))
+ uri
+ (when query-string (str "?" query-string)))
+ req (assoc req :http-url http-url)
+ proxy-ignore-hosts (or proxy-ignore-hosts
+ #{"localhost" "127.0.0.1"})
+ ^HttpUriRequest http-req (maybe-force-proxy
+ http-client
+ (http-request-for request-method
+ http-url body)
+ proxy-host
+ proxy-port
+ proxy-ignore-hosts)]
+ (when response-interceptor
+ (.addResponseInterceptor
+ http-client
+ (proxy [HttpResponseInterceptor] []
+ (process [resp ctx]
+ (response-interceptor resp ctx)))))
+ (when-not (conn/reusable? conn-mgr)
+ (.addHeader http-req "Connection" "close"))
+ (doseq [[header-n header-v] headers]
+ (if (coll? header-v)
+ (doseq [header-vth header-v]
+ (.addHeader http-req header-n header-vth))
+ (.addHeader http-req header-n (str header-v))))
+ (if multipart
+ (.setEntity ^HttpEntityEnclosingRequest http-req
+ (mp/create-multipart-entity multipart req))
+ (when (and body (instance? HttpEntityEnclosingRequest http-req))
+ (if (instance? HttpEntity body)
+ (.setEntity ^HttpEntityEnclosingRequest http-req body)
+ (.setEntity ^HttpEntityEnclosingRequest http-req
+ (if (string? body)
+ (StringEntity. ^String body "UTF-8")
+ (ByteArrayEntity. body))))))
+ (when (opt req :debug) (print-debug! req http-req))
+ (try
+ (let [http-resp (.execute http-client http-req)
+ http-entity (.getEntity http-resp)
+ resp {:status (.getStatusCode (.getStatusLine http-resp))
+ :headers (parse-headers
+ (.headerIterator http-resp)
+ (opt req :use-header-maps-in-response))
+ :body (coerce-body-entity req http-entity conn-mgr)}]
+ (if (opt req :save-request)
+ (-> resp
+ (assoc :request req)
+ (assoc-in [:request :body-type] (type body))
+ (update-in [:request]
+ #(if (opt req :debug-body)
+ (assoc % :body-content
+ (cond
+ (isa? (type (:body %)) String)
+ (:body %)
+
+ (isa? (type (:body %)) HttpEntity)
+ (let [baos (ByteArrayOutputStream.)]
+ (.writeTo ^HttpEntity (:body %) baos)
+ (.toString baos "UTF-8"))
+
+ :else nil))
+ %))
+ (assoc-in [:request :http-req] http-req)
+ (dissoc :save-request?))
+ resp))
+ (catch Throwable e
+ (when-not (conn/reusable? conn-mgr)
+ (conn/shutdown-manager conn-mgr))
+ (throw e))))))
diff --git a/src/clj_http/headers.clj b/src/clj_http/headers.clj
index 3ee7820..d229320 100644
--- a/src/clj_http/headers.clj
+++ b/src/clj_http/headers.clj
@@ -8,8 +8,8 @@
\"Accept-Encoding\")."
(:require [clojure.string :as s]
[potemkin :as potemkin])
- (:import (java.util Locale)
- (org.apache.http Header HeaderIterator)))
+ (:import java.util.Locale
+ [org.apache.http Header HeaderIterator]))
(def special-cases
"A collection of HTTP headers that do not follow the normal
@@ -119,9 +119,14 @@
mta)
(with-meta [_ mta]
(HeaderMap. m mta))
+
clojure.lang.Associative
(containsKey [_ k]
(contains? m (normalize k)))
+ (entryAt [_ k]
+ (if (contains? m (normalize k))
+ (clojure.lang.MapEntry. k (get _ k))))
+
(empty [_]
(HeaderMap. {} nil)))
@@ -131,13 +136,19 @@
(into (HeaderMap. {} nil)
(apply array-map keyvals)))
+(defn- header-map-request
+ [req]
+ (let [req-headers (:headers req)]
+ (if req-headers
+ (-> req (assoc :headers (into (header-map) req-headers)
+ :use-header-maps-in-response? true))
+ req)))
+
(defn wrap-header-map
"Middleware that converts headers from a map into a header-map."
[client]
- (fn [req]
- (let [req-headers (:headers req)
- req (if req-headers
- (-> req (assoc :headers (into (header-map) req-headers)
- :use-header-maps-in-response? true))
- req)]
- (client req))))
+ (fn
+ ([req]
+ (client (header-map-request req)))
+ ([req respond raise]
+ (client (header-map-request req) respond raise))))
diff --git a/src/clj_http/links.clj b/src/clj_http/links.clj
index bd45c2d..a1665a6 100644
--- a/src/clj_http/links.clj
+++ b/src/clj_http/links.clj
@@ -38,6 +38,17 @@
(map read-link-value)
(into {})))
+(defn- links-response
+ [response]
+ (if-let [link-headers (get-in response [:headers "link"])]
+ (let [link-headers (if (coll? link-headers)
+ link-headers
+ [link-headers])]
+ (assoc response
+ :links
+ (into {} (map read-link-headers link-headers))))
+ response))
+
(defn wrap-links
"Add a :links key to the response map that contains parsed Link headers. The
links will be represented as a map, with the 'rel' value being the key. The
@@ -47,13 +58,8 @@
=> {:links {:next {:href \"http://example.com/page2.html\"
:title \"Page 2\"}}}"
[client]
- (fn [request]
- (let [response (client request)]
- (if-let [link-headers (get-in response [:headers "link"])]
- (let [link-headers (if (coll? link-headers)
- link-headers
- [link-headers])]
- (assoc response
- :links
- (into {} (map read-link-headers link-headers))))
- response))))
+ (fn
+ ([request]
+ (links-response (client request)))
+ ([request respond raise]
+ (client request #(respond (links-response %)) raise))))
diff --git a/src/clj_http/multipart.clj b/src/clj_http/multipart.clj
index 3090084..e4381ac 100644
--- a/src/clj_http/multipart.clj
+++ b/src/clj_http/multipart.clj
@@ -1,22 +1,18 @@
(ns clj-http.multipart
"Namespace used for clj-http to create multipart entities and bodies."
- (:import (java.io File InputStream)
- (org.apache.http.entity ContentType)
- (org.apache.http.entity.mime MultipartEntity)
- (org.apache.http.entity.mime.content ContentBody
- ByteArrayBody
- FileBody
- InputStreamBody
- StringBody)
- (org.apache.http Consts)))
+ (:import [java.io File InputStream]
+ org.apache.http.Consts
+ org.apache.http.entity.ContentType
+ [org.apache.http.entity.mime HttpMultipartMode MultipartEntityBuilder]
+ [org.apache.http.entity.mime.content ByteArrayBody ContentBody FileBody InputStreamBody StringBody]))
;; we don't need to make a fake byte-array every time, only once
(def byte-array-type (type (byte-array 0)))
(defmulti
make-multipart-body
- "Create a body object from the given map, dispatching on the type of its content.
- By default supported content body types are:
+ "Create a body object from the given map, dispatching on the type of its
+ content. By default supported content body types are:
- String
- byte array (requires providing name)
- InputStream (requires providing name)
@@ -30,10 +26,11 @@
(defmethod make-multipart-body :default
[multipart]
- (throw (Exception. (str "Unsupported type for multipart content: " (type (:content multipart))))))
+ (throw (Exception. (str "Unsupported type for multipart content: "
+ (type (:content multipart))))))
(defmethod make-multipart-body File
- ; Create a FileBody object from the given map, requiring at least :content
+ ;; Create a FileBody object from the given map, requiring at least :content
[{:keys [^String name ^String mime-type ^File content ^String encoding]}]
(cond
(and name mime-type content encoding)
@@ -55,10 +52,10 @@
(throw (Exception. "Multipart file body must contain at least :content"))))
(defmethod make-multipart-body InputStream
- ; Create an InputStreamBody object from the given map, requiring at least
- ; :content and :name. If no :length is specified, clj-http will use
- ; chunked transfer-encoding, if :length is specified, clj-http will
- ; workaround things be proxying the InputStreamBody to return a length.
+ ;; Create an InputStreamBody object from the given map, requiring at least
+ ;; :content and :name. If no :length is specified, clj-http will use
+ ;; chunked transfer-encoding, if :length is specified, clj-http will
+ ;; workaround things be proxying the InputStreamBody to return a length.
[{:keys [^String name ^String mime-type ^InputStream content length]}]
(cond
(and content name length)
@@ -81,8 +78,8 @@
"at least :content and :name")))))
(defmethod make-multipart-body byte-array-type
- ; Create a ByteArrayBody object from the given map, requiring at least :content
- ; and :name.
+ ;; Create a ByteArrayBody object from the given map, requiring at least
+ ;; :content and :name.
[{:keys [^String name ^String mime-type ^bytes content]}]
(cond
(and content name mime-type)
@@ -95,33 +92,69 @@
(throw (Exception. (str "Multipart byte array body must contain "
"at least :content and :name")))))
+(defmulti ^java.nio.charset.Charset encoding-to-charset class)
+(defmethod encoding-to-charset nil [encoding] nil)
+(defmethod encoding-to-charset java.nio.charset.Charset [encoding] encoding)
+(defmethod encoding-to-charset java.lang.String [encoding]
+ (java.nio.charset.Charset/forName encoding))
+
(defmethod make-multipart-body String
- ; Create a StringBody object from the given map, requiring at least :content.
- ; If :encoding is specified, it will be created using the Charset for
- ; that encoding.
- [{:keys [mime-type ^String content encoding]}]
+ ;; Create a StringBody object from the given map, requiring at least :content.
+ ;; If :encoding is specified, it will be created using the Charset for that
+ ;; encoding.
+ [{:keys [^String mime-type ^String content encoding]}]
(cond
(and content mime-type encoding)
- (StringBody. content (ContentType/create mime-type encoding))
+ (StringBody.
+ content (ContentType/create mime-type (encoding-to-charset encoding)))
(and content encoding)
- (StringBody. content (ContentType/create "text/plain" encoding))
+ (StringBody.
+ content (ContentType/create "text/plain" (encoding-to-charset encoding)))
content
- (StringBody. content (ContentType/create "text/plain" Consts/ASCII))))
+ (StringBody. content (ContentType/create "text/plain" Consts/UTF_8))))
(defmethod make-multipart-body ContentBody
- ; Use provided org.apache.http.entity.mime.content.ContentBody directly
+ ;; Use provided org.apache.http.entity.mime.content.ContentBody directly
[{:keys [^ContentBody content]}]
content)
+(defn- multipart-workaround
+ "Workaround for AsyncHttpClient to bypass 25kb restriction on getContent.
+
+ See https://github.com/dakrone/clj-http/issues/560.
+ "
+ [^org.apache.http.entity.mime.MultipartFormEntity mp-entity]
+ (reify org.apache.http.HttpEntity
+ (isRepeatable [_] (.isRepeatable mp-entity))
+ (isChunked [_] (.isChunked mp-entity))
+ (isStreaming [_] (.isStreaming mp-entity))
+ (getContentLength [_] (.getContentLength mp-entity))
+ (getContentType [_] (.getContentType mp-entity))
+ (getContentEncoding [_] (.getContentEncoding mp-entity))
+ (consumeContent [_] (.consumeContent mp-entity))
+ (getContent [_]
+ (let [os (java.io.ByteArrayOutputStream.)]
+ (.writeTo mp-entity os)
+ (.flush os)
+ (java.io.ByteArrayInputStream. (.toByteArray os))))
+ (writeTo [_ output-stream] (.writeTo mp-entity output-stream))))
+
(defn create-multipart-entity
"Takes a multipart vector of maps and creates a MultipartEntity with each
map added as a part, depending on the type of content."
- [multipart]
- (let [mp-entity (MultipartEntity.)]
+ [multipart {:keys [mime-subtype multipart-mode multipart-charset]
+ :or {mime-subtype "form-data"
+ multipart-mode HttpMultipartMode/STRICT}}]
+ (let [mp-entity (doto (MultipartEntityBuilder/create)
+ (.setMode multipart-mode)
+ (.setMimeSubtype mime-subtype))]
+ (when multipart-charset
+ (.setCharset mp-entity (encoding-to-charset multipart-charset)))
(doseq [m multipart]
(let [name (or (:part-name m) (:name m))
part (make-multipart-body m)]
(.addPart mp-entity name part)))
- mp-entity))
+ (multipart-workaround
+ (.build mp-entity))))
diff --git a/src/clj_http/util.clj b/src/clj_http/util.clj
index 63b3dcb..f45ded6 100644
--- a/src/clj_http/util.clj
+++ b/src/clj_http/util.clj
@@ -2,13 +2,11 @@
"Helper functions for the HTTP client."
(:require [clojure.string :refer [blank? lower-case split trim]]
[clojure.walk :refer [postwalk]])
- (:import (org.apache.commons.codec.binary Base64)
- (org.apache.commons.io IOUtils)
- (java.io BufferedInputStream ByteArrayInputStream
- ByteArrayOutputStream)
- (java.net URLEncoder URLDecoder)
- (java.util.zip InflaterInputStream DeflaterInputStream
- GZIPInputStream GZIPOutputStream)))
+ (:import [java.io BufferedInputStream ByteArrayInputStream ByteArrayOutputStream EOFException InputStream PushbackInputStream]
+ [java.net URLDecoder URLEncoder]
+ [java.util.zip DeflaterInputStream GZIPInputStream GZIPOutputStream InflaterInputStream]
+ org.apache.commons.codec.binary.Base64
+ org.apache.commons.io.IOUtils))
(defn utf8-bytes
"Returns the encoding's bytes corresponding to the given string. If no
@@ -25,12 +23,12 @@
(defn url-decode
"Returns the form-url-decoded version of the given string, using either a
specified encoding or UTF-8 by default."
- [encoded & [encoding]]
+ [^String encoded & [^String encoding]]
(URLDecoder/decode encoded (or encoding "UTF-8")))
(defn url-encode
"Returns an UTF-8 URL encoded version of the given string."
- [unencoded & [encoding]]
+ [^String unencoded & [^String encoding]]
(URLEncoder/encode unencoded (or encoding "UTF-8")))
(defn base64-encode
@@ -43,8 +41,13 @@
[b]
(when b
(cond
- (instance? java.io.InputStream b)
- (GZIPInputStream. b)
+ (instance? InputStream b)
+ (let [^PushbackInputStream b (PushbackInputStream. b)
+ first-byte (int (try (.read b) (catch EOFException _ -1)))]
+ (case first-byte
+ -1 b
+ (do (.unread b first-byte)
+ (GZIPInputStream. b))))
:else
(IOUtils/toByteArray (GZIPInputStream. (ByteArrayInputStream. b))))))
@@ -58,14 +61,41 @@
(.close gos)
(.toByteArray baos))))
+(defn force-stream
+ "Force b as InputStream if it is a ByteArray."
+ ^InputStream [b]
+ (if (instance? InputStream b)
+ b
+ (ByteArrayInputStream. b)))
+
(defn force-byte-array
"force b as byte array if it is an InputStream, also close the stream"
^bytes [b]
- (if (instance? java.io.InputStream b)
- (try (IOUtils/toByteArray ^java.io.InputStream b)
- (finally (.close ^java.io.InputStream b)))
+ (if (instance? InputStream b)
+ (let [^PushbackInputStream bs (PushbackInputStream. b)]
+ (try
+ (let [first-byte (int (try (.read bs) (catch EOFException _ -1)))]
+ (case first-byte
+ -1 (byte-array 0)
+ (do (.unread bs first-byte)
+ (IOUtils/toByteArray bs))))
+ (finally (.close bs))))
b))
+(defn force-string
+ "Convert s (a ByteArray or InputStream) to String."
+ ^String [s ^String charset]
+ (if (instance? InputStream s)
+ (let [^PushbackInputStream bs (PushbackInputStream. s)]
+ (try
+ (let [first-byte (int (try (.read bs) (catch EOFException _ -1)))]
+ (case first-byte
+ -1 ""
+ (do (.unread bs first-byte)
+ (IOUtils/toString bs charset))))
+ (finally (.close bs))))
+ (IOUtils/toString ^"[B" s charset)))
+
(defn inflate
"Returns a zlib inflate'd version of the given byte array or InputStream."
[b]
@@ -73,7 +103,7 @@
;; This weirdness is because HTTP servers lie about what kind of deflation
;; they're using, so we try one way, then if that doesn't work, reset and
;; try the other way
- (let [stream (BufferedInputStream. (if (instance? java.io.InputStream b)
+ (let [stream (BufferedInputStream. (if (instance? InputStream b)
b
(ByteArrayInputStream. b)))
_ (.mark stream 512)
@@ -113,15 +143,17 @@
false
(or v1 v2)))))
+(defn- trim-quotes [s]
+ (clojure.string/replace s #"^\s*(\"(.*)\"|(.*?))\s*$" "$2$3"))
+
(defn parse-content-type
"Parse `s` as an RFC 2616 media type."
[s]
- (if-let [m (re-matches #"\s*(([^/]+)/([^ ;]+))\s*(\s*;.*)?" (str s))]
+ (when-let [m (re-matches #"\s*(([^/]+)/([^ ;]+))\s*(\s*;.*)?" (str s))]
{:content-type (keyword (nth m 1))
:content-type-params
(->> (split (str (nth m 4)) #"\s*;\s*")
- (identity)
(remove blank?)
(map #(split % #"="))
- (mapcat (fn [[k v]] [(keyword (lower-case k)) (trim v)]))
+ (mapcat (fn [[k v]] [(keyword (lower-case k)) (trim-quotes v)]))
(apply hash-map))}))
diff --git a/test-resources/big_array_json.json b/test-resources/big_array_json.json
new file mode 100644
index 0000000..51ccef7
--- /dev/null
+++ b/test-resources/big_array_json.json
@@ -0,0 +1,102 @@
+[
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]},
+ {"foo": "bar", "baz": "qux", "values": [1, 2, 3, 4, 5]}
+]
diff --git a/test-resources/client-keystore b/test-resources/client-keystore
index d30af3d..55bdc0c 100644
Binary files a/test-resources/client-keystore and b/test-resources/client-keystore differ
diff --git a/test-resources/keystore b/test-resources/keystore
index 13d78bf..2944ca2 100644
Binary files a/test-resources/keystore and b/test-resources/keystore differ
diff --git a/test-resources/m.txt b/test-resources/m.txt
new file mode 100644
index 0000000..8e6ce25
--- /dev/null
+++ b/test-resources/m.txt
@@ -0,0 +1,4 @@
+this
+is
+some
+file.
diff --git a/test-resources/small.jpg b/test-resources/small.jpg
new file mode 100644
index 0000000..71911bf
Binary files /dev/null and b/test-resources/small.jpg differ
diff --git a/test/clj_http/test/client.clj b/test/clj_http/test/client.clj
deleted file mode 100644
index b8e8828..0000000
--- a/test/clj_http/test/client.clj
+++ /dev/null
@@ -1,856 +0,0 @@
-(ns clj-http.test.client
- (:require [cheshire.core :as json]
- [clj-http.client :as client]
- [clj-http.conn-mgr :as conn]
- [clj-http.test.core :refer [run-server]]
- [clj-http.util :as util]
- [clojure.string :as str]
- [clojure.java.io :refer [resource]]
- [clojure.test :refer :all]
- [cognitect.transit :as transit]
- [ring.util.codec :refer [form-decode-str]]
- [ring.middleware.nested-params :refer [parse-nested-keys]])
- (:import (java.net UnknownHostException)
- (java.io ByteArrayInputStream)
- (org.apache.http HttpEntity)))
-
-(def base-req
- {:scheme :http
- :server-name "localhost"
- :server-port 18080})
-
-(defn request [req]
- (client/request (merge base-req req)))
-
-(defn parse-form-params [s]
- (->> (str/split (form-decode-str s) #"&")
- (map #(str/split % #"="))
- (map #(vector
- (map keyword (parse-nested-keys (first %)))
- (second %)))
- (reduce (fn [m [ks v]]
- (assoc-in m ks v)) {})))
-
-(deftest ^:integration roundtrip
- (run-server)
- ;; roundtrip with scheme as a keyword
- (let [resp (request {:uri "/get" :method :get})]
- (is (= 200 (:status resp)))
- (is (= "close" (get-in resp [:headers "connection"])))
- (is (= "get" (:body resp))))
- ;; roundtrip with scheme as a string
- (let [resp (request {:uri "/get" :method :get
- :scheme "http"})]
- (is (= 200 (:status resp)))
- (is (= "close" (get-in resp [:headers "connection"])))
- (is (= "get" (:body resp))))
- (let [params {:a "1" :b {:c "2"}}]
- (doseq [[content-type read-fn]
- [[nil (comp parse-form-params slurp)]
- [:x-www-form-urlencoded (comp parse-form-params slurp)]
- [:edn (comp read-string slurp)]
- [:transit+json #(client/parse-transit % :json)]
- [:transit+msgpack #(client/parse-transit % :msgpack)]]]
- (let [resp (request {:uri "/post"
- :as :stream
- :method :post
- :content-type content-type
- :form-params params})]
- (is (= 200 (:status resp)))
- (is (= "close" (get-in resp [:headers "connection"])))
- (is (= params (read-fn (:body resp))))))))
-
-(deftest ^:integration nil-input
- (is (thrown-with-msg? Exception #"Host URL cannot be nil"
- (client/get nil)))
- (is (thrown-with-msg? Exception #"Host URL cannot be nil"
- (client/post nil)))
- (is (thrown-with-msg? Exception #"Host URL cannot be nil"
- (client/put nil)))
- (is (thrown-with-msg? Exception #"Host URL cannot be nil"
- (client/delete nil))))
-
-(defn is-passed [middleware req]
- (let [client (middleware identity)]
- (is (= req (client req)))))
-
-(defn is-applied [middleware req-in req-out]
- (let [client (middleware identity)]
- (is (= req-out (client req-in)))))
-
-(deftest redirect-on-get
- (let [client (fn [req]
- (if (= "foo.com" (:server-name req))
- {:status 302
- :headers {"location" "http://bar.com/bat"}}
- {:status 200
- :req req}))
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method :get})]
- (is (= 200 (:status resp)))
- (is (= :get (:request-method (:req resp))))
- (is (= :http (:scheme (:req resp))))
- (is (= ["http://foo.com" "http://bar.com/bat"] (:trace-redirects resp)))
- (is (= "/bat" (:uri (:req resp))))))
-
-(deftest relative-redirect-on-get
- (let [client (fn [req]
- (if (:redirects-count req)
- {:status 200
- :req req}
- {:status 302
- :headers {"location" "/bat"}}))
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method :get})]
- (is (= 200 (:status resp)))
- (is (= :get (:request-method (:req resp))))
- (is (= :http (:scheme (:req resp))))
- (is (= ["http://foo.com" "http://foo.com/bat"] (:trace-redirects resp)))
- (is (= "/bat" (:uri (:req resp))))))
-
-(deftest trace-redirects-using-uri
- (let [client (fn [req] {:status 200 :req req})
- r-client (-> client client/wrap-redirects)
- resp (r-client {:scheme :http :server-name "foo.com" :uri "/"
- :request-method :get})]
- (is (= 200 (:status resp)))
- (is (= :get (:request-method (:req resp))))
- (is (= :http (:scheme (:req resp))))
- (is (= [] (:trace-redirects resp)))))
-
-(deftest redirect-without-location-header
- (let [client (fn [req]
- {:status 302 :body "no redirection here"})
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method :get})]
- (is (= 302 (:status resp)))
- (is (= ["http://foo.com"] (:trace-redirects resp)))
- (is (= "no redirection here" (:body resp)))))
-
-(deftest redirect-with-query-string
- (let [client (fn [req]
- (if (= "foo.com" (:server-name req))
- {:status 302
- :headers {"location" "http://bar.com/bat?x=y"}}
- {:status 200
- :req req}))
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method :get :query-params {:x "z"}})]
- (is (= 200 (:status resp)))
- (is (= :get (:request-method (:req resp))))
- (is (= :http (:scheme (:req resp))))
- (is (= ["http://foo.com" "http://bar.com/bat?x=y"] (:trace-redirects resp)))
- (is (= "/bat" (:uri (:req resp))))
- (is (= "x=y" (:query-string (:req resp))))
- (is (nil? (:query-params (:req resp))))))
-
-(deftest max-redirects
- (let [client (fn [req]
- (if (= "foo.com" (:server-name req))
- {:status 302
- :headers {"location" "http://bar.com/bat"}}
- {:status 200
- :req req}))
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method :get :max-redirects 0})]
- (is (= 302 (:status resp)))
- (is (= ["http://foo.com"] (:trace-redirects resp)))
- (is (= "http://bar.com/bat" (get (:headers resp) "location")))))
-
-(deftest redirect-303-to-get-on-any-method
- (doseq [method [:get :head :post :delete :put :option]]
- (let [client (fn [req]
- (if (= "foo.com" (:server-name req))
- {:status 303
- :headers {"location" "http://bar.com/bat"}}
- {:status 200
- :req req}))
- r-client (-> client client/wrap-url client/wrap-redirects)
- resp (r-client {:server-name "foo.com" :url "http://foo.com"
- :request-method method})]
- (is (= 200 (:status resp)))
- (is (= :get (:request-method (:req resp))))
- (is (= :http (:scheme (:req resp))))
- (is (= ["http://foo.com" "http://bar.com/bat"] (:trace-redirects resp)))
- (is (= "/bat" (:uri (:req resp)))))))
-
-(deftest pass-on-non-redirect
- (let [client (fn [req] {:status 200 :body (:body req)})
- r-client (client/wrap-redirects client)
- resp (r-client {:body "ok" :url "http://foo.com"})]
- (is (= 200 (:status resp)))
- (is (= ["http://foo.com"] (:trace-redirects resp)))
- (is (= "ok" (:body resp)))))
-
-(deftest pass-on-non-redirectable-methods
- (doseq [method [:put :post :delete]
- status [301 302 307]]
- (let [client (fn [req] {:status status :body (:body req)
- :headers {"location" "http://foo.com/bat"}})
- r-client (client/wrap-redirects client)
- resp (r-client {:body "ok" :url "http://foo.com"
- :request-method method})]
- (is (= status (:status resp)))
- (is (= ["http://foo.com"] (:trace-redirects resp)))
- (is (= {"location" "http://foo.com/bat"} (:headers resp)))
- (is (= "ok" (:body resp))))))
-
-(deftest force-redirects-on-non-redirectable-methods
- (doseq [method [:put :post :delete]
- [status expected-method] [[301 :get] [302 :get] [307 method]]]
- (let [client (fn [{:keys [trace-redirects body] :as req}]
- (if trace-redirects
- {:status 200 :body body :trace-redirects trace-redirects
- :req req}
- {:status status :body body :req req
- :headers {"location" "http://foo.com/bat"}}))
- r-client (client/wrap-redirects client)
- resp (r-client {:body "ok" :url "http://foo.com"
- :request-method method
- :force-redirects true})]
- (is (= 200 (:status resp)))
- (is (= ["http://foo.com" "http://foo.com/bat"] (:trace-redirects resp)))
- (is (= "ok" (:body resp)))
- (is (= expected-method (:request-method (:req resp)))))))
-
-(deftest pass-on-follow-redirects-false
- (let [client (fn [req] {:status 302 :body (:body req)})
- r-client (client/wrap-redirects client)
- resp (r-client {:body "ok" :follow-redirects false})]
- (is (= 302 (:status resp)))
- (is (= "ok" (:body resp)))
- (is (nil? (:trace-redirects resp)))))
-
-(deftest throw-on-exceptional
- (let [client (fn [req] {:status 500})
- e-client (client/wrap-exceptions client)]
- (is (thrown-with-msg? Exception #"500"
- (e-client {}))))
- (let [client (fn [req] {:status 500 :body "foo"})
- e-client (client/wrap-exceptions client)]
- (is (thrown-with-msg? Exception #":body"
- (e-client {:throw-entire-message? true})))))
-
-(deftest pass-on-non-exceptional
- (let [client (fn [req] {:status 200})
- e-client (client/wrap-exceptions client)
- resp (e-client {})]
- (is (= 200 (:status resp)))))
-
-(deftest pass-on-exceptional-when-surpressed
- (let [client (fn [req] {:status 500})
- e-client (client/wrap-exceptions client)
- resp (e-client {:throw-exceptions false})]
- (is (= 500 (:status resp)))))
-
-(deftest apply-on-compressed
- (let [client (fn [req]
- (is (= "gzip, deflate"
- (get-in req [:headers "accept-encoding"])))
- {:body (util/gzip (util/utf8-bytes "foofoofoo"))
- :headers {"content-encoding" "gzip"}})
- c-client (client/wrap-decompression client)
- resp (c-client {})]
- (is (= "foofoofoo" (util/utf8-string (:body resp))))
- (is (= "gzip" (:orig-content-encoding resp)))
- (is (= nil (get-in resp [:headers "content-encoding"])))))
-
-(deftest apply-on-deflated
- (let [client (fn [req]
- (is (= "gzip, deflate"
- (get-in req [:headers "accept-encoding"])))
- {:body (util/deflate (util/utf8-bytes "barbarbar"))
- :headers {"content-encoding" "deflate"}})
- c-client (client/wrap-decompression client)
- resp (c-client {})]
- (is (= "barbarbar" (-> resp :body util/force-byte-array util/utf8-string))
- "string correctly inflated")
- (is (= "deflate" (:orig-content-encoding resp)))
- (is (= nil (get-in resp [:headers "content-encoding"])))))
-
-(deftest t-disabled-body-decompression
- (let [client (fn [req]
- (is (not= "gzip, deflate"
- (get-in req [:headers "accept-encoding"])))
- {:body (util/deflate (util/utf8-bytes "barbarbar"))
- :headers {"content-encoding" "deflate"}})
- c-client (client/wrap-decompression client)
- resp (c-client {:decompress-body false})]
- (is (= (slurp (util/inflate (util/deflate (util/utf8-bytes "barbarbar"))))
- (slurp (util/inflate (-> resp :body util/force-byte-array))))
- "string not inflated")
- (is (= nil (:orig-content-encoding resp)))
- (is (= "deflate" (get-in resp [:headers "content-encoding"])))))
-
-(deftest t-weird-non-known-compression
- (let [client (fn [req]
- (is (= "gzip, deflate"
- (get-in req [:headers "accept-encoding"])))
- {:body (util/utf8-bytes "foofoofoo")
- :headers {"content-encoding" "pig-latin"}})
- c-client (client/wrap-decompression client)
- resp (c-client {})]
- (is (= "foofoofoo" (util/utf8-string (:body resp))))
- (is (= "pig-latin" (:orig-content-encoding resp)))
- (is (= "pig-latin" (get-in resp [:headers "content-encoding"])))))
-
-(deftest pass-on-non-compressed
- (let [c-client (client/wrap-decompression (fn [req] {:body "foo"}))
- resp (c-client {:uri "/foo"})]
- (is (= "foo" (:body resp)))))
-
-(deftest apply-on-accept
- (is-applied client/wrap-accept
- {:accept :json}
- {:headers {"accept" "application/json"}})
- (is-applied client/wrap-accept
- {:accept :transit+json}
- {:headers {"accept" "application/transit+json"}})
- (is-applied client/wrap-accept
- {:accept :transit+msgpack}
- {:headers {"accept" "application/transit+msgpack"}}))
-
-(deftest pass-on-no-accept
- (is-passed client/wrap-accept
- {:uri "/foo"}))
-
-(deftest apply-on-accept-encoding
- (is-applied client/wrap-accept-encoding
- {:accept-encoding [:identity :gzip]}
- {:headers {"accept-encoding" "identity, gzip"}}))
-
-(deftest pass-on-no-accept-encoding
- (is-passed client/wrap-accept-encoding
- {:uri "/foo"}))
-
-(deftest apply-on-output-coercion
- (let [client (fn [req] {:body (util/utf8-bytes "foo")})
- o-client (client/wrap-output-coercion client)
- resp (o-client {:uri "/foo"})]
- (is (= "foo" (:body resp)))))
-
-(deftest pass-on-no-output-coercion
- (let [client (fn [req] {:body nil})
- o-client (client/wrap-output-coercion client)
- resp (o-client {:uri "/foo"})]
- (is (nil? (:body resp))))
- (let [the-stream (ByteArrayInputStream. (byte-array []))
- client (fn [req] {:body the-stream})
- o-client (client/wrap-output-coercion client)
- resp (o-client {:uri "/foo" :as :stream})]
- (is (= the-stream (:body resp))))
- (let [client (fn [req] {:body :thebytes})
- o-client (client/wrap-output-coercion client)
- resp (o-client {:uri "/foo" :as :byte-array})]
- (is (= :thebytes (:body resp)))))
-
-(deftest apply-on-input-coercion
- (let [i-client (client/wrap-input-coercion identity)
- resp (i-client {:body "foo"})
- resp2 (i-client {:body "foo2" :body-encoding "ASCII"})
- data (slurp (.getContent ^HttpEntity (:body resp)))
- data2 (slurp (.getContent ^HttpEntity (:body resp2)))]
- (is (= "UTF-8" (:character-encoding resp)))
- (is (= "foo" data))
- (is (= "ASCII" (:character-encoding resp2)))
- (is (= "foo2" data2))))
-
-(deftest pass-on-no-input-coercion
- (is-passed client/wrap-input-coercion
- {:body nil}))
-
-(deftest no-length-for-input-stream
- (let [i-client (client/wrap-input-coercion identity)
- resp1 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))})
- resp2 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))
- :length 3})
- ^HttpEntity body1 (:body resp1)
- ^HttpEntity body2 (:body resp2)]
- (is (= -1 (.getContentLength body1)))
- (is (= 3 (.getContentLength body2)))))
-
-(deftest apply-on-content-type
- (is-applied client/wrap-content-type
- {:content-type :json}
- {:headers {"content-type" "application/json"}
- :content-type :json})
- (is-applied client/wrap-content-type
- {:content-type :json :character-encoding "UTF-8"}
- {:headers {"content-type" "application/json; charset=UTF-8"}
- :content-type :json :character-encoding "UTF-8"})
- (is-applied client/wrap-content-type
- {:content-type :transit+json}
- {:headers {"content-type" "application/transit+json"}
- :content-type :transit+json})
- (is-applied client/wrap-content-type
- {:content-type :transit+msgpack}
- {:headers {"content-type" "application/transit+msgpack"}
- :content-type :transit+msgpack}))
-
-(deftest pass-on-no-content-type
- (is-passed client/wrap-content-type
- {:uri "/foo"}))
-
-(deftest apply-on-query-params
- (is-applied client/wrap-query-params
- {:query-params {"foo" "bar" "dir" "<<"}}
- {:query-string "foo=bar&dir=%3C%3C"})
- (is-applied client/wrap-query-params
- {:query-string "foo=1"
- :query-params {"foo" ["2" "3"]}}
- {:query-string "foo=1&foo=2&foo=3"}))
-
-(deftest pass-on-no-query-params
- (is-passed client/wrap-query-params
- {:uri "/foo"}))
-
-(deftest apply-on-basic-auth
- (is-applied client/wrap-basic-auth
- {:basic-auth ["Aladdin" "open sesame"]}
- {:headers {"authorization"
- "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="}}))
-
-(deftest pass-on-no-basic-auth
- (is-passed client/wrap-basic-auth
- {:uri "/foo"}))
-
-(deftest apply-on-oauth
- (is-applied client/wrap-oauth
- {:oauth-token "my-token"}
- {:headers {"authorization"
- "Bearer my-token"}}))
-
-(deftest pass-on-no-oauth
- (is-passed client/wrap-oauth
- {:uri "/foo"}))
-
-(deftest apply-on-method
- (let [m-client (client/wrap-method identity)
- echo (m-client {:key :val :method :post})]
- (is (= :val (:key echo)))
- (is (= :post (:request-method echo)))
- (is (not (:method echo)))))
-
-(deftest pass-on-no-method
- (let [m-client (client/wrap-method identity)
- echo (m-client {:key :val})]
- (is (= :val (:key echo)))
- (is (not (:request-method echo)))))
-
-(deftest apply-on-url
- (let [u-client (client/wrap-url identity)
- resp (u-client {:url "http://google.com:8080/baz foo?bar=bat bit?"})]
- (is (= :http (:scheme resp)))
- (is (= "google.com" (:server-name resp)))
- (is (= 8080 (:server-port resp)))
- (is (= "/baz%20foo" (:uri resp)))
- (is (= "bar=bat%20bit?" (:query-string resp)))))
-
-(deftest pass-on-no-url
- (let [u-client (client/wrap-url identity)
- resp (u-client {:uri "/foo"})]
- (is (= "/foo" (:uri resp)))))
-
-(deftest provide-default-port
- (is (= nil (-> "http://example.com/" client/parse-url :server-port)))
- (is (= 8080 (-> "http://example.com:8080/" client/parse-url :server-port)))
- (is (= nil (-> "https://example.com/" client/parse-url :server-port)))
- (is (= 8443 (-> "https://example.com:8443/" client/parse-url :server-port))))
-
-(deftest decode-credentials-from-url
- (is (= "fred's diner:fred's password"
- (-> "http://fred%27s%20diner:fred%27s%20password@example.com/foo"
- client/parse-url
- :user-info))))
-
-(defrecord Point [x y])
-
-(def write-point
- "Write a point in Transit format."
- (transit/write-handler
- (constantly "point")
- (fn [point] [(:x point) (:y point)])
- (constantly nil)))
-
-(def read-point
- "Read a point in Transit format."
- (transit/read-handler
- (fn [[x y]]
- (->Point x y))))
-
-(def transit-opts
- "Transit read and write options."
- {:encode {:handlers {Point write-point}}
- :decode {:handlers {"point" read-point}}})
-
-(def transit-opts-deprecated
- "Deprecated Transit read and write options."
- {:handlers {Point write-point "point" read-point}})
-
-(deftest apply-on-form-params
- (testing "With form params"
- (let [param-client (client/wrap-form-params identity)
- resp (param-client {:request-method :post
- :form-params (sorted-map :param1 "value1"
- :param2 "value2")})]
- (is (= "param1=value1¶m2=value2" (:body resp)))
- (is (= "application/x-www-form-urlencoded" (:content-type resp)))
- (is (not (contains? resp :form-params))))
- (let [param-client (client/wrap-form-params identity)
- resp (param-client {:request-method :put
- :form-params (sorted-map :param1 "value1"
- :param2 "value2")})]
- (is (= "param1=value1¶m2=value2" (:body resp)))
- (is (= "application/x-www-form-urlencoded" (:content-type resp)))
- (is (not (contains? resp :form-params)))))
-
- (testing "With json form params"
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 "value2"}
- resp (param-client {:request-method :post
- :content-type :json
- :form-params params})]
- (is (= (json/encode params) (:body resp)))
- (is (= "application/json" (:content-type resp)))
- (is (not (contains? resp :form-params))))
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 "value2"}
- resp (param-client {:request-method :put
- :content-type :json
- :form-params params})]
- (is (= (json/encode params) (:body resp)))
- (is (= "application/json" (:content-type resp)))
- (is (not (contains? resp :form-params))))
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 "value2"}
- resp (param-client {:request-method :patch
- :content-type :json
- :form-params params})]
- (is (= (json/encode params) (:body resp)))
- (is (= "application/json" (:content-type resp)))
- (is (not (contains? resp :form-params))))
- (let [param-client (client/wrap-form-params identity)
- params {:param1 (java.util.Date. (long 0))}
- resp (param-client {:request-method :put
- :content-type :json
- :form-params params
- :json-opts {:date-format "yyyy-MM-dd"}})]
- (is (= (json/encode params {:date-format "yyyy-MM-dd"}) (:body resp)))
- (is (= "application/json" (:content-type resp)))
- (is (not (contains? resp :form-params)))))
-
- (testing "With EDN form params"
- (doseq [method [:post :put :patch]]
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 (Point. 1 2)}
- resp (param-client {:request-method method
- :content-type :edn
- :form-params params})]
- (is (= (pr-str params) (:body resp)))
- (is (= "application/edn" (:content-type resp)))
- (is (not (contains? resp :form-params))))))
-
- (testing "With Transit/JSON form params"
- (doseq [method [:post :put :patch]]
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 (Point. 1 2)}
- resp (param-client {:request-method method
- :content-type :transit+json
- :form-params params
- :transit-opts transit-opts})]
- (is (= params (client/parse-transit
- (ByteArrayInputStream. (:body resp))
- :json transit-opts)))
- (is (= "application/transit+json" (:content-type resp)))
- (is (not (contains? resp :form-params))))))
-
- (testing "With Transit/MessagePack form params"
- (doseq [method [:post :put :patch]]
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 "value2"}
- resp (param-client {:request-method method
- :content-type :transit+msgpack
- :form-params params
- :transit-opts transit-opts})]
- (is (= params (client/parse-transit
- (ByteArrayInputStream. (:body resp))
- :msgpack transit-opts)))
- (is (= "application/transit+msgpack" (:content-type resp)))
- (is (not (contains? resp :form-params))))))
-
- (testing "With Transit/JSON form params and deprecated options"
- (let [param-client (client/wrap-form-params identity)
- params {:param1 "value1" :param2 (Point. 1 2)}
- resp (param-client {:request-method :post
- :content-type :transit+json
- :form-params params
- :transit-opts transit-opts-deprecated})]
- (is (= params (client/parse-transit
- (ByteArrayInputStream. (:body resp))
- :json transit-opts-deprecated)))
- (is (= "application/transit+json" (:content-type resp)))
- (is (not (contains? resp :form-params)))))
-
- (testing "Ensure it does not affect GET requests"
- (let [param-client (client/wrap-form-params identity)
- resp (param-client {:request-method :get
- :body "untouched"
- :form-params {:param1 "value1"
- :param2 "value2"}})]
- (is (= "untouched" (:body resp)))
- (is (not (contains? resp :content-type)))))
-
- (testing "with no form params"
- (let [param-client (client/wrap-form-params identity)
- resp (param-client {:body "untouched"})]
- (is (= "untouched" (:body resp)))
- (is (not (contains? resp :content-type))))))
-
-(deftest apply-on-nested-params
- (testing "nested parameter maps"
- (are [in out] (is-applied client/wrap-nested-params
- {:query-params in :form-params in}
- {:query-params out :form-params out})
- {"foo" "bar"} {"foo" "bar"}
- {"x" {"y" "z"}} {"x[y]" "z"}
- {"a" {"b" {"c" "d"}}} {"a[b][c]" "d"}
- {"a" "b", "c" "d"} {"a" "b", "c" "d"}))
-
- (testing "not creating empty param maps"
- (is-applied client/wrap-query-params {} {})))
-
-(deftest t-ignore-unknown-host
- (is (thrown? UnknownHostException (client/get "http://aorecuf892983a.com")))
- (is (nil? (client/get "http://aorecuf892983a.com"
- {:ignore-unknown-host? true}))))
-
-(deftest test-status-predicates
- (testing "2xx statuses"
- (doseq [s (range 200 299)]
- (is (client/success? {:status s}))
- (is (not (client/redirect? {:status s})))
- (is (not (client/client-error? {:status s})))
- (is (not (client/server-error? {:status s})))))
- (testing "3xx statuses"
- (doseq [s (range 300 399)]
- (is (not (client/success? {:status s})))
- (is (client/redirect? {:status s}))
- (is (not (client/client-error? {:status s})))
- (is (not (client/server-error? {:status s})))))
- (testing "4xx statuses"
- (doseq [s (range 400 499)]
- (is (not (client/success? {:status s})))
- (is (not (client/redirect? {:status s})))
- (is (client/client-error? {:status s}))
- (is (not (client/server-error? {:status s})))))
- (testing "5xx statuses"
- (doseq [s (range 500 599)]
- (is (not (client/success? {:status s})))
- (is (not (client/redirect? {:status s})))
- (is (not (client/client-error? {:status s})))
- (is (client/server-error? {:status s}))))
- (testing "409 Conflict"
- (is (client/conflict? {:status 409}))
- (is (not (client/conflict? {:status 201})))
- (is (not (client/conflict? {:status 404})))))
-
-(deftest test-wrap-lower-case-headers
- (is (= {:status 404} ((client/wrap-lower-case-headers
- (fn [r] r)) {:status 404})))
- (is (= {:headers {"content-type" "application/json"}}
- ((client/wrap-lower-case-headers
- #(do (is (= {:headers {"accept" "application/json"}} %1))
- {:headers {"Content-Type" "application/json"}}))
- {:headers {"Accept" "application/json"}}))))
-
-(deftest t-request-timing
- (is (pos? (:request-time ((client/wrap-request-timing
- (fn [r] (Thread/sleep 15) r)) {})))))
-
-(deftest t-wrap-additional-header-parsing
- (let [^String text (slurp (resource "header-test.html"))
- client (fn [req] {:body (.getBytes text)})
- new-client (client/wrap-additional-header-parsing client)
- resp (new-client {:decode-body-headers true})
- resp2 (new-client {:decode-body-headers false})
- resp3 ((client/wrap-additional-header-parsing
- (fn [req] {:body nil})) {:decode-body-headers true})
- resp4 ((client/wrap-additional-header-parsing
- (fn [req] {:headers {"content-type" "application/pdf"}
- :body (.getBytes text)}))
- {:decode-body-headers true})]
- (is (= {"content-type" "text/html; charset=Shift_JIS"
- "content-style-type" "text/css"
- "content-script-type" "text/javascript"}
- (:headers resp)))
- (is (nil? (:headers resp2)))
- (is (nil? (:headers resp3)))
- (is (= {"content-type" "application/pdf"} (:headers resp4)))))
-
-(deftest t-wrap-additional-header-parsing-html5
- (let [^String text (slurp (resource "header-html5-test.html"))
- client (fn [req] {:body (.getBytes text)})
- new-client (client/wrap-additional-header-parsing client)
- resp (new-client {:decode-body-headers true})]
- (is (= {"content-type" "text/html; charset=UTF-8"}
- (:headers resp)))))
-
-(deftest ^:integration t-request-without-url-set
- (run-server)
- ;; roundtrip with scheme as a keyword
- (let [resp (request {:uri "/redirect-to-get"
- :method :get})]
- (is (= 200 (:status resp)))
- (is (= "close" (get-in resp [:headers "connection"])))
- (is (= "get" (:body resp)))))
-
-(deftest ^:integration t-reusable-conn-mgrs
- (run-server)
- (let [cm (conn/make-reusable-conn-manager {:timeout 10 :insecure? false})
- resp1 (request {:uri "/redirect-to-get"
- :method :get
- :connection-manager cm})
- resp2 (request {:uri "/redirect-to-get"
- :method :get})]
- (is (= 200 (:status resp1) (:status resp2)))
- (is (nil? (get-in resp1 [:headers "connection"]))
- "connection should remain open")
- (is (= "close" (get-in resp2 [:headers "connection"]))
- "connection should be closed")
- (.shutdown cm)))
-
-(deftest test-url-encode-path
- (is (= (client/url-encode-illegal-characters "?foo bar+baz[]75")
- "?foo%20bar+baz%5B%5D75"))
- (is (= {:uri (str "/:@-._~!$&'()*+,="
- ";"
- ":@-._~!$&'()*+,"
- "="
- ":@-._~!$&'()*+,==")
- :query-string (str "/?:@-._~!$'()*+,;"
- "="
- "/?:@-._~!$'()*+,;==")}
- ;; This URL sucks, yes, it's actually a valid URL
- (select-keys (client/parse-url
- (str "http://example.com/:@-._~!$&'()*+,=;:@-._~!$&'()*+"
- ",=:@-._~!$&'()*+,==?/?:@-._~!$'()*+,;=/?:@-._~!$'("
- ")*+,;==#/?:@-._~!$&'()*+,;="))
- [:uri :query-string])))
- (let [all-chars (apply str (map char (range 256)))
- all-legal (client/url-encode-illegal-characters all-chars)]
- (is (= all-legal
- (client/url-encode-illegal-characters all-legal)))))
-
-(deftest t-coercion-methods
- (let [json-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
- auto-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
- edn-body (ByteArrayInputStream. (.getBytes "{:foo \"bar\"}"))
- transit-json-body (ByteArrayInputStream.
- (.getBytes "[\"^ \",\"~:foo\",\"bar\"]"))
- transit-msgpack-body (->> (map byte [-127 -91 126 58 102 111
- 111 -93 98 97 114])
- (byte-array 11)
- (ByteArrayInputStream.))
- www-form-urlencoded-body (ByteArrayInputStream. (.getBytes "foo=bar"))
- auto-www-form-urlencoded-body
- (ByteArrayInputStream. (.getBytes "foo=bar"))
- json-resp {:body json-body :status 200
- :headers {"content-type" "application/json"}}
- auto-resp {:body auto-body :status 200
- :headers {"content-type" "application/json"}}
- edn-resp {:body edn-body :status 200
- :headers {"content-type" "application/edn"}}
- transit-json-resp {:body transit-json-body :status 200
- :headers {"content-type" "application/transit-json"}}
- transit-msgpack-resp {:body transit-msgpack-body :status 200
- :headers {"content-type"
- "application/transit-msgpack"}}
- www-form-urlencoded-resp
- {:body www-form-urlencoded-body :status 200
- :headers {"content-type"
- "application/x-www-form-urlencoded"}}
- auto-www-form-urlencoded-resp
- {:body auto-www-form-urlencoded-body :status 200
- :headers {"content-type"
- "application/x-www-form-urlencoded"}}]
- (is (= {:foo "bar"}
- (:body (client/coerce-response-body {:as :json} json-resp))
- (:body (client/coerce-response-body {:as :clojure} edn-resp))
- (:body (client/coerce-response-body {:as :auto} auto-resp))
- (:body (client/coerce-response-body {:as :transit+json}
- transit-json-resp))
- (:body (client/coerce-response-body {:as :transit+msgpack}
- transit-msgpack-resp))
- (:body (client/coerce-response-body {:as :auto}
- auto-www-form-urlencoded-resp))
- (:body (client/coerce-response-body {:as :x-www-form-urlencoded}
- www-form-urlencoded-resp))))))
-
-(deftest ^:integration t-with-middleware
- (run-server)
- (is (:request-time (request {:uri "/get" :method :get})))
- (is (= client/*current-middleware* client/default-middleware))
- (client/with-middleware [client/wrap-url
- client/wrap-method
- #'client/wrap-request-timing]
- (is (:request-time (request {:uri "/get" :method :get})))
- (is (= client/*current-middleware* [client/wrap-url
- client/wrap-method
- #'client/wrap-request-timing])))
- (client/with-middleware (->> client/default-middleware
- (remove #{client/wrap-request-timing}))
- (is (not (:request-time (request {:uri "/get" :method :get}))))
- (is (not (contains? (set client/*current-middleware*)
- client/wrap-request-timing)))
- (is (contains? (set client/default-middleware)
- client/wrap-request-timing))))
-
-(deftest t-detect-charset-by-content-type
- (is (= "UTF-8" (client/detect-charset nil)))
- (is (= "UTF-8"(client/detect-charset "application/json")))
- (is (= "UTF-8"(client/detect-charset "text/html")))
- (is (= "GBK"(client/detect-charset "application/json; charset=GBK")))
- (is (= "ISO-8859-1" (client/detect-charset
- "application/json; charset=ISO-8859-1")))
- (is (= "ISO-8859-1" (client/detect-charset
- "application/json; charset = ISO-8859-1")))
- (is (= "GB2312" (client/detect-charset "text/html; Charset=GB2312"))))
-
-(deftest ^:integration multi-valued-query-params
- (run-server)
- (testing "default (repeating) multi-valued query params"
- (let [resp (request {:uri "/query-string"
- :method :get
- :query-params {:a [1 2 3]
- :b ["x" "y" "z"]}})
- query-string (-> resp :body form-decode-str)]
- (is (= 200 (:status resp)))
- (is (.contains query-string "a=1&a=2&a=3") query-string)
- (is (.contains query-string "b=x&b=y&b=z") query-string)))
-
- (testing "multi-valued query params in indexed-style"
- (let [resp (request {:uri "/query-string"
- :method :get
- :multi-param-style :indexed
- :query-params {:a [1 2 3]
- :b ["x" "y" "z"]}})
- query-string (-> resp :body form-decode-str)]
- (is (= 200 (:status resp)))
- (is (.contains query-string "a[0]=1&a[1]=2&a[2]=3") query-string)
- (is (.contains query-string "b[0]=x&b[1]=y&b[2]=z") query-string)))
-
- (testing "multi-valued query params in array-style"
- (let [resp (request {:uri "/query-string"
- :method :get
- :multi-param-style :array
- :query-params {:a [1 2 3]
- :b ["x" "y" "z"]}})
- query-string (-> resp :body form-decode-str)]
- (is (= 200 (:status resp)))
- (is (.contains query-string "a[]=1&a[]=2&a[]=3") query-string)
- (is (.contains query-string "b[]=x&b[]=y&b[]=z") query-string))))
diff --git a/test/clj_http/test/client_test.clj b/test/clj_http/test/client_test.clj
new file mode 100644
index 0000000..d2c2c59
--- /dev/null
+++ b/test/clj_http/test/client_test.clj
@@ -0,0 +1,1791 @@
+(ns clj-http.test.client-test
+ (:require [cheshire.core :as json]
+ [clj-http.client :as client]
+ [clj-http.conn-mgr :as conn]
+ [clj-http.test.core-test :refer [run-server]]
+ [clj-http.util :as util]
+ [clojure.java.io :refer [resource]]
+ [clojure.string :as str]
+ [clojure.test :refer :all]
+ [cognitect.transit :as transit]
+ [ring.middleware.nested-params :refer [parse-nested-keys]]
+ [ring.util.codec :refer [form-decode-str]]
+ [slingshot.slingshot :refer [try+]])
+ (:import java.io.ByteArrayInputStream
+ java.io.PipedInputStream
+ java.io.PipedOutputStream
+ java.net.UnknownHostException
+ org.apache.http.HttpEntity
+ org.apache.logging.log4j.LogManager))
+
+(defonce logger (LogManager/getLogger "clj-http.test.client-test"))
+
+(def base-req
+ {:scheme :http
+ :server-name "localhost"
+ :server-port 18080})
+
+(defn request
+ ([req]
+ (client/request (merge base-req req)))
+ ([req respond raise]
+ (client/request (merge base-req req) respond raise)))
+
+(defn parse-form-params [s]
+ (->> (str/split (form-decode-str s) #"&")
+ (map #(str/split % #"="))
+ (map #(vector
+ (map keyword (parse-nested-keys (first %)))
+ (second %)))
+ (reduce (fn [m [ks v]]
+ (assoc-in m ks v)) {})))
+
+(deftest ^:integration roundtrip
+ (run-server)
+ ;; roundtrip with scheme as a keyword
+ (let [resp (request {:uri "/get" :method :get})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= "get" (:body resp))))
+ ;; roundtrip with scheme as a string
+ (let [resp (request {:uri "/get" :method :get
+ :scheme "http"})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= "get" (:body resp))))
+ (let [params {:a "1" :b "2"}]
+ (doseq [[content-type read-fn]
+ [[nil (comp parse-form-params slurp)]
+ [:x-www-form-urlencoded (comp parse-form-params slurp)]
+ [:edn (comp read-string slurp)]
+ [:transit+json #(client/parse-transit % :json)]
+ [:transit+msgpack #(client/parse-transit % :msgpack)]]]
+ (let [resp (request {:uri "/post"
+ :as :stream
+ :method :post
+ :content-type content-type
+ :form-params params})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= params (read-fn (:body resp)))
+ (str "failed with content-type [" content-type "]"))))))
+
+(deftest ^:integration roundtrip-async
+ (run-server)
+ ;; roundtrip with scheme as a keyword
+ (let [resp (promise)
+ exception (promise)
+ _ (request {:uri "/get" :method :get
+ :async? true} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= "close" (get-in @resp [:headers "connection"])))
+ (is (= "get" (:body @resp)))
+ (is (not (realized? exception))))
+ ;; roundtrip with scheme as a string
+ (let [resp (promise)
+ exception (promise)
+ _ (request {:uri "/get" :method :get
+ :scheme "http"
+ :async? true} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= "close" (get-in @resp [:headers "connection"])))
+ (is (= "get" (:body @resp)))
+ (is (not (realized? exception))))
+
+ (let [params {:a "1" :b "2"}]
+ (doseq [[content-type read-fn]
+ [[nil (comp parse-form-params slurp)]
+ [:x-www-form-urlencoded (comp parse-form-params slurp)]
+ [:edn (comp read-string slurp)]
+ [:transit+json #(client/parse-transit % :json)]
+ [:transit+msgpack #(client/parse-transit % :msgpack)]]]
+ (let [resp (promise)
+ exception (promise)
+ _ (request {:uri "/post"
+ :as :stream
+ :method :post
+ :content-type content-type
+ :flatten-nested-keys []
+ :form-params params
+ :async? true} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= "close" (get-in @resp [:headers "connection"])))
+ (is (= params (read-fn (:body @resp))))
+ (is (not (realized? exception)))))))
+
+(def ^:dynamic *test-dynamic-var* nil)
+
+(deftest ^:integration async-preserves-dynamic-variable-bindings
+ (run-server)
+ (let [expected-var "cat"]
+ (binding [*test-dynamic-var* expected-var]
+ (let [test-fn (fn [uri success-p fail-p]
+ (request {:uri uri
+ :method :get
+ :scheme "http"
+ :async? true}
+ (fn [_]
+ (deliver success-p *test-dynamic-var*)
+ (deliver fail-p :success))
+ (fn [_]
+ (deliver success-p :fail)
+ (deliver fail-p *test-dynamic-var*))))]
+ (testing "dynamic variables on success responses"
+ (let [success-p (promise)
+ fail-p (promise)]
+ (test-fn "/get" success-p fail-p)
+ (is (= @success-p expected-var *test-dynamic-var*))
+ (is (= @fail-p :success)
+ "Verify that we went through the success path, not the failure")))
+
+ (testing "dynamic variables on fail responses"
+ (let [success-p (promise)
+ fail-p (promise)]
+ (test-fn "/json-bad" success-p fail-p)
+ (is (= @success-p :fail)
+ "Verify that we went through the failure path, not the success")
+ (is (= @fail-p expected-var *test-dynamic-var*))))))))
+
+(deftest ^:integration multipart-async
+ (run-server)
+ (let [resp (promise)
+ exception (promise)
+ _ (request {:uri "/post" :method :post
+ :async? true
+ :multipart [{:name "title" :content "some-file"}
+ {:name "Content/Type" :content "text/plain"}
+ {:name "file"
+ :content (clojure.java.io/file
+ "test-resources/m.txt")}]}
+ resp
+ exception
+ )]
+ (is (= 200 (:status @resp)))
+ (is (not (realized? exception)))
+ #_(when (realized? exception) (prn @exception)))
+
+ ;; Regression Testing https://github.com/dakrone/clj-http/issues/560
+ (testing "multipart uploads larger than 25kb"
+ (let [resp (promise)
+ exception (promise)
+ ;; assumption: file > 5kb
+ file (clojure.java.io/file "test-resources/big_array_json.json")
+
+ _ (request {:uri "/post" :method :post
+ :async? true
+ :multipart [{:name "part-1" :content file}
+ {:name "part-2" :content file}
+ {:name "part-3" :content file}
+ {:name "part-4" :content file}
+ {:name "part-5" :content file}]}
+ resp
+ exception)]
+ (is (= 200 (:status (deref resp 500 :failed))))
+ (is (not (realized? exception))))))
+
+(deftest ^:integration nil-input
+ (is (thrown-with-msg? Exception #"Host URL cannot be nil"
+ (client/get nil)))
+ (is (thrown-with-msg? Exception #"Host URL cannot be nil"
+ (client/post nil)))
+ (is (thrown-with-msg? Exception #"Host URL cannot be nil"
+ (client/put nil)))
+ (is (thrown-with-msg? Exception #"Host URL cannot be nil"
+ (client/delete nil))))
+
+(defn async-identity-client
+ "A async client which simply respond the request"
+ [request respond raise]
+ (respond request))
+
+(defn is-passed [middleware req]
+ (let [client (middleware identity)]
+ (is (= req (client req)))))
+
+(defn is-passed-async [middleware req]
+ (let [client (middleware async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (client req resp exception)]
+ (is (= req @resp))
+ (is (not (realized? exception)))))
+
+(defn is-applied [middleware req-in req-out]
+ (let [client (middleware identity)]
+ (is (= req-out (client req-in)))))
+
+(defn is-applied-async [middleware req-in req-out]
+ (let [client (middleware async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (client req-in resp exception)]
+ (is (= req-out @resp))
+ (is (not (realized? exception)))))
+
+(deftest redirect-on-get
+ (let [client (fn [req]
+ (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get})]
+ (is (= 200 (:status resp)))
+ (is (= :get (:request-method (:req resp))))
+ (is (= :http (:scheme (:req resp))))
+ (is (= ["http://example.com" "http://example.net/bat"]
+ (:trace-redirects resp)))
+ (is (= "/bat" (:uri (:req resp))))))
+
+(deftest redirect-on-get-async
+ (let [client (fn [req respond raise]
+ (respond (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req})))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= :get (:request-method (:req @resp))))
+ (is (= :http (:scheme (:req @resp))))
+ (is (= ["http://example.com" "http://example.net/bat"]
+ (:trace-redirects @resp)))
+ (is (= "/bat" (:uri (:req @resp))))
+ (is (not (realized? exception)))))
+
+(deftest relative-redirect-on-get
+ (let [client (fn [req]
+ (if (:redirects-count req)
+ {:status 200
+ :req req}
+ {:status 302
+ :headers {"location" "/bat"}}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get})]
+ (is (= 200 (:status resp)))
+ (is (= :get (:request-method (:req resp))))
+ (is (= :http (:scheme (:req resp))))
+ (is (= ["http://example.com" "http://example.com/bat"]
+ (:trace-redirects resp)))
+ (is (= "/bat" (:uri (:req resp))))))
+
+(deftest relative-redirect-on-get-async
+ (let [client (fn [req respond raise]
+ (respond (if (:redirects-count req)
+ {:status 200
+ :req req}
+ {:status 302
+ :headers {"location" "/bat"}})))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= :get (:request-method (:req @resp))))
+ (is (= :http (:scheme (:req @resp))))
+ (is (= ["http://example.com" "http://example.com/bat"]
+ (:trace-redirects @resp)))
+ (is (= "/bat" (:uri (:req @resp))))
+ (is (not (realized? exception)))))
+
+(deftest trace-redirects-using-uri
+ (let [client (fn [req] {:status 200 :req req})
+ r-client (-> client client/wrap-redirects)
+ resp (r-client {:scheme :http :server-name "example.com" :uri "/"
+ :request-method :get})]
+ (is (= 200 (:status resp)))
+ (is (= :get (:request-method (:req resp))))
+ (is (= :http (:scheme (:req resp))))
+ (is (= [] (:trace-redirects resp)))))
+
+(deftest trace-redirects-using-uri-async
+ (let [client (fn [req respond raise] (respond {:status 200 :req req}))
+ r-client (-> client client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:scheme :http :server-name "example.com" :uri "/"
+ :request-method :get} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= :get (:request-method (:req @resp))))
+ (is (= :http (:scheme (:req @resp))))
+ (is (= [] (:trace-redirects @resp)))
+ (is (not (realized? exception)))))
+
+(deftest redirect-without-location-header
+ (let [client (fn [req]
+ {:status 302 :body "no redirection here"})
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get})]
+ (is (= 302 (:status resp)))
+ (is (= ["http://example.com"] (:trace-redirects resp)))
+ (is (= "no redirection here" (:body resp)))))
+
+(deftest redirect-without-location-header-async
+ (let [client (fn [req respond raise]
+ (respond {:status 302 :body "no redirection here"}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get} resp exception)]
+ (is (= 302 (:status @resp)))
+ (is (= ["http://example.com"] (:trace-redirects @resp)))
+ (is (= "no redirection here" (:body @resp)))
+ (is (not (realized? exception)))))
+
+(deftest redirect-with-query-string
+ (let [client (fn [req]
+ (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat?x=y"}}
+ {:status 200
+ :req req}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get :query-params {:x "z"}})]
+ (is (= 200 (:status resp)))
+ (is (= :get (:request-method (:req resp))))
+ (is (= :http (:scheme (:req resp))))
+ (is (= ["http://example.com" "http://example.net/bat?x=y"]
+ (:trace-redirects resp)))
+ (is (= "/bat" (:uri (:req resp))))
+ (is (= "x=y" (:query-string (:req resp))))
+ (is (nil? (:query-params (:req resp))))))
+
+(deftest redirect-with-query-string-async
+ (let [client (fn [req respond raise]
+ (respond (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat?x=y"}}
+ {:status 200
+ :req req})))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get :query-params {:x "z"}}
+ resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= :get (:request-method (:req @resp))))
+ (is (= :http (:scheme (:req @resp))))
+ (is (= ["http://example.com" "http://example.net/bat?x=y"]
+ (:trace-redirects @resp)))
+ (is (= "/bat" (:uri (:req @resp))))
+ (is (= "x=y" (:query-string (:req @resp))))
+ (is (nil? (:query-params (:req @resp))))
+ (is (not (realized? exception)))))
+
+(deftest max-redirects
+ (let [client (fn [req]
+ (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get :max-redirects 0})]
+ (is (= 302 (:status resp)))
+ (is (= ["http://example.com"] (:trace-redirects resp)))
+ (is (= "http://example.net/bat" (get (:headers resp) "location")))))
+
+(deftest max-redirects-async
+ (let [client (fn [req respond raise]
+ (respond (if (= "example.com" (:server-name req))
+ {:status 302
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req})))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method :get :max-redirects 0}
+ resp exception)]
+ (is (= 302 (:status @resp)))
+ (is (= ["http://example.com"] (:trace-redirects @resp)))
+ (is (= "http://example.net/bat" (get (:headers @resp) "location")))
+ (is (not (realized? exception)))))
+
+(deftest redirect-303-to-get-on-any-method
+ (doseq [method [:get :head :post :delete :put :option]]
+ (let [client (fn [req]
+ (if (= "example.com" (:server-name req))
+ {:status 303
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req}))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method method})]
+ (is (= 200 (:status resp)))
+ (is (= :get (:request-method (:req resp))))
+ (is (= :http (:scheme (:req resp))))
+ (is (= ["http://example.com" "http://example.net/bat"]
+ (:trace-redirects resp)))
+ (is (= "/bat" (:uri (:req resp)))))))
+
+(deftest redirect-303-to-get-on-any-method-async
+ (doseq [method [:get :head :post :delete :put :option]]
+ (let [client (fn [req respond raise]
+ (respond (if (= "example.com" (:server-name req))
+ {:status 303
+ :headers {"location" "http://example.net/bat"}}
+ {:status 200
+ :req req})))
+ r-client (-> client client/wrap-url client/wrap-redirects)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:server-name "example.com" :url "http://example.com"
+ :request-method method}
+ resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= :get (:request-method (:req @resp))))
+ (is (= :http (:scheme (:req @resp))))
+ (is (= ["http://example.com" "http://example.net/bat"]
+ (:trace-redirects @resp)))
+ (is (= "/bat" (:uri (:req @resp))))
+ (is (not (realized? exception))))))
+
+(deftest pass-on-non-redirect
+ (let [client (fn [req] {:status 200 :body (:body req)})
+ r-client (client/wrap-redirects client)
+ resp (r-client {:body "ok" :url "http://example.com"})]
+ (is (= 200 (:status resp)))
+ (is (= ["http://example.com"] (:trace-redirects resp)))
+ (is (= "ok" (:body resp)))))
+
+(deftest pass-on-non-redirect-async
+ (let [client (fn [req respond raise]
+ (respond {:status 200 :body (:body req)}))
+ r-client (client/wrap-redirects client)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:body "ok" :url "http://example.com"} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= ["http://example.com"] (:trace-redirects @resp)))
+ (is (= "ok" (:body @resp)))
+ (is (not (realized? exception)))))
+
+(deftest pass-on-non-redirectable-methods
+ (doseq [method [:put :post :delete]
+ status [301 302 307 308]]
+ (let [client (fn [req] {:status status :body (:body req)
+ :headers {"location" "http://example.com/bat"}})
+ r-client (client/wrap-redirects client)
+ resp (r-client {:body "ok" :url "http://example.com"
+ :request-method method})]
+ (is (= status (:status resp)))
+ (is (= ["http://example.com"] (:trace-redirects resp)))
+ (is (= {"location" "http://example.com/bat"} (:headers resp)))
+ (is (= "ok" (:body resp))))))
+
+(deftest pass-on-non-redirectable-methods-async
+ (doseq [method [:put :post :delete]
+ status [301 302 307 308]]
+ (let [client (fn [req respond raise]
+ (respond {:status status :body (:body req)
+ :headers {"location" "http://example.com/bat"}}))
+ r-client (client/wrap-redirects client)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:body "ok" :url "http://example.com"
+ :request-method method} resp exception)]
+ (is (= status (:status @resp)))
+ (is (= ["http://example.com"] (:trace-redirects @resp)))
+ (is (= {"location" "http://example.com/bat"} (:headers @resp)))
+ (is (= "ok" (:body @resp)))
+ (is (not (realized? exception))))))
+
+(deftest force-redirects-on-non-redirectable-methods
+ (doseq [method [:put :post :delete]
+ [status expected-method] [[301 :get] [302 :get] [307 method]]]
+ (let [client (fn [{:keys [trace-redirects body] :as req}]
+ (if trace-redirects
+ {:status 200 :body body :trace-redirects trace-redirects
+ :req req}
+ {:status status :body body :req req
+ :headers {"location" "http://example.com/bat"}}))
+ r-client (client/wrap-redirects client)
+ resp (r-client {:body "ok" :url "http://example.com"
+ :request-method method
+ :force-redirects true})]
+ (is (= 200 (:status resp)))
+ (is (= ["http://example.com" "http://example.com/bat"]
+ (:trace-redirects resp)))
+ (is (= "ok" (:body resp)))
+ (is (= expected-method (:request-method (:req resp)))))))
+
+(deftest force-redirects-on-non-redirectable-methods-async
+ (doseq [method [:put :post :delete]
+ [status expected-method] [[301 :get] [302 :get] [307 method]]]
+ (let [client (fn [{:keys [trace-redirects body] :as req} respond raise]
+ (respond (if trace-redirects
+ {:status 200 :body body
+ :trace-redirects trace-redirects
+ :req req}
+ {:status status :body body :req req
+ :headers {"location"
+ "http://example.com/bat"}})))
+ r-client (client/wrap-redirects client)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:body "ok" :url "http://example.com"
+ :request-method method
+ :force-redirects true} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (= ["http://example.com" "http://example.com/bat"]
+ (:trace-redirects @resp)))
+ (is (= "ok" (:body @resp)))
+ (is (= expected-method (:request-method (:req @resp))))
+ (is (not (realized? exception))))))
+
+(deftest pass-on-follow-redirects-false
+ (let [client (fn [req] {:status 302 :body (:body req)})
+ r-client (client/wrap-redirects client)
+ resp (r-client {:body "ok" :follow-redirects false})]
+ (is (= 302 (:status resp)))
+ (is (= "ok" (:body resp)))
+ (is (nil? (:trace-redirects resp)))))
+
+(deftest pass-on-follow-redirects-false-async
+ (let [client (fn [req respond raise]
+ (respond {:status 302 :body (:body req)}))
+ r-client (client/wrap-redirects client)
+ resp (promise)
+ exception (promise)
+ _ (r-client {:body "ok" :follow-redirects false} resp exception)]
+ (is (= 302 (:status @resp)))
+ (is (= "ok" (:body @resp)))
+ (is (nil? (:trace-redirects @resp)))
+ (is (not (realized? exception)))))
+
+(deftest throw-on-exceptional
+ (let [client (fn [req] {:status 500})
+ e-client (client/wrap-exceptions client)]
+ (is (thrown-with-msg? Exception #"500"
+ (e-client {}))))
+ (let [client (fn [req] {:status 500 :body "foo"})
+ e-client (client/wrap-exceptions client)]
+ (is (thrown-with-msg? Exception #":body"
+ (e-client {:throw-entire-message? true})))))
+
+(deftest throw-on-custom-exceptional
+ (let [client (fn [req] {:status 201})
+ e-client (client/wrap-exceptions client)]
+ (is (thrown-with-msg? Exception #"201"
+ (e-client {:unexceptional-status #{200}})))))
+
+(deftest throw-type-field
+ (let [client (fn [req] {:status 500})
+ e-client (client/wrap-exceptions client)]
+ (try+
+ (e-client {})
+ (catch [:type :clj-http.client/unexceptional-status] _
+ (is true))
+ (catch Object _
+ (is false ":type selector was not caught.")))))
+
+(deftest throw-on-exceptional-async
+ (let [client (fn [req respond raise]
+ (try
+ (respond {:status 500})
+ (catch Throwable ex
+ (raise ex))))
+ e-client (client/wrap-exceptions client)
+ resp (promise)
+ exception (promise)
+ _ (e-client {} resp exception)]
+ (is (thrown-with-msg? Exception #"500"
+ (throw @exception))))
+ (let [client (fn [req respond raise]
+ (try
+ (respond {:status 500 :body "foo"})
+ (catch Throwable ex
+ (raise ex))))
+ e-client (client/wrap-exceptions client)
+ resp (promise)
+ exception (promise)
+ _ (e-client {:throw-entire-message? true} resp exception)]
+ (is (thrown-with-msg? Exception #":body"
+ (throw @exception)))))
+
+(deftest pass-on-non-exceptional
+ (let [client (fn [req] {:status 200})
+ e-client (client/wrap-exceptions client)
+ resp (e-client {})]
+ (is (= 200 (:status resp)))))
+
+(deftest pass-on-custom-non-exceptional
+ (let [client (fn [req] {:status 500})
+ e-client (client/wrap-exceptions client)
+ resp (e-client {:unexceptional-status #{200 500}})]
+ (is (= 500 (:status resp)))))
+
+(deftest pass-on-non-exceptional-async
+ (let [client (fn [req respond raise] (respond {:status 200}))
+ e-client (client/wrap-exceptions client)
+ resp (promise)
+ exception (promise)
+ _ (e-client {} resp exception)]
+ (is (= 200 (:status @resp)))
+ (is (not (realized? exception)))))
+
+(deftest pass-on-exceptional-when-surpressed
+ (let [client (fn [req] {:status 500})
+ e-client (client/wrap-exceptions client)
+ resp (e-client {:throw-exceptions false})]
+ (is (= 500 (:status resp)))))
+
+(deftest pass-on-exceptional-when-surpressed-async
+ (let [client (fn [req respond raise] (respond {:status 500}))
+ e-client (client/wrap-exceptions client)
+ resp (promise)
+ exception (promise)
+ _ (e-client {:throw-exceptions false} resp exception)]
+ (is (= 500 (:status @resp)))
+ (is (not (realized? exception)))))
+
+(deftest apply-on-compressed
+ (let [client (fn [req]
+ (is (= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ {:body (util/gzip (util/utf8-bytes "foofoofoo"))
+ :headers {"content-encoding" "gzip"}})
+ c-client (client/wrap-decompression client)
+ resp (c-client {})]
+ (is (= "foofoofoo" (util/utf8-string (:body resp))))
+ (is (= "gzip" (:orig-content-encoding resp)))
+ (is (= nil (get-in resp [:headers "content-encoding"])))))
+
+(deftest apply-on-compressed-async
+ (let [client (fn [req respond raise]
+ (is (= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ (respond {:body (util/gzip (util/utf8-bytes "foofoofoo"))
+ :headers {"content-encoding" "gzip"}}))
+ c-client (client/wrap-decompression client)
+ resp (promise)
+ exception (promise)
+ _ (c-client {} resp exception)]
+ (is (= "foofoofoo" (util/utf8-string (:body @resp))))
+ (is (= "gzip" (:orig-content-encoding @resp)))
+ (is (= nil (get-in @resp [:headers "content-encoding"])))))
+
+(deftest apply-on-deflated
+ (let [client (fn [req]
+ (is (= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ {:body (util/deflate (util/utf8-bytes "barbarbar"))
+ :headers {"content-encoding" "deflate"}})
+ c-client (client/wrap-decompression client)
+ resp (c-client {})]
+ (is (= "barbarbar" (-> resp :body util/force-byte-array util/utf8-string))
+ "string correctly inflated")
+ (is (= "deflate" (:orig-content-encoding resp)))
+ (is (= nil (get-in resp [:headers "content-encoding"])))))
+
+(deftest apply-on-deflated-async
+ (let [client (fn [req respond raise]
+ (is (= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ (respond {:body (util/deflate (util/utf8-bytes "barbarbar"))
+ :headers {"content-encoding" "deflate"}}))
+ c-client (client/wrap-decompression client)
+ resp (promise)
+ exception (promise)
+ _ (c-client {} resp exception)]
+ (is (= "barbarbar" (-> @resp :body util/force-byte-array util/utf8-string))
+ "string correctly inflated")
+ (is (= "deflate" (:orig-content-encoding @resp)))
+ (is (= nil (get-in @resp [:headers "content-encoding"])))))
+
+(deftest t-disabled-body-decompression
+ (let [client (fn [req]
+ (is (not= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ {:body (util/deflate (util/utf8-bytes "barbarbar"))
+ :headers {"content-encoding" "deflate"}})
+ c-client (client/wrap-decompression client)
+ resp (c-client {:decompress-body false})]
+ (is (= (slurp (util/inflate (util/deflate (util/utf8-bytes "barbarbar"))))
+ (slurp (util/inflate (-> resp :body util/force-byte-array))))
+ "string not inflated")
+ (is (= nil (:orig-content-encoding resp)))
+ (is (= "deflate" (get-in resp [:headers "content-encoding"])))))
+
+(deftest t-weird-non-known-compression
+ (let [client (fn [req]
+ (is (= "gzip, deflate"
+ (get-in req [:headers "accept-encoding"])))
+ {:body (util/utf8-bytes "foofoofoo")
+ :headers {"content-encoding" "pig-latin"}})
+ c-client (client/wrap-decompression client)
+ resp (c-client {})]
+ (is (= "foofoofoo" (util/utf8-string (:body resp))))
+ (is (= "pig-latin" (:orig-content-encoding resp)))
+ (is (= "pig-latin" (get-in resp [:headers "content-encoding"])))))
+
+(deftest pass-on-non-compressed
+ (let [c-client (client/wrap-decompression (fn [req] {:body "foo"}))
+ resp (c-client {:uri "/foo"})]
+ (is (= "foo" (:body resp)))))
+
+(deftest apply-on-accept
+ (is-applied client/wrap-accept
+ {:accept :json}
+ {:headers {"accept" "application/json"}})
+ (is-applied client/wrap-accept
+ {:accept :transit+json}
+ {:headers {"accept" "application/transit+json"}})
+ (is-applied client/wrap-accept
+ {:accept :transit+msgpack}
+ {:headers {"accept" "application/transit+msgpack"}}))
+
+(deftest apply-on-accept-async
+ (is-applied-async client/wrap-accept
+ {:accept :json}
+ {:headers {"accept" "application/json"}})
+ (is-applied-async client/wrap-accept
+ {:accept :transit+json}
+ {:headers {"accept" "application/transit+json"}})
+ (is-applied-async client/wrap-accept
+ {:accept :transit+msgpack}
+ {:headers {"accept" "application/transit+msgpack"}}))
+
+(deftest pass-on-no-accept
+ (is-passed client/wrap-accept
+ {:uri "/foo"}))
+
+(deftest pass-on-no-accept-async
+ (is-passed-async client/wrap-accept
+ {:uri "/foo"}))
+
+(deftest apply-on-accept-encoding
+ (is-applied client/wrap-accept-encoding
+ {:accept-encoding [:identity :gzip]}
+ {:headers {"accept-encoding" "identity, gzip"}}))
+
+(deftest apply-custom-accept-encoding
+ (testing "no custom encodings to accept"
+ (is-applied (comp client/wrap-accept-encoding
+ client/wrap-decompression)
+ {}
+ {:headers {"accept-encoding" "gzip, deflate"}
+ :orig-content-encoding nil}))
+ (testing "accept some custom encodings, but still include gzip and deflate"
+ (is-applied (comp client/wrap-accept-encoding
+ client/wrap-decompression)
+ {:accept-encoding [:foo :bar]}
+ {:headers {"accept-encoding" "foo, bar, gzip, deflate"}
+ :orig-content-encoding nil}))
+ (testing "accept some custom encodings, but exclude gzip and deflate"
+ (is-applied (comp client/wrap-accept-encoding
+ client/wrap-decompression)
+ {:accept-encoding [:foo :bar] :decompress-body false}
+ {:headers {"accept-encoding" "foo, bar"}
+ :decompress-body false})))
+
+(deftest pass-on-no-accept-encoding
+ (is-passed client/wrap-accept-encoding
+ {:uri "/foo"}))
+
+(deftest apply-on-output-coercion
+ (let [client (fn [req] {:body (util/utf8-bytes "foo")})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo"})]
+ (is (= "foo" (:body resp)))))
+
+(deftest apply-on-output-coercion-async
+ (let [client (fn [req respond raise]
+ (respond {:body (util/utf8-bytes "foo")}))
+ o-client (client/wrap-output-coercion client)
+ resp (promise)
+ exception (promise)
+ _ (o-client {:uri "/foo"} resp exception)]
+ (is (= "foo" (:body @resp)))
+ (is (not (realized? exception)))))
+
+(deftest pass-on-no-output-coercion
+ (let [client (fn [req] {:body nil})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo"})]
+ (is (nil? (:body resp))))
+ (let [the-stream (ByteArrayInputStream. (byte-array []))
+ client (fn [req] {:body the-stream})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo" :as :stream})]
+ (is (= the-stream (:body resp))))
+ (let [client (fn [req] {:body :thebytes})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo" :as :byte-array})]
+ (is (= :thebytes (:body resp)))))
+
+(deftest pass-on-no-output-coercion-async
+ (let [client (fn [req] {:body nil})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo"})]
+ (is (nil? (:body resp))))
+ (let [the-stream (ByteArrayInputStream. (byte-array []))
+ client (fn [req] {:body the-stream})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo" :as :stream})]
+ (is (= the-stream (:body resp))))
+ (let [client (fn [req] {:body :thebytes})
+ o-client (client/wrap-output-coercion client)
+ resp (o-client {:uri "/foo" :as :byte-array})]
+ (is (= :thebytes (:body resp)))))
+
+(deftest apply-on-input-coercion
+ (let [i-client (client/wrap-input-coercion identity)
+ resp (i-client {:body "foo"})
+ resp2 (i-client {:body "foo2" :body-encoding "ASCII"})
+ data (slurp (.getContent ^HttpEntity (:body resp)))
+ data2 (slurp (.getContent ^HttpEntity (:body resp2)))]
+ (is (= "UTF-8" (:character-encoding resp)))
+ (is (= "foo" data))
+ (is (= "ASCII" (:character-encoding resp2)))
+ (is (= "foo2" data2))))
+
+(deftest apply-on-input-coercion-async
+ (let [i-client (client/wrap-input-coercion (fn [request respond raise]
+ (respond request)))
+ resp (promise)
+ _ (i-client {:body "foo"} resp nil)
+ resp2 (promise)
+ _ (i-client {:body "foo2" :body-encoding "ASCII"} resp2 nil)
+ data (slurp (.getContent ^HttpEntity (:body @resp)))
+ data2 (slurp (.getContent ^HttpEntity (:body @resp2)))]
+ (is (= "UTF-8" (:character-encoding @resp)))
+ (is (= "foo" data))
+ (is (= "ASCII" (:character-encoding @resp2)))
+ (is (= "foo2" data2))))
+
+(deftest pass-on-no-input-coercion
+ (is-passed client/wrap-input-coercion
+ {:body nil}))
+
+(deftest pass-on-no-input-coercion
+ (is-passed-async client/wrap-input-coercion
+ {:body nil}))
+
+(deftest no-length-for-input-stream
+ (let [i-client (client/wrap-input-coercion identity)
+ resp1 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))})
+ resp2 (i-client {:body (ByteArrayInputStream. (util/utf8-bytes "foo"))
+ :length 3})
+ ^HttpEntity body1 (:body resp1)
+ ^HttpEntity body2 (:body resp2)]
+ (is (= -1 (.getContentLength body1)))
+ (is (= 3 (.getContentLength body2)))))
+
+(deftest apply-on-content-type
+ (is-applied client/wrap-content-type
+ {:content-type :json}
+ {:headers {"content-type" "application/json"}
+ :content-type :json})
+ (is-applied client/wrap-content-type
+ {:content-type :json :character-encoding "UTF-8"}
+ {:headers {"content-type" "application/json; charset=UTF-8"}
+ :content-type :json :character-encoding "UTF-8"})
+ (is-applied client/wrap-content-type
+ {:content-type :transit+json}
+ {:headers {"content-type" "application/transit+json"}
+ :content-type :transit+json})
+ (is-applied client/wrap-content-type
+ {:content-type :transit+msgpack}
+ {:headers {"content-type" "application/transit+msgpack"}
+ :content-type :transit+msgpack}))
+
+(deftest apply-on-content-type-async
+ (is-applied-async client/wrap-content-type
+ {:content-type :json}
+ {:headers {"content-type" "application/json"}
+ :content-type :json})
+ (is-applied-async client/wrap-content-type
+ {:content-type :json :character-encoding "UTF-8"}
+ {:headers {"content-type" "application/json; charset=UTF-8"}
+ :content-type :json :character-encoding "UTF-8"})
+ (is-applied-async client/wrap-content-type
+ {:content-type :transit+json}
+ {:headers {"content-type" "application/transit+json"}
+ :content-type :transit+json})
+ (is-applied-async client/wrap-content-type
+ {:content-type :transit+msgpack}
+ {:headers {"content-type" "application/transit+msgpack"}
+ :content-type :transit+msgpack}))
+
+(deftest pass-on-no-content-type
+ (is-passed client/wrap-content-type
+ {:uri "/foo"}))
+
+(deftest apply-on-query-params
+ (is-applied client/wrap-query-params
+ {:query-params {"foo" "bar" "dir" "<<"}}
+ {:query-string "foo=bar&dir=%3C%3C"})
+ (is-applied client/wrap-query-params
+ {:query-string "foo=1"
+ :query-params {"foo" ["2" "3"]}}
+ {:query-string "foo=1&foo=2&foo=3"}))
+
+(deftest apply-on-query-params-async
+ (is-applied-async client/wrap-query-params
+ {:query-params {"foo" "bar" "dir" "<<"}}
+ {:query-string "foo=bar&dir=%3C%3C"})
+ (is-applied-async client/wrap-query-params
+ {:query-string "foo=1"
+ :query-params {"foo" ["2" "3"]}}
+ {:query-string "foo=1&foo=2&foo=3"}))
+
+(deftest pass-on-no-query-params
+ (is-passed client/wrap-query-params
+ {:uri "/foo"}))
+
+(deftest apply-on-basic-auth
+ (is-applied client/wrap-basic-auth
+ {:basic-auth ["Aladdin" "open sesame"]}
+ {:headers {"authorization"
+ "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="}}))
+
+(deftest apply-on-basic-auth-async
+ (is-applied-async client/wrap-basic-auth
+ {:basic-auth ["Aladdin" "open sesame"]}
+ {:headers {"authorization"
+ "Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ=="}}))
+
+(deftest pass-on-no-basic-auth
+ (is-passed client/wrap-basic-auth
+ {:uri "/foo"}))
+
+(deftest apply-on-oauth
+ (is-applied client/wrap-oauth
+ {:oauth-token "my-token"}
+ {:headers {"authorization"
+ "Bearer my-token"}}))
+
+(deftest apply-on-oauth-async
+ (is-applied-async client/wrap-oauth
+ {:oauth-token "my-token"}
+ {:headers {"authorization"
+ "Bearer my-token"}}))
+
+(deftest pass-on-no-oauth
+ (is-passed client/wrap-oauth
+ {:uri "/foo"}))
+
+(deftest apply-on-method
+ (let [m-client (client/wrap-method identity)
+ echo (m-client {:key :val :method :post})]
+ (is (= :val (:key echo)))
+ (is (= :post (:request-method echo)))
+ (is (not (:method echo)))))
+
+(deftest apply-on-method-async
+ (let [m-client (client/wrap-method async-identity-client)
+ echo (promise)
+ exception (promise)
+ _ (m-client {:key :val :method :post} echo exception)]
+ (is (= :val (:key @echo)))
+ (is (= :post (:request-method @echo)))
+ (is (not (:method @echo)))))
+
+(deftest pass-on-no-method
+ (let [m-client (client/wrap-method identity)
+ echo (m-client {:key :val})]
+ (is (= :val (:key echo)))
+ (is (not (:request-method echo)))))
+
+(deftest apply-on-url
+ (let [u-client (client/wrap-url identity)
+ resp (u-client {:url "http://google.com:8080/baz foo?bar=bat bit?"})]
+ (is (= :http (:scheme resp)))
+ (is (= "google.com" (:server-name resp)))
+ (is (= 8080 (:server-port resp)))
+ (is (= "/baz%20foo" (:uri resp)))
+ (is (= "bar=bat%20bit?" (:query-string resp)))))
+
+(deftest apply-on-url
+ (let [u-client (client/wrap-url async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (u-client {:url "http://google.com:8080/baz foo?bar=bat bit?"}
+ resp exception)]
+ (is (= :http (:scheme @resp)))
+ (is (= "google.com" (:server-name @resp)))
+ (is (= 8080 (:server-port @resp)))
+ (is (= "/baz%20foo" (:uri @resp)))
+ (is (= "bar=bat%20bit?" (:query-string @resp)))
+ (is (not (realized? exception)))))
+
+(deftest pass-on-no-url
+ (let [u-client (client/wrap-url identity)
+ resp (u-client {:uri "/foo"})]
+ (is (= "/foo" (:uri resp)))))
+
+(deftest provide-default-port
+ (is (= nil (-> "http://example.com/" client/parse-url :server-port)))
+ (is (= 8080 (-> "http://example.com:8080/" client/parse-url :server-port)))
+ (is (= nil (-> "https://example.com/" client/parse-url :server-port)))
+ (is (= 8443 (-> "https://example.com:8443/" client/parse-url :server-port)))
+ (is (= "https://example.com:8443/"
+ (-> "https://example.com:8443/" client/parse-url :url))))
+
+(deftest decode-credentials-from-url
+ (is (= "fred's diner:fred's password"
+ (-> "http://fred%27s%20diner:fred%27s%20password@example.com/foo"
+ client/parse-url
+ :user-info))))
+
+(deftest unparse-url
+ (is (= "http://fred's diner:fred's password@example.com/foo"
+ (-> "http://fred%27s%20diner:fred%27s%20password@example.com/foo"
+ client/parse-url client/unparse-url)))
+ (is (= "https://foo:bar@example.org:8080"
+ (-> "https://foo:bar@example.org:8080"
+ client/parse-url client/unparse-url)))
+ (is (= "ftp://example.org?foo"
+ (-> "ftp://example.org?foo"
+ client/parse-url client/unparse-url))))
+
+(defrecord Point [x y])
+
+(def write-point
+ "Write a point in Transit format."
+ (transit/write-handler
+ (constantly "point")
+ (fn [point] [(:x point) (:y point)])
+ (constantly nil)))
+
+(def read-point
+ "Read a point in Transit format."
+ (transit/read-handler
+ (fn [[x y]]
+ (->Point x y))))
+
+(def transit-opts
+ "Transit read and write options."
+ {:encode {:handlers {Point write-point}}
+ :decode {:handlers {"point" read-point}}})
+
+(def transit-opts-deprecated
+ "Deprecated Transit read and write options."
+ {:handlers {Point write-point "point" read-point}})
+
+(deftest apply-on-form-params
+ (testing "With form params"
+ (let [param-client (client/wrap-form-params identity)
+ resp (param-client {:request-method :post
+ :form-params (sorted-map :param1 "value1"
+ :param2 "value2")})]
+ (is (= "param1=value1¶m2=value2" (:body resp)))
+ (is (= "application/x-www-form-urlencoded" (:content-type resp)))
+ (is (not (contains? resp :form-params))))
+ (let [param-client (client/wrap-form-params identity)
+ resp (param-client {:request-method :put
+ :form-params (sorted-map :param1 "value1"
+ :param2 "value2")})]
+ (is (= "param1=value1¶m2=value2" (:body resp)))
+ (is (= "application/x-www-form-urlencoded" (:content-type resp)))
+ (is (not (contains? resp :form-params)))))
+
+ (testing "With json form params"
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 "value2"}
+ resp (param-client {:request-method :post
+ :content-type :json
+ :form-params params})]
+ (is (= (json/encode params) (:body resp)))
+ (is (= "application/json" (:content-type resp)))
+ (is (not (contains? resp :form-params))))
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 "value2"}
+ resp (param-client {:request-method :put
+ :content-type :json
+ :form-params params})]
+ (is (= (json/encode params) (:body resp)))
+ (is (= "application/json" (:content-type resp)))
+ (is (not (contains? resp :form-params))))
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 "value2"}
+ resp (param-client {:request-method :patch
+ :content-type :json
+ :form-params params})]
+ (is (= (json/encode params) (:body resp)))
+ (is (= "application/json" (:content-type resp)))
+ (is (not (contains? resp :form-params))))
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 (java.util.Date. (long 0))}
+ resp (param-client {:request-method :put
+ :content-type :json
+ :form-params params
+ :json-opts {:date-format "yyyy-MM-dd"}})]
+ (is (= (json/encode params {:date-format "yyyy-MM-dd"}) (:body resp)))
+ (is (= "application/json" (:content-type resp)))
+ (is (not (contains? resp :form-params)))))
+
+ (testing "With EDN form params"
+ (doseq [method [:post :put :patch]]
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 (Point. 1 2)}
+ resp (param-client {:request-method method
+ :content-type :edn
+ :form-params params})]
+ (is (= (pr-str params) (:body resp)))
+ (is (= "application/edn" (:content-type resp)))
+ (is (not (contains? resp :form-params))))))
+
+ (testing "With Transit/JSON form params"
+ (doseq [method [:post :put :patch]]
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 (Point. 1 2)}
+ resp (param-client {:request-method method
+ :content-type :transit+json
+ :form-params params
+ :transit-opts transit-opts})]
+ (is (= params (client/parse-transit
+ (ByteArrayInputStream. (:body resp))
+ :json transit-opts)))
+ (is (= "application/transit+json" (:content-type resp)))
+ (is (not (contains? resp :form-params))))))
+
+ (testing "With Transit/MessagePack form params"
+ (doseq [method [:post :put :patch]]
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 "value2"}
+ resp (param-client {:request-method method
+ :content-type :transit+msgpack
+ :form-params params
+ :transit-opts transit-opts})]
+ (is (= params (client/parse-transit
+ (ByteArrayInputStream. (:body resp))
+ :msgpack transit-opts)))
+ (is (= "application/transit+msgpack" (:content-type resp)))
+ (is (not (contains? resp :form-params))))))
+
+ (testing "With Transit/JSON form params and deprecated options"
+ (let [param-client (client/wrap-form-params identity)
+ params {:param1 "value1" :param2 (Point. 1 2)}
+ resp (param-client {:request-method :post
+ :content-type :transit+json
+ :form-params params
+ :transit-opts transit-opts-deprecated})]
+ (is (= params (client/parse-transit
+ (ByteArrayInputStream. (:body resp))
+ :json transit-opts-deprecated)))
+ (is (= "application/transit+json" (:content-type resp)))
+ (is (not (contains? resp :form-params)))))
+
+ (testing "Ensure it does not affect GET requests"
+ (let [param-client (client/wrap-form-params identity)
+ resp (param-client {:request-method :get
+ :body "untouched"
+ :form-params {:param1 "value1"
+ :param2 "value2"}})]
+ (is (= "untouched" (:body resp)))
+ (is (not (contains? resp :content-type)))))
+
+ (testing "with no form params"
+ (let [param-client (client/wrap-form-params identity)
+ resp (param-client {:body "untouched"})]
+ (is (= "untouched" (:body resp)))
+ (is (not (contains? resp :content-type))))))
+
+(deftest apply-on-form-params-async
+ (testing "With form params"
+ (let [param-client (client/wrap-form-params async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (param-client {:request-method :post
+ :form-params (sorted-map :param1 "value1"
+ :param2 "value2")}
+ resp exception)]
+ (is (= "param1=value1¶m2=value2" (:body @resp)))
+ (is (= "application/x-www-form-urlencoded" (:content-type @resp)))
+ (is (not (contains? @resp :form-params)))
+ (is (not (realized? exception))))
+ (let [param-client (client/wrap-form-params async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (param-client {:request-method :put
+ :form-params (sorted-map :param1 "value1"
+ :param2 "value2")}
+ resp exception)]
+ (is (= "param1=value1¶m2=value2" (:body @resp)))
+ (is (= "application/x-www-form-urlencoded" (:content-type @resp)))
+ (is (not (contains? @resp :form-params)))
+ (is (not (realized? exception)))))
+
+ (testing "Ensure it does not affect GET requests"
+ (let [param-client (client/wrap-form-params async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (param-client {:request-method :get
+ :body "untouched"
+ :form-params {:param1 "value1"
+ :param2 "value2"}}
+ resp exception)]
+ (is (= "untouched" (:body @resp)))
+ (is (not (contains? @resp :content-type)))
+ (is (not (realized? exception)))))
+
+ (testing "with no form params"
+ (let [param-client (client/wrap-form-params async-identity-client)
+ resp (promise)
+ exception (promise)
+ _ (param-client {:body "untouched"} resp exception)]
+ (is (= "untouched" (:body @resp)))
+ (is (not (contains? @resp :content-type)))
+ (is (not (realized? exception))))))
+
+(deftest apply-on-nested-params
+ (testing "nested parameter maps"
+ (is-applied (comp client/wrap-form-params
+ client/wrap-nested-params)
+ {:query-params {"foo" "bar"}
+ :form-params {"foo" "bar"}
+ :flatten-nested-keys [:query-params :form-params]}
+ {:query-params {"foo" "bar"}
+ :form-params {"foo" "bar"}
+ :flatten-nested-keys [:query-params :form-params]})
+ (is-applied (comp client/wrap-form-params
+ client/wrap-nested-params)
+ {:query-params {"x" {"y" "z"}}
+ :form-params {"x" {"y" "z"}}
+ :flatten-nested-keys [:query-params]}
+ {:query-params {"x[y]" "z"}
+ :form-params {"x" {"y" "z"}}
+ :flatten-nested-keys [:query-params]})
+ (is-applied (comp client/wrap-form-params
+ client/wrap-nested-params)
+ {:query-params {"a" {"b" {"c" "d"}}}
+ :form-params {"a" {"b" {"c" "d"}}}
+ :flatten-nested-keys [:form-params]}
+ {:query-params {"a" {"b" {"c" "d"}}}
+ :form-params {"a[b][c]" "d"}
+ :flatten-nested-keys [:form-params]})
+ (is-applied (comp client/wrap-form-params
+ client/wrap-nested-params)
+ {:query-params {"a" {"b" {"c" "d"}}}
+ :form-params {"a" {"b" {"c" "d"}}}
+ :flatten-nested-keys [:query-params :form-params]}
+ {:query-params {"a[b][c]" "d"}
+ :form-params {"a[b][c]" "d"}
+ :flatten-nested-keys [:query-params :form-params]}))
+
+ (testing "not creating empty param maps"
+ (is-applied client/wrap-query-params {} {})))
+
+(deftest t-ignore-unknown-host
+ (is (thrown? UnknownHostException (client/get "http://example.invalid")))
+ (is (nil? (client/get "http://example.invalid"
+ {:ignore-unknown-host? true}))))
+
+(deftest t-ignore-unknown-host-async
+ (let [resp (promise) exception (promise)]
+ (client/get "http://example.invalid"
+ {:async? true} resp exception)
+ (is (thrown? UnknownHostException (throw @exception))))
+ (let [resp (promise) exception (promise)]
+ (client/get "http://example.invalid"
+ {:ignore-unknown-host? true
+ :async? true} resp exception)
+ (is (nil? @resp))))
+
+(deftest test-status-predicates
+ (testing "2xx statuses"
+ (doseq [s (range 200 299)]
+ (is (client/success? {:status s}))
+ (is (not (client/redirect? {:status s})))
+ (is (not (client/client-error? {:status s})))
+ (is (not (client/server-error? {:status s})))))
+ (testing "3xx statuses"
+ (doseq [s (range 300 399)]
+ (is (not (client/success? {:status s})))
+ (is (client/redirect? {:status s}))
+ (is (not (client/client-error? {:status s})))
+ (is (not (client/server-error? {:status s})))))
+ (testing "4xx statuses"
+ (doseq [s (range 400 499)]
+ (is (not (client/success? {:status s})))
+ (is (not (client/redirect? {:status s})))
+ (is (client/client-error? {:status s}))
+ (is (not (client/server-error? {:status s})))))
+ (testing "5xx statuses"
+ (doseq [s (range 500 599)]
+ (is (not (client/success? {:status s})))
+ (is (not (client/redirect? {:status s})))
+ (is (not (client/client-error? {:status s})))
+ (is (client/server-error? {:status s}))))
+ (testing "409 Conflict"
+ (is (client/conflict? {:status 409}))
+ (is (not (client/conflict? {:status 201})))
+ (is (not (client/conflict? {:status 404})))))
+
+(deftest test-wrap-lower-case-headers
+ (is (= {:status 404} ((client/wrap-lower-case-headers
+ (fn [r] r)) {:status 404})))
+ (is (= {:headers {"content-type" "application/json"}}
+ ((client/wrap-lower-case-headers
+ #(do (is (= {:headers {"accept" "application/json"}} %1))
+ {:headers {"Content-Type" "application/json"}}))
+ {:headers {"Accept" "application/json"}}))))
+
+(deftest t-request-timing
+ (is (pos? (:request-time ((client/wrap-request-timing
+ (fn [r] (Thread/sleep 15) r)) {})))))
+
+(deftest t-wrap-additional-header-parsing
+ (let [^String text (slurp (resource "header-test.html"))
+ client (fn [req] {:body (.getBytes text)})
+ new-client (client/wrap-additional-header-parsing client)
+ resp (new-client {:decode-body-headers true})
+ resp2 (new-client {:decode-body-headers false})
+ resp3 ((client/wrap-additional-header-parsing
+ (fn [req] {:body nil})) {:decode-body-headers true})
+ resp4 ((client/wrap-additional-header-parsing
+ (fn [req] {:headers {"content-type" "application/pdf"}
+ :body (.getBytes text)}))
+ {:decode-body-headers true})]
+ (is (= {"content-type" "text/html; charset=Shift_JIS"
+ "content-style-type" "text/css"
+ "content-script-type" "text/javascript"}
+ (:headers resp)))
+ (is (nil? (:headers resp2)))
+ (is (nil? (:headers resp3)))
+ (is (= {"content-type" "application/pdf"} (:headers resp4)))))
+
+(deftest t-wrap-additional-header-parsing-html5
+ (let [^String text (slurp (resource "header-html5-test.html"))
+ client (fn [req] {:body (.getBytes text)})
+ new-client (client/wrap-additional-header-parsing client)
+ resp (new-client {:decode-body-headers true})]
+ (is (= {"content-type" "text/html; charset=UTF-8"}
+ (:headers resp)))))
+
+(deftest ^:integration t-request-without-url-set
+ (run-server)
+ ;; roundtrip with scheme as a keyword
+ (let [resp (request {:uri "/redirect-to-get"
+ :method :get})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= "get" (:body resp)))))
+
+(deftest ^:integration t-reusable-conn-mgrs
+ (run-server)
+ (let [cm (conn/make-reusable-conn-manager {:timeout 10 :insecure? false})
+ resp1 (request {:uri "/redirect-to-get"
+ :method :get
+ :connection-manager cm})
+ resp2 (request {:uri "/redirect-to-get"
+ :method :get})]
+ (is (= 200 (:status resp1) (:status resp2)))
+ (is (nil? (get-in resp1 [:headers "connection"]))
+ "connection should remain open")
+ (is (= "close" (get-in resp2 [:headers "connection"]))
+ "connection should be closed")
+ (.shutdown cm)))
+
+(deftest ^:integration t-reusable-async-conn-mgrs
+ (run-server)
+ (let [cm (conn/make-reuseable-async-conn-manager {:timeout 10 :insecure? false})
+ resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise)]
+ (request {:async? true :uri "/redirect-to-get" :method :get :connection-manager cm}
+ resp1
+ exce1)
+ (request {:async? true :uri "/redirect-to-get" :method :get}
+ resp2
+ exce2)
+ (is (= 200 (:status @resp1) (:status @resp2)))
+ (is (nil? (get-in @resp1 [:headers "connection"]))
+ "connection should remain open")
+ (is (= "close" (get-in @resp2 [:headers "connection"]))
+ "connection should be closed")
+ (is (not (realized? exce2)))
+ (is (not (realized? exce1)))
+ (.shutdown cm)))
+
+(deftest ^:integration t-with-async-pool
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise)]
+ (request {:async? true :uri "/get" :method :get} resp1 exce1)
+ (request {:async? true :uri "/get" :method :get} resp2 exce2)
+ (is (= 200 (:status @resp1) (:status @resp2)))
+ (is (not (realized? exce2)))
+ (is (not (realized? exce1))))))
+
+(deftest ^:integration t-with-async-pool-sleep
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise)]
+ (request {:async? true :uri "/get" :method :get} resp1 exce1)
+ (Thread/sleep 500)
+ (request {:async? true :uri "/get" :method :get} resp2 exce2)
+ (is (= 200 (:status @resp1) (:status @resp2)))
+ (is (not (realized? exce2)))
+ (is (not (realized? exce1))))))
+
+(deftest ^:integration t-async-pool-wrap-exception
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise) count (atom 2)]
+ (request {:async? true :uri "/error" :method :get} resp1 exce1)
+ (Thread/sleep 500)
+ (request {:async? true :uri "/get" :method :get} resp2 exce2)
+ (is (realized? exce1))
+ (is (not (realized? exce2)))
+ (is (= 200 (:status @resp2))))))
+
+(deftest ^:integration t-async-pool-exception-when-start
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise)
+ middleware (fn [client]
+ (fn [req resp raise] (throw (Exception.))))]
+ (client/with-additional-middleware
+ [middleware]
+ (try (request {:async? true :uri "/error" :method :get} resp1 exce1)
+ (catch Throwable ex))
+ (Thread/sleep 500)
+ (try (request {:async? true :uri "/get" :method :get} resp2 exce2)
+ (catch Throwable ex))
+ (is (not (realized? exce1)))
+ (is (not (realized? exce2)))
+ (is (not (realized? resp1)))
+ (is (not (realized? resp2)))))))
+
+(deftest ^:integration t-reuse-async-pool
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp1 (promise) resp2 (promise)
+ exce1 (promise) exce2 (promise)]
+ (request {:async? true :uri "/get" :method :get}
+ (fn [resp]
+ (resp1 resp)
+ (request (client/reuse-pool
+ {:async? true
+ :uri "/get"
+ :method :get}
+ resp)
+ resp2
+ exce2))
+ exce1)
+ (is (= 200 (:status @resp1) (:status @resp2)))
+ (is (not (realized? exce2)))
+ (is (not (realized? exce1))))))
+
+(deftest ^:integration t-async-pool-redirect-to-get
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp (promise)
+ exce (promise)]
+ (request {:async? true :uri "/redirect-to-get"
+ :method :get :redirect-strategy :default} resp exce)
+ (is (= 200 (:status @resp)))
+ (is (not (realized? exce))))))
+
+(deftest ^:integration t-async-pool-max-redirect
+ (run-server)
+ (client/with-async-connection-pool {}
+ (let [resp (promise)
+ exce (promise)]
+ (request {:async? true :uri "/redirect" :method :get
+ :redirect-strategy :default
+ :throw-exceptions true} resp exce)
+ (is @exce)
+ (is (not (realized? resp))))))
+
+(deftest test-url-encode-path
+ (is (= (client/url-encode-illegal-characters "?foo bar+baz[]75")
+ "?foo%20bar+baz%5B%5D75"))
+ (is (= {:uri (str "/:@-._~!$&'()*+,="
+ ";"
+ ":@-._~!$&'()*+,"
+ "="
+ ":@-._~!$&'()*+,==")
+ :query-string (str "/?:@-._~!$'()*+,;"
+ "="
+ "/?:@-._~!$'()*+,;==")}
+ ;; This URL sucks, yes, it's actually a valid URL
+ (select-keys (client/parse-url
+ (str "http://example.com/:@-._~!$&'()*+,=;:@-._~!$&'()*+"
+ ",=:@-._~!$&'()*+,==?/?:@-._~!$'()*+,;=/?:@-._~!$'("
+ ")*+,;==#/?:@-._~!$&'()*+,;="))
+ [:uri :query-string])))
+ (let [all-chars (apply str (map char (range 256)))
+ all-legal (client/url-encode-illegal-characters all-chars)]
+ (is (= all-legal
+ (client/url-encode-illegal-characters all-legal)))))
+
+(defmethod client/coerce-response-body :json+ms949
+ [req resp]
+ (client/coerce-json-body req resp true "MS949"))
+
+(deftest t-coercion-methods
+ (let [json-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
+ json-ms949-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"안뇽\"}" "MS949"))
+ auto-body (ByteArrayInputStream. (.getBytes "{\"foo\":\"bar\"}"))
+ edn-body (ByteArrayInputStream. (.getBytes "{:foo \"bar\"}"))
+ transit-json-body (ByteArrayInputStream.
+ (.getBytes "[\"^ \",\"~:foo\",\"bar\"]"))
+ transit-msgpack-body (->> (map byte [-127 -91 126 58 102 111
+ 111 -93 98 97 114])
+ (byte-array 11)
+ (ByteArrayInputStream.))
+ www-form-urlencoded-body (ByteArrayInputStream. (.getBytes "foo=bar"))
+ auto-www-form-urlencoded-body
+ (ByteArrayInputStream. (.getBytes "foo=bar"))
+ json-resp {:body json-body :status 200
+ :headers {"content-type" "application/json"}}
+ json-ms949-resp {:body json-ms949-body :status 200
+ :headers {"content-type" "application/json; charset=ms949"}}
+ auto-resp {:body auto-body :status 200
+ :headers {"content-type" "application/json"}}
+ edn-resp {:body edn-body :status 200
+ :headers {"content-type" "application/edn"}}
+ transit-json-resp {:body transit-json-body :status 200
+ :headers {"content-type" "application/transit-json"}}
+ transit-msgpack-resp {:body transit-msgpack-body :status 200
+ :headers {"content-type"
+ "application/transit-msgpack"}}
+ www-form-urlencoded-resp
+ {:body www-form-urlencoded-body :status 200
+ :headers {"content-type"
+ "application/x-www-form-urlencoded"}}
+ auto-www-form-urlencoded-resp
+ {:body auto-www-form-urlencoded-body :status 200
+ :headers {"content-type"
+ "application/x-www-form-urlencoded"}}]
+ (is (= {:foo "bar"}
+ (:body (client/coerce-response-body {:as :json} json-resp))
+ (:body (client/coerce-response-body {:as :clojure} edn-resp))
+ (:body (client/coerce-response-body {:as :auto} auto-resp))
+ (:body (client/coerce-response-body {:as :transit+json}
+ transit-json-resp))
+ (:body (client/coerce-response-body {:as :transit+msgpack}
+ transit-msgpack-resp))
+ (:body (client/coerce-response-body {:as :auto}
+ auto-www-form-urlencoded-resp))
+ (:body (client/coerce-response-body {:as :x-www-form-urlencoded}
+ www-form-urlencoded-resp))))
+ (is (= {:foo "안뇽"}
+ (:body (client/coerce-response-body {:as :json+ms949} json-ms949-resp))))
+
+ (testing "throws AssertionError when optional libraries are not loaded"
+ (with-redefs [client/json-enabled? false]
+ (is (thrown? AssertionError (client/coerce-response-body {:as :json} json-resp)))
+ (is (thrown? AssertionError (client/coerce-response-body {:as :auto} json-resp))))
+ (with-redefs [client/transit-enabled? false]
+ (is (thrown? AssertionError (client/coerce-response-body {:as :transit+json} transit-json-resp)))
+ (is (thrown? AssertionError (client/coerce-response-body {:as :transit+msgpack} transit-msgpack-resp))))
+ (with-redefs [client/ring-codec-enabled? false]
+ (is (thrown? AssertionError (client/coerce-response-body {:as :x-www-form-urlencoded} www-form-urlencoded-resp)))
+ (is (thrown? AssertionError (client/coerce-response-body {:as :auto} auto-www-form-urlencoded-resp)))))))
+
+
+(deftest t-reader-coercion
+ (let [read-lines (fn [reader] (vec (take-while not-empty (repeatedly #(.readLine reader)))))
+ reader-body (ByteArrayInputStream. (.getBytes "foo\nbar\n"))
+ reader-resp {:body reader-body :status 200 :headers {"content-type" "text/plain; charset=utf-8"}}
+ encoded-body (ByteArrayInputStream. (byte-array [0xA9]))
+ encoded-resp {:body encoded-body :status 200 :headers {"content-type" "text/plain; charset=iso-8859-1"}}
+ utf8-body (ByteArrayInputStream. (byte-array [0xC2 0xA9]))
+ utf8-resp {:body utf8-body :status 200 :headers {"content-type" "text/plain; charset=utf-8"}}]
+ (is (= ["foo" "bar"]
+ (read-lines (:body (client/coerce-response-body {:as :reader} reader-resp)))))
+
+ (is (= "©"
+ (.readLine (:body (client/coerce-response-body {:as :reader} encoded-resp)))
+ (.readLine (:body (client/coerce-response-body {:as :reader} utf8-resp)))))))
+
+(deftest ^:integration t-with-middleware
+ (run-server)
+ (is (:request-time (request {:uri "/get" :method :get})))
+ (is (= client/*current-middleware* client/default-middleware))
+ (client/with-middleware [client/wrap-url
+ client/wrap-method
+ #'client/wrap-request-timing]
+ (is (:request-time (request {:uri "/get" :method :get})))
+ (is (= client/*current-middleware* [client/wrap-url
+ client/wrap-method
+ #'client/wrap-request-timing])))
+ (client/with-middleware (->> client/default-middleware
+ (remove #{client/wrap-request-timing}))
+ (is (not (:request-time (request {:uri "/get" :method :get}))))
+ (is (not (contains? (set client/*current-middleware*)
+ client/wrap-request-timing)))
+ (is (contains? (set client/default-middleware)
+ client/wrap-request-timing))))
+
+(deftest t-detect-charset-by-content-type
+ (is (= "UTF-8" (client/detect-charset nil)))
+ (is (= "UTF-8"(client/detect-charset "application/json")))
+ (is (= "UTF-8"(client/detect-charset "text/html")))
+ (is (= "GBK"(client/detect-charset "application/json; charset=GBK")))
+ (is (= "ISO-8859-1" (client/detect-charset
+ "application/json; charset=ISO-8859-1")))
+ (is (= "ISO-8859-1" (client/detect-charset
+ "application/json; charset = ISO-8859-1")))
+ (is (= "GB2312" (client/detect-charset "text/html; Charset=GB2312"))))
+
+(deftest ^:integration customMethodTest
+ (run-server)
+ (let [resp (request {:uri "/propfind" :method "PROPFIND"})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= "propfind" (:body resp))))
+ (let [resp (request {:uri "/propfind-with-body"
+ :method "PROPFIND"
+ :body "propfindbody"})]
+ (is (= 200 (:status resp)))
+ (is (= "close" (get-in resp [:headers "connection"])))
+ (is (= "propfindbody" (:body resp)))))
+
+(deftest ^:integration status-line-parsing
+ (run-server)
+ (let [resp (request {:uri "/get" :method :get})
+ protocol-version (:protocol-version resp)]
+ (is (= 200 (:status resp)))
+ (is (= "HTTP" (:name protocol-version)))
+ (is (= 1 (:major protocol-version)))
+ (is (= 1 (:minor protocol-version)))
+ (is (= "OK" (:reason-phrase resp)))))
+
+(deftest ^:integration multi-valued-query-params
+ (run-server)
+ (testing "default (repeating) multi-valued query params"
+ (let [resp (request {:uri "/query-string"
+ :method :get
+ :query-params {:a [1 2 3]
+ :b ["x" "y" "z"]}})
+ query-string (-> resp :body form-decode-str)]
+ (is (= 200 (:status resp)))
+ (is (.contains query-string "a=1&a=2&a=3") query-string)
+ (is (.contains query-string "b=x&b=y&b=z") query-string)))
+
+ (testing "multi-valued query params in indexed-style"
+ (let [resp (request {:uri "/query-string"
+ :method :get
+ :multi-param-style :indexed
+ :query-params {:a [1 2 3]
+ :b ["x" "y" "z"]}})
+ query-string (-> resp :body form-decode-str)]
+ (is (= 200 (:status resp)))
+ (is (.contains query-string "a[0]=1&a[1]=2&a[2]=3") query-string)
+ (is (.contains query-string "b[0]=x&b[1]=y&b[2]=z") query-string)))
+
+ (testing "multi-valued query params in array-style"
+ (let [resp (request {:uri "/query-string"
+ :method :get
+ :multi-param-style :array
+ :query-params {:a [1 2 3]
+ :b ["x" "y" "z"]}})
+ query-string (-> resp :body form-decode-str)]
+ (is (= 200 (:status resp)))
+ (is (.contains query-string "a[]=1&a[]=2&a[]=3") query-string)
+ (is (.contains query-string "b[]=x&b[]=y&b[]=z") query-string)))
+ (testing "multi-valued query params in comma-separated"
+ (let [resp (request {:uri "/query-string"
+ :method :get
+ :multi-param-style :comma-separated
+ :query-params {:a [1 2 3]
+ :b ["x" "y" "z"]}})
+ query-string (-> resp :body form-decode-str)]
+ (is (= 200 (:status resp)))
+ (is (.contains query-string "a=1,2,3") query-string)
+ (is (.contains query-string "b=x,y,z") query-string))))
+
+(deftest t-wrap-flatten-nested-params
+ (is-applied client/wrap-flatten-nested-params
+ {}
+ {:flatten-nested-keys [:query-params]})
+ (is-applied client/wrap-flatten-nested-params
+ {:flatten-nested-keys []}
+ {:flatten-nested-keys []})
+ (is-applied client/wrap-flatten-nested-params
+ {:flatten-nested-keys [:foo]}
+ {:flatten-nested-keys [:foo]})
+ (is-applied client/wrap-flatten-nested-params
+ {:ignore-nested-query-string true}
+ {:ignore-nested-query-string true
+ :flatten-nested-keys []})
+ (is-applied client/wrap-flatten-nested-params
+ {}
+ {:flatten-nested-keys '(:query-params)})
+ (is-applied client/wrap-flatten-nested-params
+ {:flatten-nested-form-params true}
+ {:flatten-nested-form-params true
+ :flatten-nested-keys '(:query-params :form-params)})
+ (is-applied client/wrap-flatten-nested-params
+ {:flatten-nested-form-params true
+ :ignore-nested-query-string true}
+ {:ignore-nested-query-string true
+ :flatten-nested-form-params true
+ :flatten-nested-keys '(:form-params)})
+ (try
+ ((client/wrap-flatten-nested-params identity)
+ {:flatten-nested-form-params true
+ :ignore-nested-query-string true
+ :flatten-nested-keys [:thing :bar]})
+ (is false "should have thrown exception")
+ (catch IllegalArgumentException e
+ (is (= (.getMessage e)
+ (str "only :flatten-nested-keys or :ignore-nested-query-string/"
+ ":flatten-nested-keys may be specified, not both")))))
+ (try
+ ((client/wrap-flatten-nested-params identity)
+ {:ignore-nested-query-string true
+ :flatten-nested-keys [:thing :bar]})
+ (is false "should have thrown exception")
+ (catch IllegalArgumentException e
+ (is (= (.getMessage e)
+ (str "only :flatten-nested-keys or :ignore-nested-query-string/"
+ ":flatten-nested-keys may be specified, not both"))))))
+
+(defn transit-resp [body]
+ {:body body
+ :status 200
+ :headers {"content-type" "application/transit-json"}})
+
+(deftest issue-609-empty-transit-response
+ (testing "Body is available right away"
+ (is (= {:foo "bar"}
+ (:body (client/coerce-response-body
+ {:as :transit+json}
+ (transit-resp (ByteArrayInputStream.
+ (.getBytes "[\"^ \",\"~:foo\",\"bar\"]"))))))))
+
+ (testing "Empty body is read as nil"
+ (is (nil? (:body (client/coerce-response-body
+ {:as :transit+json}
+ (transit-resp (ByteArrayInputStream. (.getBytes ""))))))))
+
+ (testing "Body is read correctly even if the data becomes available later"
+ ;; Ensure both streams are closed (normally done inside future).
+ (with-open [o (PipedOutputStream.)
+ i (PipedInputStream.)]
+ (.connect i o)
+ (future
+ (Thread/sleep 10)
+ (.write o (.getBytes "[\"^ \",\"~:foo\",\"bar\"]"))
+ ;; Close right now, with-open will wait until test is done.
+ (.close o))
+ (is (= {:foo "bar"}
+ (:body (client/coerce-response-body
+ {:as :transit+json}
+ (transit-resp i))))))))
diff --git a/test/clj_http/test/conn_mgr.clj b/test/clj_http/test/conn_mgr.clj
deleted file mode 100644
index 7209af0..0000000
--- a/test/clj_http/test/conn_mgr.clj
+++ /dev/null
@@ -1,95 +0,0 @@
-(ns clj-http.test.conn-mgr
- (:require [clj-http.conn-mgr :as conn-mgr]
- [clj-http.core :as core]
- [clj-http.test.core :refer [run-server]]
- [clojure.test :refer :all]
- [ring.adapter.jetty :as ring])
- (:import (java.security KeyStore)
- (org.apache.http.conn.ssl SSLSocketFactory)
- (org.apache.http.impl.conn BasicClientConnectionManager)))
-
-(def client-ks "test-resources/client-keystore")
-(def client-ks-pass "keykey")
-(def secure-request {:request-method :get :uri "/"
- :server-port 18084 :scheme :https
- :keystore client-ks :keystore-pass client-ks-pass
- :trust-store client-ks :trust-store-pass client-ks-pass
- :server-name "localhost" :insecure? true})
-
-(defn secure-handler [req]
- (if (nil? (:ssl-client-cert req))
- {:status 403}
- {:status 200}))
-
-(deftest load-keystore
- (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")]
- (is (instance? KeyStore ks))
- (is (> (.size ks) 0))))
-
-(deftest use-existing-keystore
- (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")
- ks (conn-mgr/get-keystore ks)]
- (is (instance? KeyStore ks))
- (is (> (.size ks) 0))))
-
-(deftest load-keystore-with-nil-pass
- (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil nil)]
- (is (instance? KeyStore ks))))
-
-(deftest keystore-scheme-factory
- (let [sr (conn-mgr/get-keystore-scheme-registry
- {:keystore client-ks :keystore-pass client-ks-pass
- :trust-store client-ks :trust-store-pass client-ks-pass})
- socket-factory (.getSchemeSocketFactory (.get sr "https"))]
- (is (instance? SSLSocketFactory socket-factory))))
-
-(deftest ^:integration ssl-client-cert-get
- (let [server (ring/run-jetty secure-handler
- {:port 18083 :ssl-port 18084
- :ssl? true
- :join? false
- :keystore "test-resources/keystore"
- :key-password "keykey"
- :client-auth :want})]
- (try
- (let [resp (core/request {:request-method :get :uri "/get"
- :server-port 18084 :scheme :https
- :insecure? true :server-name "localhost"})]
- (is (= 403 (:status resp))))
- (let [resp (core/request secure-request)]
- (is (= 200 (:status resp))))
- (finally
- (.stop server)))))
-
-(deftest ^:integration t-closed-conn-mgr-for-as-stream
- (run-server)
- (let [shutdown? (atom false)
- cm (proxy [BasicClientConnectionManager] []
- (shutdown []
- (reset! shutdown? true)))]
- (try
- (core/request {:request-method :get :uri "/timeout"
- :server-port 18080 :scheme :http
- :server-name "localhost"
- ;; timeouts forces an exception being thrown
- :socket-timeout 1
- :conn-timeout 1
- :connection-manager cm
- :as :stream})
- (is false "request should have thrown an exception")
- (catch Exception e))
- (is @shutdown? "Connection manager has been shut down")))
-
-(deftest ^:integration t-closed-conn-mgr-for-empty-body
- (run-server)
- (let [shutdown? (atom false)
- cm (proxy [BasicClientConnectionManager] []
- (shutdown []
- (reset! shutdown? true)))
- response (core/request {:request-method :get :uri "/unmodified-resource"
- :server-port 18080 :scheme :http
- :server-name "localhost"
- :connection-manager cm })]
- (is (nil? (:body response)) "response shouldn't have body")
- (is (= 304 (:status response)))
- (is @shutdown? "connection manager should be shut downed")))
diff --git a/test/clj_http/test/conn_mgr_test.clj b/test/clj_http/test/conn_mgr_test.clj
new file mode 100644
index 0000000..1cc7dba
--- /dev/null
+++ b/test/clj_http/test/conn_mgr_test.clj
@@ -0,0 +1,152 @@
+(ns clj-http.test.conn-mgr-test
+ (:require [clj-http.conn-mgr :as conn-mgr]
+ [clj-http.core :as core]
+ [clj-http.test.core-test :refer [run-server]]
+ [clojure.test :refer :all]
+ [ring.adapter.jetty :as ring])
+ (:import java.security.KeyStore
+ [javax.net.ssl KeyManagerFactory TrustManagerFactory]
+ org.apache.http.impl.conn.BasicHttpClientConnectionManager))
+
+(def client-ks "test-resources/client-keystore")
+(def client-ks-pass "keykey")
+(def secure-request {:request-method :get :uri "/"
+ :server-port 18084 :scheme :https
+ :keystore client-ks :keystore-pass client-ks-pass
+ :trust-store client-ks :trust-store-pass client-ks-pass
+ :server-name "localhost" :insecure? true})
+
+(defn secure-handler [req]
+ (if (nil? (:ssl-client-cert req))
+ {:status 403}
+ {:status 200}))
+
+(deftest load-keystore
+ (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")]
+ (is (instance? KeyStore ks))
+ (is (> (.size ks) 0))))
+
+(deftest use-existing-keystore
+ (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")
+ ks (conn-mgr/get-keystore ks)]
+ (is (instance? KeyStore ks))
+ (is (> (.size ks) 0))))
+
+(deftest load-keystore-with-nil-pass
+ (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil nil)]
+ (is (instance? KeyStore ks))))
+
+(def array-of-trust-manager
+ (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")
+ tmf (doto (TrustManagerFactory/getInstance (TrustManagerFactory/getDefaultAlgorithm))
+ (.init ks))]
+ (.getTrustManagers tmf)))
+
+(def array-of-key-manager
+ (let [ks (conn-mgr/get-keystore "test-resources/keystore" nil "keykey")
+ tmf (doto (KeyManagerFactory/getInstance (KeyManagerFactory/getDefaultAlgorithm))
+ (.init ks (.toCharArray "keykey")))]
+ (.getKeyManagers tmf)))
+
+(deftest ^:integration ssl-client-cert-get
+ (let [server (ring/run-jetty secure-handler
+ {:port 18083 :ssl-port 18084
+ :ssl? true
+ :join? false
+ :keystore "test-resources/keystore"
+ :key-password "keykey"
+ :client-auth :want})]
+ (try
+ (let [resp (core/request {:request-method :get :uri "/get"
+ :server-port 18084 :scheme :https
+ :insecure? true :server-name "localhost"})]
+ (is (= 403 (:status resp))))
+ (let [resp (core/request secure-request)]
+ (is (= 200 (:status resp))))
+ (finally
+ (.stop server)))))
+
+(deftest ^:integration ssl-client-cert-get-async
+ (let [server (ring/run-jetty secure-handler
+ {:port 18083 :ssl-port 18084
+ :ssl? true
+ :join? false
+ :keystore "test-resources/keystore"
+ :key-password "keykey"
+ :client-auth :want})]
+ (try
+ (let [resp (promise)
+ exception (promise)
+ _ (core/request {:request-method :get :uri "/get"
+ :server-port 18084 :scheme :https
+ :insecure? true :server-name "localhost"
+ :async? true} resp exception)]
+ (is (= 403 (:status (deref resp 1000 {:status :timeout})))))
+ (let [resp (promise)
+ exception (promise)
+ _ (core/request (assoc secure-request :async? true) resp exception)]
+ (is (= 200 (:status (deref resp 1000 {:status :timeout})))))
+
+ (testing "with reusable connection pool"
+ (let [pool (conn-mgr/make-reusable-async-conn-manager {:timeout 10000
+ :keystore client-ks :keystore-pass client-ks-pass
+ :trust-store client-ks :trust-store-pass client-ks-pass
+ :insecure? true})]
+ (try
+ (let [resp (promise) exception (promise)
+ _ (core/request {:request-method :get :uri "/get"
+ :server-port 18084 :scheme :https
+ :server-name "localhost"
+ :connection-manager pool :async? true} resp exception)]
+ (is (= 200 (:status (deref resp 1000 {:status :timeout}))))
+ (is (:body @resp))
+ (is (not (realized? exception))))
+ (finally
+ (conn-mgr/shutdown-manager pool)))))
+ (finally
+ (.stop server)))))
+
+(deftest ^:integration t-closed-conn-mgr-for-as-stream
+ (run-server)
+ (let [shutdown? (atom false)
+ cm (proxy [BasicHttpClientConnectionManager] []
+ (shutdown []
+ (reset! shutdown? true)))]
+ (try
+ (core/request {:request-method :get :uri "/timeout"
+ :server-port 18080 :scheme :http
+ :server-name "localhost"
+ ;; timeouts forces an exception being thrown
+ :socket-timeout 1
+ :connection-timeout 1
+ :connection-manager cm
+ :as :stream})
+ (is false "request should have thrown an exception")
+ (catch Exception e))
+ (is @shutdown? "Connection manager has been shutdown")))
+
+(deftest ^:integration t-closed-conn-mgr-for-empty-body
+ (run-server)
+ (let [shutdown? (atom false)
+ cm (proxy [BasicHttpClientConnectionManager] []
+ (shutdown []
+ (reset! shutdown? true)))
+ response (core/request {:request-method :get :uri "/unmodified-resource"
+ :server-port 18080 :scheme :http
+ :server-name "localhost"
+ :connection-manager cm})]
+ (is (nil? (:body response)) "response shouldn't have body")
+ (is (= 304 (:status response)))
+ (is @shutdown? "connection manager should be shutdown")))
+
+(deftest t-reusable-conn-mgrs
+ (let [regular (conn-mgr/make-regular-conn-manager {})
+ regular-reusable (conn-mgr/make-reusable-conn-manager {})
+ async (conn-mgr/make-regular-async-conn-manager {})
+ async-reusable (conn-mgr/make-reusable-async-conn-manager {})
+ async-reuseable (conn-mgr/make-reuseable-async-conn-manager {})]
+ (is (false? (conn-mgr/reusable? regular)))
+ (is (true? (conn-mgr/reusable? regular-reusable)))
+ (is (false? (conn-mgr/reusable? async)))
+ (is (true? (conn-mgr/reusable? async-reusable)))
+ (is (true? (conn-mgr/reusable? async-reuseable)))))
diff --git a/test/clj_http/test/cookies.clj b/test/clj_http/test/cookies_test.clj
similarity index 98%
rename from test/clj_http/test/cookies.clj
rename to test/clj_http/test/cookies_test.clj
index 2232179..8dd2b35 100644
--- a/test/clj_http/test/cookies.clj
+++ b/test/clj_http/test/cookies_test.clj
@@ -1,8 +1,7 @@
-(ns clj-http.test.cookies
+(ns clj-http.test.cookies-test
(:require [clj-http.cookies :refer :all]
- [clj-http.util :refer :all]
[clojure.test :refer :all])
- (:import (org.apache.http.impl.cookie BasicClientCookie BasicClientCookie2)))
+ (:import [org.apache.http.impl.cookie BasicClientCookie BasicClientCookie2]))
(defn refer-private [ns]
(doseq [[symbol var] (ns-interns ns)]
diff --git a/test/clj_http/test/core.clj b/test/clj_http/test/core.clj
deleted file mode 100644
index 6a74916..0000000
--- a/test/clj_http/test/core.clj
+++ /dev/null
@@ -1,589 +0,0 @@
-(ns clj-http.test.core
- (:require [cheshire.core :as json]
- [clj-http.client :as client]
- [clj-http.core :as core]
- [clj-http.util :as util]
- [clojure.java.io :refer [file]]
- [clojure.pprint :as pp]
- [clojure.test :refer :all]
- [ring.adapter.jetty :as ring])
- (:import (java.io ByteArrayInputStream)
- (org.apache.http.params CoreConnectionPNames CoreProtocolPNames)
- (org.apache.http.message BasicHeader BasicHeaderIterator)
- (org.apache.http.client.methods HttpPost)
- (org.apache.http.client.params CookiePolicy ClientPNames)
- (org.apache.http HttpRequest HttpResponse HttpConnection HttpInetConnection
- HttpVersion)
- (org.apache.http.protocol HttpContext ExecutionContext)
- (org.apache.http.impl.client DefaultHttpClient)
- (org.apache.http.client.params ClientPNames)
- (java.net SocketTimeoutException)
- (sun.security.provider.certpath SunCertPathBuilderException)))
-
-(defn handler [req]
- (condp = [(:request-method req) (:uri req)]
- [:get "/get"]
- {:status 200 :body "get"}
- [:get "/empty"]
- {:status 200 :body nil}
- [:get "/clojure"]
- {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
- :headers {"content-type" "application/clojure"}}
- [:get "/edn"]
- {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
- :headers {"content-type" "application/edn"}}
- [:get "/clojure-bad"]
- {:status 200 :body "{:foo \"bar\" :baz #=(+ 1 1)}"
- :headers {"content-type" "application/clojure"}}
- [:get "/json"]
- {:status 200 :body "{\"foo\":\"bar\"}"
- :headers {"content-type" "application/json"}}
- [:get "/json-array"]
- {:status 200 :body "[\"foo\", \"bar\"]"
- :headers {"content-type" "application/json"}}
- [:get "/json-bad"]
- {:status 400 :body "{\"foo\":\"bar\"}"}
- [:get "/redirect"]
- {:status 302
- :headers {"location" "http://localhost:18080/redirect"}}
- [:get "/redirect-to-get"]
- {:status 302
- :headers {"location" "http://localhost:18080/get"}}
- [:get "/unmodified-resource"]
- {:status 304}
- [:get "/transit-json"]
- {:status 200 :body (str "[\"^ \",\"~:eggplant\",[\"^ \",\"~:quux\","
- "[\"~#set\",[1,3,2]]],\"~:baz\",\"~f7\","
- "\"~:foo\",\"bar\"]")
- :headers {"content-type" "application/transit+json"}}
- [:get "/transit-msgpack"]
- {:status 200
- :body (->> [-125 -86 126 58 101 103 103 112 108 97 110 116 -127 -90 126
- 58 113 117 117 120 -110 -91 126 35 115 101 116 -109 1 3 2
- -91 126 58 98 97 122 -93 126 102 55 -91 126 58 102 111 111
- -93 98 97 114]
- (map byte)
- (byte-array)
- (ByteArrayInputStream.))
- :headers {"content-type" "application/transit+msgpack"}}
- [:head "/head"]
- {:status 200}
- [:get "/content-type"]
- {:status 200 :body (:content-type req)}
- [:get "/header"]
- {:status 200 :body (get-in req [:headers "x-my-header"])}
- [:post "/post"]
- {:status 200 :body (:body req)}
- [:get "/error"]
- {:status 500 :body "o noes"}
- [:get "/timeout"]
- (do
- (Thread/sleep 10)
- {:status 200 :body "timeout"})
- [:delete "/delete-with-body"]
- {:status 200 :body "delete-with-body"}
- [:post "/multipart"]
- {:status 200 :body (:body req)}
- [:get "/get-with-body"]
- {:status 200 :body (:body req)}
- [:options "/options"]
- {:status 200 :body "options"}
- [:copy "/copy"]
- {:status 200 :body "copy"}
- [:move "/move"]
- {:status 200 :body "move"}
- [:patch "/patch"]
- {:status 200 :body "patch"}
- [:get "/headers"]
- {:status 200 :body (json/encode (:headers req))}
- [:get "/query-string"]
- {:status 200 :body (:query-string req)}))
-
-(defn run-server
- []
- (defonce server
- (ring/run-jetty #'handler {:port 18080 :join? false})))
-
-(defn localhost [path]
- (str "http://localhost:18080" path))
-
-(def base-req
- {:scheme :http
- :server-name "localhost"
- :server-port 18080})
-
-(defn request [req]
- (core/request (merge base-req req)))
-
-(defn slurp-body [req]
- (slurp (:body req)))
-
-(deftest ^:integration makes-get-request
- (run-server)
- (let [resp (request {:request-method :get :uri "/get"})]
- (is (= 200 (:status resp)))
- (is (= "get" (slurp-body resp)))))
-
-(deftest ^:integration makes-head-request
- (run-server)
- (let [resp (request {:request-method :head :uri "/head"})]
- (is (= 200 (:status resp)))
- (is (nil? (:body resp)))))
-
-(deftest ^:integration sets-content-type-with-charset
- (run-server)
- (let [resp (client/request {:scheme :http
- :server-name "localhost"
- :server-port 18080
- :request-method :get :uri "/content-type"
- :content-type "text/plain"
- :character-encoding "UTF-8"})]
- (is (= "text/plain; charset=UTF-8" (:body resp)))))
-
-(deftest ^:integration sets-content-type-without-charset
- (run-server)
- (let [resp (client/request {:scheme :http
- :server-name "localhost"
- :server-port 18080
- :request-method :get :uri "/content-type"
- :content-type "text/plain"})]
- (is (= "text/plain" (:body resp)))))
-
-(deftest ^:integration sets-arbitrary-headers
- (run-server)
- (let [resp (request {:request-method :get :uri "/header"
- :headers {"x-my-header" "header-val"}})]
- (is (= "header-val" (slurp-body resp)))))
-
-(deftest ^:integration sends-and-returns-byte-array-body
- (run-server)
- (let [resp (request {:request-method :post :uri "/post"
- :body (util/utf8-bytes "contents")})]
- (is (= 200 (:status resp)))
- (is (= "contents" (slurp-body resp)))))
-
-(deftest ^:integration returns-arbitrary-headers
- (run-server)
- (let [resp (request {:request-method :get :uri "/get"})]
- (is (string? (get-in resp [:headers "date"])))
- (is (nil? (get-in resp [:headers "Date"])))))
-
-(deftest ^:integration returns-status-on-exceptional-responses
- (run-server)
- (let [resp (request {:request-method :get :uri "/error"})]
- (is (= 500 (:status resp)))))
-
-(deftest ^:integration sets-socket-timeout
- (run-server)
- (try
- (is (thrown? SocketTimeoutException
- (client/request {:scheme :http
- :server-name "localhost"
- :server-port 18080
- :request-method :get :uri "/timeout"
- :socket-timeout 1})))))
-
-(deftest ^:integration delete-with-body
- (run-server)
- (let [resp (request {:request-method :delete :uri "/delete-with-body"
- :body (.getBytes "foo bar")})]
- (is (= 200 (:status resp)))))
-
-(deftest ^:integration self-signed-ssl-get
- (let [server (ring/run-jetty handler
- {:port 8081 :ssl-port 18082
- :ssl? true
- :join? false
- :keystore "test-resources/keystore"
- :key-password "keykey"})]
- (try
- (is (thrown? SunCertPathBuilderException
- (client/request {:scheme :https
- :server-name "localhost"
- :server-port 18082
- :request-method :get :uri "/get"})))
- (let [resp (request {:request-method :get :uri "/get" :server-port 18082
- :scheme :https :insecure? true})]
- (is (= 200 (:status resp)))
- (is (= "get" (String. (util/force-byte-array (:body resp))))))
- (finally
- (.stop server)))))
-
-(deftest ^:integration multipart-form-uploads
- (run-server)
- (let [bytes (util/utf8-bytes "byte-test")
- stream (ByteArrayInputStream. bytes)
- resp (request {:request-method :post :uri "/multipart"
- :multipart [{:name "a" :content "testFINDMEtest"
- :encoding "UTF-8"
- :mime-type "application/text"}
- {:name "b" :content bytes
- :mime-type "application/json"}
- {:name "d"
- :content (file "test-resources/keystore")
- :encoding "UTF-8"
- :mime-type "application/binary"}
- {:name "c" :content stream
- :mime-type "application/json"}
- {:name "e" :part-name "eggplant"
- :content "content"
- :mime-type "application/text"}]})
- resp-body (apply str (map #(try (char %) (catch Exception _ ""))
- (util/force-byte-array (:body resp))))]
- (is (= 200 (:status resp)))
- (is (re-find #"testFINDMEtest" resp-body))
- (is (re-find #"application/json" resp-body))
- (is (re-find #"application/text" resp-body))
- (is (re-find #"UTF-8" resp-body))
- (is (re-find #"byte-test" resp-body))
- (is (re-find #"name=\"c\"" resp-body))
- (is (re-find #"name=\"d\"" resp-body))
- (is (re-find #"name=\"eggplant\"" resp-body))
- (is (re-find #"content" resp-body))))
-
-(deftest ^:integration multipart-inputstream-length
- (run-server)
- (let [bytes (util/utf8-bytes "byte-test")
- stream (ByteArrayInputStream. bytes)
- resp (request {:request-method :post :uri "/multipart"
- :multipart [{:name "c" :content stream :length 9
- :mime-type "application/json"}]})
- resp-body (apply str (map #(try (char %) (catch Exception _ ""))
- (util/force-byte-array (:body resp))))]
- (is (= 200 (:status resp)))
- (is (re-find #"byte-test" resp-body))))
-
-(deftest ^:integration t-save-request-obj
- (run-server)
- (let [resp (request {:request-method :post :uri "/post"
- :body "foo bar"
- :save-request? true
- :debug-body true})]
- (is (= 200 (:status resp)))
- (is (= {:scheme :http
- :http-url (localhost "/post")
- :request-method :post
- :save-request? true
- :debug-body true
- :uri "/post"
- :server-name "localhost"
- :server-port 18080
- :body-content "foo bar"
- :body-type String}
- (dissoc (:request resp) :body :http-req)))
- (is (instance? HttpPost (-> resp :request :http-req)))))
-
-(deftest parse-headers
- (are [headers expected]
- (let [iterator (BasicHeaderIterator.
- (into-array BasicHeader
- (map (fn [[name value]]
- (BasicHeader. name value))
- headers)) nil)]
- (is (= (core/parse-headers iterator) expected)))
-
- [] {}
-
- [["Set-Cookie" "one"]] {"set-cookie" "one"}
-
- [["Set-Cookie" "one"] ["set-COOKIE" "two"]]
- {"set-cookie" ["one" "two"]}
-
- [["Set-Cookie" "one"] ["serVer" "some-server"] ["set-cookie" "two"]]
- {"set-cookie" ["one" "two"] "server" "some-server"}))
-
-(deftest ^:integration t-streaming-response
- (run-server)
- (let [stream (:body (request {:request-method :get :uri "/get" :as :stream}))
- body (slurp stream)]
- (is (= "get" body))))
-
-(deftest throw-on-invalid-body
- (is (thrown-with-msg? IllegalArgumentException #"Invalid request method :bad"
- (client/request {:url "http://example.org"
- :method :bad}))))
-
-(deftest ^:integration throw-on-too-many-redirects
- (run-server)
- (let [resp (client/get (localhost "/redirect")
- {:max-redirects 2 :throw-exceptions false})]
- (is (= 302 (:status resp)))
- (is (= (apply vector (repeat 3 "http://localhost:18080/redirect"))
- (:trace-redirects resp))))
- (is (thrown-with-msg? Exception #"Too many redirects: 3"
- (client/get (localhost "/redirect")
- {:max-redirects 2 :throw-exceptions true})))
- (is (thrown-with-msg? Exception #"Too many redirects: 21"
- (client/get (localhost "/redirect")
- {:throw-exceptions true}))))
-
-(deftest ^:integration get-with-body
- (run-server)
- (let [resp (request {:request-method :get :uri "/get-with-body"
- :body (.getBytes "foo bar")})]
- (is (= 200 (:status resp)))
- (is (= "foo bar" (String. (util/force-byte-array (:body resp)))))))
-
-(deftest ^:integration head-with-body
- (run-server)
- (let [resp (request {:request-method :head :uri "/head" :body "foo"})]
- (is (= 200 (:status resp)))))
-
-(deftest ^:integration t-clojure-output-coercion
- (run-server)
- (let [resp (client/get (localhost "/clojure") {:as :clojure})]
- (is (= 200 (:status resp)))
- (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}} (:body resp))))
- (let [clj-resp (client/get (localhost "/clojure") {:as :auto})
- edn-resp (client/get (localhost "/edn") {:as :auto})]
- (is (= 200 (:status clj-resp) (:status edn-resp)))
- (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
- (:body clj-resp)
- (:body edn-resp)))))
-
-(deftest ^:integration t-transit-output-coercion
- (run-server)
- (let [transit-json-resp (client/get (localhost "/transit-json") {:as :auto})
- transit-msgpack-resp (client/get (localhost "/transit-msgpack")
- {:as :auto})]
- (is (= 200
- (:status transit-json-resp)
- (:status transit-msgpack-resp)))
- (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
- (:body transit-json-resp)
- (:body transit-msgpack-resp)))))
-
-(deftest ^:integration t-json-output-coercion
- (run-server)
- (let [resp (client/get (localhost "/json") {:as :json})
- resp-array (client/get (localhost "/json-array") {:as :json-strict})
- resp-str (client/get (localhost "/json")
- {:as :json :coerce :exceptional})
- resp-str-keys (client/get (localhost "/json") {:as :json-string-keys})
- resp-strict-str-keys (client/get (localhost "/json")
- {:as :json-strict-string-keys})
- resp-auto (client/get (localhost "/json") {:as :auto})
- bad-resp (client/get (localhost "/json-bad")
- {:throw-exceptions false :as :json})
- bad-resp-json (client/get (localhost "/json-bad")
- {:throw-exceptions false :as :json
- :coerce :always})
- bad-resp-json2 (client/get (localhost "/json-bad")
- {:throw-exceptions false :as :json
- :coerce :unexceptional})]
- (is (= 200
- (:status resp)
- (:status resp-array)
- (:status resp-str)
- (:status resp-str-keys)
- (:status resp-strict-str-keys)
- (:status resp-auto)))
- (is (= {:foo "bar"}
- (:body resp)
- (:body resp-auto)))
- (is (= ["foo", "bar"]
- (:body resp-array)))
- (is (= {"foo" "bar"}
- (:body resp-strict-str-keys)
- (:body resp-str-keys)))
- ;; '("foo" "bar") and ["foo" "bar"] compare as equal with =.
- (is (vector? (:body resp-array)))
- (is (= "{\"foo\":\"bar\"}" (:body resp-str)))
- (is (= 400
- (:status bad-resp)
- (:status bad-resp-json)
- (:status bad-resp-json2)))
- (is (= "{\"foo\":\"bar\"}" (:body bad-resp))
- "don't coerce on bad response status by default")
- (is (= {:foo "bar"} (:body bad-resp-json)))
- (is (= "{\"foo\":\"bar\"}" (:body bad-resp-json2)))))
-
-(deftest ^:integration t-ipv6
- (run-server)
- (let [resp (client/get "http://[::1]:18080/get")]
- (is (= 200 (:status resp)))
- (is (= "get" (:body resp)))))
-
-(deftest t-custom-retry-handler
- (let [called? (atom false)]
- (is (thrown? Exception
- (client/post "http://localhost"
- {:multipart [{:name "title" :content "Foo"}
- {:name "Content/type"
- :content "text/plain"}
- {:name "file"
- :content (file "/tmp/missingfile")}]
- :retry-handler (fn [ex try-count http-context]
- (reset! called? true)
- false)})))
- (is @called?)))
-
-;; super-basic test for methods that aren't used that often
-(deftest ^:integration t-copy-options-move
- (run-server)
- (let [resp1 (client/options (localhost "/options"))
- resp2 (client/move (localhost "/move"))
- resp3 (client/copy (localhost "/copy"))
- resp4 (client/patch (localhost "/patch"))]
- (is (= #{200} (set (map :status [resp1 resp2 resp3 resp4]))))
- (is (= "options" (:body resp1)))
- (is (= "move" (:body resp2)))
- (is (= "copy" (:body resp3)))
- (is (= "patch" (:body resp4)))))
-
-(deftest ^:integration t-json-encoded-form-params
- (run-server)
- (let [params {:param1 "value1" :param2 {:foo "bar"}}
- resp (client/post (localhost "/post") {:content-type :json
- :form-params params})]
- (is (= 200 (:status resp)))
- (is (= (json/encode params) (:body resp)))))
-
-(deftest ^:integration t-request-interceptor
- (run-server)
- (let [req-ctx (atom [])
- {:keys [status trace-redirects] :as resp}
- (client/get
- (localhost "/get")
- {:request-interceptor
- (fn [^HttpRequest req ^HttpContext ctx]
- (reset! req-ctx {:method (.getMethod req) :uri (.getURI req)}))})]
- (is (= 200 status))
- (is (= "GET" (:method @req-ctx)))
- (is (= "/get" (.getPath (:uri @req-ctx))))))
-
-
-(deftest ^:integration t-response-interceptor
- (run-server)
- (let [saved-ctx (atom [])
- {:keys [status trace-redirects] :as resp}
- (client/get
- (localhost "/redirect-to-get")
- {:response-interceptor
- (fn [^HttpResponse resp ^HttpContext ctx]
- (let [^HttpInetConnection conn
- (.getAttribute ctx ExecutionContext/HTTP_CONNECTION)]
- (swap! saved-ctx conj {:remote-port (.getRemotePort conn)
- :http-conn conn})))})]
- (is (= 200 status))
- (is (= 2 (count @saved-ctx)))
- (is (= (count trace-redirects) (count @saved-ctx)))
- (is (every? #(= 18080 (:remote-port %)) @saved-ctx))
- (is (every? #(instance? HttpConnection (:http-conn %)) @saved-ctx))))
-
-(deftest ^:integration t-send-input-stream-body
- (run-server)
- (let [b1 (:body (client/post "http://localhost:18080/post"
- {:body (ByteArrayInputStream. (.getBytes "foo"))
- :length 3}))
- b2 (:body (client/post "http://localhost:18080/post"
- {:body (ByteArrayInputStream.
- (.getBytes "foo"))}))
- b3 (:body (client/post "http://localhost:18080/post"
- {:body (ByteArrayInputStream.
- (.getBytes "apple"))
- :length 2}))]
- (is (= b1 "foo"))
- (is (= b2 "foo"))
- (is (= b3 "ap"))))
-
-(deftest t-add-client-params
- (testing "Using add-client-params!"
- (let [ps {"http.conn-manager.timeout" 100
- "http.socket.timeout" 250
- "http.protocol.allow-circular-redirects" false
- "http.protocol.version" HttpVersion/HTTP_1_0
- "http.useragent" "clj-http"}
- setps (.getParams (doto (DefaultHttpClient.)
- (core/add-client-params! ps)))]
- (doseq [[k v] ps]
- (is (= v (.getParameter setps k)))))))
-
-;; Regression, get notified if something changes
-(deftest ^:integration t-known-client-params-are-unchanged
- (let [params ["http.socket.timeout" CoreConnectionPNames/SO_TIMEOUT
- "http.connection.timeout"
- CoreConnectionPNames/CONNECTION_TIMEOUT
- "http.protocol.version" CoreProtocolPNames/PROTOCOL_VERSION
- "http.useragent" CoreProtocolPNames/USER_AGENT
- "http.conn-manager.timeout" ClientPNames/CONN_MANAGER_TIMEOUT
- "http.protocol.allow-circular-redirects"
- ClientPNames/ALLOW_CIRCULAR_REDIRECTS]]
- (doseq [[plaintext constant] (partition 2 params)]
- (is (= plaintext constant)))))
-
-;; If you don't explicitly set a :cookie-policy, use
-;; CookiePolicy/BROWSER_COMPATIBILITY
-(deftest t-add-client-params-default-cookie-policy
- (testing "Using add-client-params! to get a default cookie policy"
- (let [setps (.getParams (doto (DefaultHttpClient.)
- (core/add-client-params! {})))]
- (is (= CookiePolicy/BROWSER_COMPATIBILITY
- (.getParameter setps ClientPNames/COOKIE_POLICY))))))
-
-;; If you set a :cookie-policy, the name of the policy is registered
-;; as (str (type cookie-policy))
-(deftest t-add-client-params-cookie-policy
- (testing "Using add-client-params! to get an explicitly set :cookie-policy"
- (let [setps (.getParams (doto (DefaultHttpClient.)
- (core/add-client-params!
- {:cookie-policy (constantly nil)})))]
- (is (.startsWith ^String (.getParameter setps ClientPNames/COOKIE_POLICY)
- "class ")))))
-
-
-;; This relies on connections to writequit.org being slower than 1ms, if this
-;; fails, you must have very nice internet.
-(deftest ^:integration sets-conn-timeout
- (run-server)
- (try
- (is (thrown? org.apache.http.conn.ConnectTimeoutException
- (client/request {:scheme :http
- :server-name "www.writequit.org"
- :server-port 80
- :request-method :get :uri "/"
- :conn-timeout 1})))))
-
-(deftest ^:integration connection-pool-timeout
- (run-server)
- (client/with-connection-pool {:timeout 1 :threads 1 :default-per-route 1}
- (let [async-request #(future (client/request {:scheme :http
- :server-name "localhost"
- :server-port 18080
- :request-method :get
- :conn-timeout 1
- :uri "/timeout"}))
- is-pool-timeout-error?
- (fn [req-fut]
- (instance? org.apache.http.conn.ConnectionPoolTimeoutException
- (try @req-fut (catch Exception e (.getCause e)))))
- req1 (async-request)
- req2 (async-request)
- timeout-error1 (is-pool-timeout-error? req1)
- timeout-error2 (is-pool-timeout-error? req2)]
- (is (or timeout-error1 timeout-error2)))))
-
-(deftest ^:integration t-header-collections
- (run-server)
- (let [headers (-> (client/get "http://localhost:18080/headers"
- {:headers {"foo" ["bar" "baz"]
- "eggplant" "quux"}})
- :body
- json/decode)]
- (is (= {"eggplant" "quux" "foo" "bar,baz"}
- (select-keys headers ["foo" "eggplant"])))))
-
-(deftest ^:integration t-clojure-no-read-eval
- (run-server)
- (is (thrown? Exception (client/get (localhost "/clojure-bad") {:as :clojure}))
- "Should throw an exception when reading clojure eval components"))
-
-(deftest ^:integration t-numeric-headers
- (run-server)
- (client/request {:method :get :url (localhost "/get") :headers {"foo" 2}}))
-
-;; Currently failing, see: https://github.com/dakrone/clj-http/issues/257
-;; (deftest ^:integration t-empty-response-coercion
-;; (run-server)
-;; (let [resp (client/get (localhost "/empty") {:as :clojure})]
-;; (is (= (:body resp) ""))))
diff --git a/test/clj_http/test/core_test.clj b/test/clj_http/test/core_test.clj
new file mode 100644
index 0000000..2567d79
--- /dev/null
+++ b/test/clj_http/test/core_test.clj
@@ -0,0 +1,995 @@
+(ns clj-http.test.core-test
+ (:require [cheshire.core :as json]
+ [clj-http.client :as client]
+ [clj-http.conn-mgr :as conn]
+ [clj-http.core :as core]
+ [clj-http.util :as util]
+ [clojure.java.io :refer [file]]
+ [clojure.test :refer :all]
+ [ring.adapter.jetty :as ring])
+ (:import java.io.ByteArrayInputStream
+ [java.net InetAddress SocketTimeoutException]
+ [java.util.concurrent TimeoutException TimeUnit]
+ [org.apache.http HttpConnection HttpInetConnection HttpRequest HttpResponse ProtocolException]
+ org.apache.http.client.config.RequestConfig
+ org.apache.http.client.params.ClientPNames
+ org.apache.http.client.protocol.HttpClientContext
+ org.apache.http.impl.conn.InMemoryDnsResolver
+ org.apache.http.impl.cookie.RFC6265CookieSpecProvider
+ [org.apache.http.message BasicHeader BasicHeaderIterator]
+ [org.apache.http.params CoreConnectionPNames CoreProtocolPNames]
+ [org.apache.http.protocol ExecutionContext HttpContext]
+ org.apache.logging.log4j.LogManager
+ sun.security.provider.certpath.SunCertPathBuilderException))
+
+(defonce logger (LogManager/getLogger "clj-http.test.core-test"))
+
+(defn handler [req]
+ (condp = [(:request-method req) (:uri req)]
+ [:get "/get"]
+ {:status 200 :body "get"}
+ [:get "/dont-cache"]
+ {:status 200 :body "nocache"
+ :headers {"cache-control" "private"}}
+ [:get "/empty"]
+ {:status 200 :body nil}
+ [:get "/empty-gzip"]
+ {:status 200 :body nil
+ :headers {"content-encoding" "gzip"}}
+ [:get "/clojure"]
+ {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
+ :headers {"content-type" "application/clojure"}}
+ [:get "/edn"]
+ {:status 200 :body "{:foo \"bar\" :baz 7M :eggplant {:quux #{1 2 3}}}"
+ :headers {"content-type" "application/edn"}}
+ [:get "/clojure-bad"]
+ {:status 200 :body "{:foo \"bar\" :baz #=(+ 1 1)}"
+ :headers {"content-type" "application/clojure"}}
+ [:get "/json"]
+ {:status 200 :body "{\"foo\":\"bar\"}"
+ :headers {"content-type" "application/json"}}
+ [:get "/json-array"]
+ {:status 200 :body "[\"foo\", \"bar\"]"
+ :headers {"content-type" "application/json"}}
+ [:get "/json-large-array"]
+ {:status 200 :body (file "test-resources/big_array_json.json")
+ :headers {"content-type" "application/json"}}
+ [:get "/json-bad"]
+ {:status 400 :body "{\"foo\":\"bar\"}"}
+ [:get "/redirect"]
+ {:status 302
+ :headers {"location" "http://localhost:18080/redirect"}}
+ [:get "/bad-redirect"]
+ {:status 301 :headers {"location" "https:///"}}
+ [:get "/redirect-to-get"]
+ {:status 302
+ :headers {"location" "http://localhost:18080/get"}}
+ [:get "/unmodified-resource"]
+ {:status 304}
+ [:get "/transit-json"]
+ {:status 200 :body (str "[\"^ \",\"~:eggplant\",[\"^ \",\"~:quux\","
+ "[\"~#set\",[1,3,2]]],\"~:baz\",\"~f7\","
+ "\"~:foo\",\"bar\"]")
+ :headers {"content-type" "application/transit+json"}}
+ [:get "/transit-json-bad"]
+ {:status 400 :body "[\"^ \", \"~:foo\",\"bar\"]"}
+ [:get "/transit-json-empty"]
+ {:status 200
+ :headers {"content-type" "application/transit+json"}}
+ [:get "/transit-msgpack"]
+ {:status 200
+ :body (->> [-125 -86 126 58 101 103 103 112 108 97 110 116 -127 -90 126
+ 58 113 117 117 120 -110 -91 126 35 115 101 116 -109 1 3 2
+ -91 126 58 98 97 122 -93 126 102 55 -91 126 58 102 111 111
+ -93 98 97 114]
+ (map byte)
+ (byte-array)
+ (ByteArrayInputStream.))
+ :headers {"content-type" "application/transit+msgpack"}}
+ [:head "/head"]
+ {:status 200}
+ [:get "/content-type"]
+ {:status 200 :body (:content-type req)}
+ [:get "/header"]
+ {:status 200 :body (get-in req [:headers "x-my-header"])}
+ [:post "/post"]
+ {:status 200 :body (:body req)}
+ [:get "/error"]
+ {:status 500 :body "o noes"}
+ [:get "/timeout"]
+ (do
+ (Thread/sleep 10)
+ {:status 200 :body "timeout"})
+ [:delete "/delete-with-body"]
+ {:status 200 :body "delete-with-body"}
+ [:post "/multipart"]
+ {:status 200 :body (:body req)}
+ [:head "/head-with-body"]
+ {:status 200 :headers {"body" (slurp (:body req))}}
+ [:get "/get-with-body"]
+ {:status 200 :body (:body req)}
+ [:options "/options"]
+ {:status 200 :body "options"}
+ [:copy "/copy"]
+ {:status 200 :body "copy"}
+ [:move "/move"]
+ {:status 200 :body "move"}
+ [:patch "/patch"]
+ {:status 200 :body "patch"}
+ [:get "/headers"]
+ {:status 200 :body (json/encode (:headers req))}
+ [:propfind "/propfind"]
+ {:status 200 :body "propfind"}
+ [:propfind "/propfind-with-body"]
+ {:status 200 :body (:body req)}
+ [:get "/query-string"]
+ {:status 200 :body (:query-string req)}
+ [:get "/cookie"]
+ {:status 200 :body "yay" :headers {"Set-Cookie" "foo=bar"}}
+ [:get "/bad-cookie"]
+ {:status 200 :body "yay"
+ :headers
+ {"Set-Cookie"
+ (str "DD-PSHARD=3; expires=\"Thu, 12-Apr-2018 06:40:25 GMT\"; "
+ "Max-Age=604800; Path=/; secure; HttpOnly")}}))
+
+(defn add-headers-if-requested [client]
+ (fn [req]
+ (let [resp (client req)
+ add-all (-> req :headers (get "add-headers"))
+ headers (:headers resp)]
+ (if add-all
+ (assoc resp :headers (assoc headers "got" (pr-str (:headers req))))
+ resp))))
+
+(defn run-server
+ []
+ (defonce server
+ (ring/run-jetty (add-headers-if-requested #'handler) {:port 18080 :join? false})))
+
+(defn localhost [path]
+ (str "http://localhost:18080" path))
+
+(def base-req
+ {:scheme :http
+ :server-name "localhost"
+ :server-port 18080})
+
+(defn request [req]
+ (core/request (merge base-req req)))
+
+(defn slurp-body [req]
+ (slurp (:body req)))
+
+(deftest ^:integration makes-get-request
+ (run-server)
+ (let [resp (request {:request-method :get :uri "/get"})]
+ (is (= 200 (:status resp)))
+ (is (= "get" (slurp-body resp)))))
+
+(deftest ^:integration dns-resolver
+ (run-server)
+ (let [custom-dns-resolver (doto (InMemoryDnsResolver.)
+ (.add "foo.bar.com" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))
+ resp (request {:request-method :get :uri "/get"
+ :server-name "foo.bar.com"
+ :dns-resolver custom-dns-resolver})]
+ (is (= 200 (:status resp)))
+ (is (= "get" (slurp-body resp)))))
+
+(deftest ^:integration dns-resolver-unknown-host
+ (run-server)
+ (let [custom-dns-resolver (doto (InMemoryDnsResolver.)
+ (.add "foo.bar.com" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))]
+ (is (thrown? java.net.UnknownHostException (request {:request-method :get :uri "/get"
+ :server-name "www.google.com"
+ :dns-resolver custom-dns-resolver})))))
+
+(deftest ^:integration dns-resolver-reusable-connection-manager
+ (run-server)
+ (let [custom-dns-resolver (doto (InMemoryDnsResolver.)
+ (.add "totallynonexistant.google.com"
+ (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))
+ cm (conn/make-reuseable-async-conn-manager {:dns-resolver custom-dns-resolver})
+ hc (core/build-async-http-client {} cm)]
+ (client/get "http://totallynonexistant.google.com:18080/json"
+ {:connection-manager cm
+ :http-client hc
+ :as :json
+ :async true}
+ (fn [resp]
+ (is (= 200 (:status resp)))
+ (is (= {:foo "bar"} (:body resp))))
+ (fn [e] (is false (str "failed with " e)))))
+ (let [custom-dns-resolver (doto (InMemoryDnsResolver.)
+ (.add "nonexistant.google.com" (into-array[(InetAddress/getByAddress (byte-array [127 0 0 1]))])))
+ cm (conn/make-reusable-conn-manager {:dns-resolver custom-dns-resolver})
+ hc (:http-client (client/get "http://nonexistant.google.com:18080/get"
+ {:connection-manager cm}))
+ resp (client/get "http://nonexistant.google.com:18080/json"
+ {:connection-manager cm
+ :http-client hc
+ :as :json})]
+ (is (= 200 (:status resp)))
+ (is (= {:foo "bar"} (:body resp)))))
+
+(deftest ^:integration save-request-option
+ (run-server)
+ (let [resp (request {:request-method :post
+ :uri "/post"
+ :body (util/utf8-bytes "contents")
+ :save-request? true})]
+ (is (= "/post" (-> resp :request :uri)))))
+
+(deftest ^:integration makes-head-request
+ (run-server)
+ (let [resp (request {:request-method :head :uri "/head"})]
+ (is (= 200 (:status resp)))
+ (is (nil? (:body resp)))))
+
+(deftest ^:integration sets-content-type-with-charset
+ (run-server)
+ (let [resp (client/request {:scheme :http
+ :server-name "localhost"
+ :server-port 18080
+ :request-method :get :uri "/content-type"
+ :content-type "text/plain"
+ :character-encoding "UTF-8"})]
+ (is (= "text/plain; charset=UTF-8" (:body resp)))))
+
+(deftest ^:integration sets-content-type-without-charset
+ (run-server)
+ (let [resp (client/request {:scheme :http
+ :server-name "localhost"
+ :server-port 18080
+ :request-method :get :uri "/content-type"
+ :content-type "text/plain"})]
+ (is (= "text/plain" (:body resp)))))
+
+(deftest ^:integration sets-arbitrary-headers
+ (run-server)
+ (let [resp (request {:request-method :get :uri "/header"
+ :headers {"x-my-header" "header-val"}})]
+ (is (= "header-val" (slurp-body resp)))))
+
+(deftest ^:integration sends-and-returns-byte-array-body
+ (run-server)
+ (let [resp (request {:request-method :post :uri "/post"
+ :body (util/utf8-bytes "contents")})]
+ (is (= 200 (:status resp)))
+ (is (= "contents" (slurp-body resp)))))
+
+(deftest ^:integration returns-arbitrary-headers
+ (run-server)
+ (let [resp (request {:request-method :get :uri "/get"})]
+ (is (string? (get-in resp [:headers "date"])))
+ (is (nil? (get-in resp [:headers "Date"])))))
+
+(deftest ^:integration returns-status-on-exceptional-responses
+ (run-server)
+ (let [resp (request {:request-method :get :uri "/error"})]
+ (is (= 500 (:status resp)))))
+
+(deftest ^:integration sets-socket-timeout
+ (run-server)
+ (try
+ (is (thrown? SocketTimeoutException
+ (client/request {:scheme :http
+ :server-name "localhost"
+ :server-port 18080
+ :request-method :get :uri "/timeout"
+ :socket-timeout 1})))))
+
+(deftest ^:integration delete-with-body
+ (run-server)
+ (let [resp (request {:request-method :delete :uri "/delete-with-body"
+ :body (.getBytes "foo bar")})]
+ (is (= 200 (:status resp)))))
+
+(deftest ^:integration self-signed-ssl-get
+ (let [server (ring/run-jetty handler
+ {:port 8081 :ssl-port 18082
+ :ssl? true
+ :join? false
+ :keystore "test-resources/keystore"
+ :key-password "keykey"})]
+ (try
+ (is (thrown? SunCertPathBuilderException
+ (client/request {:scheme :https
+ :server-name "localhost"
+ :server-port 18082
+ :request-method :get :uri "/get"})))
+ (let [resp (request {:request-method :get :uri "/get" :server-port 18082
+ :scheme :https :insecure? true})]
+ (is (= 200 (:status resp)))
+ (is (= "get" (String. (util/force-byte-array (:body resp))))))
+ (finally
+ (.stop server)))))
+
+(deftest ^:integration multipart-form-uploads
+ (run-server)
+ (let [bytes (util/utf8-bytes "byte-test")
+ stream (ByteArrayInputStream. bytes)
+ resp (request {:request-method :post :uri "/multipart"
+ :multipart [{:name "a" :content "testFINDMEtest"
+ :encoding "UTF-8"
+ :mime-type "application/text"}
+ {:name "b" :content bytes
+ :mime-type "application/json"}
+ {:name "d"
+ :content (file "test-resources/keystore")
+ :encoding "UTF-8"
+ :mime-type "application/binary"}
+ {:name "c" :content stream
+ :mime-type "application/json"}
+ {:name "e" :part-name "eggplant"
+ :content "content"
+ :mime-type "application/text"}]})
+ resp-body (apply str (map #(try (char %) (catch Exception _ ""))
+ (util/force-byte-array (:body resp))))]
+ (is (= 200 (:status resp)))
+ (is (re-find #"testFINDMEtest" resp-body))
+ (is (re-find #"application/json" resp-body))
+ (is (re-find #"application/text" resp-body))
+ (is (re-find #"UTF-8" resp-body))
+ (is (re-find #"byte-test" resp-body))
+ (is (re-find #"name=\"c\"" resp-body))
+ (is (re-find #"name=\"d\"" resp-body))
+ (is (re-find #"name=\"eggplant\"" resp-body))
+ (is (re-find #"content" resp-body))))
+
+(deftest ^:integration multipart-inputstream-length
+ (run-server)
+ (let [bytes (util/utf8-bytes "byte-test")
+ stream (ByteArrayInputStream. bytes)
+ resp (request {:request-method :post :uri "/multipart"
+ :multipart [{:name "c" :content stream :length 9
+ :mime-type "application/json"}]})
+ resp-body (apply str (map #(try (char %) (catch Exception _ ""))
+ (util/force-byte-array (:body resp))))]
+ (is (= 200 (:status resp)))
+ (is (re-find #"byte-test" resp-body))))
+
+(deftest parse-headers
+ (are [headers expected]
+ (let [iterator (BasicHeaderIterator.
+ (into-array BasicHeader
+ (map (fn [[name value]]
+ (BasicHeader. name value))
+ headers)) nil)]
+ (is (= (core/parse-headers iterator) expected)))
+
+ [] {}
+
+ [["Set-Cookie" "one"]] {"set-cookie" "one"}
+
+ [["Set-Cookie" "one"] ["set-COOKIE" "two"]]
+ {"set-cookie" ["one" "two"]}
+
+ [["Set-Cookie" "one"] ["serVer" "some-server"] ["set-cookie" "two"]]
+ {"set-cookie" ["one" "two"] "server" "some-server"}))
+
+(deftest ^:integration t-streaming-response
+ (run-server)
+ (let [stream (:body (request {:request-method :get :uri "/get" :as :stream}))
+ body (slurp stream)]
+ (is (= "get" body))))
+
+
+(deftest ^:integration throw-on-too-many-redirects
+ (run-server)
+ (let [resp (client/get (localhost "/redirect")
+ {:max-redirects 2 :throw-exceptions false
+ :redirect-strategy :none
+ :allow-circular-redirects true})]
+ (is (= 302 (:status resp))))
+
+ (let [resp (client/get (localhost "/redirect")
+ {:max-redirects 3
+ :redirect-strategy :graceful
+ :allow-circular-redirects true})]
+ (is (= 302 (:status resp)))
+ (is (= 3 (count (:trace-redirects resp))))
+ (is (= ["http://localhost:18080/redirect"
+ "http://localhost:18080/redirect"
+ "http://localhost:18080/redirect"]
+ (:trace-redirects resp))))
+
+ (is (thrown-with-msg? Exception #"Maximum redirects \(2\) exceeded"
+ (client/get (localhost "/redirect")
+ {:max-redirects 2
+ :throw-exceptions true
+ :allow-circular-redirects true})))
+ (is (thrown-with-msg? Exception #"Maximum redirects \(50\) exceeded"
+ (client/get (localhost "/redirect")
+ {:throw-exceptions true
+ :allow-circular-redirects true}))))
+
+(deftest ^:integration get-with-body
+ (run-server)
+ (let [resp (request {:request-method :get :uri "/get-with-body"
+ :body (.getBytes "foo bar")})]
+ (is (= 200 (:status resp)))
+ (is (= "foo bar" (String. (util/force-byte-array (:body resp)))))))
+
+(deftest ^:integration head-with-body
+ (run-server)
+ (let [resp (request {:request-method :head :uri "/head-with-body"
+ :body (.getBytes "foo")})]
+ (is (= 200 (:status resp)))
+ (is (= "foo" (get-in resp [:headers "body"])))))
+
+(deftest ^:integration t-clojure-output-coercion
+ (run-server)
+ (let [resp (client/get (localhost "/clojure") {:as :clojure})]
+ (is (= 200 (:status resp)))
+ (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}} (:body resp))))
+ (let [clj-resp (client/get (localhost "/clojure") {:as :auto})
+ edn-resp (client/get (localhost "/edn") {:as :auto})]
+ (is (= 200 (:status clj-resp) (:status edn-resp)))
+ (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
+ (:body clj-resp)
+ (:body edn-resp)))))
+
+(deftest ^:integration t-transit-output-coercion
+ (run-server)
+ (let [transit-json-resp (client/get (localhost "/transit-json") {:as :auto})
+ transit-msgpack-resp (client/get (localhost "/transit-msgpack")
+ {:as :auto})
+ bad-status-resp-default
+ (client/get (localhost "/transit-json-bad")
+ {:throw-exceptions false :as :transit+json})
+ bad-status-resp-always
+ (client/get (localhost "/transit-json-bad")
+ {:throw-exceptions false :as :transit+json
+ :coerce :always})
+ bad-status-resp-exceptional
+ (client/get (localhost "/transit-json-bad")
+ {:throw-exceptions false :as :transit+json
+ :coerce :exceptional})
+ empty-resp (client/get (localhost "/transit-json-empty")
+ {:throw-exceptions false :as :transit+json})]
+ (is (= 200
+ (:status transit-json-resp)
+ (:status transit-msgpack-resp)
+ (:status empty-resp)))
+ (is (= 400
+ (:status bad-status-resp-default)
+ (:status bad-status-resp-always)
+ (:status bad-status-resp-exceptional)))
+ (is (= {:foo "bar" :baz 7M :eggplant {:quux #{1 2 3}}}
+ (:body transit-json-resp)
+ (:body transit-msgpack-resp)))
+
+ (is (nil? (:body empty-resp)))
+
+ (is (= "[\"^ \", \"~:foo\",\"bar\"]"
+ (:body bad-status-resp-default)))
+ (is (= {:foo "bar"}
+ (:body bad-status-resp-always)))
+ (is (= {:foo "bar"}
+ (:body bad-status-resp-exceptional)))))
+
+(deftest ^:integration t-json-output-coercion
+ (run-server)
+ (let [resp (client/get (localhost "/json") {:as :json})
+ resp-array (client/get (localhost "/json-array") {:as :json})
+ resp-array-strict (client/get (localhost "/json-array") {:as :json-strict})
+ resp-large-array (client/get (localhost "/json-large-array") {:as :json})
+ resp-large-array-strict (client/get (localhost "/json-large-array") {:as :json-strict})
+ resp-str (client/get (localhost "/json")
+ {:as :json :coerce :exceptional})
+ resp-str-keys (client/get (localhost "/json") {:as :json-string-keys})
+ resp-strict-str-keys (client/get (localhost "/json")
+ {:as :json-strict-string-keys})
+ resp-auto (client/get (localhost "/json") {:as :auto})
+ bad-resp (client/get (localhost "/json-bad")
+ {:throw-exceptions false :as :json})
+ bad-resp-json (client/get (localhost "/json-bad")
+ {:throw-exceptions false :as :json
+ :coerce :always})
+ bad-resp-json2 (client/get (localhost "/json-bad")
+ {:throw-exceptions false :as :json
+ :coerce :unexceptional})]
+ (is (= 200
+ (:status resp)
+ (:status resp-array)
+ (:status resp-array-strict)
+ (:status resp-large-array)
+ (:status resp-large-array-strict)
+ (:status resp-str)
+ (:status resp-str-keys)
+ (:status resp-strict-str-keys)
+ (:status resp-auto)))
+ (is (= {:foo "bar"}
+ (:body resp)
+ (:body resp-auto)))
+ (is (= ["foo", "bar"]
+ (:body resp-array)))
+ (is (= {"foo" "bar"}
+ (:body resp-strict-str-keys)
+ (:body resp-str-keys)))
+ ;; '("foo" "bar") and ["foo" "bar"] compare as equal with =.
+ (is (vector? (:body resp-array)))
+ (is (vector? (:body resp-array-strict)))
+ (is (= "{\"foo\":\"bar\"}" (:body resp-str)))
+ (is (= 400
+ (:status bad-resp)
+ (:status bad-resp-json)
+ (:status bad-resp-json2)))
+ (is (= "{\"foo\":\"bar\"}" (:body bad-resp))
+ "don't coerce on bad response status by default")
+ (is (= {:foo "bar"} (:body bad-resp-json)))
+ (is (= "{\"foo\":\"bar\"}" (:body bad-resp-json2)))
+
+ (testing "lazily parsed stream completes parsing."
+ (is (= 100 (count (:body resp-large-array)))))
+ (is (= 100 (count (:body resp-large-array-strict))))))
+
+(deftest ^:integration t-ipv6
+ (run-server)
+ (let [resp (client/get "http://[::1]:18080/get")]
+ (is (= 200 (:status resp)))
+ (is (= "get" (:body resp)))))
+
+(deftest t-custom-retry-handler
+ (let [called? (atom false)]
+ (is (thrown? Exception
+ (client/post "http://localhost"
+ {:multipart [{:name "title" :content "Foo"}
+ {:name "Content/type"
+ :content "text/plain"}
+ {:name "file"
+ :content (file "/tmp/missingfile")}]
+ :retry-handler (fn [ex try-count http-context]
+ (reset! called? true)
+ false)})))
+ (is @called?)))
+
+;; super-basic test for methods that aren't used that often
+(deftest ^:integration t-copy-options-move
+ (run-server)
+ (let [resp1 (client/options (localhost "/options"))
+ resp2 (client/move (localhost "/move"))
+ resp3 (client/copy (localhost "/copy"))
+ resp4 (client/patch (localhost "/patch"))]
+ (is (= #{200} (set (map :status [resp1 resp2 resp3 resp4]))))
+ (is (= "options" (:body resp1)))
+ (is (= "move" (:body resp2)))
+ (is (= "copy" (:body resp3)))
+ (is (= "patch" (:body resp4)))))
+
+(deftest ^:integration t-json-encoded-form-params
+ (run-server)
+ (let [params {:param1 "value1" :param2 {:foo "bar"}}
+ resp (client/post (localhost "/post") {:content-type :json
+ :form-params params})]
+ (is (= 200 (:status resp)))
+ (is (= (json/encode params) (:body resp)))))
+
+(deftest ^:integration t-request-interceptor
+ (run-server)
+ (let [req-ctx (atom [])
+ {:keys [status trace-redirects] :as resp}
+ (client/get
+ (localhost "/get")
+ {:request-interceptor
+ (fn [^HttpRequest req ^HttpContext ctx]
+ (reset! req-ctx {:method (.getMethod req) :uri (.getURI req)}))})]
+ (is (= 200 status))
+ (is (= "GET" (:method @req-ctx)))
+ (is (= "/get" (.getPath (:uri @req-ctx))))))
+
+(deftest ^:integration t-response-interceptor
+ (run-server)
+ (let [saved-ctx (atom [])
+ {:keys [status trace-redirects] :as resp}
+ (client/get
+ (localhost "/redirect-to-get")
+ {:response-interceptor
+ (fn [^HttpResponse resp ^HttpContext ctx]
+ (let [^HttpInetConnection conn
+ (.getAttribute ctx ExecutionContext/HTTP_CONNECTION)]
+ (swap! saved-ctx conj {:remote-port (.getRemotePort conn)
+ :http-conn conn})))})]
+ (is (= 200 status))
+ (is (= 2 (count @saved-ctx)))
+ #_(is (= (count trace-redirects) (count @saved-ctx)))
+ (is (every? #(= 18080 (:remote-port %)) @saved-ctx))
+ (is (every? #(instance? HttpConnection (:http-conn %)) @saved-ctx))))
+
+(deftest ^:integration t-send-input-stream-body
+ (run-server)
+ (let [b1 (:body (client/post "http://localhost:18080/post"
+ {:body (ByteArrayInputStream. (.getBytes "foo"))
+ :length 3}))
+ b2 (:body (client/post "http://localhost:18080/post"
+ {:body (ByteArrayInputStream.
+ (.getBytes "foo"))}))
+ b3 (:body (client/post "http://localhost:18080/post"
+ {:body (ByteArrayInputStream.
+ (.getBytes "apple"))
+ :length 2}))]
+ (is (= b1 "foo"))
+ (is (= b2 "foo"))
+ (is (= b3 "ap"))))
+
+;; (deftest t-add-client-params
+;; (testing "Using add-client-params!"
+;; (let [ps {"http.conn-manager.timeout" 100
+;; "http.socket.timeout" 250
+;; "http.protocol.allow-circular-redirects" false
+;; "http.protocol.version" HttpVersion/HTTP_1_0
+;; "http.useragent" "clj-http"}
+;; setps (.getParams (doto (DefaultHttpClient.)
+;; (core/add-client-params! ps)))]
+;; (doseq [[k v] ps]
+;; (is (= v (.getParameter setps k)))))))
+
+;; Regression, get notified if something changes
+(deftest ^:integration t-known-client-params-are-unchanged
+ (let [params ["http.socket.timeout" CoreConnectionPNames/SO_TIMEOUT
+ "http.connection.timeout"
+ CoreConnectionPNames/CONNECTION_TIMEOUT
+ "http.protocol.version" CoreProtocolPNames/PROTOCOL_VERSION
+ "http.useragent" CoreProtocolPNames/USER_AGENT
+ "http.conn-manager.timeout" ClientPNames/CONN_MANAGER_TIMEOUT
+ "http.protocol.allow-circular-redirects"
+ ClientPNames/ALLOW_CIRCULAR_REDIRECTS]]
+ (doseq [[plaintext constant] (partition 2 params)]
+ (is (= plaintext constant)))))
+
+;; If you don't explicitly set a :cookie-policy, use
+;; CookiePolicy/BROWSER_COMPATIBILITY
+;; (deftest t-add-client-params-default-cookie-policy
+;; (testing "Using add-client-params! to get a default cookie policy"
+;; (let [setps (.getParams (doto (DefaultHttpClient.)
+;; (core/add-client-params! {})))]
+;; (is (= CookiePolicy/BROWSER_COMPATIBILITY
+;; (.getParameter setps ClientPNames/COOKIE_POLICY))))))
+
+;; If you set a :cookie-policy, the name of the policy is registered
+;; as (str (type cookie-policy))
+;; (deftest t-add-client-params-cookie-policy
+;; (testing "Using add-client-params! to get an explicitly set :cookie-policy"
+;; (let [setps (.getParams (doto (DefaultHttpClient.)
+;; (core/add-client-params!
+;; {:cookie-policy (constantly nil)})))]
+;; (is (.startsWith ^String
+;; (.getParameter setps ClientPNames/COOKIE_POLICY)
+;; "class ")))))
+
+
+;; This relies on connections to writequit.org being slower than 10ms, if this
+;; fails, you must have very nice internet.
+(deftest ^:integration sets-connection-timeout
+ (run-server)
+ (try
+ (is (thrown? SocketTimeoutException
+ (client/request {:scheme :http
+ :server-name "writequit.org"
+ :server-port 80
+ :request-method :get :uri "/"
+ :connection-timeout 10})))))
+
+(deftest ^:integration connection-pool-timeout
+ (run-server)
+ (client/with-connection-pool {:threads 1 :default-per-route 1}
+ (let [async-request #(future (client/request {:scheme :http
+ :server-name "localhost"
+ :server-port 18080
+ :request-method :get
+ :connection-timeout 1
+ :connection-request-timeout 1
+ :uri "/timeout"}))
+ is-pool-timeout-error?
+ (fn [req-fut]
+ (instance? org.apache.http.conn.ConnectionPoolTimeoutException
+ (try @req-fut (catch Exception e (.getCause e)))))
+ req1 (async-request)
+ req2 (async-request)
+ timeout-error1 (is-pool-timeout-error? req1)
+ timeout-error2 (is-pool-timeout-error? req2)]
+ (is (or timeout-error1 timeout-error2)))))
+
+(deftest ^:integration t-header-collections
+ (run-server)
+ (let [headers (-> (client/get "http://localhost:18080/headers"
+ {:headers {"foo" ["bar" "baz"]
+ "eggplant" "quux"}})
+ :body
+ json/decode)]
+ (is (= {"eggplant" "quux" "foo" "bar,baz"}
+ (select-keys headers ["foo" "eggplant"])))))
+
+(deftest ^:integration t-clojure-no-read-eval
+ (run-server)
+ (is (thrown? Exception (client/get (localhost "/clojure-bad") {:as :clojure}))
+ "Should throw an exception when reading clojure eval components"))
+
+(deftest ^:integration t-numeric-headers
+ (run-server)
+ (client/request {:method :get :url (localhost "/get") :headers {"foo" 2}}))
+
+(deftest ^:integration t-empty-response-coercion
+ (run-server)
+ (let [resp (client/get (localhost "/empty") {:as :clojure})]
+ (is (= (:body resp) nil)))
+ (let [resp (client/get (localhost "/empty") {:as :json})]
+ (is (= (:body resp) nil)))
+ (let [resp (client/get (localhost "/empty-gzip")
+ {:as :clojure})]
+ (is (= (:body resp) nil)))
+ (let [resp (client/get (localhost "/empty-gzip")
+ {:as :json})]
+ (is (= (:body resp) nil))))
+
+(deftest ^:integration t-trace-redirects
+ (run-server)
+ (let [resp-with-redirects
+ (client/request {:method :get
+ :url (localhost "/redirect-to-get")})
+
+ resp-with-graceful-redirects
+ (client/request {:method :get
+ :url (localhost "/redirect-to-get")
+ :redirect-strategy :graceful})
+
+ resp-without-redirects
+ (client/request {:method :get
+ :url (localhost "/redirect-to-get")
+ :redirect-strategy :none})]
+
+ (is (= (:trace-redirects resp-with-redirects)
+ ["http://localhost:18080/get"]))
+
+ (is (= (:trace-redirects resp-with-graceful-redirects)
+ ["http://localhost:18080/get"]))
+
+ (is (= (:trace-redirects resp-without-redirects) []))))
+
+(deftest t-request-config
+ (let [params {:conn-timeout 100 ;; deprecated
+ :connection-timeout 200 ;; takes precedence over `:conn-timeout`
+ :conn-request-timeout 300 ;; deprecated
+ :connection-request-timeout 400 ;; takes precedence over `:conn-request-timeout`
+ :socket-timeout 500
+ :max-redirects 600
+ :cookie-spec "foo"
+ :normalize-uri false}
+ request-config (core/request-config params)]
+ (is (= 200 (.getConnectTimeout request-config)))
+ (is (= 400 (.getConnectionRequestTimeout request-config)))
+ (is (= 500 (.getSocketTimeout request-config)))
+ (is (= 600 (.getMaxRedirects request-config)))
+ (is (= core/CUSTOM_COOKIE_POLICY (.getCookieSpec request-config)))
+ (is (false? (.isNormalizeUri request-config)))))
+
+(deftest ^:integration t-override-request-config
+ (run-server)
+ (let [called-args (atom [])
+ real-http-client core/build-http-client
+ http-context (HttpClientContext/create)
+ request-config (.build (RequestConfig/custom))]
+ (with-redefs
+ [core/build-http-client
+ (fn [& args]
+ (proxy [org.apache.http.impl.client.CloseableHttpClient] []
+ (execute [http-req context]
+ (swap! called-args conj [http-req context])
+ (.execute (apply real-http-client args) http-req context))))]
+ (client/request {:method :get
+ :url "http://localhost:18080/get"
+ :http-client-context http-context
+ :http-request-config request-config})
+
+ (let [context-for-request (last (last @called-args))]
+ (is (= http-context context-for-request))
+ (is (= request-config (.getRequestConfig context-for-request)))))))
+
+(deftest ^:integration test-custom-http-builder-fns
+ (run-server)
+ (let [resp (client/get (localhost "/get")
+ {:headers {"add-headers" "true"}
+ :http-builder-fns
+ [(fn [builder req]
+ (.setDefaultHeaders builder (:hdrs req)))]
+ :hdrs [(BasicHeader. "foo" "bar")]})]
+ (is (= 200 (:status resp)))
+ (is (.contains (get-in resp [:headers "got"]) "\"foo\" \"bar\"")
+ "Headers should have included the new default headers"))
+ (let [resp (promise)
+ error (promise)
+ f (client/get (localhost "/get")
+ {:async true
+ :headers {"add-headers" "true"}
+ :async-http-builder-fns
+ [(fn [builder req]
+ (.setDefaultHeaders builder (:hdrs req)))]
+ :hdrs [(BasicHeader. "foo" "bar")]}
+ resp error)]
+ (.get f)
+ (is (= 200 (:status @resp)))
+ (is (.contains (get-in @resp [:headers "got"]) "\"foo\" \"bar\"")
+ "Headers should have included the new default headers")
+ (is (not (realized? error)))))
+
+(deftest ^:integration test-custom-http-client-builder
+ (run-server)
+ (let [methods (atom nil)
+ resp (client/get
+ (localhost "/get")
+ {:http-client-builder
+ (-> (org.apache.http.impl.client.HttpClientBuilder/create)
+ (.setRequestExecutor
+ (proxy [org.apache.http.protocol.HttpRequestExecutor] []
+ (execute [request connection context]
+ (->> request
+ .getRequestLine
+ .getMethod
+ (swap! methods conj))
+ (proxy-super execute request connection context)))))})]
+ (is (= ["GET"] @methods))))
+
+(deftest ^:integration test-bad-redirects
+ (run-server)
+ (try
+ (client/get (localhost "/bad-redirect"))
+ (is false "should have thrown an exception")
+ (catch ProtocolException e
+ (is (.contains
+ (.getMessage e)
+ "Redirect URI does not specify a valid host name: https:///"))))
+ ;; async version
+ (let [e (atom nil)
+ latch (promise)]
+ (try
+ (.get
+ (client/get (localhost "/bad-redirect") {:async true}
+ (fn [resp]
+ (is false
+ (str "should not have been called but got" resp)))
+ (fn [err]
+ (reset! e err)
+ (deliver latch true)
+ nil)))
+ (catch Exception error
+ (is (.contains
+ (.getMessage error)
+ "Redirect URI does not specify a valid host name: https:///"))))
+ @latch
+ (is (.contains
+ (.getMessage @e)
+ "Redirect URI does not specify a valid host name: https:///")))
+ (try
+ (.get (client/get
+ (localhost "/bad-redirect")
+ {:async true
+ :validate-redirects false}
+ (fn [resp]
+ (is false
+ (str "should not have been called but got" resp)))
+ (fn [err]
+ (is false
+ (str "should not have been called but got" err))))
+ 1 TimeUnit/SECONDS)
+ (is false "should have thrown a timeout exception")
+ (catch TimeoutException te)))
+
+(deftest ^:integration test-reusable-http-client
+ (run-server)
+ (let [cm (conn/make-reuseable-async-conn-manager {})
+ hc (core/build-async-http-client {} cm)]
+ (client/get (localhost "/json")
+ {:connection-manager cm
+ :http-client hc
+ :as :json
+ :async true}
+ (fn [resp]
+ (is (= 200 (:status resp)))
+ (is (= {:foo "bar"} (:body resp)))
+ (is (= hc (:http-client resp))
+ "http-client is correctly reused"))
+ (fn [e] (is false (str "failed with " e)))))
+ (let [cm (conn/make-reusable-conn-manager {})
+ hc (:http-client (client/get (localhost "/get")
+ {:connection-manager cm}))
+ resp (client/get (localhost "/json")
+ {:connection-manager cm
+ :http-client hc
+ :as :json})]
+ (is (= 200 (:status resp)))
+ (is (= {:foo "bar"} (:body resp)))
+ (is (= hc (:http-client resp))
+ "http-client is correctly reused")))
+
+(deftest ^:integration t-cookies-spec
+ (run-server)
+ (try
+ (client/get (localhost "/bad-cookie"))
+ (is false "should have failed")
+ (catch org.apache.http.cookie.MalformedCookieException e))
+ (client/get (localhost "/bad-cookie") {:decode-cookies false})
+ (let [validated (atom false)
+ spec-provider (RFC6265CookieSpecProvider.)
+ resp (client/get (localhost "/cookie")
+ {:cookie-spec
+ (fn [http-context]
+ (proxy [org.apache.http.impl.cookie.CookieSpecBase] []
+ ;; Version and version header
+ (getVersion [] 0)
+ (getVersionHeader [] nil)
+ ;; parse headers into cookie objects
+ (parse [header cookie-origin]
+ (.parse (.create spec-provider http-context)
+ header cookie-origin))
+ ;; Validate a cookie, throwing MalformedCookieException if the
+ ;; cookies isn't valid
+ (validate [cookie cookie-origin]
+ (reset! validated true))
+ ;; Determine if a cookie matches the target location
+ (match [cookie cookie-origin] true)
+ ;; Format a list of cookies into a list of headers
+ (formatCookies [cookies] (java.util.ArrayList.))))})]
+ (is (= @validated true))))
+
+
+(deftest t-cache-config
+ (let [cc (core/build-cache-config
+ {:cache-config {:allow-303-caching true
+ :asynchronous-worker-idle-lifetime-secs 10
+ :asynchronous-workers-core 2
+ :asynchronous-workers-max 3
+ :heuristic-caching-enabled true
+ :heuristic-coefficient 1.5
+ :heuristic-default-lifetime 12
+ :max-cache-entries 100
+ :max-object-size 123
+ :max-update-retries 3
+ :revalidation-queue-size 2
+ :shared-cache false
+ :weak-etag-on-put-delete-allowed true}})]
+ (is (= true (.is303CachingEnabled cc)))
+ (is (= 10 (.getAsynchronousWorkerIdleLifetimeSecs cc)))
+ (is (= 2 (.getAsynchronousWorkersCore cc)))
+ (is (= 3 (.getAsynchronousWorkersMax cc)))
+ (is (= true (.isHeuristicCachingEnabled cc)))
+ (is (= 1.5 (.getHeuristicCoefficient cc)))
+ (is (= 12 (.getHeuristicDefaultLifetime cc)))
+ (is (= 100 (.getMaxCacheEntries cc)))
+ (is (= 123 (.getMaxObjectSize cc)))
+ (is (= 3 (.getMaxUpdateRetries cc)))
+ (is (= 2 (.getRevalidationQueueSize cc)))
+ (is (= false (.isSharedCache cc)))
+ (is (= true (.isWeakETagOnPutDeleteAllowed cc)))))
+
+(deftest ^:integration t-client-caching
+ (run-server)
+ (let [cm (conn/make-reusable-conn-manager {})
+ r1 (client/get (localhost "/get")
+ {:connection-manager cm :cache true})
+ client (:http-client r1)
+ r2 (client/get (localhost "/get")
+ {:connection-manager cm :http-client client :cache true})
+ r3 (client/get (localhost "/get")
+ {:connection-manager cm :http-client client :cache true})
+ r4 (client/get (localhost "/get")
+ {:connection-manager cm :http-client client :cache true})]
+ (is (= :CACHE_MISS (:cached r1)))
+ (is (= :VALIDATED (:cached r2)))
+ (is (= :VALIDATED (:cached r3)))
+ (is (= :VALIDATED (:cached r4))))
+ (let [cm (conn/make-reusable-conn-manager {})
+ r1 (client/get (localhost "/dont-cache")
+ {:connection-manager cm :cache true})
+ client (:http-client r1)
+ r2 (client/get (localhost "/dont-cache")
+ {:connection-manager cm :http-client client :cache true})
+ r3 (client/get (localhost "/dont-cache")
+ {:connection-manager cm :http-client client :cache true})
+ r4 (client/get (localhost "/dont-cache")
+ {:connection-manager cm :http-client client :cache true})]
+ (is (= :CACHE_MISS (:cached r1)))
+ (is (= :CACHE_MISS (:cached r2)))
+ (is (= :CACHE_MISS (:cached r3)))
+ (is (= :CACHE_MISS (:cached r4)))))
diff --git a/test/clj_http/test/headers.clj b/test/clj_http/test/headers_test.clj
similarity index 92%
rename from test/clj_http/test/headers.clj
rename to test/clj_http/test/headers_test.clj
index 6925797..a2c5e83 100644
--- a/test/clj_http/test/headers.clj
+++ b/test/clj_http/test/headers_test.clj
@@ -1,12 +1,11 @@
-(ns clj-http.test.headers
+(ns clj-http.test.headers-test
(:require [clj-http.client :as client]
[clj-http.headers :refer :all]
[clj-http.util :refer [lower-case-keys]]
[clojure.test :refer :all])
- (:import (javax.servlet.http HttpServletRequest
- HttpServletResponse)
- (org.eclipse.jetty.server Request Server)
- (org.eclipse.jetty.server.handler AbstractHandler)))
+ (:import [javax.servlet.http HttpServletRequest HttpServletResponse]
+ [org.eclipse.jetty.server Request Server]
+ org.eclipse.jetty.server.handler.AbstractHandler))
(deftest test-special-case
(are [expected given]
@@ -66,7 +65,13 @@
(is (= "baz" (:foo (merge (header-map :foo "bar")
{"Foo" "baz"}))))
(let [m-with-meta (with-meta m {:withmeta-test true})]
- (is (= (:withmeta-test (meta m-with-meta)) true)))))
+ (is (= (:withmeta-test (meta m-with-meta)) true)))
+
+ (testing "select-keys"
+ (are [expected keyset] (= expected (select-keys m keyset))
+ {"foo" "bar"} ["foo"]
+ {"foo" "bar"} ["foo" "non-existent-key"]
+ {"foo" "bar" "Foo" "bar" :foo "bar"} ["foo" "Foo" :foo]))))
(deftest test-empty
(testing "an empty header-map is a header-map"
diff --git a/test/clj_http/test/links.clj b/test/clj_http/test/links_test.clj
similarity index 81%
rename from test/clj_http/test/links.clj
rename to test/clj_http/test/links_test.clj
index 8b884f4..0e8fd41 100644
--- a/test/clj_http/test/links.clj
+++ b/test/clj_http/test/links_test.clj
@@ -1,4 +1,4 @@
-(ns clj-http.test.links
+(ns clj-http.test.links-test
(:require [clj-http.links :refer :all]
[clojure.test :refer :all]))
@@ -30,9 +30,9 @@
(is (not (contains? response :links))))))
(deftest t-multiple-link-headers
- (let [handler (link-handler ["<http://tmblr.co/Zl_A>; rel=shorturl"
- "<http://25.media.com/foo.png>; rel=icon"])
+ (let [handler (link-handler ["<http://example.com/Zl_A>; rel=shorturl"
+ "<http://example.com/foo.png>; rel=icon"])
resp (handler {})]
(is (= (:links resp)
- {:shorturl {:href "http://tmblr.co/Zl_A"}
- :icon {:href "http://25.media.com/foo.png"}}))))
+ {:shorturl {:href "http://example.com/Zl_A"}
+ :icon {:href "http://example.com/foo.png"}}))))
diff --git a/test/clj_http/test/multipart.clj b/test/clj_http/test/multipart_test.clj
similarity index 78%
rename from test/clj_http/test/multipart.clj
rename to test/clj_http/test/multipart_test.clj
index 9e888bf..7c309cf 100644
--- a/test/clj_http/test/multipart.clj
+++ b/test/clj_http/test/multipart_test.clj
@@ -1,9 +1,10 @@
-(ns clj-http.test.multipart
+(ns clj-http.test.multipart-test
(:require [clj-http.multipart :refer :all]
[clojure.test :refer :all])
- (:import (java.io File ByteArrayOutputStream ByteArrayInputStream)
- (org.apache.http.entity.mime.content FileBody StringBody ContentBody ByteArrayBody InputStreamBody)
- (java.nio.charset Charset)))
+ (:import [java.io ByteArrayInputStream ByteArrayOutputStream File]
+ java.nio.charset.Charset
+ [org.apache.http.entity.mime.content ByteArrayBody ContentBody FileBody InputStreamBody StringBody]
+ org.apache.http.util.EntityUtils))
(defn body-str [^StringBody body]
(-> body .getReader slurp))
@@ -28,12 +29,15 @@
(make-multipart-body {:content nil}))))
(testing "unsupported content type throws exception"
- (is (thrown-with-msg? Exception #"Unsupported type for multipart content: class java.lang.Object"
- (make-multipart-body {:content (Object.)}))))
+ (is (thrown-with-msg?
+ Exception
+ #"Unsupported type for multipart content: class java.lang.Object"
+ (make-multipart-body {:content (Object.)}))))
(testing "ContentBody content direct usage"
(let [contentBody (StringBody. "abc")]
- (is (identical? contentBody (make-multipart-body {:content contentBody})))))
+ (is (identical? contentBody
+ (make-multipart-body {:content contentBody})))))
(testing "StringBody"
@@ -49,7 +53,9 @@
(is (= (Charset/forName "ascii") (body-charset body)))))
(testing "can create StringBody with content and mime-type and encoding"
- (let [body (make-multipart-body {:content "abc" :mime-type "stream-body" :encoding "ascii"})]
+ (let [body (make-multipart-body {:content "abc"
+ :mime-type "stream-body"
+ :encoding "ascii"})]
(is (instance? StringBody body))
(is (= "abc" (body-str body)))
(is (= (Charset/forName "ascii") (body-charset body)))
@@ -58,11 +64,14 @@
(testing "ByteArrayBody"
(testing "exception thrown on missing name"
- (is (thrown-with-msg? Exception #"Multipart byte array body must contain at least :content and :name"
- (make-multipart-body {:content (byte-array [0 1 2])}))))
+ (is (thrown-with-msg?
+ Exception
+ #"Multipart byte array body must contain at least :content and :name"
+ (make-multipart-body {:content (byte-array [0 1 2])}))))
(testing "can create ByteArrayBody with name only"
- (let [body (make-multipart-body {:content (byte-array [0 1 2]) :name "testname"})]
+ (let [body (make-multipart-body {:content (byte-array [0 1 2])
+ :name "testname"})]
(is (instance? ByteArrayBody body))
(is (= "testname" (.getFilename body)))
(is (= [0 1 2] (vec (body-bytes body))))))
@@ -79,10 +88,12 @@
(testing "InputStreamBody"
(testing "exception thrown on missing name"
- (is (thrown-with-msg?
- Exception
- #"Multipart input stream body must contain at least :content and :name"
- (make-multipart-body {:content (ByteArrayInputStream. (byte-array [0 1 2]))}))))
+ (is
+ (thrown-with-msg?
+ Exception
+ #"Multipart input stream body must contain at least :content and :name"
+ (make-multipart-body
+ {:content (ByteArrayInputStream. (byte-array [0 1 2]))}))))
(testing "can create InputStreamBody with name and content"
(let [input-stream (make-input-stream 1 2 3)
@@ -102,7 +113,8 @@
(is (= "input-stream-body" (body-mime-type body)))
(is (identical? input-stream (.getInputStream body)))))
- (testing "can create input InputStreamBody name, content, mime-type and length"
+ (testing
+ "can create input InputStreamBody name, content, mime-type and length"
(let [input-stream (make-input-stream 1 2 3)
body (make-multipart-body {:content input-stream
:name "testname"
@@ -161,3 +173,11 @@
(is (= (Charset/forName "ascii") (body-charset body)))
(is (= test-file (.getFile body) ))
(is (= "testname" (.getFilename body)))))))
+
+(deftest test-multipart-content-charset
+ (testing "charset is nil if no multipart-charset is supplied"
+ (let [mp-entity (create-multipart-entity [] nil)]
+ (is (nil? (EntityUtils/getContentCharSet mp-entity)))))
+ (testing "charset is set if a multipart-charset is supplied"
+ (let [mp-entity (create-multipart-entity [] {:multipart-charset "UTF-8"})]
+ (is (= "UTF-8" (EntityUtils/getContentCharSet mp-entity))))))
diff --git a/test/clj_http/test/util.clj b/test/clj_http/test/util_test.clj
similarity index 55%
rename from test/clj_http/test/util.clj
rename to test/clj_http/test/util_test.clj
index 3dc1648..e886c4f 100644
--- a/test/clj_http/test/util.clj
+++ b/test/clj_http/test/util_test.clj
@@ -1,6 +1,9 @@
-(ns clj-http.test.util
+(ns clj-http.test.util-test
(:require [clj-http.util :refer :all]
- [clojure.test :refer :all]))
+ [clojure.java.io :as io]
+ [clojure.test :refer :all])
+ (:import org.apache.commons.io.input.NullInputStream
+ org.apache.commons.io.IOUtils))
(deftest test-lower-case-keys
(are [map expected]
@@ -38,6 +41,27 @@
" application/json; charset=UTF-8 "
{:content-type :application/json
:content-type-params {:charset "UTF-8"}}
+ " application/json; charset=\"utf-8\" "
+ {:content-type :application/json
+ :content-type-params {:charset "utf-8"}}
"text/html; charset=ISO-8859-4"
{:content-type :text/html
:content-type-params {:charset "ISO-8859-4"}}))
+
+(deftest test-force-byte-array
+ (testing "empty InputStream returns empty byte-array"
+ (is (= 0 (alength (force-byte-array (NullInputStream. 0))))))
+ (testing "InputStream contain bytes for JPEG file is coereced properly"
+ (let [jpg-path "test-resources/small.jpg"]
+ ;; coerce to seq to force byte-by-byte comparison
+ (is (= (seq (IOUtils/toByteArray (io/input-stream jpg-path)))
+ (seq (force-byte-array (io/input-stream jpg-path))))))))
+
+(deftest test-gunzip
+ (testing "with input streams"
+ (testing "with empty stream, does not apply gunzip stream"
+ (is (= "" (slurp (gunzip (force-stream (byte-array 0)))))))
+ (testing "with non-empty stream, gunzip decompresses data"
+ (let [data "hello world"]
+ (is (= data
+ (slurp (gunzip (force-stream (gzip (.getBytes data)))))))))))
diff --git a/test/log4j.properties b/test/log4j.properties
deleted file mode 100755
index de3a290..0000000
--- a/test/log4j.properties
+++ /dev/null
@@ -1,26 +0,0 @@
-#############
-# Appenders #
-#############
-
-# standard out appender
-log4j.appender.C = org.apache.log4j.ConsoleAppender
-log4j.appender.C.layout = org.apache.log4j.PatternLayout
-log4j.appender.C.layout.ConversionPattern = %d | ES | %-5p | [%t] | %c | %m%n
-
-# daily rolling file appender
-log4j.appender.F = org.apache.log4j.FileAppender
-log4j.appender.F.File = http.log
-log4j.appender.F.Append = true
-log4j.appender.F.layout = org.apache.log4j.PatternLayout
-log4j.appender.F.layout.ConversionPattern = %d | CLJ-HTTP | %-5p | [%t] | %c | %m%n
-
-###########
-# Loggers #
-###########
-
-# default
-log4j.rootLogger = DEBUG, F
-
-# Things
-log4j.logger.org.apache.http = DEBUG
-log4j.logger.org.apache.http.wire = INFO
diff --git a/test/log4j2.properties b/test/log4j2.properties
new file mode 100755
index 0000000..56f6f01
--- /dev/null
+++ b/test/log4j2.properties
@@ -0,0 +1,19 @@
+status = error
+dest = err
+name = PropertiesConfig
+
+filter.threshold.type = ThresholdFilter
+filter.threshold.level = debug
+
+appender.console.type = Console
+appender.console.name = STDOUT
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d | %-5p | [%t] | %c | %m%n
+
+rootLogger.level = info
+rootLogger.appenderRef.stdout.ref = STDOUT
+
+# Set this to debug to log all data to/from server
+# See https://hc.apache.org/httpcomponents-client-4.5.x/logging.html
+logger.wire.name = org.apache.http.wire
+logger.wire.level = info
\ No newline at end of file
Debdiff
[The following lists of changes regard files as different if they have different names, permissions or owners.]
Files in second set of .debs but not in first
-rw-r--r-- root/root /usr/share/doc/libclj-http-clojure/CONTRIBUTING.md -rw-r--r-- root/root /usr/share/java/clj-http-3.12.3+git20221108.0.1cc8be6.jar lrwxrwxrwx root/root /usr/share/java/clj-http.jar -> clj-http-3.12.3+git20221108.0.1cc8be6.jar
Files in first set of .debs but not in second
-rw-r--r-- root/root /usr/share/java/clj-http-2.3.0.jar lrwxrwxrwx root/root /usr/share/java/clj-http.jar -> clj-http-2.3.0.jar
No differences were encountered in the control files