Awesome
Manticore
Note: While I'll continue to maintain the library here, I've moved the canonical copy to Gitlab at https://gitlab.com/cheald/manticore - it is preferred that you submit issues and PRs there.
Manticore is a fast, robust HTTP client built on the Apache HTTPClient libraries. It is only compatible with JRuby.
Installation
Add this line to your application's Gemfile:
gem 'manticore', platform: :jruby
And then execute:
$ bundle
Or install it yourself as:
$ gem install manticore
Documentation
Documentation is available at rubydoc.info.
Performance
Manticore is very fast.
Major Features
As it's built on the Apache Commons HTTP components, Manticore is very rich. It includes support for:
- Keepalive connections (and connection pooling)
- Transparent gzip and deflate handling
- Transparent cookie handling
- Both synchronous and asynchronous execution models
- Lazy evaluation
- Authentication
- Proxy support
- SSL
Usage
Quick Start
If you don't want to worry about setting up and maintaining client pools, Manticore comes with a facade that you can use to start making requests right away:
get_body = Manticore.get("http://www.google.com/", query: {q: "kittens"}).body
post_body = Manticore.post("http://www.google.com/", params: {q: "kittens"}).body
# Or
get_body = Manticore.http(:get, "http://www.google.com/").body
This is threadsafe and automatically backed with a pool, so you can execute Manticore.get
in multiple threads without harming performance.
Alternately, you can mix the Manticore::Facade
into your own class for similar behavior:
class MyClient
include Manticore::Facade
include_http_client user_agent: "MyClient/1.0"
end
response_code = MyClient.get("http://www.google.com/").code
Mixing the client into a class will create a new pool. If you want to share a single pool between clients, specify the shared_pool
option:
class MyClient
include Manticore::Facade
include_http_client shared_pool: true
end
class MyOtherClient
include Manticore::Facade
include_http_client shared_pool: true
end
For detailed documentation, see the full Manticore::Client documentation.
Configuring clients
Rather than using the Facade, you can create your own standalone Client instances. When you create a Client
, you will pass various parameters that it will use to set up the pool.
client = Manticore::Client.new(request_timeout: 5,
connect_timeout: 5,
socket_timeout: 5,
pool_max: 10,
pool_max_per_route: 2)
Then, you can make requests from the client. Pooling and route maximum constraints are automatically managed:
response = client.get("http://www.google.com/")
body = response.body
It is recommend that you instantiate a client once, then re-use it, rather than instantiating a new client per request.
Additionally, if you pass a block to the initializer, the underlying HttpClientBuilder and RequestConfig.Builder will be yielded so that you can operate on them directly:
client = Manticore::Client.new(socket_timeout: 5) do |http_client_builder, request_builder|
http_client_builder.disable_redirect_handling
end
Pools
You've seen "pools" mentioned a few times. Manticore creates and configures a PoolingHttpClientConnectionManager
which all requests are run through. The advantage here is that configuration and setup is performed once, and this lets clients take advantage of things like keepalive,
per-route concurrency limits, and other neat things. In general, you should create one Manticore::Client
instance per unique configuration needed. For example, you might have an app that performs 2 functions:
- General HTTP requesting from the internet-at-large
- Communication with a backend service over SSL, using a custom trust store
To set this up, you might create 2 pools, each configured for the task:
general_http_client = Manticore::Client.new(connect_timeout: 10,
socket_timeout: 10,
request_timeout: 10,
follow_redirects: true,
max_per_route: 2)
# With an OpenSSL CA store
proxied_backend_client = Manticore::Client.new(proxy: "https://backend.internal:4242",
ssl: { ca_file: "my_certs.pem" })
# Or with a .jks truststore
proxied_backend_client = Manticore::Client.new(proxy: "https://backend.internal:4242",
ssl: {
truststore: "./truststore.jks",
truststore_password: "s3cr3t"
})
This would create 2 separate request pools; the first would be configured with generous timeouts and redirect following, and would use the system default trust stores (ie, the normal certs used to verify SSL certificates with the normal certificate authorities). Additionally, it will only permit 2 concurrent requests to a given domain ("route") at a time; this can be nice for web crawling or fetching against rate-limited APIs, to help you stay under your rate limits even when executing in a parallel context. The second client would use a custom trust store to recognize certs signed with your internal CA, and would proxy all requests through an internal server.
Creating pools is expensive, so you don't want to be doing it for each request. Instead, you should set up your pools once and then re-use them. Clients and their backing pools are thread-safe, so feel free to set them up once before you start performing parallel operations.
Background requests
You might want to fire off requests without blocking your calling thread. You can do this with Client#background
:
response = client.background.get("http://google.com")
.on_success {|response| puts response.code }
response.call # The request is now running, but the calling thread isn't blocked. The on_success handler will be evaluated whenever
# the request completes.
Parallel execution
Manticore can perform concurrent execution of multiple requests. In previous versions of Manticore, this was called "async". We now call these
"parallel" or "batch" requests. Client#async
, Client#parallel
, and Client#batch
are equivalent.
client = Manticore::Client.new
# These aren't actually executed until #execute! is called.
# You can define response handlers on the not-yet-resolved response object:
response = client.parallel.get("http://www.yahoo.com")
response.on_success do |response|
puts "The length of the Yahoo! homepage is #{response.body.length}"
end
response.on_failure do |response|
puts "http://www.nooooooooooooooo.com/"
end
# ...or even by chaining them onto the call
client.parallel.get("http://bing.com")
.on_success {|r| puts r.code }
.on_failure {|e| puts "on noes!" }
.on_complete { puts "Job's done!" }
client.execute!
Lazy Evaluation
Manticore attempts to avoid doing any actual work until right before you need results. As a result, responses are lazy-evaluated as late as possible. The following rules apply:
- Synchronous and parallel/batch responses are synchronously evaluted when you call an accessor on them, like
#body
or#headers
, or invoke them with#call
- Synchronous and background responses which pass a handler block are evaluated immediately. Sync responses will block the calling thread until complete, while background responses will not block the calling thread.
- Parallel/batch responses are evaluated when you call
Client#execute!
. Responses which have been previously evaluted by calling an accessor like#body
or which have been manually called with#call
will not be re-requested. - Background responses are evaluated when you call
#call
and return aFuture
, on which you can call#get
to synchronously get the resolved response.
As a result, this allows you to attach handlers to synchronous, background, and parallel responses in the same fashion:
## Standard/sync requests
response = client.get("http://google.com")
response.on_success {|r| puts "Success!" } # Because the response isn't running yet, we can attach a handler
body = response.body # When you access information from the response, the request finally runs
## Parallel requests
response1 = client.parallel.get("http://google.com")
response2 = client.parallel.get("http://yahoo.com")
response1.on_success {|response| puts "Yay!" }
response2.on_failure {|exception| puts "Whoops!" }
client.execute! # Nothing runs until we call Client#execute!
body = response1.body # Now the responses are resolved and we can get information from them
## Background requests
request = client.background.get("http://google.com")
request.on_success {|r| puts "Success!" } # We can attach handlers before the request is kicked off
future = request.call # We invoke #call on it to fire it off.
response = future.get # You can get the Response via Future#get. This will block the calling thread until it resolves, though.
If you want to immediately evaluate a request, you can either pass a handler block to it, or you can call #call
to fire it off:
# This will evaluate immediately
client.get("http://google.com") {|r| r.body }
# As will this, via explicit invocation of #call
client.get("http://google.com").call
For batch/parallel/async requests, a passed block will be treated as the on_success handler, but will not cause the request to be immediately invoked:
# This will not evaluate yet
client.batch.get("http://google.com") {|r| puts "Fetched Google" }
# ...but now it does.
client.execute!
Stubbing
Manticore provides a stubbing interface somewhat similar to Typhoeus'
client.stub("http://google.com", body: "response body", code: 200)
client.get("http://google.com") do |response|
response.body.should == "response body"
end
client.clear_stubs!
This works for parallel/batch/async requests as well:
client.stub("http://google.com", body: "response body", code: 200)
# The request to google.com returns a stub as expected
client.parallel.get("http://google.com").on_success do |response|
response.should be_a Manticore::ResponseStub
end
# Since yahoo.com isn't stubbed, a full request will be performed
client.parallel.get("http://yahoo.com").on_success do |response|
response.should be_a Manticore::Response
end
client.clear_stubs!
If you don't want to worry about stub teardown, you can just use #respond_with
, which will stub the next
response the client makes with a ResponseStub rather than permitting it to execute a remote request.
client.respond_with(body: "body").get("http://google.com") do |response|
response.body.should == "body"
end
You can also chain proxies to, say, stub an parallel request:
response = client.parallel.respond_with(body: "response body").get("http://google.com")
client.execute!
response.body.should == "response body"
Additionally, you can stub and unstub individual URLs as desired:
client.stub("http://google.com", body: "response body", code: 200)
client.stub("http://yahoo.com", body: "response body", code: 200)
# The request to google.com returns a stub as expected
client.get("http://google.com") do |response|
response.should be_a Manticore::ResponseStub
end
# After this point, yahoo will remain stubbed, while google will not.
client.unstub("http://google.com")
Faraday Adapter
Manticore includes a Faraday adapter. To use it:
require 'faraday/adapter/manticore'
Faraday.new(...) do |faraday|
faraday.adapter :manticore
end
Contributing
- Fork it
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request