Home

Awesome

Build Status

Trapperkeeper Webserver Service

This project provides a webserver service for use with the trapperkeeper service framework To use this service in your trapperkeeper application, simply add this project as a dependency in your leiningen project file:

Clojars Project

Then add the webserver service to your bootstrap.cfg file, via:

puppetlabs.trapperkeeper.services.webserver.jetty9-service/jetty9-service

Note that this implementation of the :WebserverService interface is based on Jetty 9, which contains performance improvements over previous versions of Jetty that may be significant depending on your application. This service requires JRE 1.7 or greater; however, the interface is intended to be agnostic to the underlying web server implementation. We also provide a Jetty 7 version of the service, which can be used interchangeably with this one and will support older JDKs. You should only need to change your lein dependencies and your bootstrap.cfg file--no code changes.

The web server is configured via the trapperkeeper configuration service; so, you can control various properties of the server (ports, SSL, etc.) by adding a webserver section to one of your Trapperkeeper configuration files, and setting various properties therein. For more info, see Configuring the Webserver. It is possible to configure both a single webserver or multiple webservers.

The webserver-service currently supports web applications built using Clojure's Ring library and Java's Servlet API. There is also an experimental webserver service that supports loading ruby Rack applications via JRuby; for more info, see the trapperkeeper-ruby project.

Example code

Four examples are included with this project:

Service Protocol

This is the protocol for the current implementation of the :WebserverService:

(defprotocol WebserverService
  (add-context-handler [this base-path context-path] [this base-path context-path options])
  (add-ring-handler [this handler path] [this handler path options])
  (add-websocket-handler [this handlers path] [this handler path options])
  (add-servlet-handler [this servlet path] [this servlet path options])
  (add-war-handler [this war path] [this war path options])
  (add-proxy-route [this target path] [this target path options])
  (override-webserver-settings! [this overrides] [this server-id overrides])
  (get-registered-endpoints [this] [this server-id])
  (log-registered-endpoints [this] [this server-id])
  (join [this] [this server-id])

Here is a bit more info about each of these functions:

add-ring-handler

add-ring-handler takes two arguments: [handler path]. The handler argument is just a normal Ring application (the same as what you would pass to run-jetty if you were using the ring-jetty-adapter). The path is a URL prefix / context string that will be prepended to all your handler's URLs; this is key to allowing the registration of multiple handlers in the same web server without the possibility of URL collisions. So, for example, if your ring handler has routes /foo and /bar, and you call:

(add-ring-handler my-app "/my-app")

Then your routes will be served at /my-app/foo and my-app/bar.

You may specify "" as the value for path if you are only registering a single handler and do not need to prefix the URL.

There is also a three argument version of this function which takes these arguments: [handler path options]. options is a map containing optional keys.

Here's an example of how to use the :WebserverService:

(defservice MyWebService
   [[:WebserverService add-ring-handler]]
   ;; initialization
   (init [this context]
      (add-ring-handler my-app "/my-app")
      context))

This would add your ring handler to the default server at endpoint "/my-app". Alternatively, if you did this:

(defservice MyWebService
   [[:WebserverService add-ring-handler]]
   ;; initialization
   (init [this context]
      (add-ring-handler my-app "/my-app" {:server-id :foo})
      context))

it would add your ring handler to the server with id :foo at endpoint "/my-app", rather than the default server.

NOTE FOR COMPOJURE APPS: If you are using compojure, it's important to note that compojure requires use of the context macro in order to support nested routes. So, if you're not already using context, you will need to do something like this:

(ns foo
   (:require [compojure.core :as c]
   ;;...
   ))

(defservice MyWebService
   [[:WebserverService add-ring-handler]]
   ;; initialization
   (init [this svc-context]
        (let [context-path "/my-app"
              context-app  (c/context context-path [] my-compojure-app)]
            (add-ring-handler context-app context-path))
        svc-context))
Request URI Normalization

The :normalize-request-uri setting, which can be provided in the options argument for an add handler call, controls whether or not the path component from the HTTP request URI is normalized. The value for the setting is expected to be a boolean.

When set to false (the default value), the "raw" path component will be used by the webserver when evaluating a request to the handler.

When set to true, the path component that the webserver evaluates for a request to the handler will have been "normalized". For a Ring request handler, the "normalized" value (instead of the "raw" value) will be associated with the :uri key in the Ring request map. For a Servlet request handler, the "normalized" value (instead of the "raw" value) will be returned from a call made to the getRequestURI method on the HttpServletRequest object.

The following steps, in order, are performed against the raw path component when the :normalize-request-uri setting is true:

  1. URL (percent) decode the path, assuming any percent-encodings represent UTF-8 characters.

For example:

/foo//bar/%2E%2E/ba%7A => /foo//bar/../baz

If a non-percent encoded semicolon character, U+003B, is found in the path during the percent decoding step, that character and all following characters will be removed from the resulting path, unless there is another forward slash, in which case the characters from the semicolon to the next forward slash (including the semicolon) will be removed.

For example:

/foo//bar/%2E%2E/ba%7A;bim => /foo//bar/../baz

/foo/bar;bar=chocolate/baz;baz=bim => /foo/bar/baz

Requests intending to include a semicolon in the path should percent-encode the semicolon. In this case, the server will preserve the semicolon after the decoding step.

For example:

/foo//bar/%2E%2E/ba%7A%3Bbim => /foo//bar/../baz;bim

If the request has malformed content, e.g., partially-formed percent-encoded characters like '%A%B', an HTTP 400 (Bad Request) error will be returned.

If the request has invalid % encoded UTF-8 characters, the path will be decoded as an ISO-8859-1 encoded string.

  1. Check the percent-decoded path for any relative path segments ('..' or '.').

If one or more relative path segments are found, an HTTP 400 (Bad Request) error will be returned.

For example, an error would be returned for any of the following paths:

.
..
/foo//bar/../baz
/foo//./bar/baz

The following paths would not be considered to contain relative paths:

/foo//bar/baz
/foo//bar/.../baz
/foo//bar/a.b/baz
/foo//bar/a..b/baz
  1. Compact any repeated forward slash characters in a path.

For example:

/foo//bar/baz => /foo/bar/baz
/foo/bar////baz => /foo/bar/baz

The following example shows the result after normalization of a URI request path which includes repeated forward slash characters which have been percent-encoded:

/foo%2F%2Fbar/ba%7A => /foo/bar/baz

add-context-handler

add-context-handler takes two arguments: [base-path context-path]. The base-path argument is a URL string pointing to a location containing static resources which are made accessible at the context-path URL prefix.

For example, to make your CSS files stored in the resources/css directory available at /css:

(defservice MyWebService
   [[:WebserverService add-context-handler]]
   ;; initialization
   (init [this context]
      (add-context-handler "resources/css" "/css")
      context))

There is also a three argument version of the function which takes these arguments: [base-path context-path options], where the first two arguments are the same as in the two argument version and options is a map containing five optional keys, :server-id, :redirect-if-no-trailing-slash, :normalize-request-uri, :follow-links, and :context-listeners.

The value stored in :server-id specifies which server to add the context handler to, similar to how it is done in add-ring-handler. Again, like add-ring-handler, if this key is absent or the two argument version is called, the context handler will be added to the default server. Calling the two-argument version or leaving out :server-id will not work in a multiserver set-up if no default server is specified.

The value stored in :redirect-if-no-trailing-slash is a boolean indicating whether or not to redirect when a request is made to this handler without a trailing slash, just like with add-ring-handler. Again, this defaults to false.

The value stored in :normalize-request-uri is a boolean indicating whether or not the request URI should be normalized before it is made available to the handler. See the Request URI Normalization section for more information on the normalization process.

The value stored in :follow-links is a boolean indicating whether or not symbolic links should be served. The service does NOT serve symbolic links by default.

The value stored in :context-listeners is a list of objects implementing the [ServletContextListener] (http://docs.oracle.com/javaee/7/api/javax/servlet/ServletContextListener.html) interface. These listeners are registered with the context created for serving the static content and receive notifications about the lifecycle events in the context as defined in the ServletContextListener interface. Of particular interest is the contextInitialized event notification as it provides access to the configuration of the context through the methods defined in the [ServletContext] (http://docs.oracle.com/javaee/7/api/javax/servlet/ServletContext.html) interface. This opens up wide possibilities for customizing the context - in an extreme case the context originally capable of serving just the static content can be changed through this mechanism to a fully dynamic web application (in fact this very mechanism is used in the [trapperkeeper-ruby] (https://github.com/puppetlabs/trapperkeeper-ruby) project to turn the context into a container for hosting an arbitrary ruby rack application - see [here] (https://github.com/puppetlabs/trapperkeeper-ruby/blob/master/src/clojure/puppetlabs/trapperkeeper/services/rack_jetty/rack_jetty_service.clj)).

add-servlet-handler

add-servlet-handler takes two arguments: [servlet path]. The servlet argument is a normal Java Servlet. The path is the URL prefix at which the servlet will be registered.

There is also a three argument version of the function which takes these arguments: [servlet path options], where the first two arguments are the same as in the two argument version and options is a map containing four optional keys, :server-id, :redirect-if-no-trailing-slash, normalize-request-uri, and :servlet-init-params.

As in add-ring-handler, :server-id specifies which server to add the handler to. If :server-id is absent or the two-argument function is called, the servlet handler will be added to the default server. Calling the two-argument version or leaving out :server-id will not work in a multiserver set-up if no default server is specified.

The value stored in :redirect-if-no-trailing-slash is a boolean indicating whether or not to redirect when a request is made to this handler without a trailing slash, just like with add-ring-handler. Again, this defaults to false.

The value stored in :normalize-request-uri is a boolean indicating whether or not the request URI should be normalized before it is made available to the handler. See the Request URI Normalization section for more information on the normalization process.

The value stored at the :servlet-init-params key is a map of servlet init parameters.

For example, to host a servlet at /my-app:

(ns foo
    ;; ...
    (:import [bar.baz SomeServlet]))

(defservice MyWebService
  [[:WebserverService add-servlet-handler]]
  ;; initialization
  (init [this context]
    (add-servlet-handler (SomeServlet. "some config") "/my-app")
    context))

For more information see the example servlet app.

add-websocket-handler

NOTE: Websockets support is currently an experimental feature; the API for websockets support may be subject to minor changes in a future release.

add-websocket-handler takes two arguments: [handlers path]. The handlers is a map of callbacks to invoke when handling a websocket session. The path is the URL prefix where this websocket servlet will be registered.

The possible callbacks for the handlers map are:

{:on-connect (fn [ws])
 :on-error   (fn [ws error])
 :on-close   (fn [ws status-code reason])
 :on-text    (fn [ws text])
 :on-bytes   (fn [ws bytes offset len])}

Querying data or sending messages over the websocket is supported by the functions of WebSocketProtocol protocol from the puppetlabs.experimental.websocket.client namespace:

(connected? [this]
  "Returns a boolean indicating if the session is currently connected")
(send! [this msg]
  "Send a message to the websocket client")
(close! [this] [this code reason]
  "Close the websocket session")
(remote-addr [this]
  "Find the remote address of a websocket client")
(ssl? [this]
  "Returns a boolean indicating if the session was established by wss://")
(peer-certs [this]
  "Returns an array of X509Certs presented by the ssl peer, if any")
(request-path [this]
  "Returns the URI path used in the websocket upgrade request to the server"))

For example, to provide a simple websockets echo service as /wsecho:

(ns foo
   (:require [puppetlabs.experimental.websockets.client :as ws-client]))

(def echo-handlers
  {:on-text (fn [ws text] (ws-client/send! ws text))})

(defservice wsecho-webservice
  [[:WebserverService add-websocket-handler]]
  (init [this context]
    (add-websocket-handler echo-handlers "/wsecho")
    context))

add-war-handler

add-war-handler takes two arguments: [war path]. The war is the file path or the URL to a WAR file. The path is the URL prefix at which the WAR will be registered.

For example, to host resources/cas.war WAR at /cas:

(defservice cas-webservice
  [[:WebserverService add-war-handler]]
  (init [this context]
    (add-war-handler "resources/cas.war" "/cas")
    context))

There is also a three-argument version that takes these parameters: [war path options]. options is a map containing three optional keys, :server-id, :redirect-if-no-trailing-slash, and :normalize-request-uri.

As with add-ring-handler, this determines which server the handler is added to. If this key is absent or the two argument version is called, the handler will be added to the default server. Calling the two-argument version or leaving out :server-id will not work in a multiserver set-up if no default server is specified.

The value stored in :redirect-if-no-trailing-slash is a boolean indicating whether or not to redirect when a request is made to this handler without a trailing slash, just like with add-ring-handler. Again, this defaults to false.

The value stored in :normalize-request-uri is a boolean indicating whether or not the request URI should be normalized before it is made available to the handler. See the Request URI Normalization section for more information on the normalization process.

add-proxy-route

add-proxy-route is used to configure certain the server as a reverse proxy for certain routes. This function will accept two or three arguments: [target path], or [target path options].

path is the URL prefix for requests that you wish to proxy.

target is a map that controls how matching requests will be proxied; here are the keys required in the target map:

options, if provided, is a map containing optional configuration for the proxy route:

Simple example:

(defservice foo-service
  [[:WebserverService add-proxy-route]]
  (init [this context]
    (add-proxy-route
        {:host "localhost"
         :port 10000
         :path "/bar"}
        "/foo")
    context))

In this example, all incoming requests with a prefix of /foo will be proxied to localhost:10000, with a prefix of /bar, using the same scheme (HTTP/HTTPS) that the original request used, and using the SSL context of the main webserver.

So, e.g., an HTTPS request to the main webserver at /foo/hello-world would be proxied to https://localhost:10000/bar/hello-world.

A slightly more complex example:

(defservice foo-service
  [[:WebserverService add-proxy-route]]
  (init [this context]
    (add-proxy-route
        {:host "localhost"
         :port 10000
         :path "/bar"}
        "/foo"
        {:scheme :https
         :ssl-config {:ssl-cert    "/tmp/cert.pem"
                      :ssl-key     "/tmp/key.pem"
                      :ssl-ca-cert "/tmp/ca.pem"}})
    context))

In this example, all incoming requests with a prefix of foo will be proxied to https://localhost:10000/bar. We'll proxy using HTTPS even if the original request was HTTP, and we'll use the three pem files in /tmp to configure the HTTPS client, regardless of the SSL configuration of the main web server.

#####:rewrite-uri-callback-fn

This option lets you provide a function to manipulate the rewritten target URI. The function is called in the overridden implementation of rewriteURI method after the target URI is computed. It must take two arguments, [target-uri req], where target-uri is a URI and req is an HttpServletRequest. target-uri will be modified and returned by the function.

An example with a rewrite URI callback function:

(defservice foo-service
  [[:WebserverService add-proxy-route]]
  (init [this context]
    (add-proxy-route
        {:host "localhost"
         :port 10000
         :path "/bar"}
        "/foo"
        {:rewrite-uri-callback-fn (fn [target-uri req]
          (if-not (= "GET" (.getMethod req))
            (URI. "http://localhost:11111/errors/unsupported-method")
            target-uri))})
    context))

In this example, all incoming requests with a method other than GET will be proxied to http://localhost:11111/errors/unsupported-method.

#####:callback-fn

This option lets you provide a function to manipulate the request object. The function will be passed to the customizeProxyRequest method. It must take two arguments, [proxy-req req], where proxy-req is a Request and req is an HttpServletRequest. proxy-req will be modified and returned by the function.

An example with a callback function:

(defservice foo-service
  [[:WebserverService add-proxy-route]]
  (init [this context]
    (add-proxy-route
        {:host "localhost"
         :port 10000
         :path "/bar"}
        "/foo"
        {:callback-fn (fn [proxy-req req]
          (.header proxy-req "x-example" "baz"))})
    context))

In this example, all incoming requests with a prefix of foo will be proxied to https://localhost:10000/bar, using the same scheme (HTTP/HTTPS) that the original request used, and using the SSL context of the main webserver. In addition, a header "x-example" with the value "baz" will be added to the request before it is proxied, using the header method.

#####:failure-callback-fn

This option lets you provide a function to manipulate the response object in case of failure. It must take four arguments, [req resp proxy-resp failure], where req is the original HttpServletRequest, resp is an HttpServletResponse, proxy-req a Response and failure is a Throwable explaining the cause of the problem. resp may be modified, the function does not return any value.

An example with an on-failure function:

(defservice foo-service
  [[:WebserverService add-proxy-route]]
  (init [this context]
    (add-proxy-route
        {:host "localhost"
         :port 10000
         :path "/bar"}
        "/foo"
        {:failure-callback-fn (fn [req resp proxy-resp failure]
          (.println (.getWriter resp) (str "Proxying failed: " (.getMessage failure))))})
    context))

In this example, in case of proxying failure the response body will be augmented by an error message explaining what the cause of the problem was.

override-webserver-settings!

override-webserver-settings! is used to override settings in the webserver section of the webserver service's config file. This function will accept one argument, [overrides]. overrides is a map which should contain a key/value pair for each setting to be overridden. The name of the setting to override should be expressed as a Clojure keyword. For any setting expressed in the service config which is not overridden, the setting value from the config will be used.

For example, the webserver config may contain:

webserver {
    ssl-host:    0.0.0.0
    ssl-port:    9001
    ssl-cert:    mycert.pem
    ssl-key:     mykey.pem
    ssl-ca-cert: myca.pem
}

Overrides may be supplied from the service using code like the following:

(defservice foo-service
  [[:WebserverService override-webserver-settings!]]
  (init [this context]
    (override-webserver-settings!
      {:ssl-port    9002
       :ssl-cert    "myoverriddencert.pem"
       :ssl-key     "myoverriddenkey.pem"})
    context))

For this example, the effective settings used during webserver startup would be:

{:ssl-host    "0.0.0.0"
 :ssl-port    9002
 :ssl-cert    "myoverriddencert.pem"
 :ssl-key     "myoverriddenkey.pem"
 :ssl-ca-cert "myca.pem"}

The overridden webserver settings will be considered only at the point the webserver is being started -- during the start lifecycle phase of the webserver service. For this reason, a call to this function must be made during a service's init lifecycle phase in order for the overridden settings to be considered.

Only one call from a service may be made to this function during application startup.

If a call is made to this function after webserver startup or after another call has already been made to this function (e.g., from other service), a java.lang.IllegalStateException will be thrown.

A three argument version is available which takes these parameters: [server-id overrides]. server-id is the id of the server for which you wish to override the settings. If the two argument version is called, they will be overridden for the default server. The one-argument version of this function will not work in a multiserver set-up if no default server is specified.

get-registered-endpoints

This function returns a map containing information on each URL endpoint registered by the Jetty9 service on the default server. Each key in the map is a URL endpoint, with each value being an array of maps containing information on each handler registered at that URL endpoint. The possible keys appearing in these maps are:

The schema for the various types of handler maps can be viewed here.

There is also a version that takes one argument, [server-id], which specifies which server for which you want to pull the endpoints. If this parameter is absent, the endpoints will be pulled for the default server. The no-argument version of this function will not work in a multiserver set-up if no default server is specified.

log-registered-endpoints

This function logs the data returned by get-registered-endpoints at the info level.

There is a version of this function that takes a single argument, [server-id]. This specifies which server for which you want to log the endpoints. If this is absent, the endpoints registered on the default server will be logged. The no-argument version of this function will not work in a multiserver set-up if no default server is specified.

join

This function is not recommended for normal use, but is provided for compatibility with the ring-jetty-adapter. ring-jetty-adapter/run-jetty, by default, calls join on the underlying Jetty server instance. This allows your thread to block until Jetty shuts down. This should not be necessary for normal trapperkeeper usage, because trapperkeeper already blocks the main thread and waits for a termination condition before allowing the process to exit. However, if you do need this functionality for some reason, you can simply call (join) to cause your thread to wait for the Jetty server to shut down.

There is another version of this function that takes a single argument, [server-id]. This is the id of the server you want to join. If this is not specified, then the default server will be joined. The no-argument version of this function will not work in a multi-server set-up if no default server is specified.

Service lifecycle phases

The Trapperkeeper service manipulates the Java Jetty code in the following ways during these lifecycle phases.

init

A ContextHandlerCollection is created during the init lifecycle which allows for consumers to use the add-*-handler and add-proxy-route functions, but the Jetty server itself has not started yet. This allows the service consumer to setup SSL keys and perform other operations needed before the server is started.

start

In the start lifecycle phase the Jetty server object is created, the ContextHandlerCollection is added to it, and the server is then started. Adding handlers after this phase should still work fine, but it is recommended that handlers be added during the consuming service's init phase.

Webrouting Service

This project provides a secondary Webrouting Service, which in many cases is preferable for use over the Webserver Service. Documentation is available for it here.

TrapperKeeper Webserver Service Test Utils

This project provides some utility code for testing. Documentation on these test utils is available here.

Support

We use the Trapperkeeper project on JIRA for tickets on the Trapperkeeper Webserver Service, although Github issues are welcome too.