Home

Awesome

<img align="right" src="optimus.png"> Optimus Build Status

A Ring middleware for frontend performance optimization.

It serves your static assets:

In other words: Develop with ease. Optimize in production.

Features

Depending on how you use it, Optimus:

You might also be interested in:

Install

Add [optimus "2023.11.21"] to :dependencies in your project.clj.

This project no longer uses Semantic Versioning. Instead we're aiming to never break the API. Feel free to check out the change log.

There were breaking changes in 0.16, 0.17, 0.19 and 2022-02-13. If you're upgrading, you might want to read more about them.

Usage

Let's look at an example:

(ns my-app.example
  (:require [optimus.prime :as optimus]
            [optimus.assets :as assets] ;; 1
            [optimus.optimizations :as optimizations] ;; 2
            [optimus.strategies :as strategies])) ;; 3

(defn get-assets [] ;; 4
  (concat ;; 5
   (assets/load-bundle "public" ;; 6
                       "styles.css" ;; 7
                       ["/styles/reset.css" ;; 8
                        "/styles/main.css"]) ;; 9
   (assets/load-bundles "public" ;; 10
                        {"lib.js" ["/scripts/ext/angular.js"
                                   #"/scripts/ext/.+\.js$"] ;; 11
                         "app.js" ["/scripts/controllers.js"
                                   "/scripts/directives.js"]})
   (assets/load-assets "public" ;; 12
                       ["/images/logo.png"
                        "/images/photo.jpg"])
   [{:path "/init.js" ;; 13
     :contents (str "var contextPath = " (:context-path env))
     :bundle "app.js"}]))

(-> app
    (optimus/wrap ;; 14
     get-assets ;; 15
     (if (= :dev (:env config)) ;; 16
       optimizations/none ;; 17
       optimizations/all) ;; 18
     (if (= :dev (:env config)) ;; 19
       strategies/serve-live-assets ;; 20
       strategies/serve-frozen-assets)) ;; 21
    (ring.middleware.content-type/wrap-content-type) ;; 22
    (ring.middleware.not-modified/wrap-not-modified)) ;; 23

  1. Assets are scripts, stylesheets, images, fonts and other static resources your webapp uses.

  2. You can mix and match optimizations.

  3. You can choose different strategies for how you want to serve your assets.

  4. Declare how to get your assets in a function.

  5. It returns a list of assets.

  6. The helpers in optimus.assets load files from a given directory on the classpath (normally in the resources directory). So in this case, the files are loaded from resources/public/.

  7. The name of this bundle is styles.css.

  8. It takes a list of paths. These paths double as URLs to the assets, and paths to the files in the public directory.

  9. The contents are concatenated together in the order specified in the bundle.

  10. You can declare several bundles at once with load-bundles.

  11. You can use regexen to find multiple files without specifying each individually. Make sure you're specific enough to avoid including weird things out of other jars on the class path.

    Notice that angular.js is included first, even tho it is included by the regex. This way you can make sure dependencies are loaded before their dependents.

  12. You can add individual assets that aren't part of a bundle, but should be optimized and served through Optimus. This is useful to add cache busters and far future Expires headers to images served straight from your HTML.

    If you use the optimus.assets helpers, you don't have to list images and fonts referenced in your CSS files - those are added along with the stylesheet.

  13. Assets don't have to be files on disk. This example creates an asset on the path /init.js that is bundled along with the app.js bundle.

  14. Add optimus/wrap as a Ring middleware.

  15. Pass in the function that loads all your assets.

  16. Pass in the function that optimizes your assets. You can choose from those in optimus.optimizations, or write your own asset transformation functions.

  17. optimizations/none does nothing and returns your assets unharmed.

  18. When you use optimizations/all you get everything that Optimus provides. But you can easily exchange this for a function that executes only the transformations that you need.

  19. Pass in your chosen strategy. Set up properly with environment variables of some kind.

  20. In development you want the assets to be served live. No need to restart the app just to see changes or new files.

  21. In production you want the assets to be frozen. They're loaded and optimized when the application starts.

    Take note: You're free to serve optimized, live assets. It'll be a little slow, but what if your javascript doesn't minify well? How do you reproduce it? It's damn annoying having to restart the server for each change. Here's a way that optimizes just like production, but still serves fresh changes without restarts.

  22. Since Ring comes with content type middleware, Optimus doesn't worry about it. Just make sure to put it after Optimus.

  23. The same goes for responding with 304 Not Modified. Since Optimus adds Last-Modified headers, Ring handles the rest.

Using the new URLs

Since we're rewriting URLs to include cache busters, we need to access them through Optimus.

Notice that we use map, since there is likely more than one URL in development mode.

(ns my-app.view
  (:require [optimus.link :as link]))

(defn my-page
  [request]
  (hiccup.core/html
   [:html
    [:head
     (map (fn [url] [:link {:rel "stylesheet" :href url}])
          (link/bundle-paths request ["styles.css"]))]
    [:body
     (map (fn [url] [:script {:src url}])
          (link/bundle-paths request ["lib.js" "app.js"]))]]))

There's even some sugar available:

(defn my-page
  [request]
  (hiccup.core/html
   [:html
    [:head
     (optimus.html/link-to-css-bundles request ["styles.css"])]
    [:body
     (optimus.html/link-to-js-bundles request ["lib.js" "app.js"])]]))

These link-to-*-bundles will return a string of HTML that includes several script/link tags in development, and a single tag in production.

Specifying the optimizations

If you want to mix and match optimizations, here's how you do that:

(defn my-optimize [assets options]
  (-> assets
      (optimizations/minify-js-assets options)
      (optimizations/minify-css-assets options)
      (optimizations/inline-css-imports)
      (optimizations/concatenate-bundles)
      (optimizations/add-cache-busted-expires-headers)
      (optimizations/add-last-modified-headers)))

(-> app
    (optimus/wrap
     get-assets
     (if (= :dev (:env config))
       optimizations/none
       my-optimize)
     the-strategy))

Just remember that you should always add cache busters after concatenating bundles.

Adding your own asset transformation functions is fair game too. In fact, it's encouraged. Let's say you need to serve all assets from a Content Delivery Network ...

Yeah, we are using a Content Delivery Network. How does that work?

To serve the files from a different host, add a :base-url to the assets:

(defn add-cdn-base-url-to-assets [assets]
  (map #(assoc % :base-url "http://cdn.example.com") assets))

(defn my-optimize [assets options]
  (-> assets
      (optimizations/all options)
      (add-cdn-base-url-to-assets)))

This supposes that your CDN will pull assets from your app server on cache misses. If you need to push files to the CDN, you also need to save them to disk. Like this:

(defn export-assets []
  (-> (get-assets)
      (my-optimize (get-optimisation-options))
      (optimus.export/save-assets "./cdn-export/")))

You can even add an alias to your project.clj:

:aliases {"export-assets" ["run" "-m" "my-app.example/export-assets"]}

And run lein export-assets from the command line. Handy.

Hey, I'm serving my app from a sub-directory, how can I avoid referencing it everywhere?

Locally, your app is known as http://localhost:3000/, but sometimes in production it must share the limelight with others like it. Maybe it'll go live at http://limelight.com/myapp/. Wouldn't it be nice if you could do that without adding the extra folder and referencing it everywhere?

To serve the files from a directory/context path add a :context-path to the assets:

(defn add-context-path-to-assets [assets]
  (map #(assoc % :context-path "/myapp/") assets))

(defn my-optimize [assets options]
  (-> assets
      (add-context-path-to-assets)
      (optimizations/all options)))

Now your links to your assets (including those in CSS files) will reference assets with the context path + the file path.

(Note that you need to add context paths before optimizing assets, otherwise absolute references in CSS files will not include the context path).

Those are a whole lot of files being exported.

Yeah, two reasons for that:

Like this:

(defn export-assets []
  (as-> (get-assets) assets
        (optimizations/all assets options)
        (remove :bundled assets)
        (remove :outdated assets)
        (optimus.export/save-assets assets "./cdn-export/")))

But now it takes more time when the app starts in production!

Yes, doing work takes time. If you want to avoid optimizing all the assets on every app startup, you can do it during build time. The build server will build and optimize the assets, write the assets to disk, and also write an asset manifest that your app will read during startup. Your uberjar should include the assets and the asset manifest.

Now, hold tight while I dump this code on you. You've already seen bits and pieces of it above. The build and get-prebuilt-assets functions are the two sister functions that work in tandem to produce and consume the asset manifest during build and runtime, respectively:

(ns my-app.assets
  (:require
    [clojure.edn :as edn]
    [clojure.java.io :as io]
    [clojure.string :as string]
    [optimus.export]
    [optimus.optimizations]))

;; Functions used during build

(defn get-optimized-assets [get-assets-fn optimize-assets-fn]
  (->> (get-assets-fn)
       (optimize-assets-fn)
       (into []
             (comp (remove :outdated)
                   (remove :bundled)))))

(defn get-asset-manifest [assets]
  (for [asset assets] (dissoc asset :contents :resource)))

(defn build
  "Build and optimize assets using Optimus. Saves the optimized assets to the target dir, and writes an
   asset-manifest.edn to the target dir."
  [get-assets-fn optimize-assets-fn target]
  (let [optimized (get-optimized-assets get-assets-fn optimize-assets-fn)]
    (optimus.export/save-assets optimized target)
    (spit (str target "/asset-manifest.edn")
          (pr-str (get-asset-manifest optimized)))))

;; Functions used during app startup

(defn add-asset-resource [asset]
  (if-let [resource (-> (:path asset)
                        (string/replace-first \/ "")
                        (io/resource))]
    (assoc asset :resource resource)
    (throw (ex-info "Could not read resource for asset" {:asset asset}))))

(defn get-prebuilt-assets
  "Reads the asset-manifest.edn file and returns a vector of Optimus bundle assets to be used by Optimus middleware.

   Throws if asset-manifest.edn or any of the referenced assets couldn't be read."
  []
  (if-let [assets (edn/read-string (slurp (io/resource "asset-manifest.edn")))]
    (mapv add-asset-resource assets)
    (throw (Exception. "Could not read assets from asset-manifest.edn"))))

;; Your app-specific assets and optimizations

(defn get-assets []
  ;; Your vanilla get-assets function, used for dev and prod both.
  )

(defn optimize-assets [assets]
  ;; Your production optimizations
  (-> assets
      (optimus.optimizations/all {})))
      
(defn build-assets!
  [target-dir]
  (println "Building assets into" target-dir)
  (build get-assets optimize-assets target-dir))

From Leiningen you can build your assets like this:

  1. Have a task alias that invokes the build-assets! function. In your project.clj:

      :profiles {:uberjar {:resource-paths ["build-resources"]}}
  :aliases {"build-assets" ["run" "-m" "my-app.assets/build-assets!" "build-resources"]}

  2. Augment the command for building your uberjar thus:

    lein do clean, build-assets, uberjar

If you're using tools.build you must call build-assets! from your uberjar-building function.

That was the build. Now your optimized bundles are part of the uberjar. During startup your app must now read the asset manifest instead of going off and optimizing all the assets all over again. So the middleware must be set up somewhat differently than what we have seen so far:

(optimus/wrap
  (if (= :dev (:env config)) ;; 1
    assets/get-assets
    assets/get-prebuilt-assets)
  optimus.optimizations/none ;; 2
  (if (= :dev (:env config)) ;;3
    optimus.strategies/serve-live-assets
    optimus.strategies/serve-frozen-assets))
  1. In prod, get the assets from the asset manifest.
  2. Never optimize anything - prod assets will be pre-optimized, and in dev you don't want to optimize.
  3. Choose you strategy - this is the same as above.

Serving bundles under a custom URL prefix

optimizations/concatenate-bundles generates a bundled asset with the "/bundle" prefix. If you need to serve assets with a different prefix, provide the :bundle-url-prefix config option to either optimizations/all or optimizations/concatenate-bundles:

(-> app
    (optimus/wrap
     get-assets
     (optimizations/all {:bundle-url-prefix "/assets/bundles"})
     the-strategy))

So how does all this work in development mode?

The paths are used unchanged. So given this example:

(-> app
    (optimus/wrap
     #(assets/load-bundle "public" "app.js"
                          ["/app/some.js"
                           "/app/cool.js"
                           "/app/code.js"])
     optimizations/none
     strategies/serve-live-assets))

When you call

(optimus.link/bundle-paths request ["app.js"])

it returns

["/app/some.js"
 "/app/cool.js"
 "/app/code.js"]

And those are served from resources/public/, or more specifically on eg. public/app/some.js on the classpath.

What about production mode?

When you use the serve-frozen-assets strategy, all the contents for each bundle is read at startup. And with optimizations/all, the URLs are generated from the hash of the contents and the identifier of the bundle.

So when you call (link/bundle-paths request ["app.js"]), it now returns:

["/bundles/d131dd02c5e6/app.js"]

and the middleware handles this URL by returning the concatenated file contents in the order given by the bundle.

What if the contents have changed?

All the contents are read at startup, and then never checked again. To read in new contents, the app has to be restarted.

No, I mean, what if someone requests an old version of app.js?

With a different hash? Yeah, then they get a 404. In production, you should serve the files through Nginx or Varnish to avoid this problem while doing rolling restarts of app servers.

Why not just ignore the hash and return the current contents?

Because then the user might be visiting an old app server with a new URL, and suddenly she is caching stale contents. Or worse, your Nginx or Varnish cache picks up on it and is now serving out old shit in a new wrapping. Not cool.

This of course depends on how your machines are set up, and how you do your rolling restarts, but it's a source of bugs that are hard to track down.

What if I need to share static files with someone else?

Well, they have no way of knowing the cache buster hash, of course. Luckily the files are still available on their original URLs.

When you're serving optimized assets, the bundles are also available. For instance: /bundles/d131dd02c5e6/app.js can also be accessed on /bundles/app.js.

Please note: You have to make extra sure these URLs are not served with far future expires headers, or you'll be in trouble when updating.

How do I handle cache busters on images?

CSS files that reference images are rewritten so that they point to cache busting URLs.

If you're using static images in your HTML, then you'll add a list of these files with optimus.assets/load-assets like point 11 in the big example.

And then grab the cache buster URL like so:

(link/file-path request "/images/logo.png")

You can also add a fallback image, if the given one doesn't exists.

(link/file-path request (str "/images/members/" (:id member) ".png")
                :fallback "/images/anonymous.png")

Sam Ritchie has written this HTML transformation using Enlive that rewrites all your image tags with no extra work. That is pretty cool!

Can I tweak how Optimus behaves?

There are some options to be tuned, but if you're planning on doing major things there's nothing wrong with writing your own strategies or optimizations. A pull request is welcome too.

Now, for the options. You pass them to the wrapper after the strategy:

(-> app
    (optimus/wrap
     get-assets optimize the-strategy
     {:cache-live-assets 2000
      :uglify-js {:mangle-names true
                  :transpile-es6? false}
      :clean-css {:level 2}}))

Values in this example are all defaults, so it's just a verbose noop.

:uglify-js

:clean-css

These options are passed straight to clean-css. Please see the clean-css documentation for available options.

In earlier versions of Optimus, this was a curated set of options. These old options will still work (we're trying not to break your stuff), but it is probably a good idea to take a look at all the available settings in clean-css.

Automatic compilation when assets source files change

Sometimes in development mode serving live assets may be too slow even with caching (for example if you're looking up lots of regexen on the class path). In this case you can use serve-live-assets-autorefresh strategy. This strategy will watch for changes in assets source files and recompile assets in the background whenever it's needed. Compiled assets are then cached until the next change in the source files. By default it assumes that assets sources are located in resources directory, but it can be customized with :assets-dirs config option.

(-> app
    (optimus/wrap
     get-assets optimize serve-live-assets-autorefresh
     {:assets-dirs ["resources/public"]}))

What are these assets anyway? They seem magical to me.

Luckily they're just data. The most basic operation of Optimus is serving assets from a list, with this minimal structure:

[{:path :contents}]

It serves the :contents if the request :uri matches :path.

In addition to :path and :contents, the asset map may contain:

There's also the case that some assets may be binary. Some of them might be large. Instead of keeping those :contents in memory, they have a :resource instead, which is a URL to be served as a stream.

Built on top of that is a bunch of operations that either help you:

If you want to know more, the tests are a good place to start reading. They go in to all the details of how Optimus works and even has some commentary on reasoning and reasons.

Are there any working examples to look at?

Take a look at these:

Are you using Optimus in an open-source project? Please do let me know, and I'll add it to the list.

Why not split Optimus into a bunch of middlewares?

I set out to create a suite of middlewares for frontend optimization. The first was Catenate, which concerned itself with concatenation into bundles. So I certainly agree with your point. You'd be hard pressed to think otherwise in the Clojure community, I think, with its focus on "decomplecting". The reason I gave up on that idea is two-fold:

I'll try to elaborate.

First class assets

Let's look at two examples:

In other words, both the bundle middleware and the cache-busting middleware either needs to own the list of assets, or it needs to rest on a first class asset concept in the stack.

Now add the ability to serve WebP images to browsers that support it. Not only do you have to change the image URLs, but you also have to serve a different set of CSS to use these new images. So this middleware would have to know which CSS files reference which files, and rewrite them.

All of these could be fixed with a well-thought out Asset concept in the Ring middleware stack. Which is what Optimus is an attempt at. It adds a list of assets to the request, with enough information for the linking functions to figure out which versions of which files to link.

Orthogonality

Different transformations aren't orthogonal. Some examples:

So ordering matters. You can't just throw in another middleware, you have to order it just so-and-so. I started writing documentation for this in Catenate. It would say "If you're also using cache-busting middleware, make sure to place it after Catenate." After writing a few of those sentences, I came to the conclusion that they were not entirely separate things. Since they're so dependent on each other, they should live together.

There's also the case of when to optimize. In production you want to optimize once - either as a build step, or when starting the application. In development you don't want any optimization (unless you're debugging), but you still need to create the list of assets so you're able to link to it. This is something all the optimization middlewares would have to tackle on their own - basically each layer freezing their optimized assets on server start, and all but the last one doing so in vain.

Optimus' solution

Optimus solves this by creating a separate middleware stack for optimizations, that work on assets (not requests), and that can be done at different times by different asset-serving strategies.

So yes, the optimizations have been split into several middlewares. But not middlewares for the Ring stack. They are Asset-specific middlewares.

Optimus is open for extension

Even tho Optimus itself doesn't do transpiling, building a transpiler to use with Optimus is pretty nice. I created optimus-less as an example implementation.

You create a custom asset loader and plug it into Optimus' load-asset multimethod. Let :original-url be the original "styles.less", so the linking features can find it, replace the :contents with the compiled CSS, and serve it under the :path "styles.css".

It's about 20 lines of code, including requires. And adding support for more transpilers require no changes to Optimus itself.

Can I switch JavaScript engines?

Yes.

Optimus relies on the javax.script (JSR-223) API to load a JS engine from the classpath. Optimus only adds GraalJS to the classpath, as an explicit dependency, to ensure there is a sensible default.

So it is possible to swap out GraalJS for any other javax.script-compatible JS engine that is available on the classpath. Simply add your alternative JS engine as a dependency using your project's build system. (Note: if doing so, you may wish to add Optimus' GraalJS dependencies to its :exclusions.)

Once another JS engine is made available to javax.script, Optimus can be instructed to prefer it using the environ setting :optimus-js-engines.

This can be declared in various possible ways:

The value my-engine-name can be a single preferred engine, or a comma-separated list of engine names, e.g. nashorn,rhino, ordered left-to-right from most-preferred to least-preferred. Only the first, most-preferred available engine gets loaded and used.

Example: Rhino

To use Rhino, first add a dependency on a JSR-223 API for Rhino such as [cat.inspiracio/rhino-js-engine "1.7.10"] (this uses bunkenberg/rhino-js-engine).

Then if using Leiningen, add {:env {:optimus-js-engines "rhino"}} to project.clj. (You may want to add this to a profile, e.g. if switching between engines.)

Alternatively, define a Java system property in your run context, such as -Doptimus.js.engines=rhino. Or set the shell environment variable OPTIMUS_JS_ENGINES=rhino for the execution context.

Example: Nashorn

Warning: Nashorn is slow and deprecated beyond Java 9.

Using a suitable JDK that ships with Nashorn, just declare:

What about V8 or other engines?

In past Optimus versions before we switched to javax.script, V8 was loaded using clj-v8, but clj-v8 does not expose a javax.script API. At the time of writing, there is no maintained V8 binding for general-purpose use from Java, let alone a maintained javax.script binding for V8.

If such a project should emerge, it should be possible to add it to your dependencies and declare its short name in :optimus-js-engines to load it, without any changes to Optimus itself.

Likewise, for any other JS engine that implements javax.script interfaces.

Change log

There were breaking changes in 0.16, 0.17 and 0.19. If you're upgrading, you might want to read more about them.

From 2023.10.13 to 2023.11.21

From 2023-10-03 to 2023.10.13

From 2023-02-08 to 2023-10-03

From 2022-02-13 to 2023-02-08

From 0.20.2 to 2022-02-13

This is what we have been waiting for to release the big 1.0, except that we no longer belive in semantic versioning - or to be more precise: We aim to never again break the API, removing the need.

Also:

From 0.20.0 to 0.20.2

From 0.19.0 to 0.20

From 0.18.5 to 0.19

From 0.18 to 0.18.5

From 0.17 to 0.18

From 0.16 to 0.17

From 0.15 to 0.16

From 0.14 to 0.15

From 0.13 to 0.14

From 0.12 to 0.13

From 0.11 to 0.12

From 0.10 to 0.11

From 0.9 to 0.10

Contribute

Yes, please do. And add tests for your feature or fix, or I'll certainly break it later.

Installing dependencies

You need npm installed to fetch the JavaScript dependencies. The actual fetching is automated however.

Running the tests

lein midje will run all tests.

lein midje namespace.* will run only tests beginning with "namespace.".

lein midje :autotest will run all the tests indefinitely. It sets up a watcher on the code files. If they change, only the relevant tests will be run again.

lein with-profile +rhino midje [...] will fetch and load Rhino dependencies, and execute JS code in tests with it.

lein with-profile +nashorn midje [...] will use Nashorn to execute JS code in tests, only on JDKs which ship with it.

Contributors

Thanks!

License

Copyright © Magnar Sveen, since 2013

Distributed under the Eclipse Public License, the same as Clojure.