Home

Awesome

tools.analyzer

An analyzer for host agnostic Clojure code, written in Clojure and producing AST in EDN.

I gave a talk on tools.analyzer[.jvm] at ClojureX in December 2015. Video here

Timothy Baldridge gave a talk on tools.analyzer[.jvm] at Clojure/West in March 2014. Video here.

Note that the analyzer in this library should not to be used directly as it lacks any knowledge about host-specific special forms and it should only be considered as a building platform for host-specific analyzers. Currently the following platform specific analyzers written on top of tools.analyzer exist: tools.analyzer.jvm, tools.analyzer.js

Quickref

Example Usage

clojure.tools.analyzer/analyze will not work out of the box, as it requires a number of entry-points to be set. Here's what could happen trying to use clojure.tools.analyzer/analyze directly:

clojure.tools.analyzer> (analyze 'a {})
Attempting to call unbound fn: #'clojure.tools.analyzer/macroexpand-1
  [Thrown class java.lang.IllegalStateException]

At the moment there exist two official analyzers written on top of tools.analyzer: tools.analyzer.jvm for clojure on the JVM and tools.analyzer.js for clojurescript. We will use tools.analyzer.jvm for those examples.

Here's a simplified version of how clojure.tools.analyzer.jvm/analyze is defined:

(require '[clojure.tools.analyzer :as ana])
(require '[clojure.tools.analyzer.env :as env])
(defn analyze [form env]
  (binding [ana/macroexpand-1 macroexpand-1
            ana/create-var    create-var
            ana/parse         parse
            ana/var?          var?]
       (env/ensure (global-env)
         (run-passes (-analyze form env))))))

Here, -analyze is a multimethod that defaults to ana/analyze and defines analysis methods for the JVM specific special forms, global-env is a function that returns a global environment for the JVM analyzer and run-passes is a function that takes an AST and applies a number of passes to it.

The tools.analyzer.jvm README contains more examples on how the analyze function works as well as a reference for all the nodes it can return.

One of the most important features of tools.analyzer is the ability to walk generically through the AST nodes, this has been immensely useful to write most of the passes used by the various analyzers. The tools.analyzer.ast namespace provides a number of functions that implement various generic AST walking strategies.

The children function returns a vector of the children nodes of the current node (the output has been elided and pretty-printed for clarity):

clojure.tools.analyzer.jvm> (require '[clojure.tools.analyzer.ast :as ast])
nil
clojure.tools.analyzer.jvm> (ast/children (analyze '(do 1 2 :foo)))
[{:op   :const,
  :id   0,
  :type :number,
  :val  1,
  :form 1,
  ...}
 {:op   :const,
  :id   1,
  :type :number,
  :val  2,
  :form 2,
  ...}
 {:op   :const,
  :id   3,
  :type :keyword,
  :val  :foo,
  :form :foo,
  ...}]

If we want to access a flattened view of all the nodes of an AST, we can use the nodes function:

clojure.tools.analyzer.jvm> (ast/nodes (analyze '[1 (+ 1 2)]))
({:op        :vector,
  :top-level true,
  :items
  [{:op   :const,
    :type :number,
    :val  1,
    ...}
   {:op     :static-call,
    :class  clojure.lang.Numbers,
    :method add,
    :form   (. clojure.lang.Numbers (add 1 2)),
    :args   [{:op  :const,
              :val 1,
              ...}
             {:op  :const,
              :val 2,
              ...}],
   ...}]
  :form [1 (+ 1 2)],
  ...}
 {:op   :const,
  :type :number,
  :val  1,
  ...}
 {:op    :static-call,
  :class  clojure.lang.Numbers,
  :method add,
  :form   (. clojure.lang.Numbers (add 1 2)),
  :args [{:op  :const,
          :val 1,
          ...}
         {:op  :const,
          :val 2,
          ...}],
  ...}
  ..)

The update-children function takes an AST node and a function and replaces the children nodes of the given node with the result of applying the function to each children node.

clojure.tools.analyzer.jvm> (ast/update-children (analyze '(do 1 (+ 1 2) :foo))
                               #(assoc % :visited true))
{:op :do
 :statements
 [{:op      :const,
   :val     1,
   :visited true,
   ...}
  {:op      :static-call,
   :class   clojure.lang.Numbers,
   :method  add,
   :visited true,
   :args   [{:op  :const
             :val 1,
             ...}
            {:op  :const,
             :val 2,
             ...}],
   ...}]
 :ret
 {:op      :const,
  :val     :foo,
  :visited true,
  ...},
 ...}

If it's desirable to walk all the AST applying a function to all the nodes and the children nodes, one of walk, prewalk or postwalk should be used, read the docstrings of the three functions to understand the differences. Here's the previous example using prewalk instead of update-children:

clojure.tools.analyzer.jvm> (ast/prewalk (analyze '(do 1 (+ 1 2) :foo))
                               #(assoc % :visited true))
{:op      :do
 :visited true,
 :statements
 [{:op      :const,
   :val     1,
   :visited true,
   ...}
  {:op      :static-call,
   :class   clojure.lang.Numbers,
   :method  add,
   :visited true,
   :args   [{:op      :const
             :val     1,
             :visited true,
             ...}
            {:op     :const,
             :val     2,
             :visited true,
             ...}],
   ...}]
 :ret
 {:op      :const,
  :val     :foo,
  :visited true,
  ...},
 ...}

As you can see, this time all the nodes have been marked :visited.

Since version 0.6.0, passes can be scheduled automatically using clojure.tools.analyzer.passes/schedule rather than having to compose them and sort out pass dependencies manually, refer to its docstrings and examples from tools.analyzer.jvm for more info.

SPONSORSHIP

YourKit

YourKit has given an open source license for their profiler, greatly simplifying the profiling of tools.analyzer performance.

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products:

Releases and Dependency Information

Latest stable release: 1.2.0

Leiningen dependency information:

[org.clojure/tools.analyzer "1.2.0"]

Maven dependency information:

<dependency>
  <groupId>org.clojure</groupId>
  <artifactId>tools.analyzer</artifactId>
  <version>1.2.0</version>
</dependency>

Changelog

API Index

Developer Information

License

Copyright © Nicola Mometto, Rich Hickey & contributors.

Distributed under the Eclipse Public License, the same as Clojure.