Home

Awesome

perf-map-agent

Join the chat at https://gitter.im/jrudolph/perf-map-agent Build Status

A java agent to generate /tmp/perf-<pid>.map files for just-in-time(JIT)-compiled methods for use with the Linux perf tools.

Build

Make sure JAVA_HOME is configured to point to a JDK. You need cmake >= 2.8.6 (see #30). Then run the following on the command line:

cmake .
make

# will create links to run scripts in <somedir>
bin/create-links-in <somedir>

Architecture

Linux perf tools will expect symbols for code executed from unknown memory regions at /tmp/perf-<pid>.map. This allows runtimes that generate code on the fly to supply dynamic symbol mappings to be used with the perf suite of tools.

perf-map-agent is an agent that will generate such a mapping file for Java applications. It consists of a Java agent written C and a small Java bootstrap application which attaches the agent to a running Java process.

When the agent is attached it instructs the JVM to report code blobs generated by the JVM at runtime for various purposes. Most importantly, this includes JIT-compiled methods but also various dynamically-generated infrastructure parts like the dynamically created interpreter, adaptors, and jump tables for virtual dispatch (see vtable and itable entries). The agent creates a /tmp/perf-<pid>.map file which it fills with one line per code blob that maps a memory location to a code blob name.

The Java application takes the PID of a Java process as an argument and an arbitrary number of additional arguments which it passes to the agent. It then attaches to the target process and instructs it to load the agent library.

Command line scripts

The bin directory contains a set of shell scripts to combine common perf / dtrace perations with creating the map file.

Environment variables:

Options

You can add a comma separated list of options to perf-java (or the AttachOnce runner). These options are currently supported:

Known Issues

Skid

You should be aware that instruction level profiling is not absolutely accurate but suffers from 'skid'. 'skid' means that the actual instruction pointer may already have moved a bit further when a sample is recorded. In that case, (possibly hot) code is reported at an address shortly after the actual hot instruction. See this sample from one of Brendan's presentations demonstrating this issue.

If using unfold, perf-map-agent will report sections that contain code inlined from other methods as separate entries. Unfolded entries can be quite short, e.g. an inlined getter may only consist of a few instructions that now lives inside of another method's JITed code. The next few instructions may then already belong to another entry. In such a case, it is more likely that skid will not only affect the instruction pointer inside of a method entry but may affect which entry is chosen in the first place.

Skid that occurs inside a method is only visible when analyzing the actual assembler code (as with perf annotate). Skid that affects the actual symbol resolution to choose a wrong entry will be much more visible as wrong entries will be reported with tools that operate on the symbol level like the standard views of perf report, perf top, or in flame graphs.

So, while it is tempting to enable unfolded entries for the perceived extra resolution, this extra information is sometimes just noise which will not only clutter the overall view but may also be misleading or wrong.

Inaccurate mappings using the unfold* options

Hotspot does not retain line number and other debug information for inlined code at other places than safepoints. This makes sense because you don't usually observe code running between safepoints from the JVM's perspective. This is different when observing a process from the outside like with perf. For observed code locations outside of safepoints, the JVM will not report any inlining information and perf-map-agent will assign those areas to the host method of the inlining.

For more fidelity, Hotspot can be instructed to include debug information for non-safepoints as well. Use -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints when running the target process. Note, however, that this will produce a lot more information with the generated perf-<pid>.map file potentially growing to MBs of size.

Agent Library Unloading

Unloading or reloading of a changed agent library is not supported by the JVM (but re-attaching is). Therefore, if you make changes to the agent and recompile it you need to restart a target process that has an older version loaded to use the newer version.

Missing symbols for libjvm.so

libjvm.so is the runtime component of the JVM. It is not covered by perf-map-agent but perf will use debug symbols as provided by the distribution. If symbols for libjvm.so are missing see instructions for your Linux distribution to install debug symbols for the JVM. See also issue #39 which contains a few pointers about how to install these.

Disclaimer

I'm not a professional C code writer. The code is very "experimental", and it is e.g. missing checks for error conditions etc.. Use it at your own risk. You have been warned!

License

This library is licensed under GPLv2. See the LICENSE file.