Awesome
HdrLogProcessing
Utilities for HDR Histogram logs manipulation. This repo currently includes utilities for summarizing and unioning of logs.
- Requires Maven to build and JDK8 to build/run.
- Released under BSD licence.
For brevity in the following examples, lets assume you built the project and added the following alias:
alias hodor=java -jar processor.jar
Summary tool
Using the above alias run:
$ hodor summarize [...]
SummarizeHistogramLogsRange supports the following options:
-start (-s) N : relative log start time in seconds, (default: 0.0)
-end (-e) N : relative log end time in seconds, (default: MAX_DOUBLE)
-ignoreTag (-it) : summary should not be split by tag, (default: false)
-inputFile (-if) VAL : add an input hdr log from input path, also takes regexp
-inputFilePath (-ifp) VAL : add an input file by path relative to working dir or absolute
-inputPath (-ip) VAL : set path to use for input files, defaults to current folder
-outputBucketSize (-obs) N : csv output bucket size, (default: 100)
-outputFile (-of) VAL : set an output file destination, default goes to sysout
-outputValueUnitRatio (-ovr) N : output value unit ratio, (default: 1.0)
-percentilesOutputTicksPerHalf (-tph) N: ticks per half percentile, used for hgrm output, (default: 5)
-summaryType (-st) [CSV | PERCENTILES | HGRM]: : summary type: csv, percentiles, hgrm
-verbose (-v) : verbose logging, (default: false)
This is useful when for example you are face with a histogram log you have collected from your application over time and you wish to summarize the percentiles from the full run:
$ hodor summarize -if my-awesome-app-latencies.hdr
TotalCount=27663673
Period(ms)=205823
Throughput(ops/sec)=134405.16
Min=263
Mean=6561.99
50.000ptile=5491
90.000ptile=8887
99.000ptile=49023
99.900ptile=72767
99.990ptile=92927
99.999ptile=116415
Max=145151
Now perhaps the first 200 seconds of this run are an unstable warmup period I wish to exclude from my summary:
$ hodor summarize -if my-awesome-app-latencies.hdr -s 200
Or maybe I got several logs, from several runs and I want an overall summary, excluding the first 60 seconds of the run and saving the output into a file:
$ hodor summarize -if run1.hdr -if run2.hdr -if run3.hdr -s 60 -of runs-summary.out
-OR you could use a regexp to get all the files-
$ hodor summarize -if ^run.*.hdr -s 60 -of runs-summary.out
The default output is percentiles as shown above. We support HGRM output if you wish to plot the result with the useful plotter in HdrHistogram, and a CSV format to enable statistical analysis with other tools. The HGRM output with an output file will result in a file per tagged summary with the convention of: outputfile.tag.hgrm
The summary tool supports tags, and if your logs contains histograms of different tags they will get summarized separately. You can use the '-it|ignoreTag' option to summarize all tags together.
Union tool
Using the above alias run:
$ hodor union [...]
UnionHistogramLogs supports the following options:
-end (-e) N : relative log end time in seconds, (default: MAX_DOUBLE)
-inputFile (-if) VAL : add an input hdr log from input path, also takes regexp
-inputPath (-ip) VAL : set path to use for input files, defaults to current folder
-outputFile (-of) VAL : set an output file destination, default goes to sysout
-relative (-r) : relative timeline merge, (default: true)
-start (-s) N : relative log start time in seconds, (default: 0.0)
-taggedInputFile (-tif) VAL : a <tag>=<filename> add an input file, tag all
histograms from this file with tag. If histograms
have a tag it will be conactanated to file tag
<file-tag>::<histogram-tag>.
-verbose (-v) : verbose logging, (default: false)
Sometimes you got lots of files, and you really wish you could just throw them all into one file. For example, lets say you used 3 separate load-generating clients to measure your server latencies. You can union all the logs into a single log as follows:
$ hodor union -if ^load-gen.*.hdr -of union-load-gens.hdr
The above union will use absolute time so the result will be as if all load generators were logged from a single source (assuming the clocks are reasonablely in sync). You may want to collect multiple runs timelines into a single union. This is possible using the '-r' option.
If each load generator represents a different operation you could use tags to differentiate them in the union:
$ hodor union -tif READ=load-gen1.hdr -tif READ=load-gen2.hdr -tif WRITE=load-gen3.hdr -of union-load-gens.hdr
Split tool
Using the above alias run:
$ hodor union [...]
SplitHistogramLogs supports the following options:
-end (-e) N : relative log end time in seconds, (default: MAX_DOUBLE)
-excludeTag (-et) VAL : add a tag to exclude from input, 'default' is a special tag for the null tag.
-includeTag (-it) VAL : when include tags are used only the explicitly included will be split out, 'default' is a special tag for the null tag.
-inputFile (-if) VAL : set the input hdr log from input path
-inputPath (-ip) VAL : set path to use for input files, defaults to current folder
-start (-s) N : relative log start time in seconds, (default: 0.0)
-verbose (-v) : verbose logging, (default: false)
Some tools do not support tags yet, so you may want to split a log into several logs for post processing.
$ hodor split -if taggyLog.hdr
Will result in the creation of a log file per tag, with the default tag going to the 'default' file. So tags A,B,C will end up in the files A.taggyLog.hdr, B.taggyLog.hdr, C.taggyLog.hdr respectively. If you need only certain tags A,B:
$ hodor split -if taggyLog.hdr -it A -it B
HDR to CSV tool
Using the above alias, run:
$ hodor to-csv -i INPUT_FILE
It will result in a strict transformation of a log file to CSV. Intervals will be preserved. For each interval, important percentiles will be written in dedicated columns. The resulting CSV is printed on stdout.
Example usage:
$ hodor to-csv -i input.hgrm | tee output.csv
#Timestamp,Throughput,Min,Avg,p50,p90,p95,p99,p999,p9999,Max
1523292112.000,2364,60608,143515,113599,221823,268543,442367,1638399,6348799,6348799
1523292113.000,192,64672,130923,116287,188031,205823,260351,366591,366591,366591
1523292114.000,384,67520,144460,118527,213759,264703,414463,1793023,1793023,1793023
...