Home

Awesome

potiron

potiron - Normalize, Index, Enrich and Visualize Network Capture

potiron is a tool to analyze a series of network capture (pcap) files, parse these with standard tools and normalize it in JSON format. Then the JSON format is imported into a Redis database to visualize the normalized information.

The current version potiron supports ipsumdump and tshark.

Potiron web interface

Install

IMPORTANT: Use pipenv

NOTE: It requires python3.6+

Various packages required:

Install redis

git clone https://github.com/antirez/redis.git
cd redis
git checkout 5.0
make
make test
cd ..

Install potiron

git clone https://github.com/CIRCL/potiron
cd potiron
pipenv install
echo POTIRON_HOME="'`pwd`'" > .env
pipenv shell
npm install -g phantomjs-prebuilt
cd ./var/www
bash ./update_thirdparty.sh

Usage

Start/Stop the redis instance(s)

run_redis.py --start --status

Alternatively you can specify which instance to run, by specifying its name using the -n parameter.

For instance: run_redis.py --start -n standard

Note that all the required files to run 2 redis instances are provided by default: standard and isn

# Obviously, following the same model:
run_redis.py --stop
# Which uses the same parameter to specify any instance name as well

Create, delete or flush redis instances

# Create a new redis instance, which will be identified by the name you give to it
manage_redis.py -c -n new_redis_instance

# Delete any existing instance
manage_redis.py -d -n any_instance_name
# Be aware that without specifying any name, it will delete all the existing instances

# Flush any running redis instance
manage_redis.py -f -n any_running_instance_name
# Be aware that without specifying any name, it will flush all the running instances

Store data from pcap files in a redis instance

parse_pcap_files.py -u redis_backends/standard/standard.sock -i PATH_TO_ANY_PCAP_FILES

Parameters:

Create interactive graphics

/!\ REWORK STILL IN PROGRESS, DOCUMENTATION TO COME ONCE IT IS DONE, SOON /!\

NOTE: On the other hand, GNU plot visualization provided with the server's part is working as expected

For all the following graphs, the parameters used are :

First you need to define a root directory where all the JSON files will be exported. Then you need to proceed your network capture files to generate the JSON files :

Process with ipsumdump :

./potiron-json-ipsumpdump.py -c -i /tmp/test-honeypot-1-20140826000000.cap.gz -o ../out/
potiron[24989]: [INFO] Created filename ../out/2014/08/26/test-honeypot-1-20140826000000.json

Or process with tshark :

./potiron-json-tshark.py -c -i /tmp/test-honeypot-1-20140826000000.cap.gz -o ../out/
potiron[24989]: [INFO] Created filename ../out/2014/08/26/test-honeypot-1-20140826000000.json

Additional functionalities grant you the possibility to filter data :

As a result, both field and tshark filters can be used in parallel to select a specific part of the whole data, when using combined keys is a way to separate in redis values coming from tcp packets and values coming from udp, icmp, GRE, etc packets. Please note that these two filters should be defined with wireshark / tshark typography !

Example :

    ./potiron-json-tshark.py -c -i /tmp/test-honeypot-1-20140826000000.cap.gz -o ../out/ -ff tcp.srcport tcp.dstport udp.srcport udp.dstport -tf "tcp.srcport eq 80 or tcp.dstport eq 80" -r -u /tmp/redis.sock -ck

Then the JSON file can be imported into the Redis database :

./potiron-redis.py --filename ../out/2014/08/26/test-honeypot-1-20140826000000.json --unix /tmp/redis.sock

As the tshark module uses this one to process the storage in redis, combined keys are available here as well.

But it is also possible to skip JSON storage and save directly data in Redis, using the '--disable_json' (equals to '-dj') argument. In this case, specifying an output directory is not needed but make sure the path of the unix socket is filled out. An example of command processing storage directly in Redis, skipping JSON, could then be:

  ./potiron-json-tshark.py -c -i /tmp/test-honeypot-120140826000000.cap.gz -u /tmp/redis.sock -dj

Some specific data fields can be represented into graphics (multiple values can be specified after parameter '-v') :

./bokeh_month.py -s test-honeypot-1 -f dport -v 22 -d 2017-03 -u /tmp/redis.sock -o ./out/ --logo /home/user/Pictures/logo.png

The last parameter --logo is optionnal and is the ABSOLUTE path of the logo file which will be displayed. If no argument is given, the default file is the CIRCL logo stored in the doc/ directory of potiron.

Here, the -v parameter is important and there are some different possible combinaisons:

As long as combined keys are used, it is also possible to display data following both of the two cases (as ports are good examples for which it is interesting to use combined keys, for instance "22-tcp 22" as src or dst port values parameter will show the differences between the scores of ports 22 seen with tcp protocol, and all the ports 22 seen).

In order to have working redirections in this plot, the resulting graphs should be created first. To do so, .csv files should be processed :

./export_csv_all_days_per_month.py -s test-honeypot-1 -f dport -d 2017-03 -u /tmp/redis.sock -o ./out/ -l 10 --skip -1

The -l parameter is used to define the number of most frequent values to display (default number is 20) The --skip parameter can be used to specify values to exclude in the graph.

Each one of these files can also be reprocessed separatly:

./export_csv_day.py -s test-honeypot-1 -f dport -d 2017-03-01 -u /tmp/redis.sock -o ./out/ -l 10 --skip -1

Statistics for an entire month can also be displayed as well. The data file used in this case is created with the following :

./export_csv_month.py -s test-honeypot-1 -f dport -d 2017-03 -u /tmp/redis.sock -o ./out/ -l 10 --skip -1

These files contain the data which will be used as data source in the graphs. The next step to do is creating the graphs with the data sources.

./generate.sh ./out/ /home/user/Pictures/logo.png

The script will simply generate all the .html files using template.html to build the graphs. Having both generate.sh and template.html in the same path is recommanded. The first parameter used here is the location of the .csv files, and the .html output files will be created in the same directory. The second parameter is optionnal and is the absolute path of the logo file which will be displayed. If no argument is given, the default path is the same used for the bokeh graph, which is the CIRCL logo stored in the doc/ directory of potiron.

In a default usage, combined keys are deployed in the redis database and data is stored by protocol. However, it is possible to display only the grouped values of all the dataset (i.e for all the protocols) by using "-p" parameter. This will set a variable to False in order to avoid treating each protocol separatly. On any case, when combined key are not used, the variable is automatically set to false to avoid errors.

In order to automatize some operations, it is possible to use the "--links" parameter in bokeh-export module in order to auto-generate all the bubble charts usefull to have all the links working. The same parameter is also available in all the export-csv modules, generating the plots with bokeh for each value in bubbles. Finaly, bubble charts can be generated as well directly from the csv files using the "--generate" or "-g" parameter in any export-csv module. When auto-generation parameter is called with bokeh module, both auto-generation parameters in export-csv module are called as well.

Example:

./export_csv_all_days_per_month.py -s test-honeypot-1 -f dport -d 2017-03 -u /tmp/redis.sock -o ./out/ -g -p

First, this command produces the same visual result regardless of the usage or not of the "-p" parameter, and only the name will be different. Top destination ports example

But the difference relies upon the links on each bubble. For users, using "-p" means "without protocols":

Summary (deprecated version)