Home

Awesome

lib-tcpdump-processing

lib-tcpdump-processing is a library designed to process .pcap(ng) tcpdump or Wireshark trace files and extract SRT packets of interest for further analysis.

It also helps to process a network trace file and generate a report with SRT related statistics. In particular, the following statistics are calculated at the receiver side:

See the get-traffic-stats script for details and report examples.

Important: Currently, trace files containing only one flow of data are supported. To support several data flows adjustments will be required.

1. Getting Started

Requirements

Install the library with pip

For development, it is recommended to:

As soon as the library is installed, you can run modules directly:

venv/bin/python -m tcpdump_processing.extract_packets --help

or use preinstalled executable scripts:

venv/bin/extract-packets --help

Install the library to import in another project

Install with pip (a venv is recommended), using the pip VCS requirement specifier:

pip install 'git+https://github.com/mbakholdina/lib-tcpdump-processing.git@v0.1#egg=tcpdump_processing'

or simply put the following row in requirements.txt:

git+https://github.com/mbakholdina/lib-tcpdump-processing.git@v0.1#egg=tcpdump_processing

Remember to quote the full URL to avoid shell expansion in the case of direct installation.

This installs the version corresponding to the git tag 'v0.1'. You can replace that with a branch name, a commit hash, or a git ref as necessary. See the pip documentation for details.

To install the latest master, use:

git+https://github.com/mbakholdina/lib-tcpdump-processing.git@master#egg=tcpdump_processing

As soon as the library is installed, you can import the whole library:

import tcpdump_processing

or a particular module:

import tcpdump_processing.extract_packets as extract_packets

2. Executable Scripts

To use the following scripts, please install the library first (see the Install the library with pip section).

extract-packets

This script parses .pcap(ng) trace file, saves the output in .csv format in the same directory as the original file, extracts packets of interest, and saves the obtained dataframe in .csv format in the same directory as the original file.

Usage:

venv/bin/extract-packets [OPTIONS] PATH

where PATH refers to .pcap(ng) file.

Options:

Options:
  --type [srt|data|control|probing|umsg_handshake|umsg_ack]
                                  Packet type to extract: SRT (both DATA and
                                  CONTROL), SRT DATA, SRT CONTROL, SRT DATA
                                  probing, SRT CONTROL UMSG_HANDSHAKE, or SRT
                                  CONTROL UMSG_ACK packets.  [default:
                                  probing]
  --overwrite / --no-overwrite    If exists, overwrite the .csv file produced
                                  out of the .pcap(ng) one at the previous
                                  iterations of running the script.  [default:
                                  no-overwrite]
  --save / --no-save              Save dataframe with extracted packets into
                                  .csv file.  [default: no-save]
  --port TEXT                     Decode packets as SRT on a specified port.
                                  This option is helpful when there is no SRT
                                  handshake in .pcap(ng) file. Should be used
                                  together with --overwrite option.
  --help                          Show this message and exit.

Here is an example of the report generated when extracting --type srt packets:

extract_packets_report

get-traffic-stats

This script helps to process a network .pcap(ng) trace file and generate a report with SRT related statistics. Intermediate data is stored in .csv format in the same directory as the original file. Both sender and receiver side dumps are supported.

Usage:

venv/bin/get-traffic-stats [OPTIONS] PATH

where PATH refers to .pcap(ng) file.

Options:

Options:
  --side [snd|rcv]              The side .pcap(ng) file was collected at.
                                [required]
  --overwrite / --no-overwrite  If exists, overwrite the .csv file produced
                                out of the .pcap(ng) one at the previous
                                iterations of running the script.  [default:
                                no-overwrite]
  --show-unrec-pkts / --no-show-unrec-pkts
                                Show sequence numbers of unrecovered at the
                                receiver side packets. Save the list of
                                sequence numbers into respective .csv file.
                                [default: no-show-unrec-pkts]
  --port TEXT                   Decode packets as SRT on a specified port.
                                This option is helpful when there is no SRT
                                handshake in .pcap(ng) file. Should be used
                                together with --overwrite option.
  --help                        Show this message and exit.

Here is an example of the report generated at the sender side:

get_traffic_stats_report_snd

Here is an example of the report generated at the receiver side:

get_traffic_stats_report_rcv

plot-snd-timing

This script parses .pcap(ng) tcpdump trace file captured at the sender side and plots the time delta between SRT packet timestamp (srt.timestamp) and packet time captured by Wireshark at the sender side (ws.time). This could be done for either SRT original DATA packets only, or both original and retransmitted DATA packets.

Usage:

venv/bin/plot-snd-timing [OPTIONS] PATH

where PATH refers to .pcap(ng) file.

Options:

Options:
  --overwrite / --no-overwrite    If exists, overwrite the .csv file produced
                                  out of the .pcap(ng) one at the previous
                                  iterations of running the script.  [default:
                                  no-overwrite]
  --with-rexmits / --without-rexmits
                                  Also show retransmitted data packets.
                                  [default: without-rexmits]
  --port TEXT                     Decode packets as SRT on a specified port.
                                  This option is helpful when there is no SRT
                                  handshake in .pcap(ng) file. Should be used
                                  together with --overwrite option.
  --latency TEXT                  SRT latency, in milliseconds, to plot on a
                                  graph.
  --help                          Show this message and exit.

dump-pkt-timestamps

The script dumps SRT timestamps (not Wireshark ws.time) of SRT data packets to a .csv file to be used by srt-xtransmit application with the --playback-csv argument.

Usage:

dump-pkt-timestamps [OPTIONS] INPUT OUTPUT

where INPUT is the .pcap(ng) file to use as an input, OUTPUT is the output .csv file to be produced.

Options:

Options:
  --overwrite / --no-overwrite  If exists, overwrite the .csv file produced
                                out of the .pcap(ng) one at the previous
                                iterations of running the script.  [default:
                                no-overwrite]
  --port TEXT                   Decode packets as SRT on a specified port.
                                This option is helpful when there is no SRT
                                handshake in .pcap(ng) file. Should be used
                                together with --overwrite option.
  --help                        Show this message and exit.

3. Data Preparation

A .pcap(ng) trace file with measurements from a specific network interface and port collected at the sender/receiver side is used as a proxy for packet data collected by SRT. This trace file is preprocessed in .csv format with timestamp, source IP address, destination IP address, protocol, and other columns and rows representing observations (sent/received packets).

This data is further cleaned and transformed using pandas in the following way:

  1. The data is filtered to extract SRT packets only (ws.protocol == SRT), which makes sense for further analysis.
  2. The dataset then is split into DATA (srt.iscontrol == 0) and CONTROL (srt.iscontrol == 1) packets.
  3. For DATA packets, timestamps are converted from seconds to microseconds using the same procedure as in the protocol. To be precise, the new variable ws.time.us is obtained as (ws.time * 1000000).astype('int64').
  4. For DATA packets, the inter-arrival time is calculated as the difference between current and previous packet timestamps and stored as a separate variable ws.iat.us. Please note that the time delta for the first SRT data packet by default is equal to 0. That's why this packet should probably be excluded from the analysis.
  5. Type conversion is performed to structure the data in appropriate formats.

The detailed description of dataset variables, tcpdump/Wireshark dissectors and other data is provided in the table below. See columns DATA and CONTROL to determine whether a variable is present () or absent (-) in a corresponding dataset.

Dataset VariableWireshark DissectorDescriptionDATACONTROLData Type
ws.no_ws.col.No.Number as registered by Wiresharkint64
frame.timeframe.timeAbsolute time when the frame was captured as registered by Wiresharkdatetime64
ws.time_ws.col.TimeRelative timestamp as registered by Wireshark (seconds)float64
ws.source_ws.col.SourceSource IP addresscategory
ws.destination_ws.col.DestinationDestination IP addresscategory
ws.protocol_ws.col.ProtocolProtocolcategory
ws.length_ws.col.LengthLength (bytes)int16
ws.info_ws.col.InfoInformationobject
udp.lengthudp.lengthUDP packet size (bytes)int16
srt.iscontrolsrt.iscontrolContent type (CONTROL if 1, DATA if 0)int8
srt.typesrt.typeMessage type (e.g. UMSG_ACK, UMSG_ACKACK)-category
srt.seqnosrt.seqnoSequence number-int64
srt.msg.rexmitsrt.msg.rexmitSent as original if 0, retransmitted if 1-int8
srt.timestampsrt.timestampTimestamp since the socket was opened (microseconds)int64
srt.idsrt.idDestination socket idcategory
srt.ack_seqnosrt.ack_seqnoFirst unacknowledged sequence number-int64
srt.rttsrt.rttRound Trip Time (RTT) estimate (microseconds)-int64
srt.rttvarsrt.rttvarThe variance of Round Trip Time (RTT) estimate (microseconds)-int64
srt.ratesrt.rateReceiving speed estimate (packets/s)-int64
srt.bwsrt.bwBandwidth estimate (packets/s)-int64
srt.rcvratesrt.rcvrateReceiving speed estimate (bytes/s)-int64
data.lendata.lenPayload size or 0 in case of control packets (bytes)-int16
ws.time.us-Relative timestamp as registered by Wireshark (microseconds)-int64
ws.iat.us-Packet inter-arrival time (microseconds)-int64

Probing DATA Packets

Probing DATA packets are extracted from the DATA packets dataset as follows:

  1. Find all the packet pairs where the latest 4 bits of their sequence numbers (srt.seqno) are 0000 and 0001. The order is important.
  2. For each packet pair, check whether both packets are sent as Original (srt.msg.rexmit == 0), not Retransmitted (srt.msg.rexmit == 1).
  3. For the remaining packet pairs, take the packet with sequence number ending with 0001 bits as a probing packet.

Here is an example of a packet pair where Frame 25 corresponds to the probing packet:

Frame 24: 1514 bytes on wire (12112 bits), 1500 bytes captured (12000 bits) on interface 0
Ethernet II, Src: 12:34:56:78:9a:bc (12:34:56:78:9a:bc), Dst: Microsof_59:95:17 (00:0d:3a:59:95:17)
Internet Protocol Version 4, Src: 51.144.160.127, Dst: 10.1.4.4
User Datagram Protocol, Src Port: 60900, Dst Port: 4200
SRT Protocol
    0... .... .... .... .... .... .... .... = Content: DATA
    .111 1111 1110 0111 0111 1000 0011 0000 = Sequence Number: 2145876016
    11.. .... .... .... .... .... .... .... = Packet Boundary: PB_SOLO (3)
    ..0. .... .... .... .... .... .... .... = In-Order Indicator: 0
    ...0 0... .... .... .... .... .... .... = Encryption Status: Not encrypted (0)
    .... .0.. .... .... .... .... .... .... = Sent as: Original
    .... ..00 0000 0000 0000 0000 0001 0001 = Message Number: 17
    Time Stamp: 449263 (0x0006daef)
    Destination Socket ID: 0x1c9ff5e1
    Data (1442 bytes)

Frame 25: 1514 bytes on wire (12112 bits), 1500 bytes captured (12000 bits) on interface 0
Ethernet II, Src: 12:34:56:78:9a:bc (12:34:56:78:9a:bc), Dst: Microsof_59:95:17 (00:0d:3a:59:95:17)
Internet Protocol Version 4, Src: 51.144.160.127, Dst: 10.1.4.4
User Datagram Protocol, Src Port: 60900, Dst Port: 4200
SRT Protocol
    0... .... .... .... .... .... .... .... = Content: DATA
    .111 1111 1110 0111 0111 1000 0011 0001 = Sequence Number: 2145876017
    11.. .... .... .... .... .... .... .... = Packet Boundary: PB_SOLO (3)
    ..0. .... .... .... .... .... .... .... = In-Order Indicator: 0
    ...0 0... .... .... .... .... .... .... = Encryption Status: Not encrypted (0)
    .... .0.. .... .... .... .... .... .... = Sent as: Original
    .... ..00 0000 0000 0000 0000 0001 0010 = Message Number: 18
    Time Stamp: 449292 (0x0006db0c)
    Destination Socket ID: 0x1c9ff5e1
    Data (1442 bytes)

UMSG_HANDSHAKE CONTROL Packets

UMSG_HANDSHAKE CONTROL packets are extracted from the CONTROL packets dataset using the following criteria: srt.type == 0x00000000.

UMSG_ACK CONTROL Packets

UMSG_ACK CONTROL packets are extracted from the CONTROL packets dataset as follows:

  1. Find all the packets with srt.type == 0x00000002.
  2. Drop rows with NaN values of srt.rate, srt.bw, and srt.rcvrate variables (so called "light acknowledgements" or "light ACKs").

RETURN TO TOP