Awesome

VoIP Patrol

Important note:

This is a fork of the original project with some breaking changes

• Changed format of JSON reports to support VOTLS
• Added possibility to record calls to have the possibility to test media after
• Reworked STUN options to support DTLS media more accurate
• Extended expected_XXX parameters for accept/call tests
• Reworked match mechanism on accept test to rely on Contact URI parameters (see account - match_account notes below)
• Added fail_on_accept parameter to control calls, that should not happen
• Code formatting :)

In general, I'm trying to follow the original project features and changes, but don't expect, that your existing advanced scenarios from the original project will work with this fork, but the simple ones will work without any modifications.

VoIP signaling and media test automaton

Designed to automate end2end and or integration tests.

VoIP patrol will follow a scenario in XML format and will output results in JSON.

Each line in the output file is a separate JSON structure, note that the entire file is not a valid JSON file, this is because VoIP patrol will output results as they become available.

It is possible to test many scenarios that are not easy to test manually like a re-invite with a new codec.

This version is extension of original project and contains changes (in reports and configuration), that are not compatible with original version

Docker quick start

quick start with docker

Linux Debian building from sources

see commands in Dockerfile

Load test example

load test example

run

./voip_patrol --help

Example: making a test call

<config>
  <actions>
    <action type="call" label="us-east-va"
            transport="tls"
            expected_cause_code="200"
            caller="15147371787@noreply.com"
            callee="12012665228@target.com"
            to_uri="+12012665228@target.com"
            max_duration="20" hangup="16"
            auth_username="VP_ENV_USERNAME"
            password="VP_ENV_PASSWORD"
            realm="target.com"
            rtp_stats="true"
    >
        <x-header name="X-Foo" value="Bar"/>
    </action>
    <!-- note: param value starting with VP_ENV_ will be replaced by environment variables -->
    <!-- note: rtp_stats will include RTP transmission statistics -->
    <!-- note: x-header tag inside an action will append an header. You can add any header like User-Agent with this method -->
    <action type="wait" complete="true"/>
  </actions>
</config>

Example: starting a TLS server

./voip_patrol \
   --port 5060 \ # TLS port 5061 +1
   --conf "xml/tls_server.xml" \
   --tls-calist "tls/ca_list.pem" \
   --tls-privkey "tls/key.pem" \
   --tls-cert "tls/certificate.pem" \
   --tls-verify-server \

<config>
  <actions>
     <!-- note: default is the "catch all" account,
          else account as to match called number -->
    <action type="accept"
            match_account="default"
            hangup="5"
            play_dtmf="0123456789#*"
            play="voice_ref_files/f.wav"
            code="200" reason="YES"
            ring_duration="5"
    />
    <!-- DTMF will be sent using RFC2833 -->
    <!-- note: wait for new incoming calls
               forever and generate test results -->
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and checking for specific header with exact match or regular expression and no match on other

<config>
  <actions>
    <action type="accept"
            match_account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-header name="Min-SE"/>
        <!-- Check that a header exists -->
        <check-header name="X-Foo" value="Bar"/>
        <!-- Check that a header exists and have a specific value -->
        <check-header name="From" regex="^.*sip:\+1234@example\.com"/>
        <!-- Check that a header exists and matches a specific regex -->
        <check-header name="To" regex="^.*sip:\+5678@example\.com" fail_on_match="true"/>
        <!-- Check that a header exists and NOT matches a specific regex -->
        <check-header name="RURI" regex="^INVITE\ sip:\d{5}@(\d{1,3}\.){3}\d{1,3}:\d{1,5};.*transport=[a-zA-Z]{3};.*"/>
        <!-- Not really a header, but allows to check the Request URI on an incoming INVITE-->
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and searching the message with a regular expression

<config>
  <actions>
    <action type="accept"
            match_account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-message method="INVITE" regex="m=audio(.*)RTP/AVP 0 8.*"/>
        <!-- searching for pcmu pcma in the SDP -->
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and searching the message with a regular expression that should not be there

<config>
  <actions>
    <action type="accept"
            match_account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-message method="INVITE" regex="m=audio(.*)RTP/AVP 0 8.*" fail_on_match="true"/>
        <!-- searching for pcmu pcma in the SDP, but this is wrong here -->
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: making tests calls with wait_until

Scenario execution is sequential and non-blocking. We can use “wait” command with previously set “wait_until” params to control parallel execution.

Call States
NULL : Before INVITE is sent or received
CALLING : After INVITE is sent
INCOMING : After INVITE is received.
EARLY : After response with To tag.
CONNECTING : After 2xx is sent/received.
CONFIRMED : After ACK is sent/received.
DISCONNECTED

<config>
  <actions>
    <action type="call" label="call#1"
            transport="udp"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="15148888888@noreply.com"
            callee="12011111111@target.com"
    />
    <!-- note: will wait until all tests pass wait_until state -->
    <action type="wait"/>
    <action type="call" label="call#2"
            transport="udp"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="15147777777@noreply.com"
            callee="12012222222@target.com"
    />
    <action type="wait" complete="true"/>
  </actions>
</config>

Example: testing registration

<config>
  <actions>
    <!-- note: proxy param to send to a proxy -->
    <action type="register" label="register target.com"
            transport="udp"
            account="VP_ENV_USERNAME"
            username="VP_ENV_USERNAME"
            auth_username="VP_ENV_USERNAME"
            password="VP_ENV_PASSWORD"
            proxy="172.16.7.1"
            realm="target.com"
            registrar="target.com"
            expected_cause_code="200"
    />
    <action type="wait" complete="true"/>
  </actions>
</config>

Example: re-invite with new codec

<config>
    <action>
        <action type="codec" disable="all"/>
        <action type="codec" enable="pcma" priority="250"/>
        <action type="codec" enable="pcmu" priority="248"/>

        <!-- call that will last 12 seconds and re-invite every 2 seconds -->
        <action type="call"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="16364990640@125.22.198.115"
            callee="12349099229@sip.mydomain.com"
            max_duration="55" hangup="12"
            auth_username="65454659288" password="adaadzWidD7T"
            realm="sip.mydomain.com"
            re_invite_interval="2"
            rtp_stats="true"
        />
        <action type="wait"/> <!-- this will wait until the call is confirmed -->
        <action type="codec" disable="pcma"/>
        <!-- re-invite will now use pcmu forcing a new session -->
        <action type="wait" ms="3000"/> <!-- this will wait 3 seconds -->
        <action type="codec" enable="pcma" priority="250"/>
        <!-- re-invite will now use pcma forcing a new session -->

        <action type="wait" complete="true"> <!-- Wait until the calls are disconnected -->
    <actions/>
<config/>

Example: Overwriting local contact header

<config><actions>
    <action type="codec" disable="all"/>
    <action type="codec" enable="pcma" priority="250"/>
    <action type="codec" enable="gsm" priority="249"/>
    <action type="codec" enable="pcmu" priority="248"/>

    <action type="call"
        transport="udp"
        caller="+15147371787@fakecustomer.xyz"
        callee="+911@edgeproxy1"
        transport="udp"
        auth_username="20255655"
        password="qntzhpbl"
        realm="sip.flowroute.com"
        rtp_stats="true"
        late_start="false"
        force_contact="sip:+15147371787@10.10.2.5:5777"
        play="/git/voip_patrol/voice_ref_files/reference_8000_12s.wav"
        hangup="5">

    <x-header name="Foo" value="Bar"/>
    </action>
    <action type="wait" complete/>
</actions></config>

Example: WAIT action

wait forever:

<action type="wait" ms="-1"/>

wait until you receive a certain amount of calls

<action type="accept" call_count="x" ... />
<action type="wait" complete="true"/>

wait 5 seconds or one call

<action type="accept" call_count="1" ... />
<action type="wait" ms="5000"/>

Sample JSON output RTP stats report with multiples sessions

one block is generated everytime a session is created

{
 "rtp_stats_0": {
      "rtt": 0,
      "remote_rtp_socket": "10.250.7.88:4028",
      "codec_name": "PCMA",
      "clock_rate": "8000",
      "Tx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 105,
        "kbytes": 16,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      },
      "Rx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 104,
        "kbytes": 16,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      }
    },
    "rtp_stats_1": {
      "rtt": 0,
      "remote_rtp_socket": "10.250.7.89:40230",
      "codec_name": "PCMU",
      "clock_rate": "8000",
      "Tx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 501,
        "kbytes": 78,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      },
      "Rx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 501,
        "kbytes": 78,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      }
    }
}

Example: email reporting

<config>
  <actions>
    <action type="alert"
     email="jchavanton+vp@gmail.com"
     email_from="test@voip-patrol.org"
     smtp_host="smtp://gmail-smtp-in.l.google.com:25"
    />
    <!-- add more test actions here ...  -->
    <action type="wait" complete="true"/>
  </actions>
</config>

accept command parameters

call command parameters

register command parameters

message command parameters

Example: sending a message

<?xml version="1.0"?>
<config>
  <actions>
    <action type="message" label="testing SIP message" transport="udp"
      expected_cause_code="202"
      text="Message in a bottle."
      from="123456@in.the.ocean"
      to_uri="15876580542@in.the.ocean"
      username="123456"
      password="pass"
     />
    <action type="wait" complete="true"/>
  </actions>
</config>

accept_message command parameters

Example: receiving a message

<?xml version="1.0"?>
<config>
  <actions>
    <action type="register" label="register" transport="udp"
      expected_cause_code="200"
      username="123456"
      password="password"
      registrar="pbx.somewhere.time"
     />
    <action type="wait" complete="true"/>
    <action type="accept_message"
      account="123456"
      message_count="1"
     />
    <action type="wait" complete="true"/>
  </actions>
</config>

wait command parameters

Example: codec configuration

<config>
  <actions>
    <action type="codec" disable="all"/>
    <action type="codec" enable="pcmu" priority="250"/>
    <!-- more actions ... -->
    <action type="wait" complete/>
  </actions>
</config>

codec command parameters

Example: TURN configuration

<config>
  <actions>
    <action type="turn" enabled="true" server="x.x.x.x:3478" username="foo" password="bar"/>
    <!-- more actions ... -->
    <action type="wait" complete/>
  </actions>
</config>

turn command parameters

using env variable in scenario actions parameters

Any value starting with VP_ENV will be replaced by the envrironment variable of the same name. Example : username="VP_ENV_USERNAME"

export VP_ENV_PASSWORD=????????
export VP_ENV_USERNAME=username

Docker

voip_patrol/docker$ tree
.
├── build.sh        # docker build command example
├── Dockerfile      # docker build file for Linux Alpine
└── voip_patrol.sh  # docker run example starting

Dependencies

PJSUA2

PJSUA2 : A C++ High Level Softphone API : built on top of PJSIP and PJMEDIA http://www.pjsip.org http://www.pjsip.org/docs/book-latest/PJSUA2Doc.pdf

External tool to test audio quality

PESQ

P.862 : Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs http://www.itu.int/rec/T-REC-P.862

./run_pesq +16000 voice_files/reference.wav voice_files/recording.wav