Awesome

VoIP Patrol

VoIP signaling and media test automaton

Designed to automate end2end and or integration tests.

VoIP patrol will follow a scenario in XML format and will output results in JSON.

Each line in the output file is a separate JSON structure, note that the entire file is not a valid JSON file, this is because VoIP patrol will output results as they become available.

It is possible to test many scenarios that are not easy to test manually like a re-invite with a new codec.

Docker quick start

quick start with docker

Linux Debian building from sources

see commands in Dockerfile

Load test example

load test example

run

./voip_patrol --help

Example: making a test call

<config>
  <actions>
    <action type="call" label="us-east-va"
            transport="tls"
            expected_cause_code="200"
            caller="15147371787@noreply.com"
            callee="12012665228@target.com"
            to_uri="+12012665228@target.com"
            max_duration="20" hangup="16"
            username="VP_ENV_USERNAME"
            password="VP_ENV_PASSWORD"
            rtp_stats="true"
    >
        <x-header name="X-Foo" value="Bar"/>
    </action>
    <!-- note: param value starting with VP_ENV_ will be replaced by environment variables -->
    <!-- note: rtp_stats will include RTP transmission statistics -->
    <!-- note: x-header tag inside an action will append an header -->
    <action type="wait" complete="true"/>
  </actions>
</config>

Sample JSON output

{
  "2": {
    "label": "us-east-va",
    "start": "17-07-2018 00:00:05",
    "end": "17-07-2018 00:00:24",
    "action": "call",
    "from": "15147371787",
    "to": "12012665228",
    "result": "PASS",
    "expected_cause_code": 200,
    "cause_code": 200,
    "reason": "Normal call clearing",
    "callid": "7iYDFukJr-9BOLOmWg.7fZyHZeZUAwao",
    "transport": "TLS",
    "peer_socket": "34.226.136.32:5061",
    "duration": 16,
    "expected_duration": 0,
    "max_duration": 20,
    "hangup_duration": 16,
    "rtp_stats_0": {
      "rtt": 0,
      "remote_rtp_socket": "10.250.7.88:4028",
      "codec_name": "PCMU",
      "clock_rate": "8000",
      "Tx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 816,
        "kbytes": 127,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      },
      "Rx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 813,
        "kbytes": 127,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      }
    }
  }
}

Example: starting a TLS server

./voip_patrol \
   --port 5060 \ # TLS port 5061 +1
   --conf "xml/tls_server.xml" \
   --tls-calist "tls/ca_list.pem" \
   --tls-privkey "tls/key.pem" \
   --tls-cert "tls/certificate.pem" \
   --tls-verify-server \

<config>
  <actions>
     <!-- note: default is the "catch all" account,
          else account as to match called number -->
    <action type="accept"
            account="default"
            hangup="5"
            play_dtmf="0123456789#*"
            play="voice_ref_files/f.wav"
            code="200" reason="YES"
            ring_duration="5"
    />
    <!-- DTMF will be sent using RFC2833 -->
    <!-- note: wait for new incoming calls
               forever and generate test results -->
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and checking for specific header

<config>
  <actions>
    <action type="accept"
            account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-header name="Min-SE"/>
        <check-header name="X-Foo" value="Bar"/>
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and checking for specific header with exact match or regular expression and no match on other

<config>
  <actions>
    <action type="accept"
            match_account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-header name="Min-SE"/>
        <!-- Chech that a header exists -->
        <check-header name="X-Foo" value="Bar"/>
        <!-- Chech that a header exists and have a specific value -->
        <check-header name="From" regex="^.*sip:\+1234@example\.com"/>
        <!-- Chech that a header exists and matches a specific regex -->
        <check-header name="To" regex="^.*sip:\+5678@example\.com" fail_on_match="true"/>
        <!-- Chech that a header exists and NOT matches a specific regex -->
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: accepting calls and searching the message with a regular expression

<config>
  <actions>
    <action type="accept"
            account="default"
            hangup="5"
            code="200" reason="OK"
    >
        <check-message method="INVITE" regex="m=audio(.*)RTP/AVP 0 8.*"/>
        <!-- searching for pcmu pcma in the SDP -->
    </action>
    <action type="wait" ms="-1"/>
  </actions>
</config>

Example: making tests calls with wait_until

Scenario execution is sequential and non-blocking. We can use “wait” command with previously set “wait_until” params to control parallel execution.

Call States
NULL : Before INVITE is sent or received
CALLING : After INVITE is sent
INCOMING : After INVITE is received.
EARLY : After response with To tag.
CONNECTING : After 2xx is sent/received.
CONFIRMED : After ACK is sent/received.
DISCONNECTED

<config>
  <actions>
    <action type="call" label="call#1"
            transport="udp"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="15148888888@noreply.com"
            callee="12011111111@target.com"
    />
    <!-- note: will wait until all tests pass wait_until state -->
    <action type="wait"/>
    <action type="call" label="call#2"
            transport="udp"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="15147777777@noreply.com"
            callee="12012222222@target.com"
    />
    <action type="wait" complete="true"/>
  </actions>
</config>

Example: testing registration

<config>
  <actions>
    <!-- note: proxy param to send to a proxy -->
    <action type="register" label="register target.com"
            transport="udp"
            account="VP_ENV_USERNAME"
            username="VP_ENV_USERNAME"
            password="VP_ENV_PASSWORD"
            proxy="172.16.7.1"
            registrar="target.com"
            expected_cause_code="200"
    />
    <action type="wait" complete="true"/>
  </actions>
</config>

Example: re-invite with new codec

<config>
    <action>
        <action type="codec" disable="all"/>
        <action type="codec" enable="pcma" priority="250"/>
        <action type="codec" enable="pcmu" priority="248"/>

        <!-- call that will last 12 seconds and re-invite every 2 seconds -->
        <action type="call"
            wait_until="CONFIRMED"
            expected_cause_code="200"
            caller="16364990640@125.22.198.115"
            callee="12349099229@sip.mydomain.com"
            max_duration="55" hangup="12"
            username="65454659288" password="adaadzWidD7T"
            re_invite_interval="2"
            rtp_stats="true"
        />
        <action type="wait"/> <!-- this will wait until the call is confirmed -->
        <action type="codec" disable="pcma"/>
        <!-- re-invite will now use pcmu forcing a new session -->
        <action type="wait" ms="3000"/> <!-- this will wait 3 seconds -->
        <action type="codec" enable="pcma" priority="250"/>
        <!-- re-invite will now use pcma forcing a new session -->

        <action type="wait" complete="true"> <!-- Wait until the calls are disconnected -->
    <actions/>
<config/>

Example: Overwriting local contact header

<config><actions>
    <action type="codec" disable="all"/>
    <action type="codec" enable="pcma" priority="250"/>
    <action type="codec" enable="gsm" priority="249"/>
    <action type="codec" enable="pcmu" priority="248"/>

    <action type="call"
        transport="udp"
        caller="+15147371787@fakecustomer.xyz"
        callee="+911@edgeproxy1"
        transport="udp"
        username="20255655"
        password="qntzhpbl"
        rtp_stats="true"
        late_start="false"
        force_contact="sip:+15147371787@10.10.2.5:5777"
        play="/git/voip_patrol/voice_ref_files/reference_8000_12s.wav"
        hangup="5">

    <x-header name="Foo" value="Bar"/>
    </action>
    <action type="wait" complete="true" />
</actions></config>

Example: WAIT action

wait forever:

<action type="wait" ms="-1"/>

wait until you receive a certain amount of calls

<action type="accept" call_count="x" ... />
<action type="wait" complete="true"/>

wait 5 seconds or one call

<action type="accept" call_count="1" ... />
<action type="wait" ms="5000"/>

Sample JSON output RTP stats report with multiples sessions

one block is generated everytime a session is created

{
 "rtp_stats_0": {
      "rtt": 0,
      "remote_rtp_socket": "10.250.7.88:4028",
      "codec_name": "PCMA",
      "clock_rate": "8000",
      "Tx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 105,
        "kbytes": 16,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      },
      "Rx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 104,
        "kbytes": 16,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      }
    },
    "rtp_stats_1": {
      "rtt": 0,
      "remote_rtp_socket": "10.250.7.89:40230",
      "codec_name": "PCMU",
      "clock_rate": "8000",
      "Tx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 501,
        "kbytes": 78,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      },
      "Rx": {
        "jitter_avg": 0,
        "jitter_max": 0,
        "pkt": 501,
        "kbytes": 78,
        "loss": 0,
        "discard": 0,
        "mos_lq": 4.5
      }
    }
}

Example: email reporting

<config>
  <actions>
    <action type="alert"
     email="jchavanton+vp@gmail.com"
     email_from="test@voip-patrol.org"
     smtp_host="smtp://gmail-smtp-in.l.google.com:25"
    />
    <!-- add more test actions here ...  -->
    <action type="wait" complete="true"/>
  </actions>
</config>

accept command parameters

call command parameters

register command parameters

message command parameters

Example: sending a message

<?xml version="1.0"?>
<config>
  <actions>
    <action type="message" label="testing SIP message" transport="udp"
      expected_cause_code="202"
      text="Message in a bottle."
      from="123456@in.the.ocean"
      to_uri="15876580542@in.the.ocean"
      username="123456"
      password="pass"
     />
    <action type="wait" complete="true"/>
  </actions>
</config>

accept_message command parameters

Example: receiving a message

<?xml version="1.0"?>
<config>
  <actions>
    <action type="register" label="register" transport="udp"
      expected_cause_code="200"
      username="123456"
      password="password"
      registrar="pbx.somewhere.time"
     />
    <action type="wait" complete="true"/>
    <action type="accept_message" 
      account="123456"
      message_count="1"
     />
    <action type="wait" complete="true"/>
  </actions>
</config>

wait command parameters

Example: codec configuration

<config>
  <actions>
    <action type="codec" disable="all"/>
    <action type="codec" enable="pcmu" priority="250"/>
    <!-- more actions ... -->
    <action type="wait" complete="true"/>
  </actions>
</config>

codec command parameters

Example: TURN configuration

<config>
  <actions>
    <action type="turn" enabled="true" server="x.x.x.x:3478" username="foo" password="bar"/>
    <!-- more actions ... -->
    <action type="wait" complete="true"/>
  </actions>
</config>

turn command parameters

using multiple accounts

When using multiple accounts, accounts can be created and selected with the following parameters.

using env variable in scenario actions parameters

Any value starting with VP_ENV will be replaced by the envrironment variable of the same name. Example : username="VP_ENV_USERNAME"

export VP_ENV_PASSWORD=????????
export VP_ENV_USERNAME=username

Docker

voip_patrol/docker$ tree
.
├── build.sh        # docker build command example
├── Dockerfile      # docker build file for Linux Alpine
└── voip_patrol.sh  # docker run example starting

Dependencies

PJSUA2

PJSUA2 : A C++ High Level Softphone API : built on top of PJSIP and PJMEDIA http://www.pjsip.org http://www.pjsip.org/docs/book-latest/PJSUA2Doc.pdf

External tool to test audio quality

PESQ

P.862 : Perceptual evaluation of speech quality (PESQ): An objective method for end-to-end speech quality assessment of narrow-band telephone networks and speech codecs http://www.itu.int/rec/T-REC-P.862

./run_pesq +16000 voice_files/reference.wav voice_files/recording.wav