Awesome

diafuzzer

Diameter fuzzer, based on specifications of Diameter applications following rfc 3588 / 6733

Overview

Diafuzzer is composed of several different tools:

• simple and accurate Diameter callflows, based on pcap traces
• script language to perform additional functions such as logging, database lookup or others
• detailed description of Diameter applications defined at 3GPP and ETSI
• runtime helpers to perform unit testing and fuzz testing

It is developped in Python, with four major components:

• Diameter.py: contains Python classes named Msg and Avp which mirror wire. Both classes implement default values for unspecified fields, and compute fields which value depends on other fields.
• Dia.py: parses dia files. It contains Python classes to model message structure and avp cardinality, plus their datatype.
• pcap2scn.py and pcap2pdu.py: shall be given a pcap file as first argument. They will both produce a Python form of Diameter PDUs contained in trace, based on Diameter.py module above. pcap2scn.py will produce a client and/or server scenario, made of interleaved send and receive operations, whereas pcap2pdu.py will only dump PDUs as objects.
• unit.py and fuzz.py: shall be given four arguments, amongst which a scenario, and a role. unit.py will replay the given scenario, and fuzz.py will use scenario as a baseline to produce fuzzing operations.

It is not compatible with Python3.

Diameter.py

Avp usage

Make sure to import Avp class from Diameter module

>>> from Diameter import Avp

To create an Avp instance:

>>> a = Avp()
>>> a
Avp(code=0,vendor=0)

The following parameters can be given when building a new instance:

For example, to create an Origin-Host Avp instance:

>>> Avp(code=264, data='hss.openims.test')
Avp(code=264, vendor=0, data='hss.openims.test')

Several ways are provided to supply values instead of raw bytes:

>>> Avp(code=266, u32=323)
Avp(code=266, vendor=0, data='\x00\x00\x01C')

The table below gives parameter names that can be used, and their format:

To create a wire-ready version of the instance:

>>> Avp(code=266, u32=323).encode()
'\x00\x00\x01\n\x00\x00\x00\x0c\x00\x00\x01C'

To create an instance from raw bytes:

>>> Avp.decode('\x00\x00\x01\n\x00\x00\x00\x0c\x00\x00\x01C')
Avp(code=266, vendor=0, data='\x00\x00\x01C')

Msg usage

Make sure to import Msg and Avp classes from Diameter module

>>> from Diameter import Msg, Avp

To create a Msg instance:

>>> m = Msg()
>>> m
Msg(code=0, app_id=0x0, avps=[
])

For example, to create a Device-Watchdog Request Msg instance:

>>> m = Msg(code=280, R=True)
>>> m
Msg(R=True, code=280, app_id=0x0, avps=[
])

In order to create a real-world Capabilities-Exchange Request Msg instance:

>>> m = Msg(R=True, code=257, app_id=0x0, avps=[
...   Avp(code=264, M=True, vendor=0, data='127.0.0.1'),
...   Avp(code=296, M=True, vendor=0, data='org.domain.com'),
...   Avp(code=257, M=True, vendor=0, data='\x00\x01\x7f\x00\x00\x01'),
...   Avp(code=266, M=True, vendor=0, data='\x00\x00\x00\x00'),
...   Avp(code=269, M=True, vendor=0, data='Mu Service Analyzer Diameter Implementation'),
...   Avp(code=299, M=True, vendor=0, data='\x00\x00\x00\x00'),
...   Avp(code=260, M=True, vendor=0, avps=[
...     Avp(code=266, M=True, vendor=0, data='\x00\x00(\xaf'),
...     Avp(code=258, M=True, vendor=0, data='\x01\x00\x00\x00'),
...   ]),
... ])
>>> m
Msg(R=True, code=257, app_id=0x0, avps=[
  Avp(code=264, M=True, vendor=0, data='127.0.0.1'),
  Avp(code=296, M=True, vendor=0, data='org.domain.com'),
  Avp(code=257, M=True, vendor=0, data='\x00\x01\x7f\x00\x00\x01'),
  Avp(code=266, M=True, vendor=0, data='\x00\x00\x00\x00'),
  Avp(code=269, M=True, vendor=0, data='Mu Service Analyzer Diameter Implementation'),
  Avp(code=299, M=True, vendor=0, data='\x00\x00\x00\x00'),
  Avp(code=260, M=True, vendor=0, avps=[
    Avp(code=266, M=True, vendor=0, data='\x00\x00(\xaf'),
    Avp(code=258, M=True, vendor=0, data='\x01\x00\x00\x00'),
  ]),
])

Encoding and decoding to/from raw bytes work the same as for Avp:

>>> m = Msg.decode('\x01\x00\x00\xbc\x80\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x08@\x00\x00\x11127.0.0.1\x00\x00\x00\x00\x00\x01(@\x00\x00\x16org.domain.com\x00\x00\x00\x00\x01\x01@\x00\x00\x0e\x00\x01\x7f\x00\x00\x01\x00\x00\x00\x00\x01\n@\x00\x00\x0c\x00\x00\x00\x00\x00\x00\x01\r@\x00\x003Mu Service Analyzer Diameter Implementation\x00\x00\x00\x01+@\x00\x00\x0c\x00\x00\x00\x00\x00\x00\x01\x04@\x00\x00 \x00\x00\x01\n@\x00\x00\x0c\x00\x00(\xaf\x00\x00\x01\x02@\x00\x00\x0c\x01\x00\x00\x00')
>>> m
Msg(R=True, code=257, app_id=0x0, avps=[
  Avp(code=264, M=True, vendor=0, data='127.0.0.1'),
  Avp(code=296, M=True, vendor=0, data='org.domain.com'),
  Avp(code=257, M=True, vendor=0, data='\x00\x01\x7f\x00\x00\x01'),
  Avp(code=266, M=True, vendor=0, data='\x00\x00\x00\x00'),
  Avp(code=269, M=True, vendor=0, data='Mu Service Analyzer Diameter Implementation'),
  Avp(code=299, M=True, vendor=0, data='\x00\x00\x00\x00'),
  Avp(code=260, M=True, vendor=0, avps=[
    Avp(code=266, M=True, vendor=0, data='\x00\x00(\xaf'),
    Avp(code=258, M=True, vendor=0, data='\x01\x00\x00\x00'),
  ]),
])
>>> m.encode()
'\x01\x00\x00\xbc\x80\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x08@\x00\x00\x11127.0.0.1\x00\x00\x00\x00\x00\x01(@\x00\x00\x16org.domain.com\x00\x00\x00\x00\x01\x01@\x00\x00\x0e\x00\x01\x7f\x00\x00\x01\x00\x00\x00\x00\x01\n@\x00\x00\x0c\x00\x00\x00\x00\x00\x00\x01\r@\x00\x003Mu Service Analyzer Diameter Implementation\x00\x00\x00\x01+@\x00\x00\x0c\x00\x00\x00\x00\x00\x00\x01\x04@\x00\x00 \x00\x00\x01\n@\x00\x00\x0c\x00\x00(\xaf\x00\x00\x01\x02@\x00\x00\x0c\x01\x00\x00\x00'

Dia.py

Dia file format

The file format is based on a sequence of sections, each section beginning with

@<keyword> ...
...

Depending on the keyword, arguments may follow, and content of section may not be empty.

The example below defines one of the S13 application message:

@id     16777252
@name   S13

@inherits       ietf-avps
@inherits       3gpp-avps

@messages
ME-Identity-Check-Request ::= <Diameter Header: 324, REQ, PXY, 16777252>
      < Session-Id >
      [ Vendor-Specific-Application-Id ]
      { Auth-Session-State }
      { Origin-Host }
      { Origin-Realm }
      [ Destination-Host ]
      { Destination-Realm }
      { Terminal-Information }
      [ User-Name ]
  *   [ AVP ]
  *   [ Proxy-Info ]
  *   [ Route-Record ]

Application are defined using an id and a name:

@id     16777252
@name   S13

Most applications will use AVPs inherited from both IETF and 3GPP:

@inherits       ietf-avps
@inherits       3gpp-avps

Messages are defined using their Command Code format:

@messages
ME-Identity-Check-Request ::= <Diameter Header: 324, REQ, PXY, 16777252>
      < Session-Id >
      [ Vendor-Specific-Application-Id ]
      { Auth-Session-State }
      { Origin-Host }
      { Origin-Realm }
      [ Destination-Host ]
      { Destination-Realm }
      { Terminal-Information }
      [ User-Name ]
  *   [ AVP ]
  *   [ Proxy-Info ]
  *   [ Route-Record ]

AVPs are defined by their code, datatype and flags:

@avp_types
User-Name                                       1               UTF8String              M
User-Password                                   2               OctetString             M
NAS-IP-Address                                  4               OctetString             M
NAS-Port                                        5               Unsigned32              M

For AVP of type Grouped, the corresponding Command Code format must be defined in a grouped section:

@grouped
Vendor-Specific-Application-Id ::= <AVP Header: 260>
      { Vendor-Id }
      [ Auth-Application-Id ]
      [ Acct-Application-Id ]

Failed-AVP ::= <AVP Header: 279>
 1*   { AVP }

For AVP of type Enumerated, the corresponding values must defined in an enum section, with AVP name in argument:

@enum Timezone-Flag
UTC                                             0
LOCAL                                           1
OFFSET                                          2

@enum QoS-Semantics
QOS_DESIRED                                     0
QOS_AVAILABLE                                   1
QOS_DELIVERED                                   2
MINIMUM_QOS                                     3
QOS_AUTHORIZED                                  4

Supported applications

Dia.py

This module holds the following classes:

• Avp: an AVP, most notably its datatype, plus its Command Code format or legitimate values when applicable.
• QualifiedAvp: an AVP and its multiplicity, in the context of a Command Code format.
• Msg: a message, most notably a command code, plus a list a QualifiedAvp instance.
• Application: an application, containing a set of messages.
• Directory: a set of applications. The file .dia-cache contains a Directory instance containing the compiled version of dia files.

Dia files are verified and compiled. This serves two purposes:

• references are resolved: for example if a message definition references an AVP, one makes sure that AVP is defined
• once verified, the whole model can be compiled to Python and pickled for later faster reuse

Typical usage of pickled model include:

• conformance verification
• fuzzing, based on associated datatype or inner structure

The set of all supported applications are grouped in a Directory instance, which is pickled into .dia-cache file.

Thus any change to dia files contained in specs directory must be followed by a generation of this .dia-cache file

The script generate-cache.py will generate this file, and print supported applications:

$ ./generate-cache.py
creating Directory instance, this might take a while ...
created in 0:00:15.830600 dumping to .dia-cache
contains the following applications:
base_rfc6733		0 (0x0)
credit_rfc4006		4 (0x4)
eap_rfc4072		5 (0x5)
mip6a_rfc5778		8 (0x8)
mip6i_rfc5778		7 (0x7)
mobipv4_rfc4004		2 (0x2)
nasreq_rfc7155		1 (0x1)
sip_rfc4740		6 (0x6)
Cx		16777216 (0x1000000)
S13		16777252 (0x1000024)
S6a		16777251 (0x1000023)
S6b		16777272 (0x1000038)
S7a		16777308 (0x100005c)
S9		16777267 (0x1000033)
Sh		16777217 (0x1000001)
SWx		16777265 (0x1000031)
Rx		16777236 (0x1000014)
Gx		16777224 (0x1000008)
Gxx		16777266 (0x1000032)
SWm		16777264 (0x1000030)
SLg		16777255 (0x1000027)
SLh		16777291 (0x100004b)
S6c		16777312 (0x1000060)
SGd		16777313 (0x1000061)

Adding or modifying an AVP

AVPs are defined in one of the files below:

• specs/ietf-avps.dia
• specs/3gpp-avps.dia
• specs/3gpp2-avps.dia
• specs/etsi-avps.dia

@avp_types
User-Name                                       1               UTF8String              M
User-Password                                   2               OctetString             M
NAS-IP-Address                                  4               OctetString             M
NAS-Port                                        5               Unsigned32              M
...
@grouped
Vendor-Specific-Application-Id ::= <AVP Header: 260>
      { Vendor-Id }
      [ Auth-Application-Id ]
      [ Acct-Application-Id ]

...
@enum QoS-Semantics
QOS_DESIRED                                     0
QOS_AVAILABLE                                   1
QOS_DELIVERED                                   2
MINIMUM_QOS                                     3
QOS_AUTHORIZED                                  4

There are up to two places to be modified:

• for Grouped AVP: declare new AVP in @avp_types section, and add Command Code format in @grouped section
• for Enumerated AVP: declare new AVP in @avp_types section, and add a new @enum section with the name of new AVP, containing the possible values and their meaning
• for the rest of datatypes: declare AVP in @avp_types section

Once the modification is done, generate .dia-cache as outlined above.

Adding a new application

When adding a new application, one needs to append the name of the new dia file in Dia.py, at the beginning of Directory __init__ method.

For example to add a fictive S42 application, one has to create a dia file called specs/S42.dia, and modify Dia.py accordingly:

class Directory:
  def __init__(self, *args):
    self.ids = {}
    self.apps = []

    if len(args) == 0:
      args = [
        # IETF applications
        'specs/base_rfc6733.dia', 'specs/credit_rfc4006.dia',
        'specs/eap_rfc4072.dia', 'specs/mip6a_rfc5778.dia',
        'specs/mip6i_rfc5778.dia', 'specs/mobipv4_rfc4004.dia',
        'specs/nasreq_rfc7155.dia', 'specs/sip_rfc4740.dia',
        # 3GPP applications
        'specs/Cx.dia', 'specs/S13.dia', 'specs/S6a.dia',
        'specs/S6b.dia', 'specs/S7a.dia', 'specs/Sh.dia',
        'specs/SWx.dia', 'specs/Rx.dia', 'specs/Gx.dia', 'specs/Gxx.dia',
        'specs/SWm.dia', 'specs/SLg.dia', 'specs/SLh.dia',
        
        # new S42 application
        'specs/S42.dia']

When supporting a new application, it can be convenient to check for AVP definitions and references. The script lean.py can be used to check definitions:

$ ./lean.py specs/S42.dia

To perform in-place formatting, use -i option:

$ ./lean.py -i specs/S42.dia

Once the modification is done, generate .dia-cache as outlined above.

Tagging

This operation allows to tag Diameter.Msg and Diameter.Avp instances with their corresponding model.

pcap2pdu.py

Goal and arguments

The goal is to extract Diameter messages contained in pcap to their Python form.

$ ./pcap2pdu.py <pcap>
[Diameter messages in their Python form follow]

Note that only pcap format is supported. In particular, pcapng and snoop formats are not supported. tshark must be in the path, and is used to dissect frames and retrieve Diameter messages.

Usage

Pcap file is processed using tshark. Pdml stream is analyzed in order to identify Diameter PDUs, which is then decoded using Diameter.Msg.decode function. Handling of IP fragmentation, or transport segmentation is done by tshark. An example of usage is given below:

$ ./pcap2pdu.py captures/Cx.pcap | head -n20
# frame 1
Msg(R=True, P=True, code=300, app_id=0x1000000, avps=[ # User-Authorization-Request
  # Session-Id
  Avp(code=263, M=True, vendor=0, data='icscf.open-ims.test;457324016;102'),
  # Origin-Host
  Avp(code=264, M=True, vendor=0, data='icscf.open-ims.test'),
  # Origin-Realm
  Avp(code=296, M=True, vendor=0, data='open-ims.test'),
  # Destination-Realm
  Avp(code=283, M=True, vendor=0, data='open-ims.test'),
  # Vendor-Specific-Application-Id
  Avp(code=260, M=True, vendor=0, avps=[
    # Vendor-Id
    Avp(code=266, M=True, vendor=0, u32=10415),
    # Auth-Application-Id
    Avp(code=258, M=True, vendor=0, u32=16777216),
  ]),
  # Auth-Session-State
  Avp(code=277, M=True, vendor=0, u32=1),

pcap2scn.py

Goal and arguments

The goal is to analyze Diameter transactions and generate smart Python scenarios made of send and receive sequences.

$ ./pcap2scn.py [--client <client scenario>] [--server <server scenario>] <pcap>

Note that only pcap format is supported. In particular, pcapng and snoop formats are not supported. tshark must be in the path, and is used to dissect frames and retrieve Diameter messages.

Usage

To generate a Ro client scenario scenarios/ro-client.scn based on a trace captures/Ro.pcap:

$ ./pcap2scn.py --client scenarios/ro-client.scn captures/Ro.pcap
detected a flow 10.201.9.245:50957 -> 10.201.9.11:3868
anchor ('/code=443/code=444', 'Subscription-Id-Data'), propagating to [(3, True, Avp(code=444, M=True, vendor=0, data='919080000016')), (5, True, Avp(code=444, M=True, vendor=0, data='919080000016'))]
anchor ('/code=283', 'Destination-Realm'), propagating to [(2, False, Avp(code=296, M=True, vendor=0, data='comverse.com')), (3, True, Avp(code=283, M=True, vendor=0, data='comverse.com')), (4, False, Avp(code=296, M=True, vendor=0, data='comverse.com')), (5, True, Avp(code=283, M=True, vendor=0, data='comverse.com')), (6, False, Avp(code=296, M=True, vendor=0, data='comverse.com'))]
...

pcap2scn.py will perform the same pcap2pdu.py plus:

• differentiate client and server sides, in order to generate send and receive operations
• detect reused AVP values
• implement copy and paste in subsequent messages

For example, an S6a server must use in the answer the Session-Id value received in the Authentication-Information Request. The reuse is detected by pcap2scn.py, and is implemented in scenario.

The copy operation takes the value from the received Session-Id:

  # frame 1
  m = Msg.recv(f)
  assert(m.code == 318)
  assert(m.R)
  tsxs[0] = (m.e2e_id, m.h2h_id)
...
  session_id = m.eval_path('/code=263').data

The paste operation is then performed as follows:

  # frame 2
  m = Msg(P=True, code=318, app_id=0x1000023, avps=[
    Avp(code=263, M=True, vendor=0, data=session_id),
...

Scenarios

A scenario must contain a run function, which will be given at runtime a stream which can be used to send and receive Diameter messages. The following scenario has been generated using pcap2scn.py:

def run(f, args={}):
  tsxs = [()]*3

  # frame 1
  m = Msg(R=True, code=272, app_id=0x4, avps=[
    Avp(code=263, M=True, vendor=0, data='nxl;api;1263278878147'),
...
  ])
  m.send(f)
  tsxs[0] = (m.e2e_id, m.h2h_id) 
  # frame 2
  m = Msg.recv(f)
  assert(m.code == 272)
  assert(not m.R)
  assert(tsxs[0] == (m.e2e_id, m.h2h_id))

  # frame 3
  m = Msg(R=True, code=272, app_id=0x4, avps=[
...

unit.py and fuzz.py

Both programs will use SCTP as transport layer.

Arguments

$ ./unit.py -h
usage: unit.py [-h] [--local-addresses LOCAL_ADDRESSES]
               [--local-port LOCAL_PORT] [--local-hostname LOCAL_HOSTNAME]
               [--local-realm LOCAL_REALM]
               {client,clientloop,server} scenario ...

positional arguments:
  {client,clientloop,server}
                        Role: client, clientloop or server
  scenario              Python scenario to run
  remote                Target IP:port

optional arguments:
  -h, --help            show this help message and exit
  --local-addresses LOCAL_ADDRESSES
                        Local IPv4 addresses that will the addresses used in
                        SCTP multihoming
  --local-port LOCAL_PORT
                        Local SCTP port
  --local-hostname LOCAL_HOSTNAME
                        Local Diameter Host, used in DWA as Origin-Host, and
                        may be used as local_hostname
  --local-realm LOCAL_REALM
                        Local Diameter realm, used in DWA as Origin-Realm, and
                        may be used as local_realm

$ ./fuzz.py -h
usage: fuzz.py [-h] [--local-addresses LOCAL_ADDRESSES]
               [--local-port LOCAL_PORT] [--local-hostname LOCAL_HOSTNAME]
               [--local-realm LOCAL_REALM]
               {client,server} scenario ...

positional arguments:
  {client,server}       Role: client or server
  scenario              Python scenario to run
  remote                Target IP:port

optional arguments:
  -h, --help            show this help message and exit
  --local-addresses LOCAL_ADDRESSES
                        Local IPv4 addresses that will the addresses used in
                        SCTP multihoming
  --local-port LOCAL_PORT
                        Local SCTP port
  --local-hostname LOCAL_HOSTNAME
                        Local Diameter Host, used in DWA as Origin-Host, and
                        may be used as local_hostname
  --local-realm LOCAL_REALM
                        Local Diameter realm, used in DWA as Origin-Realm, and
                        may be used as local_realm

Device-Watchdog handling

Diameter mandates the use of Device-Watchdog Request and Answer to verify connection states. These messages will be used after connection establishment, but they may appear during scenario, at random places.

A thin layer will shield user scenario from received Device-Watchdog Request, and will automatically send Device-Watchdog Answer.

The initial behaviour using a SOCK_SEQPACKET AF_UNIX socketpair has been modifed, in order to be more portable. It now uses a SOCK_STREAM AF_UNIX socketpair, with records being delineated via their length. It is implented in two places:

• scenario.py/pack_frame and scenario.py/unpack_frame which respectively append a 4-byte big-endian of following frame, and conversely.
• scenario.py/dwr_handler which automatically replies to received DWR, and forwards other messages to scenario instance.

However answering to DWR requires to set a suitable Origin-Host and Origin-Realm, which is clearly not dependent of the scenario. unit.py and fuzz.py both accept to set a local hostname and a local realm. Scenario can access these variables via local_hostname and local_realm.

Fuzzing process

The following pseudocode illustrates how fuzzing is performed:

• the scenario is run once, in order to capture exchanged PDU, and to ensure a single shot runs correctly
• mutations derived from model are computed
• the scenario is run as many time needed in order to test all mutations

run scenario once, and collect exchanged messages
tag exchanged messages

for each sent message:
  for each AVP contained in sent message:
    collect mutations based on datatype
    collect mutations based on qualification in context

for each mutation:
  run scenario and execute mutation

As a matter of fact, a scenario will generate a bound number of mutations. The behaviour of fuzzing process is deterministic and time needed to perform fuzzing can evaluated prior to start.

During fuzzing, each run may throw exceptions, which will be caught by fuzz.py and processed to keep on fuzzing:

• socket.timeout, which is raised by Diameter.Msg.recv when a timeout is specificed. When unspecified, this timeout will be set to 5s. When it is desirable to wait for a message without a timeout, one must use Msg.recv(f, None).
• Diameter.RecvMismatch, which may be raised by scenario when receiving an unexpected message. Scenarios generated by pcap2scn.py will check code and Request flag, and raise this exception if any does not meet the expectations.