Home

Awesome

Update (july 2017)

pycrate has just been released, intending to replace libmich. Hence, the development of libmich is stopped, and all efforts will be put on pycrate yet.

What is libmich

Libmich is a library written for Python 2, that primarily supports encoding and decoding several digital formats, making a binary file or network stream easily understandable for a human, and easier to process within applications. Many formats and routines are tailored for helping to deal with mobile-network protocols and traffic. Moreover, it has a simple ASN.1 compiler plus BER and PER encoders for dealing with all formats defined in ASN.1.

Installation

Operating systems and Python version

The library is made to work with Python 2.7. Python 3 is not supported, mainly because all binary buffers are handled with the Python str object. It works on Windows and Linux, and should work on any other operating system running Python 2.

Dependencies

There is no mandatory package dependency. There are two optional package dependencies:

Automatic installation

An installation script is available. It precompiles all ASN.1 modules before installing the whole library within your Python package directory:

python setup.py install

Manual installation

You can also install the library manually. For this, just copy the whole libmich/ sub-directory into one of your directory which is referenced within your PYTHONPATH environment variable. E.g. you can copy it in your $Python_install_dir/Lib/site-packages/ directory. If you need to use ASN.1 modules, you will have to compile them, too. Just look at the ASN.1 README for this.

Testing the library

The libmich/utils/perf.py file provides some routines for testing the speed of some encoders / decoders. It can be used to check if the library works correctly:

>>> from libmich.utils.perf import *
>>> main()
[...]
total duration: 38.8725 sec.

If something breaks here, this means something is wrong with the current code or the installation step.

License

The whole library is licensed under GPLv2: all licensed files have an header making it self-explanatory.

The few files which have been imported from external projects have not this explicit license header. Here is the list of external files, or code derived from external projects:

Contact and support

As the unique developper of the library, I am the only person to contact: michau [dot] benoit [at] gmail [dot] com

Extending the library

If you are willing to extend the library, do not hesitate to contact me by email or through the github service. Any patch or submission is very welcome! Moreover, in case you are using this library in any of your project and you find it useful, do not hesitate to drop me an email. It is always a pleasure to know where code provided on the Internet can end up...

Authors

Benoit Michau

Thanks to FlUxIuS for providing routines for SMS encoding and decoding.

A little bit of history

The initial version of the library made public after my former employer France Telecom allowed me to do so. My current employer ANSSI has also allowed me to continue extending it, providing support for more and more formats.

Usage

For the most important library objects, docstrings are provided within the code. The following list provide information of what to expect of core and important parts of the library.

Main basic core objects

The main objects for the library are defined in libmich/core/element.py. Those objects are used to specify fields of bytes or bits, that form a binary file or stream.

Here are the most basic elements:

Those 3 basic elements have common attributes and methods:

Str element has specific attributes:

Int element has specific attributes:

Bit element has specific attributes:

>>> from libmich.core.element import *
>>> from libmich.utils.repr import show, bin, hex
>>>
>>> a = Str('MyStream', Pt='azerty1234', Len=10)
>>> a
'azerty1234'
>>> a()
'azerty1234'
>>> str(a)
'azerty1234'
>>> show(a)
<[MyStream] : 'azerty1234'>
>>> a.Repr='hex'
>>> show(a)
<[MyStream] : 0x617a6572747931323334>
>>> a.Repr='bin'
>>> show(a)
<[MyStream] : 0b01100001011110100110010101110010011101000111100100110001001100100011001100110100>
>>> a.map('qsdfgh4567')
>>> a()
'qsdfgh4567'
>>> str(a)
'qsdfgh4567'
>>> show(a)
<[MyStream] : 0b01110001011100110110010001100110011001110110100000110100001101010011011000110111>
>>> a.Repr='hum'
>>> show(a)
<[MyStream] : 'qsdfgh4567'>
>>>
>>> b = Int('MyInt', Pt=25, Type='int96', Dict={0:'none', 1:'more'})
>>> b
25
>>> b()
25
>>> show(b)
<[MyInt] : 25>
>>> str(b)
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x19'
>>> b._endian = 'little'
>>> str(b)
'\x19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
>>> hex(b)
'190000000000000000000000'
>>> bin(b)
'000110010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'
>>> b.map('abcdefghijkl')
>>> show(b)
<[MyInt] : 33554238638682438954073154145>
>>> b()
33554238638682438954073154145L
>>> b.showattr()
CallName : 'MyInt'
ReprName : ''
Pt : 25
PtFunc : None
Val : 33554238638682438954073154145L
Len : 12
Type : 'int96'
Dict : <type 'dict'>
DictFunc : None
Repr : 'hum'
Trans : False
TransFunc : None
>>> b.map('\x01'+11*'\0')
>>> show(b)
<[MyInt] : '1 : more'>
>>> b()
1
>>>
>>> c = Bit('MyBits', Pt=3, BitLen=7)
>>> c
0b0000011
>>> c()
3
>>> show(c)
<[MyBits] : 0b0000011>
>>> c > 109
>>> show(c)
<[MyBits] : 0b1101101>
>>> c()
109
>>> str(c)
'\xda'
>>> c.map('\x82')
>>> show(c)
<[MyBits] : 0b1000001>
>>> c()
65
>>> c < None
>>> c > 7
>>> show(c)
<[MyBits] : 0b0000111>
>>> c()
7

Main constructed core objects

All these 3 basic core objects can be assembled into Layer objects. Each Layer element can contain Str, Int and Bit elements, plus other Layer elements (the class is a recursive object).

The main attribute of the Layer element is the constructorList for the object itself, which is converted to elementList during object instantiation. It is a list containing elements which will be handled in the given ordered and grouped way. A Layer can be enforced to be byte-aligned (default behavior), or not (in case you have byte-unaligned Bit elements inside). Lots of attributes and methods are exposed in order to manage Layer elements similarly than basic elements, but also like lists or dictionnaries. Another important method is the reautomatize one, which restores all elements to are automated within the Layer instance.

As a quick example, the testTLV class from the libmich/core/element.py can be used. Here is its definition:

class testTLV(Layer):
    _byte_aligned = True
    constructorList = [
        Int('T', ReprName='Tag', Type='uint8', Dict={0:'Reserved', 1:'Tag1', 2:'Tag2', 5:'Tag5'}),
        Bit('F1', ReprName='Flag1', Pt=0, BitLen=1),
        Bit('F2', ReprName='Flag2', Pt=1, BitLen=2),
        Bit('res', ReprName='Reserved', Pt=0, BitLen=13),
        # length in bytes (including header, excepted Tag)
        Int('L', ReprName='Length', Type='uint8' ),
        Str('V', ReprName='Value', Pt='default value'),
        ]

    def __init__(self, **kwargs):
        Layer.__init__(self, **kwargs)
        # automating the computation of Length at runtime
        self.L.Pt = self.V
        self.L.PtFunc = lambda X: len(X)+3
        # automating the parsing of Value when calling .map(buffer)
        self.V.Len = self.L
        self.V.LenFunc = lambda X: int(X)-3

Here is how it is behaving within Python:

>>> t = testTLV()
>>> t
<[testTLV]: T(Tag):None, F1(Flag1):0b0, F2(Flag2):0b01, res(Reserved):0b0000000000000, L(Length):16, V(Value):'default value'>
>>> show(t)
### [testTLV] ###
 <Tag [T] : None>
 <Flag1 [F1] : 0b0>
 <Flag2 [F2] : 0b01>
 <Reserved [res] : 0b0000000000000>
 <Length [L] : 16>
 <Value [V] : 'default value'>
>>> str(t)
'\x00 \x00\x10default value'
>>> t()
'\x00 \x00\x10default value'
>>> len(t)
17
>>> t.T() # getting the Tag value
0
>>> t.F1() # getting the F1 flag value
0
>>> t.T > 5 # setting the Tag value to 5
>>> t.F1 > 1
>>> t.V > 'this is an damned example'
>>> show(t)
### [testTLV] ###
 <Tag [T] : '5 : Tag5'>
 <Flag1 [F1] : 0b1>
 <Flag2 [F2] : 0b01>
 <Reserved [res] : 0b0000000000000>
 <Length [L] : 28>
 <Value [V] : 'this is an damned example'>
>>> str(t)
'\x05\xa0\x00\x1cthis is an damned example'
>>> hex(t)
'05a0001c7468697320697320616e2064616d6e6564206578616d706c65'
>>> bin(t)
'0000010110100000000000000001110001110100011010000110100101110011001000000110100101110011001000000110000101101110001000000110010001100001011011010110111001100101011001000010000001100101011110000110000101101101011100000110110001100101'
>>> show(t[0]) # showing the 1st contained element
<Tag [T] : 5>
>>> show(t[1]) # showing the 2nd contained element
<Flag1 [F1] : 0b1>
>>> show(t[-1]) # showing the last contained element
<Value [V] : 'this is an damned example'>
>>>
>>> t.map('\x02\x40\0\x21'+30*'A')
>>> show(t)
### [testTLV] ###
 <Tag [T] : '2 : Tag2'>
 <Flag1 [F1] : 0b0>
 <Flag2 [F2] : 0b10>
 <Reserved [res] : 0b0000000000000>
 <Length [L] : 33>
 <Value [V] : 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'>
>>> t.map('\x02\x40\0\x21'+80*'A') # the mapping of the Value buffer gets truncated due to Length
>>> show(t)
### [testTLV] ###
 <Tag [T] : '2 : Tag2'>
 <Flag1 [F1] : 0b0>
 <Flag2 [F2] : 0b10>
 <Reserved [res] : 0b0000000000000>
 <Length [L] : 33>
 <Value [V] : 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'>
>>> len(t)
34
>>> len(t.V)
30
>>> t.reautomatize() # restore the automatic computation of Length

Lastly, the Block object helps to manage multiple Layer objects with a hierarchy. It allows to define dependencies between them, like headers and payloads.

You can find more examples of how to use Layer and Block objects at the end of the libmich/core/element.py file, within the examples sub-directory (with png, IKEv2, MPEG2), and more globally in all formats definition provided within the libmich/formats/ sub-directory.

Other core objects

Other files are provided in the libmich/core/ sub-directory. Here is the list of them without much more explanation:

Formats supported

The libmich/formats/ sub-directory contains many Python files defining lots of digital formats. Here is the list of files with a tiny explanation on what it does. For further explanations, see the docstrings or directly the source code.

IP-oriented protocols:

Mobile-network-oriented protocols:

Radio-oriented protocols and formats:

File and media container formats:

ASN1 objects and modules

Libmich provides a tiny ASN.1 compiler, capable of compiling most of the ETSI / 3GPP specifications. Furthermore, PER and BER encoders are provided too. All ASN.1 protocols and Python files are in libmich/asn1/. Please refer to the dedicated ASN.1 README for further information.

Currently, the following ASN.1 protocols are supported and directly available from the library:

Other useful routines

Basic utility functions and classes are provided in the libmich/utils/ sub-directory:

The library provides routines and EPC network stacks that can be of interest to developers working with mobile network applications. The libmich/mobnet/ sub-directory provides the following files: