Awesome
pretty_j1939
python3 libs and scripts for pretty-printing J1939 candump logs.
This package can:
- pretty-print J1939 traffic captured in candump logs AND
- convert a J1939 Digital Annex (Excel) file into a JSON structure for use in the above
Some examples of pretty printing
Formatted content (one per line) next to candump data:
$ pretty_j1939.py --candata --format example.candump.txt | head
(1543509533.000838) can0 10FDA300#FFFF07FFFFFFFFFF ; {
; "DA": "All(255)",
; "PGN": "EEC6(64931)",
; "SA": "Engine #1( 0)",
; "Engine Variable Geometry Turbocharger Actuator #1": "2.8000000000000003 [%]"
; }
(1543509533.000915) can0 18FEE000#FFFFFFFFB05C6800 ; {
; "DA": "All(255)",
; "PGN": "VD(65248)",
; "SA": "Engine #1( 0)",
Single-line contents next to candump data:
$ pretty_j1939.py --candata example.candump.txt | head
(1543509533.000838) can0 10FDA300#FFFF07FFFFFFFFFF ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"EEC6(64931)","Engine Variable Geometry Turbocharger Actuator #1":"2.8000000000000003 [%]"}
(1543509533.000915) can0 18FEE000#FFFFFFFFB05C6800 ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"VD(65248)","Total Vehicle Distance":"854934.0 [m]"}
(1543509533.000991) can0 08FE6E0B#0000000000000000 ; {"SA":"Brakes - System Controller( 11)","DA":"All(255)","PGN":"HRW(65134)","Front Axle, Left Wheel Speed":"0.0 [kph]","Front axle, right wheel speed":"0.0 [kph]","Rear axle, left wheel speed":"0.0 [kph]","Rear axle, right wheel speed":"0.0 [kph]"}
(1543509533.001070) can0 18FDB255#FFFFFFFF0100FFFF ; {"SA":"Diesel Particulate Filter Controller( 85)","DA":"All(255)","PGN":"AT1IMG(64946)","Aftertreatment 1 Diesel Particulate Filter Differential Pressure":"0.1 [kPa]"}
(1543509533.001145) can0 0CF00400#207D87481400F087 ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"EEC1(61444)","Engine Torque Mode":"2 (Unknown)","Actual Engine - Percent Torque (Fractional)":"0.0 [%]","Driver's Demand Engine - Percent Torque":"0 [%]","Actual Engine - Percent Torque":"10 [%]","Engine Speed":"649.0 [rpm]","Source Address of Controlling Device for Engine Control":"0 [SA]","Engine Demand - Percent Torque":"10 [%]"}
(1543509533.001220) can0 18FF4500#6D00FA00FF00006A ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"PropB_45(65349)","Manufacturer Defined Usage (PropB_PDU2)":"0x6d00fa00ff00006a"}
(1543509533.001297) can0 18FEDF00#82FFFFFF7DE70300 ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"EEC3(65247)","Nominal Friction - Percent Torque":"5 [%]","Estimated Engine Parasitic Losses - Percent Torque":"0 [%]","Aftertreatment 1 Exhaust Gas Mass Flow Rate":"199.8 [kg/h]","Aftertreatment 1 Intake Dew Point":"0 (00 - Not exceeded the dew point)","Aftertreatment 1 Exhaust Dew Point":"0 (00 - Not exceeded the dew point)","Aftertreatment 2 Intake Dew Point":"0 (00 - Not exceeded the dew point)","Aftertreatment 2 Exhaust Dew Point":"0 (00 - Not exceeded the dew point)"}
(1543509533.001372) can0 1CFE9200#FFFFFFFFFFFFFFFF ; {"SA":"Engine #1( 0)","DA":"All(255)","PGN":"EI1(65170)"}
(1543509533.001447) can0 18F00131#FFFFFF3F00FFFFFF ; {"SA":"Cab Controller - Primary( 49)","DA":"All(255)","PGN":"EBC1(61441)","Accelerator Interlock Switch":"0 (00 - Off)","Engine Retarder Selection":"0.0 [%]"}
(1543509533.001528) can0 18FEF131#F7FFFF07CCFFFFFF ; {"SA":"Cab Controller - Primary( 49)","DA":"All(255)","PGN":"CCVS1(65265)","Cruise Control Pause Switch":"1 (01 - On)","Cruise Control Active":"0 (00 - Cruise control switched off)","Cruise Control Enable Switch":"0 (00 - Cruise control disabled)","Brake Switch":"1 (01 - Brake pedal depressed)","Cruise Control Coast (Decelerate) Switch":"0 (00 - Cruise control activator not in the position \"coast\")","Cruise Control Accelerate Switch":"0 (00 - Cruise control activator not in the position \"accelerate\")"}
Formatted contents of complete frames only.
$ pretty_j1939.py --format --no-link example.candump.txt | head
{
"PGN": "AT1HI1(64920)",
"Aftertreatment 1 Total Fuel Used": "227.5 [liters]",
"Aftertreatment 1 DPF Average Time Between Active Regenerations": "173933 [Seconds]",
"Aftertreatment 1 DPF Average Distance Between Active Regenerations": "1460.5 [m]"
}
{
"PGN": "AT1HI1(64920)",
"Aftertreatment 1 Total Fuel Used": "227.5 [liters]",
"Aftertreatment 1 DPF Average Time Between Active Regenerations": "173933 [Seconds]",
The JSON output can be used as an input to jq
to filter or format the decoded data. E.g. we can show only messages
from the "Brakes":
$ pretty_j1939.py example.candump.txt --format | jq ". | select(.SA | contains(\"Brakes\"))"
{
"PGN": "TSC1(0)",
"DA": "Retarder - Engine( 15)",
"SA": "Brakes - System Controller( 11)",
"Engine Requested Speed/Speed Limit": "8031.875 [rpm]",
"Engine Requested Torque/Torque Limit": "-125 [%]"
}
{
"PGN": "TSC1(0)",
"DA": "Retarder - Driveline( 16)",
"SA": "Brakes - System Controller( 11)",
"Engine Requested Speed/Speed Limit": "8031.875 [rpm]",
"Engine Requested Torque/Torque Limit": "-125 [%]"
}
{
"PGN": "TSC1(0)",
"DA": "Retarder, Exhaust, Engine #1( 41)",
"SA": "Brakes - System Controller( 11)",
"Engine Requested Speed/Speed Limit": "8031.875 [rpm]",
"Engine Requested Torque/Torque Limit": "-125 [%]"
}
{
"PGN": "EBC1(61441)",
"DA": "All(255)",
"SA": "Brakes - System Controller( 11)",
"ASR Brake Control Active": "0 (00 - ASR brake control passive but installed)",
"Anti-Lock Braking (ABS) Active": "0 (00 - ABS passive but installed)",
[...]
HOWTO
First, obtain a copy of the digital annex, see https://www.sae.org/standards/content/j1939da_201907/ for details.
Then, use the create_j1939db-json.py
script to convert that Digital Annex into a JSON file e.g.
create_j1939db-json.py -f tmp/J1939DA_201611.xls -w tmp/J1939DA_201611.json
Place the resulting JSON file at J1939db.json
in your working directory and use the pretty-printing script e.g.
pretty_j1939.py example.candump.txt
The pretty_j1939.py
script (and the describer
in pretty_j1939/describe.py
that it builds-on) has various levels of
verbosity available when describing J1939 traffic in candump logs:
usage: pretty_j1939.py [-h] [--da-json [DA_JSON]] [--candata] [--no-candata] [--pgn] [--no-pgn] [--spn] [--no-spn] [--transport] [--no-transport]
[--link] [--no-link] [--include-na] [--no-include-na] [--real-time] [--no-real-time] [--format] [--no-format]
candump
pretty-printing J1939 candump logs
positional arguments:
candump candump log, use - for stdin
optional arguments:
-h, --help show this help message and exit
--da-json [DA_JSON] absolute path to the input JSON DA (default="./J1939db.json")
--candata print input can data
--no-candata (default)
--pgn (default) print source/destination/type description
--no-pgn
--spn (default) print signals description
--no-spn
--transport print details of transport-layer streams found (default)
--no-transport
--link print details of link-layer frames found
--no-link (default)
--include-na include not-available (0xff) SPN values
--no-include-na (default)
--real-time emit SPNs as they are seen in transport sessions
--no-real-time (default)
--format format each structure (otherwise single-line)
--no-format (default)
To use as a library one can import the pretty_j1939 modules class as import pretty_j1939
and instantiate a describer
with describe = pretty_j1939.describe.get_describer()
. That get_describer()
function has defaults that match the
above command-line utility and accepts similar flags for customization. Then frames can be described by calling
describe(message_data.bytes, message_id.uint)
where message_data
and message_id
are both of type bitstring.Bits
created from the hex id and data strings (lsb on left).
Note that the interpretation is done per message. In case of multipacket messages, transport messages are buffered
unless real-time=True
is specified as an argument to get_describer()
Installing
pip3 install pretty_j1939
Testing
There is a very basic testing script testme.sh
which will attempt to create_j1939db-json.py
each tmp/*.xls
and
then try some pretty_j1939.py
runs with each of the resulting DA json files over all tmp/*.log
. This is
meant as a sanity test only. To test changes in create_j1939db-json.py
the contents of the resulting DA json file must
be compared to previous versions and analyzed manually; to test changes in describe.py
or pretty_j1939.py
the output
needs to be similarly analyzed manually.
There are unfortunately no *.xls
, *.json
, nor *.log
distributed with this repo, you will need to bring your own.
Notes on Digital Annex Sources
You need to obtain a J1939 Digital Annex from the SAE to create a JSON file that can be used by pretty_j1939.py
see
https://www.sae.org/standards/content/j1939da_201907/ for details.
There are multiple releases; here are a couple notes to consider when purchasing your copy of the Digital Annex.
- the 201611 Digital Annex has fewer defined SPNs in it than the 201311 Digital Annex; at some point the owners of the DA started migrating 'technical' SPNs (e.g. DMs) to other documents and out of the DA
- the 201311 Digital Annex has a couple bugs in it that the
create_j1939db-json.py
has workarounds for - the
create_j1939db-json.py
can also handle the XLS Export from isobus.net by supplying multiple excel sheets as input (with multiple-f
arguments); however, the isobus.net definitions omit almost all of the commercial vehicle SPNs and PGNs so the resultingJ1939db.json
file may not be of great use in examining candump captures from commercial vehicles.
Future Work
- port this functionality to the python-j1939 and python-can projects
- default JSON database (of limited content) based on public information
- support for J1939 aspects not encoded in the Digital Annex (ever, or anymore) e.g. Address Claim, DMs
- integrate and/or move
create_j1939-db-json.py
to canmatrix - colorize the json output (and avoid breaking pipelines)