Home

Awesome

Urlscan

main

Contributors

Scott Hansen <tech@firecat53.net> (Author and Maintainer)

Maxime Chatelle <xakz@rxsoft.eu> (Debian Maintainer)

Daniel Burrows <dburrows@debian.org> (Original Author)

Purpose and Requirements

Urlscan is a small program that is designed to integrate with the "mutt" mailreader to allow you to easily launch a Web browser for URLs contained in email messages. It is a replacement for the "urlview" program.

Requires: Python 3.7+ and the python-urwid library

Features

Urlscan parses an email message or file and scans it for URLs and email addresses. It then displays the URLs and their context within the message, and allows you to choose one or more URLs to send to your Web browser. Alternatively, it send a list of all URLs to stdout.

Relative to urlview, urlscan has the following additional features:

Installation and setup

To install urlscan, install from your distribution repositories, from Pypi, or do a local development install with pip -e:

pipx install urlscan

OR

pip install --user urlscan

OR

cd <path/to/urlscan> && pip install --user -e .

NOTE

The minimum required version of urwid is 1.2.1.

Once urlscan is installed, add the following lines to your .muttrc:

macro index,pager \cb "<pipe-message> urlscan<Enter>" "call urlscan to
extract URLs out of a message"

macro attach,compose \cb "<pipe-entry> urlscan<Enter>" "call urlscan to
extract URLs out of a message"

Once this is done, Control-b while reading mail in mutt will automatically invoke urlscan on the message.

Note for Neomutt users: As of version 2023-05-17 true color support was implemented. If you are using true color support with Neomutt, or are encountering the error setupterm: could not find terminfo database, then you should also add TERM=xterm-256color to your macro in .muttrc. See more here #135. For example: macro index,pager \cb "<pipe-message> TERM=xterm-256color urlscan<Enter>" "call urlscan to extract URLs out of a message"

To choose a particular browser, set the environment variable BROWSER. If BROWSER is not set, xdg-open will control which browser is used, if it's available.:

export BROWSER=/usr/bin/epiphany

Command Line usage

urlscan OPTIONS <file>

OPTIONS [-c, --compact]
        [-d, --dedupe]
        [-E, --regex <expression>]
        [-f, --run-safe <expression>]
        [-g, --genconf]
        [-H, --nohelp]
        [    --headers]
        [-n, --no-browser]
        [-p, --pipe]
        [-r, --run <expression>]
        [-R, --reverse]
        [-s, --single]
        [-w, --width]
        [-W  --whitespace-off]

Urlscan can extract URLs and email addresses from emails or any text file. Calling with no flags will start the curses browser. Calling with '-n' will just output a list of URLs/email addressess to stdout. The '-c' flag removes the context from around the URLs in the curses browser, and the '-d' flag removes duplicate URLs. The '-R' flag reverses the displayed order of URLs and context. Files can also be piped to urlscan using normal shell pipe mechanisms: cat <something> | urlscan or urlscan < <something>. The '-W' flag condenses the display output by suppressing blank lines and ellipses lines.

Instead of opening a web browser, the selected URL can be passed as the argument to a command using --run-safe "<command> {}" or --run "<command> {}". Note the use of {} in the command string to denote the selected URL. Alternatively, the URL can be piped to the command using --run-safe <command> --pipe (or --run). Using --run-safe with --pipe is preferred if the command supports it, as it is marginally more secure and tolerant of special characters in the URL.

Theming

Run urlscan -g to generate ~/.config/urlscan/config.json with the default color and black & white palettes. This can be edited or added to, as desired. The first palette in the list will be the default. Configure the palettes according to the Urwid display attributes.

Display width can be set with --width.

Keybindings

Run urlscan -g to generate ~/.config/urlscan/config.json. All of the keys will be listed. You can either leave in place or delete any that will not be altered.

To unset a binding, set it equal to "". For example: "P": ""

The follow actions are supported:

Update TLD list (for developers, not users)

wget https://data.iana.org/TLD/tlds-alpha-by-domain.txt

Known bugs and limitations

Build/development