Home

Awesome

ReadabiliPy

Build Status Coverage Status

ReadabiliPy contains a Python wrapper for Mozilla's Readability.js Node.js package, as well as article extraction routines written in pure Python.

This package augments the output of Readability.js to also return a list of plain text representations of article paragraphs.

ReadabiliPy comes with a handy command line application: readabilipy.

Installation

To use the Readability.js wrapper you need to have a working Node.js installation of version 14 or higher. Make sure to install Node.js before installing this package, as this ensures Readability.js will be installed. If you only want to use the Python-based article extraction, you do not need to install Node.js.

ReadabiliPy can be installed simply from PyPI:

$ pip install readabilipy

Note that to update to a new version of Readability.js you can simply reinstall ReadabiliPy.

Usage

ReadabiliPy can be used either as a command line application or as a Python library.

Command line application

The readabilipy command line application can be used to extract an article from an HTML source file.

For example, if you have the article saved as input.html in the current directory then you can run:

$ readabilipy -i ./input.html -o article.json

The extracted article can then be found in the article.json file. By default ReadabiliPy will use the Readability.js functionality to extract the article, provided this is available. If instead you'd like to use the Python-based extraction, run:

$ readabilipy -p -i ./input.html -o article.json

The complete help text of the command line application is as follows:

$ readabilipy -h
usage: readabilipy [-h] -i INPUT_FILE -o OUTPUT_FILE [-c] [-n] [-p] [-V]

Extract article data from a HTML file using either Mozilla's Readability.js
package or a simplified python-only alternative.

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input-file INPUT_FILE
                        Path to input file containing HTML.
  -o OUTPUT_FILE, --output-file OUTPUT_FILE
                        Path to file to output the article data to as JSON.
  -c, --content-digests
                        Add a 'data-content-digest' attribute containing a
                        SHA256-based digest of the element's contents to each
                        HTML element in the plain_content output.
  -n, --node-indexes    Add a 'data-node-index' attribute containing a
                        hierarchical representation of the element's position
                        in the HTML structure each HTML element in the
                        plain_content output.
  -p, --use-python-parser
                        Use the pure-python 'plain_html' parser included in
                        this project rather than Mozilla's Readability.js.
  -V, --version         Show version and exit

Library

ReadabiliPy can also be used as a Python package. The main routine is called simple_json_from_html_string and expects the HTML article as a string. Here is an example of extracting an article after downloading the page using requests:

>>> import requests
>>> from readabilipy import simple_json_from_html_string
>>> req = requests.get('https://en.wikipedia.org/wiki/Readability')
>>> article = simple_json_from_html_string(req.text, use_readability=True)

Note that you need to use the flag use_readability=True to use Readability.js, otherwise the Python-based extraction is used.

The simple_json_from_html_string function returns a dictionary with the following fields:

Note further that:

The second top-level function exported by ReadabiliPy is simple_tree_from_html_string. This returns a cleaned, parsed HTML tree of the article as a BeautifulSoup object.

Notes

License: MIT License, see the LICENSE file.

Copyright (c) 2018, The Alan Turing Institute

If you encounter any issues or have any suggestions for improvement, please open an issue on Github. You're helping to make this project better for everyone!