Home

Awesome

Pushing HTTPS :lock:

Latest Version Coverage Status Build Status

pshtt ("pushed") is a tool to scan domains for HTTPS best practices. It saves its results to a CSV (or JSON) file.

pshtt was developed to push organizations — especially large ones like the US Federal Government :us: — to adopt HTTPS across the enterprise. Federal agencies must comply with M-15-13, a 2015 memorandum from the White House Office of Management and Budget, and BOD 18-01, a 2017 directive from the Department of Homeland Security, which require federal agencies to enforce HTTPS on their public web services. Much has been done, but there's more yet to do.

pshtt is a collaboration between the Department of Homeland Security's National Cybersecurity Assessments and Technical Services (NCATS) team and the General Service Administration's 18F team, with contributions from NASA, Lawrence Livermore National Laboratory, and various non-governmental organizations.

Getting Started

pshtt requires Python 3.4+. Python 2 is not supported.

pshtt can be installed as a module, or run directly from the repository.

Installed as a module

pshtt can be installed directly via pip:

pip install pshtt

It can then be run directly:

pshtt example.com [options]

Running directly

To run the tool locally from the repository, without installing, first install the requirements:

pip install -r requirements.txt

Then run it as a module via python -m:

python -m pshtt.cli example.com [options]

Usage and examples

pshtt [options] DOMAIN...
pshtt [options] INPUT

pshtt dhs.gov
pshtt --output=homeland.csv --debug dhs.gov us-cert.gov usss.gov
pshtt --sorted current-federal.csv

Note: if INPUT ends with .csv, domains will be read from the first column of the CSV. CSV output will always be written to disk (unless --json is specified), defaulting to results.csv.

Options

  -h --help                     Show this message.
  -s --sorted                   Sort output by domain, A-Z.
  -o --output=OUTFILE           Name output file. (Defaults to "results".)
  -j --json                     Get results in JSON. (Defaults to CSV.)
  -m --markdown                 Get results in Markdown. (Defaults to CSV.)
  -d --debug                    Print debug output.
  -u --user-agent=AGENT         Override user agent.
  -t --timeout=TIMEOUT          Override timeout (in seconds).
  -c --cache-third-parties=DIR  Cache third party data, and what directory to cache it in.
  -f --ca-file=PATH             Specify custom CA bundle (PEM format)
Using your own CA Bundle

By default, pshtt relies on the root CAs that are trusted in the Mozilla root store. If you work behind a corporate proxy or have your own certificates that aren't publicly trusted, you can specify your own CA bundle:

pshtt --ca-file=/etc/ssl/ca.pem server.internal-location.gov
Using Docker (optional)
./run [opts]

opts are the same arguments that would get passed to pshtt.

What's Checked?

A domain is checked on its four endpoints:

The following values are returned in results.csv:

Domain and redirect info

Landing on HTTPS

Common errors

HSTS

Scoring

These three fields use the previous results to come to high-level conclusions about a domain's behavior.

Troubleshooting

DNS Blackhole / DNS Assist

One issue which can occur when running pshtt, particularly for home / residential networks, with standard ISPs is the use of "DNS Assist" features, a.k.a. "DNS Blackholes".

In these environments, you may see inconsistent results from pshtt owing to the fact that your ISP is attempting to detect a request for an unknown site without a DNS record and is redirecting you to a search page for that site. This means that an endpoint which should resolve as "not-alive", will instead resolve as "live", owing to the detection of the live search result page.

If you would like to disable this "feature", several ISPs offer the ability to opt out of this service, and maintain their own instructions for doing so:

Who uses pshtt?

Acknowledgements

This code was modeled after Ben Balter's site-inspector, with significant guidance from Eric Mill.

Public domain

This project is in the worldwide public domain.

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.