Home

Awesome

shavar-list-creation

Build Status Coverage

This script fetches blocklist .json from urls (such as shavar-prod-lists) and generates safebrowsing-compatible digest list files to be served by shavar.

Requirements

Run

  1. (optional) Make a virtualenv for the project and activate it:

    virtualenv -p python3.8 shavar-list-creation
    source shavar-list-creation/bin/activate
    
  2. Install required libraries:

    pip install -r requirements-test.txt
    
  3. Copy the sample_shavar_list_creation.ini file to shavar_list_creation.ini:

    cp sample_shavar_list_creation.ini shavar_list_creation.ini
    
  4. Run the unit tests (currently under development):

    python -m pytest -v --cov=. --cov-branch
    
  5. Run the lists2safebrowsing.py script:

    ./lists2safebrowsing.py
    

Usage

This is run by a Jenkins deployment job every 30 minutes that:

  1. Checks out this repository
  2. Checks out the shavar-list-creation-config repository
  3. Copies stage.ini or prod.ini to shavar_list_creation.ini
  4. Runs python lists2safebrowsing.py, which uploads updated safebrowsing list files to S3 for shavar. ⚠️ Since there is no concept of deploy for the scripts, the changes that are merged to main are immediately accessible Jenkins for both Stage and Production.