Awesome
shavar-list-creation
This script fetches blocklist .json
from urls (such as
shavar-prod-lists) and
generates safebrowsing-compatible digest list files to be served by
shavar.
Requirements
- python ≥ 3.6
- (optional) virtualenv and/or virtualenvwrapper
Run
-
(optional) Make a virtualenv for the project and activate it:
virtualenv -p python3.8 shavar-list-creation source shavar-list-creation/bin/activate
-
Install required libraries:
pip install -r requirements-test.txt
-
Copy the
sample_shavar_list_creation.ini
file toshavar_list_creation.ini
:cp sample_shavar_list_creation.ini shavar_list_creation.ini
-
Run the unit tests (currently under development):
python -m pytest -v --cov=. --cov-branch
-
Run the
lists2safebrowsing.py
script:./lists2safebrowsing.py
Usage
This is run by a Jenkins deployment job every 30 minutes that:
- Checks out this repository
- Checks out the shavar-list-creation-config repository
- Copies
stage.ini
orprod.ini
toshavar_list_creation.ini
- Runs
python lists2safebrowsing.py
, which uploads updated safebrowsing list files to S3 for shavar. ⚠️ Since there is no concept of deploy for the scripts, the changes that are merged to main are immediately accessible Jenkins for both Stage and Production.