Home

Awesome

License

Galliot now hosts all Neuralet’s content and expertise gained in three years of work and completing high-quality applications, mainly in Computer Vision and Deep Learning. More Info

Smart Social Distancing

Introduction

Smart Distancing is an open-source application to quantify social distancing measures using edge computer vision systems. Since all computation runs on the device, it requires minimal setup and minimizes privacy and security concerns. It can be used in retail, workplaces, schools, construction sites, healthcare facilities, factories, etc.

<div align="center"> <img width="100%" src="demo.gif"> </div>

You can run this application on edge devices such as NVIDIA's Jetson Nano / TX2 or Google's Coral Edge-TPU. This application measures social distancing rates and gives proper notifications each time someone ignores social distancing rules. By generating and analyzing data, this solution outputs statistics about high-traffic areas that are at high risk of exposure to COVID-19 or any other contagious virus.

If you want to understand more about the architecture you can read the following post.

Please join our slack channel or reach out to hello@galliot.us if you have any questions.

Getting Started

You can read the Get Started tutorial on Lanthorn's website. The following instructions will help you get started.

Prerequisites

Hardware

A host edge device. We currently support the following:

The features supported, the detection accuracy reached and the performance can vary from device to device.

Software

You should have Docker on your device.

Optionally, you can install docker-compose to build and run the processor containers easily. In some edge devices, such as Coral or Jetson Nano, the official installation guide can fail because there isn't in the repository an already build image for that device architecture. If this is the case, we recommend installing docker-compose using pip

Download a sample video (Optional)

If you don't have any camera to test the solution you can use any video as an input source. You can download an example with the following command.

# Download a sample video file from multiview object tracking dataset
# The video is complied from this dataset: https://researchdatafinder.qut.edu.au/display/n27416
./download_sample_video.sh

Usage

The smart social distancing app consists of two components: the frontend and the processor.

Frontend

The frontend is a public web app provided by lanthorn where you can signup for free. This web app allows you to configure some aspects of the processor (such as notifications and camera calibration) using a friendly UI. Moreover, it provides a dashboard that helps you to analyze the data that your cameras are processing.

The frontend site uses HTTPs, in order to have it communicate with the processor, the latter must be either Running with SSL enabled (See Enabling SSL on this Readme), or you must edit your site settings for https://app.lanthorn.ai in order to allow for Mixed Content (Insecure Content). Without doing any of these, communication with the local processor will fail

Running the processor

Make sure you have Docker installed on your device by following these instructions. The command that you need to execute will depend on the chosen device because each one has an independent Dockerfile.

There are three alternatives to run the processor in your device:

  1. Using git and building the docker image yourself (Follow the guide in this section).
  2. Pulling the (already built) image from Galliot's Docker Hub repository (Follow the guide in this section).
  3. Using docker-compose to build and run the processor (Follow the guide in this section).
Running a proof of concept

If you want to simply run the processor for just trying it out, then from the following steps you should only:

  1. Select your device and find its docker image. On x86, without a dedicated edge device, you should use either: a. If the device has access to an Nvidia GPU: GPU with TensorRT optimization. b. If the device has access to an Intel CPU: x86 using OpenVino. c. Otherwise: x86.
  2. Either build the image or pull it from Dockerhub. Don't forget to follow the script and download the model.
  3. Download the sample video running ./download_sample_video.sh.
  4. Run the processor using the script listed in its device.

This way you can skip security steps such as enabling HTTPs communication or oauth and get a simple version of the processor running to see if it fits your use case.

Afterwards, if you intend on running the processor while consuming from a dedicated video feed, we advise you to return to this README and read it fully.

Running the processor building the image

Make sure your system fulfills the prerequisites and then clone this repository to your local system by running this command:

git clone https://github.com/galliot-us/smart-social-distancing.git
cd smart-social-distancing

After that, checkout to the latest release:

git fetch --tags
# Checkout to the latest release tag
git checkout $(git tag | tail -1)
Run on Jetson Nano
# 1) Download TensorRT engine file built with JetPack 4.3:
./download_jetson_nano_trt.sh

# 2) Build Docker image for Jetson Nano
docker build -f jetson-nano.Dockerfile -t "galliot/smart-social-distancing:latest-jetson-nano" .

# 3) Run Docker container:
docker run -it --runtime nvidia --privileged -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-jetson-nano
Run on Jetson TX2
# 1) Download TensorRT engine file built with JetPack 4.3:
./download_jetson_tx2_trt.sh

# 2) Build Docker image for Jetson TX2
docker build -f jetson-tx2.Dockerfile -t "galliot/smart-social-distancing:latest-jetson-tx2" .

# 3) Run Docker container:
docker run -it --runtime nvidia --privileged -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-jetson-tx2
Run on Coral Dev Board
# 1) Build Docker image (This step is optional, you can skip it if you want to pull the container from galliot dockerhub)
docker build -f coral-dev-board.Dockerfile -t "galliot/smart-social-distancing:latest-coral-dev-board" .

# 2) Run Docker container:
docker run -it --privileged -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-coral-dev-board
Run on AMD64 node with a connected Coral USB Accelerator
# 1) Build Docker image
docker build -f amd64-usbtpu.Dockerfile -t "galliot/smart-social-distancing:latest-amd64" .

# 2) Run Docker container:
docker run -it --privileged -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-amd64
Run on x86

# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh

# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# 1) Build Docker image
docker build -f x86.Dockerfile -t "galliot/smart-social-distancing:latest-x86_64" .

# 2) Run Docker container:
docker run -it -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-x86_64
Run on x86 with GPU

Note that you should have Nvidia Docker Toolkit to run the app with GPU support


# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh

# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# 1) Build Docker image
docker build -f x86-gpu.Dockerfile -t "galliot/smart-social-distancing:latest-x86_64_gpu" .

# 2) Run Docker container:
Notice: you must have Docker >= 19.03 to run the container with `--gpus` flag.
docker run -it --gpus all -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-x86_64_gpu
Run on x86 with GPU using TensorRT optimization

Note that you should have Nvidia Docker Toolkit to run the app with GPU support



# 1) Build Docker image
docker build -f x86-gpu-tensorrt-openpifpaf.Dockerfile -t "galliot/smart-social-distancing:latest-x86_64_gpu_tensorrt" .

# 2) Run Docker container:
# Notice: you must have Docker >= 19.03 to run the container with `--gpus` flag.
docker run -it --gpus all -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-x86_64_gpu_tensorrt
Run on x86 using OpenVino
# download model first
./download_openvino_model.sh

# 1) Build Docker image
docker build -f x86-openvino.Dockerfile -t "galliot/smart-social-distancing:latest-x86_64_openvino" .

# 2) Run Docker container:
docker run -it -p HOST_PORT:8000 -v "$PWD":/repo  -e TZ=`./timezone.sh` galliot/smart-social-distancing:latest-x86_64_openvino
Running the processor from galliot Docker Hub repository

Before running any of the images available in the Docker repository, you need to follow these steps to have your device ready.

  1. Create a data folder.
  2. Copy the config file (available in this repository) corresponding to your device.
  3. Copy the bash script(s) (available in this repository) required to download the model(s) your device requires.
  4. Optionally, copy the script timezone.sh (available in this repository) to run the processor using your system timezone instead of UTC.

Alternatively you may simply pull the folder structure from this repository.

Run on Jetson Nano
# Download TensorRT engine file built with JetPack 4.3:
mkdir data/jetson
./download_jetson_nano_trt.sh

# Run Docker container:
docker run -it --runtime nvidia --privileged -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-jetson-nano.ini:/repo/config-jetson-nano.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-jetson-nano
Run on Jetson TX2
# Download TensorRT engine file built with JetPack 4.4
mkdir data/jetson
./download_jetson_tx2_trt.sh

# Run Docker container:
docker run -it --runtime nvidia --privileged -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-jetson-tx2.ini:/repo/config-jetson-tx2.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-jetson-tx2
Run on Coral Dev Board
# Run Docker container:
docker run -it --privileged -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-coral.ini:/repo/config-coral.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-coral-dev-board
Run on AMD64 node with a connected Coral USB Accelerator
# Run Docker container:
docker run -it --privileged -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-coral.ini:/repo/config-coral.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-amd64
Run on x86
# Download the models
mkdir data/x86
# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh
# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# Run Docker container:
docker run -it -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-x86.ini:/repo/config-x86.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64
Run on x86 with GPU

Note that you should have Nvidia Docker Toolkit to run the app with GPU support

# Download the models
mkdir data/x86
# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh
# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# Docker container:
# Notice: you must have Docker >= 19.03 to run the container with `--gpus` flag.
docker run -it --gpus all -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-x86-gpu.ini:/repo/config-x86-gpu.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64_gpu
Run on x86 with GPU using TensorRT optimization

Note that you should have Nvidia Docker Toolkit to run the app with GPU support

# Run Docker container:
# Notice: you must have Docker >= 19.03 to run the container with `--gpus` flag.
docker run -it --gpus all -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-x86-gpu-tensorrt.ini:/repo/config-x86-gpu-tensorrt.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64_gpu_tensorrt
Run on x86 using OpenVino
# Download the model
mkdir data/x86
./download_openvino_model.sh

# Run Docker container:
docker run -it -p HOST_PORT:8000 -v $PWD/data:/repo/data -v $PWD/config-x86-openvino.ini:/repo/config-x86-openvino.ini -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64_openvino
Running the processor with docker-compose
Run on Jetson Nano
# 1) Download TensorRT engine file built with JetPack 4.3:
./download_jetson_nano_trt.sh

# 2) Build Docker image for Jetson Nano (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-jetson-nano.yml build

# 3) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-jetson-nano.yml up
Run on Jetson TX2
# 1) Download TensorRT engine file built with JetPack 4.3:
./download_jetson_tx2_trt.sh

# 2) Build Docker image for Jetson TX2 (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-jetson-tx2.yml build

# 3) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-jetson-tx2.yml up
Run on Coral Dev Board
# 1) Build Docker image for Coral (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-coral-dev.yml build

# 2) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-coral-dev.yml up
Run on AMD64 node with a connected Coral USB Accelerator
# 1) Build Docker image for Coral USB Accelerator (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-amd64.yml build

# 2) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-amd64.yml up
Run on x86

# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh

# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# 2) Build Docker image for x86 (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-x86.yml build

# 3) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-x86.yml up
Run on x86 with GPU

Note that you should have Nvidia Docker Toolkit to run the app with GPU support


# If you use the OpenPifPaf model, download the model first:
./download-x86-openpifpaf-model.sh

# If you use the MobileNet model run this instead:
# ./download_x86_model.sh

# 2) Build Docker image for gpu (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-gpu.yml build

# 3) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-gpu.yml up
Run on x86 with GPU using TensorRT optimization

Note that you should have Nvidia Docker Toolkit to run the app with GPU support


# 1) Build Docker image for gpu using TensorRT (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-gpu-tensorrt.yml build

# 2) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-gpu-tensorrt.yml up
Run on x86 using OpenVino
# download model first
./download_openvino_model.sh

# 2) Build Docker image for openvino (you can omit this step and use the docker-hub images)
docker-compose -f docker-compose.yml -f docker-compose-x86-openvino.yml build

# 2) Run Docker container:
docker-compose -f docker-compose.yml -f docker-compose-x86-openvino.yml up

Processor

Optional Parameters

This is a list of optional parameters for the docker run commands. They are included in the examples of the Run the processor section.

Logging in the system's timezone

By default all docker containers use UTC as timezone, passing the flag -e TZ=`./timezone.sh` will make the container run on your system's timezone.

You may hardcode a value rather than using the timezone.sh script, such as US/Pacific. Changing the processor's timezone allows to have better control of when the reports are generated and the hours to correlate to the place where the processor is running.

Please note that the bash script may require permissions to execute (run chmod +x timezone.sh)

If you are running the processor directly from the Docker Hub repository, remember to copy/paste the script in the execution folder before adding the flag -e TZ=`./timezone.sh`.

Persisting changes

We recommend adding the projects folder as a mounted volume (-v "$PWD":/repo) if you are building the docker image. If you are using the already built one we recommend creating a directory named data and mount it (-v $PWD/data:/repo/data).

Processing historical data

If you'd like to process historical data (videos stored on the device instead of a stream), you must follow two steps:

Example using x86:

docker run -it -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64 /repo/run_historical_metrics.sh

Configuring AWS credentials

Some of the implemented features allow you to upload files into an S3 bucket. To do that you need to provide the envs AWS_BUCKET_REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. An easy way to do that is to create a .env file (following the template .env.example) and pass the flag --env-file .env when you run the processor.

Enabling SSL

We recommend exposing the processors' APIs using HTTPS. To do that, you need to create a folder named certs with a valid certificate for the processor (with its corresponding private key) and configure it in the config-*.ini file (SSLCertificateFile and SSLKeyFile configurations).

If you don't have a certificate for the processor, you can create a self-signed one using openssl and the scripts create_ca.sh and create_processor_certificate.sh.

# 1) Create your own CA (certification authority)
./create_ca.sh
# After the script execution, you should have a folder `certs/ca` with the corresponding *.key, *.pem and *.srl files

# 2) Create a certificate for the processor
./create_processor_certificate.sh <PROCESSOR_IP>
# After the script execution, you should have a folder `certs/processor` with the corresponding *.key, *.crt, *.csr and *.ext files

As you are using a self-signed certificate you will need to import the created CA (using the .pem file) in your browser as a trusted CA.

Configuring OAuth2 in the endpoints

By default, all the endpoints exposed by the processors are accessible by everyone with access to the LAN. To avoid this vulnerability, the processor includes the possibility of configuring OAuth2 to keep your API secure.

To configure OAuth2 in the processor you need to follow these steps:

  1. Enabling OAuth2 in the API by setting in True the parameter UseAuthToken (included in the API section).

  2. Set into the container the env SECRET_ACCESS_KEY. This env is used to encode the JWT token. An easy way to do that is to create a .env file (following the template .env.example) and pass the flag --env-file .env when you run the processor.

  3. Create an API user. You can do that in two ways:

    1. Using the create_api_user.py script:

    Inside the docker container, execute the script python3 create_api_user.py --user=<USER> --password=<PASSWORD>. For example, if you are using an x86 device, you can execute the following script.

    docker run -it -p HOST_PORT:8000 -v "$PWD":/repo -e TZ=`./timezone.sh` neuralet/smart-social-distancing:latest-x86_64 python3 create_api_user.py --user=<USER> --password=<PASSWORD>
    1. Using the /auth/create_api_user endpoint: Send a POST request to the endpoint http://<PROCESSOR_HOST>:<PROCESSOR_PORT>/auth/create_api_user with the following body:
    {
    "user": <USER>,
    "password": <PASSWORD>
}

    After executing one of these steps, the user and password (hashed) will be stored in the file /repo/data/auth/api_user.txt inside the container. To avoid losing that file when the container is restarted, we recommend mounting the /repo directory as a volume.

  4. Request a valid token. You can obtain one by sending a PUT request to the endpoint http://<PROCESSOR_HOST>:<PROCESSOR_PORT>/auth/access_token with the following body:

    {
    "user": <USER>,
    "password": <PASSWORD>
}

    The obtained token will be valid for 1 week (then a new one must be requested from the API) and needs to be sent as an Authorization header in all the requests. If you don't send the token (when the UseAuthToken attribute is set in True), you will receive a 401 Unauthorized response.

Supported video feeds formats

This processor uses OpenCV VideoCapture, which means that it can process:

Please note that:

If you want to integrate an IP camera that uses a private protocol, you should check with the camera provider if the device supports exporting its stream in a public protocol. For example, WYZE doesn't support RTSP as default, but you have the possibility of installing a firmware that supports it. Same goes for Google Nest Cameras, although here a token must be kept alive to access the RTSP stream

Change the default configuration

You can read and modify the configurations in config-*.ini files, accordingly:

config-jetson-nano.ini: for Jetson Nano

config-jetson-tx2.ini: for Jetson TX2

config-coral.ini: for Coral dev board / usb accelerator

config-x86.ini: for plain x86 (cpu) platforms without any acceleration

config-x86-openvino.ini: for x86 systems accelerated with Openvino

Please note that if you modify these values you should also set [App] HasBeenConfigured to "True". This allows for a client to recognize if this processor was previously configured.

You can also modify some of them using the UI. If you choose this option, make sure to mount the config file as a volume to keep the changes after any restart of the container (see section Persisting changes).

All the configurations are grouped in sections and some of them can vary depending on the chosen device.

Use different models per camera

By default, all video streams are processing running against the same ML model. When a processing threads starts running it verifies if a configuration .json file exists in the path: /repo/data/processor/config/sources/<camera_id>/ml_models/model_<device>.json If no custom configuration is detected, a file will be generated using the default values from the [Detector] section, documented above. These JSONs contain the configuration of which ML Model is used for processing said stream, and can be modified either manually or using the endpoint /ml_model documented below. Please note that models that differ in their location or name regarding the ./download_ scripts must specify their location in the field file_path.

API usage

After you run the processor on your node, you can use the exposed API to control the Processor's Core, where all the process is getting done.

The available endpoints are grouped in the following subapis:

Additionally, the API exposes 2 endpoints to stop/start the video processing

The complete list of endpoints, with a short description and the signature specification is documented (with swagger) in the url PROCESSOR_IP:PROCESSOR_PORT/docs.

NOTE Most of the endpoints update the config file given in the Dockerfile. If you don't have this file mounted (see section Persisting changes), these changes will be inside your container and will be lost after stopping it.

Interacting with the processors' generated information

Generated information

The generated information can be split into 3 categories:

Accessing and storing the information

All of the information that is generated by the processor is stored (by default) inside the edge device for security reasons. However, the processor provides features to easily export or backup the data to another system if required.

Storing the raw data

The raw data storage is managed by the SourceLogger and AreaLogger steps. By default, only the video_logger and the file_system_logger are enabled. As both steps store the data inside the processor (by default the folder /repo/data/processor/static/), we strongly recommend mounting that folder to keep the data safe when the process is restarted (Persisting changes). Moreover, we recommend keeping active these steps because the frontend and the metrics need them.

If you need to store (or process) the raw data in real-time outside the processor, you can activate the web_hook_logger and implement an endpoint that handles these events. The web_hook_logger step is configured to send an event (a PUT request) using the following format:

{
            "version": ...,
            "timestamp": ...,
            "detected_objects": ...,
            "violating_objects": ...,
            "environment_score": ...,
            "detections": ...,
            "violations_indexes": ...
        }

You only need to implement an endpoint that matches the previous signature; configure its URL in the config file and the integration will be done. We recommend this approach if you want to integrate "Smart social distancing" with another existing system with real-time data.

Another alternative is to activate the periodic task s3_backup. This task will back up all the generated data (raw data and metrics) inside the configured S3 bucket, according to the time interval defined by the BackupInterval parameter. Before enabling this feature remember to configure AWS following the steps defined in the section Configuring AWS credentials.

Accessing the metrics data

The data of aggregated metrics is stored in a set of CSV files inside the device. For now, we don't have implemented any mechanism to store these files outside the processor (the web_hook_logger only sends "raw data" events). However, if you enable the s3_backup task, the previous day's metrics files will be backed up at AWS at the beginning of the day.

You can easily visualize the metrics information in the dashboard exposed in the frontend. In addition, you can retrieve the same information through the API (see the metrics section in the API documentation exposed in http://<PROCESSOR_HOST>:<PROCESSOR_PORT>/docs#/Metrics).

Exporting the data

In addition to the previous features, the processor exposes an endpoint to export in zip format all the generated data. The signature of this endpoint can be found in http://<PROCESSOR_HOST>:<PROCESSOR_PORT>/docs#/Export.

Issues and Contributing

The project is under substantial active development; you can find our roadmap at https://github.com/galliot-us/neuralet/projects/1. Feel free to open an issue, send a Pull Request, or reach out if you have any feedback.

Contact Us

License

Most of the code in this repo is licensed under the Apache 2 license. However, some sections/classifiers include separate licenses.

These include: