Home

Awesome

GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields

This repository is the official PyTorch implementation for CVPR 2023 paper
“GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields”

https://github.com/AlessandroRuzzi/GazeNeRF/assets/76208014/7607b8fc-2aa5-45fc-8f0f-542d9dab9597

Requirements

The models in our paper are trained with Python 3.8.8, PyTorch 1.12.0, CUDA 11.3, and CentOS 7.9.2009.

To install the required packages, run:
pip install -r requirements.txt

To install the required libraries for data preprocessing, please refer to this repository.

Dataset

Our models are trained with ETH-XGaze dataset, and evaluated with ETH-XGaze, Columbia, MPIIFaceGaze, and GazeCapture datasets. The preprocessing code is mainly based on the repository of data normalization, the repository of HeadNeRF and this repository.

To preprocess the datasets, run:
python dataset_pre_precessing.py --dataset_dir=/path/to/your/dataset --dataset_name=eth_xgaze --output_dir=/path/to/your/output/directory

Training

To train the GazeNeRF model, run
python train.py --batch_size=2 --log=true --learning_rate=0.0001 --img_dir='/path/to/your/ETH-XGaze/training/dataset'

Our model was trained on a single NVIDIA A40 GPU.

Evaluation

To evaluate the trained model, run
python evaluate_metrics.py --log=true --num_epochs=75 --model_path=checkpoints/your_checkpoints.json

To generate the interpolation demos, run
python evaluate.py --model_path=checkpoints/your_checkpoints.json --img_dir='/path/to/your/ETH-XGaze/test/dataset'

Pre-trained model

You can download the pre-trained GazeNeRF here and the gaze estimator here.

Citation

@InProceedings{ruzzi2023gazenerf,
    author    = {Ruzzi, Alessandro and Shi, Xiangwei and Wang, Xi and Li, Gengyan and De Mello, Shalini and Chang, Hyung Jin and Zhang, Xucong and Hilliges, Otmar},
    title     = {GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2023},
    pages     = {9676--9685}
}