Home

Awesome

Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives

The experimental code of "Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives" in ECCV 2022.

Abstract

Recently, Implicit Neural Representations (INRs) parameterized by neural networks have emerged as a powerful and promising tool to represent different kinds of signals due to its continuous, differentiable properties, showing superiorities to classical discretized representations. However, the training of neural networks for INRs only utilizes input-output pairs, and the derivatives of the target output with respect to the input, which can be accessed in some cases, are usually ignored. In this paper, we propose a training paradigm for INRs whose target output is image pixels, to encode image derivatives in addition to image values in the neural network. Specifically, we use finite differences to approximate image derivatives. We show how the training paradigm can be leveraged to solve typical INRs problems, i.e., image regression and inverse rendering, and demonstrate this training paradigm can improve the data-efficiency and generalization capabilities of INRs.

<img src='imgs/pipeline.png'/>

Setup

Environment

Data

Reproducing Experiments

Image Regression

cd Experiments/image_regression
python main.py --config [config txt file]

For example, training with value and derivative supervision on Baby with a sine-based model:

python main.py --config configs/baby/val_der/sine.txt

Inverse Rendering

cd Experiments/inverse_rendering
python train.py --config [config txt file]

For example, train with value and derivative supervision on Fern with a ReLU-based MLP:

python train.py --config configs/fern/val_der/relu.txt

After training for 400K iterations, you can find novel view results in logs/fern/val_der/relu/testset_400000, you can evaluate results with following command and score.txt will be generated in logs/fern/val_der/relu/testset_400000/score.txt:

python eval.py --config configs/fern/val_der/relu.txt

Audio Regression

cd Experiments/audio_regression
python main.py --config [config txt file]

For example, training with value supervision and on Bach with a sine-based model:

python main.py --config configs/bach/val_sine.txt

Citation

If you find our work useful in your research, please cite:

@inproceedings{yuan2022sobolev,
  title={Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives},
  author={Wentao Yuan and Qingtian Zhu and Xiangyue Liu and Yikang Ding and Haotian Zhang and Chi Zhang},
  year={2022},
  booktitle={ECCV},

Acknowledgements

Some codes of image regression task and audio regression task are borrowed from SIREN. The implementation of inverse rendering task are based on NeRF-PyTorch, which is a PyTorch implementation of original NeRF. Thanks to these authors for releasing the code.