Awesome
ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models
This repository is a reference implementation for ViVid-1-to-3. It combines video diffusion with novel-view synthesis diffusion models for increased pose and appearace consistency.
Requirements
pip install torch "diffusers==0.24" transformers accelerate einops kornia imageio[ffmpeg] opencv-python pydantic scikit-image lpips
Run single generation task
Put the reference image to $IMAGE_PATH, and set the input_image_path in scripts/task_example.yaml to it. Then run
python run_generation.py --task_yaml_path=scripts/task_example.yaml
Run batch generation tasks
We have supported running batch generation tasks on both PC and SLURM clusters.
Prepare batch generation config yaml file
We tested our method on 100 GSO objects. The list of the objects is in scripts/gso_metadata_object_prompt_100.csv, along with our labeled text prompts if you would like to test prompt-based generation yourself. We have rendered the 100 objects beforehand. It can be downloaded here. You can decompress the content into gso-100. Then simply run the following line to prepare a batch generation job on a PC:
python -m scripts.job_config_yaml_generation
Or run the following line to prepare a batch generation job on a SLURM cluster, which will move temporary stuff to $SLURM_TMPDIR of your cluster:
python -m scripts.job_config_yaml_generation --run_on_slurm
All the yaml files will be generated in a new folder called tasks_gso.
If you want to run customized batch generation, simply add an entry in the job_specs list in the beginning of scripts/job_config_yaml_generation.py and run it with the same bash command. An example has been commented out in it.
Batch generation
For batch generation, run
python run_batch_generation.py --task_yamls_dir=tasks_gso --dataset_dir=gso-100 --output_dir=outputs --obj_csv_file=scripts/gso_metadata_object_prompt_100.csv
Tips for scheduling batch generation on SLURM clusters
It takes about 1min30s to run one generation on a v100 gpu. If the number of generations is too large for each job you can schedule on a SLURM cluster,
you can split the dataset for each job using the --run_from_obj_index and --run_to_obj_index options. For example
python run_batch_generation.py --task_yamls_dir=tasks_gso --dataset_dir=gso-100 --output_dir=outputs --obj_csv_file=scripts/gso_metadata_object_prompt_100.csv --run_from_obj_index=0 --run_to_obj_index=50
Run evaluation
Get metrics for each object
To run evaluation for a batch generation, put the experiments you want to evaluate in the eval_specs list in run_evaluation.py. Make sure the exp_name key has the same value as that of your batch generation. Also, you should modify the expdir and savedir in run_evaluation.py. Suppose you want to run the $EXP_ID-th experiment in the list, then do the following:
python run_evaluation.py --exp_id $EXP_ID
After the evaluation is run, intermediate results on PSNR, SSIM, LPIPS, FOR_8, FOR_16 for each object will be put to savedir.
Get stats for this experiment
Finally, you can use run_calculate_stats.py to get the PSNR, SSIM, LPIPS, FOR_8, FOR_16 stats for this experiment on your whole dataset. Make sure to modify the psnr_save_dir, lpips_save_dir, ssim_save_dir, for_8_save_dir, for_16_save_dir in run_calculate_stats.py to match the folder storing the intermediate results from the last step.
python run_calculate_stats.py
Acknowledgement
This repo is based on the Huggingface community implementation and converted weights of Zero-1-to-3, as well as the Huggingface community text-to-video model Zeroscope v2. Thanks for their awesome works.
Citation
If you use this code in your research, please cite our paper:
@inproceedings{kwak2024vivid,
title={Vivid-1-to-3: Novel view synthesis with video diffusion models},
author={Kwak, Jeong-gi and Dong, Erqun and Jin, Yuhe and Ko, Hanseok and Mahajan, Shweta and Yi, Kwang Moo},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={6775--6785},
year={2024}
}