Awesome
CANF-VC: Conditional Augmented Normalizing Flows for Video Compression
Update (09.11.23): CANF-VC++: Enhancing Conditional Augmented Normalizing Flows for Video Compression with Advanced Techniques
- We present CANF-VC++, an improved video compression framework from CANF-VC. CANF-VC++ demonstrates substantial Bjøntegaard-Delta rate savings of 40.2%, 38.1%, and 35.5% on UVG, HEVC Class B, and MCL-JCV datasets, respectively over CANF-VC. Please check our paper on arxiv for the details of our improvements. You can find the inference commands in full commands section.
- Performance
-
BD-rate (Anchor: CANF-VC ; GOP=32 ; only compress first 96 frames in each testing sequence)
-
R-D curves:
-
Update (08.30.22): CANF-VC with Error Propagation Aware Training Strategy
- CANF-VC-EPA is an enhanced version of CANF-VC. With exactly the same network architecture as CANF-VC, CANF-VC-EPA additionally introduces the Error Propagation Aware (EPA) training strategy from Guo et al., ECCV'20.
- Usage: Exactly the same as CANF-VC
- Performance
-
BD-rate (GOP=32 ; anchor: x265 veryslow). The best performer is marked in red and the second best in blue.
-
R-D curves:
- <img src="https://user-images.githubusercontent.com/108980934/187449482-a9fad0fe-2506-47f2-8106-f4ada30d9ef7.png" width="400"> <img src="https://user-images.githubusercontent.com/108980934/187449448-aacf46ea-801f-48f7-8ac9-ce137ecc16ca.png" width="400"> <img src="https://user-images.githubusercontent.com/108980934/187574039-7f9b9c7c-2a45-4039-b0d2-10525c5c908e.png" width="400">
-
Project Installation
- Prepare PyTorch 1.4.0 environment and correspond torchvision
- Run
sh install.sh
- (Only needed for CANF-VC*) Install
libbpg
: https://github.com/mirrorer/libbpg 3.1 Configure path tolibbpg
aslibbpg_path
indataloader.py
- Download model weights & prepare testing data
- Start evaluation:
action=test/compress/decompress
Model Weight
- Get download link after filling out this form: https://forms.gle/Twsx8NofBjGakueL6
Should be put under this project directory
Dataset
- Prepare all of your video sequence (in
.png
format), or - Download all datasets:
- Including:
U
for UVG datasetB
,C
,D
,E
for HEVC-B, -C, -D, -E datasetM
for MCL-JCV dataset
- Including:
- We provide
yuv2png.py
for you to turn.yuv
video into.png
video framespython yuv2png.py
- Please specify the path & dataset to be converted in the file
Examples
-
CANF-VC++ (PSNR):
test
:$ python3 test.py --Iframe=ANFIC --Pframe=CANFVC_PP --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_quadtree_context.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_quadtree_context.yml --dataset=U --seq=Beauty --seq_len=96 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC++/PSNR --action=test --GOP=32
compress
:$ python3 test.py --Iframe=ANFIC --Pframe=CANFVC_PP --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_quadtree_context.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_quadtree_context.yml --dataset=U --seq=Beauty --seq_len=96 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC++/PSNR --bitstream_dir=./bin/CANFVC_PP --action=compress --GOP=32
decompress
:$ python3 test.py --Iframe=ANFIC --Pframe=CANFVC_PP --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_quadtree_context.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_quadtree_context.yml --dataset=U --seq=Beauty --seq_len=96 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC++/PSNR --bitstream_dir=./bin/CANFVC_PP --action=decompress --GOP=32
-
CANF-VC (PSNR):
test
:$ python3 test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC/PSNR --action=test --GOP=32
compress
:$ python3 test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --seq=BQSquare --seq_len=100 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC/PSNR --bitstream_dir=./bin --action=compress --GOP=32
decompress
:$ python3 test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --seq=BQSquare --seq_len=100 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC/PSNR --bitstream_dir=./bin --action=decompress --GOP=32
-
CANF-VC* (PSNR):
test
:$ python3 test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC_star/PSNR --action=test --GOP=32
compress
:$ python3 test.py --Iframe=BPG --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --seq=BQSquare --seq_len=100 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC_star/PSNR --bitstream_dir=./bin --action=compress --GOP=32
decompress
:$ python3 test.py --Iframe=BPG --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset=D --seq=BQSquare --seq_len=100 --dataset_path=./video_dataset --lmda=2048 --model_dir=./models/CANF-VC_star/PSNR --bitstream_dir=./bin --action=decompress --GOP=32
Full Commands
- CANF-VC++:
test
:$ python3 test.py --Iframe=ANFIC --Pframe=CANFVC_PP --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_quadtree_context.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_quadtree_context.yml --dataset={U/B/C/D/E/M} --dataset_path=/path/to/video_dataset --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC++/{PSNR/MS-SSIM} --action=test --GOP=32
compress
/decompress
:$ python3 test.py --Iframe=ANFIC --Pframe=CANFVC_PP --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_quadtree_context.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_quadtree_context.yml --dataset={U/B/C/D/E/M} --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --dataset_path=/path/to/video_dataset --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC++/{PSNR/MS-SSIM} --bitstream_dir=./bin --action={compress/decompress} --GOP=32
- CANF-VC:
test
:$ python test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset={U/B/C/D/E/M} --dataset_path=/path/to/video_dataset --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC/{PSNR/MS-SSIM} --action=test --GOP=32 {--msssim}
compress
/decompress
:$ python test.py --Iframe=ANFIC --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset={U/B/C/D/E/M} --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --dataset_path=/path/to/video_dataset --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC/{PSNR/MS-SSIM} --bitstream_dir=./bin --action={compress/decompress} --GOP=32 {--msssim}
- CANF-VC*:
test
:$ python3 test.py --Iframe=BPG --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset={U/B/C/D/E/M} --dataset_path=/path/to/video_dataset --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC_star/{PSNR/MS-SSIM} --action=test --GOP=32 {--msssim}
compress
/decompress
:$ python3 test.py --Iframe=BPG --MENet=PWC --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder.yml --dataset={U/B/C/D/E/M} --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --dataset_path=/path/to/video_dataset --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC_star/{PSNR/MS-SSIM} --bitstream_dir=./bin --action={compress/decompress} --GOP=32 {--msssim}
- CANF-VC Lite:
test
:$ python test.py --Iframe=ANFIC --MENet=SPy --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior_Lite.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_Lite.yml --dataset={U/B/C/D/E/M} --dataset_path=/path/to/video_dataset --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC-Lite/{PSNR/MS-SSIM} --action=test --GOP=32 {--msssim}
compress
/decompress
:$ python test.py --Iframe=ANFIC --MENet=SPy --motion_coder_conf=./CANF_VC/config/DVC_motion.yml --cond_motion_coder_conf=./CANF_VC/config/CANF_motion_predprior_Lite.yml --residual_coder_conf=./CANF_VC/config/CANF_inter_coder_Lite.yml --dataset={U/B/C/D/E/M} --seq=SEQUENCE_TO_BE_COMPRESS(Optional) --seq_len=NUMBER_OF_FRAMES_TO_BE_COMPRESSED(Optional) --dataset_path=/path/to/video_dataset --lmda={2048/1024/512/256} --model_dir=/path/to/CANF-VC-Lite/{PSNR/MS-SSIM} --bitstream_dir=./bin --action={compress/decompress} --GOP=32 {--msssim}
Citation
If you find this work useful for your research, please cite:
@article{canfvc,
title={CANF-VC: Conditional Augmented Normalizing Flows for Video Compression},
author={Ho, Yung-Han and Chang, Chih-Peng and Chen, Peng-Yu and Gnutti, Alessandro and Peng, Wen-Hsiao},
journal={European Conference on Computer Vision},
year={2022}
}