Home

Awesome

Benchmark and Evaluation of "Efficient Emotional Adaptation for Audio-Driven Talking-Head Generation"

Setup

Before beginning, ensure your environment aligns with the EAT repository requirements. Install the necessary package:

pip install face-alignment

Evaluation Instructions

  1. Download Pre-trained Models:

    • Access the pre-trained models from this link.

    • After downloading, unzip the files and place them into the code folder.

      unzip code.zip -d code

  2. Download Ground Truth Videos:

    • Obtain the cropped Ground Truth videos from this link.

    • Once downloaded, unzip the files and move them into the root directory with the command:

      unzip talking_head_testing.zip -d talking_head_testing

  3. Place Your Results:

    • Position your evaluation results in the ./result folder.

  4. Execution:

    Evaluate MEAD

    • For instance, if your sampled (100) test results are located in the folder ./result/deepprompt_eam3d_all_final_313, execute the following bash command:
    bash test_psnr_ssim_sync_emoacc.sh deepprompt_eam3d_all_final_313 0
    • If you want to test the whole 985 results in MEAD test set, execute the following bash command:
    bash test_psnr_ssim_sync_emoacc_985.sh deepprompt_eam3d_all_final_313_985 0

    Evaluate LRW

    • Different from MEAD, you need to download LRW dataset from here.
    • We only use the testset of LRW.
    • Extract the wav with python extract_wav.py
    • Align and crop with PCAVS preprocess: bash preprocess_lrw_gt.sh '[VIDEO_PATH]'. Replace [VIDEO_PATH] with the absolute folder path of dataset videos, eg. '/data/lipread_test_25/video'.
    • Put your test result of EAT into './result'
    • Change the name in test_psnr_ssim_sync_lrw_100.sh. Run bash test_psnr_ssim_sync_lrw_100.sh to test LRW with 100 samples.
    • Change the name in test_psnr_ssim_sync_lrw_25k.sh. Run bash test_psnr_ssim_sync_lrw_25k.sh to test LRW with all 25k samples.