Awesome

MMPD[EMBC 2023 Oral]

📖 Abstract

Here is MMPD: Multi-Domain Mobile Video Physiology Dataset collected by Tsinghua University.
The Multi-domain Mobile Video Physiology Dataset (MMPD), comprising 11 hours(1152K frames) of recordings from mobile phones of 33 subjects. The dataset was designed to capture videos with greater representation across skin tone, body motion, and lighting conditions. MMPD is comprehensive with eight descriptive labels and can be used in conjunction with the rPPG-toolbox and PhysBench.

@INPROCEEDINGS{10340857,
  author={Tang, Jiankai and Chen, Kequan and Wang, Yuntao and Shi, Yuanchun and Patel, Shwetak and McDuff, Daniel and Liu, Xin},
  booktitle={2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)}, 
  title={MMPD: Multi-Domain Mobile Video Physiology Dataset}, 
  year={2023},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/EMBC40787.2023.10340857}}

🔥 Updates

[2024/9] Opensource Benchmark Open. Please make a PR with your result and code.
[2024/8] Contact Email is updated. Please contact tjk24@mails.tsinghua.edu.cn for the application.
[2024/1] Citation BibTex and Data Release Agreement are updated.
[2023/11] Subset of two participants' data available for educational purposes, subject to their consent. University faculty may apply for access to this subset via email.
[2023/6] size.csv file is updated, allowing users to check data integrity.
[2023/4] Code is updated in the rPPG-Toolbox_MMPD file fold, allowing users to choose any combination of multiple labels.
For those who have downloaded or are preparing to download our dataset: you are recommended to star this repo in case the dataset is updated without notice.

🔍 Samples

	LED-low	LED-high	Incandescent	Nature
Skin Tone 3<br />Stationary
Skin Tone 4<br />Rotation
Skin Tone 5<br />Talking
Skin Tone 6<br />Walking

🗝️ Access and Usage

This dataset is built for academic use. Any commercial usage is banned.
There are two kinds of datasets for your convenience: full dataset(370G, 320 x 240 resolution ), mini dataset(48G, 80 x 60 resolution ).
There are two ways for downloads： OneDrive and Baidu Netdisk for researchers of different regions. For those researchers in China, a hard disk could also be a solution. To access the dataset, you are supposed to download this data release agreement.
Please scan and dispatch the completed agreement via your institutional email to tjk24@mails.tsinghua.edu.cn and cc yuntaowang@tsinghua.edu.cn. The email should have the subject line 'MMPD Access Request - your institution.' In the email, outline your institution's website and publications for seeking access to the MMPD, including its intended application in your specific research project. The email should be sent by a faculty rather than a student.

⚙️ Experiment Procedure[Updated]

📊 Distribution

🖥️ The Dataset Structure

MMPD_Dataset
├── subject1
    ├── p1_0.mat        # px_y.mat: x refers to the order of subjects, y refers to the order of the experiments, whcich corresponds to the experiment procedure.
        ├── video       # Rendered images of the subjects at 320 x 240 resolution     [t, w, h, c]
        ├── GT_ppg      # PPG wavefrom signal                                         [t]
        ├── light       # 'LED-low','LED-high','Incandescent','Nature' 
        ├── motion      # 'Stationary','Rotation','Talking','Walking'
        ├── exercise    # True, False
        ├── skin_color  # 3,4,5,6
        ├── gender      # 'male','female'
        ├── glasser     # True, False
        ├── hair_cover  # True, False
        ├── makeup      # True, False
    ├── ... .mat
    ├── p1_19.mat
├── ...
├── subject33
├── size.csv # Each line stands for a mat file and the bytes of this mat file

Reading the data example:

import scipy.io as sio
f = sio.loadmat('p1_0.mat')
print(f.keys())

Benchmark for Open-source Models

We noticed that plenty of new papers are emerging that use MMPD as a test dataset. We are encouraging the community to open-source related codes of their methods and pull a request in the following. We will keep this updated and other researchers could use those indexes as benchmark results. If you only use part of the MMPD dataset, you should put the details on note.

📝 Results from the original paper(tested on MMPD)

Simplest scenario

In the simplest scenario, we only include the stationary, skin tone type 3, and artificial light conditions as benchmarks.

Unsupervised Signal Processing Methods（Subset）

We evaluated six traditional unsupervised methods in our dataset. In the skin tone comparison, we excluded the exercise, natural light, and walking conditions to eliminate any confounding factors and concentrate on the task. Similarly, the motion comparison experiments excluded the exercise and natural light conditions, while the light comparison experiments excluded the exercise and walking conditions. This approach enabled us to exclude cofounding factors and better understand the unique challenges posed by each task.

Supervised Deep Learning Methods（Subset）

In this paper, we investigated how state-of-the-art supervised neural networks perform on MMPD and studied the influence of skin tone, motion, and light. We used the same exclusion criteria as the evaluation on unsupervised methods.

Full Dataset Result

For the full dataset, no existing methods could accurately predict the PPG wave and heart rate. We are looking forward to algorithms that could be applied to daily scenarios. Researchers are encouraged to report their results and communicate with us.

📄 Citation

Title: MMPD: Multi-Domain Mobile Video Physiology Dataset
Jiankai Tang, Kequan Chen, Yuntao Wang, Yuanchun Shi, Shwetak Patel, Daniel McDuff, Xin Liu, "MMPD: Multi-Domain Mobile Video Physiology Dataset", IEEE EMBC, 2023

@INPROCEEDINGS{10340857,
  author={Tang, Jiankai and Chen, Kequan and Wang, Yuntao and Shi, Yuanchun and Patel, Shwetak and McDuff, Daniel and Liu, Xin},
  booktitle={2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)}, 
  title={MMPD: Multi-Domain Mobile Video Physiology Dataset}, 
  year={2023},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/EMBC40787.2023.10340857}}