Home

Awesome

HUMBI

This repository contains official code (in MATLAB) for exploring and visualizing HUMBI dataset introduced in the paper "HUMBI: A Large Multiview Dataset of Human Body Expressions" (CVPR 2020, spotlight video).

Overview

HUMBI is a large multiview dataset for human body expressions with natural clothing, captured by 107 GoPro HD cameras from 772 distinctive subjects at 60Hz. During capture, each subject was invited into the capture stage and performing a series of tasks for each expression session (i.e. gaze, face, hand, body) sequentially, guided by an instructinal video (about 2.5 minutes).

HUMBI is organized in subject-session-frame-representation hierarchy, i.e. partitioned subject by subject first -> each subject contains 4 expression sessions -> each expression session contains a sequence of frames (time instances) -> each frame contains up to 4 representations: multiview images, 3D keypoints, 3D mesh and appearance(texture) maps. For data sharing, multiple subjects of a certain session are bundled together (appearance and cloth are bundled separately but expected to be merged with image/kps/mesh if you additionally download them). You can download the data in HUMBI website.

For each human body expression session, MATLAB code for reading data of a specified subject and time instance (refered to as "frame" in the following) is provided. There is a visualize.m file in each expression folder and you can specify some options of your choice (e.g. subject index, frame index, visualize in 3d or reprojection on images, show 3d axis or not, show keypoint indices or not, etc). The purpose of the code is to help you understand the data structure and how to use them.

We will keep updating HUMBI website and github page to add more things.

Gaze

During gaze capture session, subject was asked to find and staring at the 4 requested number tag (with known location) posted on the camera stage one by one. We manually check offline and remove cases when subjects failed to look at requested tags.

Within "gaze" folder, there are up to 4 frames as well as "intrinsic.txt", "extrinsic.txt" and "project.txt" storing intrinsic parameters (K), extrinsic parameters (R and C) and projection matrices (M) respectively (see here). Each frame folder contains "image_cropped" folder and "reconstruction" foler:

Face

During face capture session, subject was asked to follow 20 distinctive dynamic facial expressions (e.g., eye rolling, frowning, and jaw opening).

Within "face" folder, there are multiple frames as well as "intrinsic.txt", "extrinsic.txt" and "project.txt" storing intrinsic parameters (K), extrinsic parameters (R and C) and projection matrices (M) respectively (see here). Each frame folder contains "image_cropped" folder and "reconstruction" foler:

Hand

During hand capture session, subject was asked to follow a series of American sign languages (e.g., counting one to ten, greeting, and daily used words).

Within "hand" folder, there are multiple frames as well as "intrinsic.txt", "extrinsic.txt" and "project.txt" storing intrinsic parameters (K), extrinsic parameters (R and C) and projection matrices (M) respectively (see here). Each frame folder contains "image_cropped" folder and "reconstruction" foler:

Body & Cloth

During body capture session, subject was asked to firstly follow a range of full-body movements, and then a slow and full speed dance performances curated by a professional choreographer.

Within "body" folder, there are multiple frames as well as "intrinsic.txt", "extrinsic.txt" and "project.txt" storing intrinsic parameters (K), extrinsic parameters (R and C) and projection matrices (M) respectively (see here). Each frame folder contains "image_cropped" folder and "reconstruction" foler:

Camera Calibration

Intrinsic parameters (stored in "intrinsic.txt") format:

Numcam: (ignore)
numframes: (number of cameras)
numP: (number of cameras)
(ignore) (camer_id, e.g., 4=image0000004.jpg)
(3x3: camera intrinsic matrix)
...iterate for each camera

Extrinsic parameters (stored in "extrinsic.txt") format:

Numcam: (ignore)
numframes: (number of cameras)
numP: (number of cameras)
(ignore) (camer_id, e.g., 4=image0000004.jpg)
(1x3:camera center, where translation matrix can be obtained by T = -RC)
(3x3: camera rotation matrix)
...iterate for each camera

Projection parameters (stored in "intrinsic.txt") format:

Numcam: (ignore)
numframes: (number of cameras)
numP: (number of cameras)
(ignore) (camer_id, e.g., 4=image0000004.jpg)
(3x4: camera projection matrix)
...iterate for each camera

Citation

@InProceedings{Yu_2020_CVPR,
author = {Yu, Zhixuan and Yoon, Jae Shin and Lee, In Kyu and Venkatesh, Prashanth and Park, Jaesik and Yu, Jihun and Park, Hyun Soo},
title = {HUMBI: A Large Multiview Dataset of Human Body Expressions},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

Question?

You can send an email to: humbi.data [at] gmail [dot] com, and we will be happy to answer your questions :blush: