Awesome

NTU-X

This repository contains details and pretrained models for the newly introduced dataset NTU-X, which is an extended version of popular NTU dataset. For additional details and results on experiments using NTU-X, take a look at the paper NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions

What is NTU-X

The original NTU dataset contains the human action skeleton which are captured using the Kinect. These skeletons have <b>25 joints</b>. However, all the current top performing models seem to be bottlenecked at certain classes which involve finer finger level movements such as, reading, writing, eat meal etc.

Hence the new NTU-X dataset, introduces a more detailed <b>118 joints</b> skeleton for the action sequences of the NTU dataset. This new dataset, along with 25 body joints, contains <b>42 finger joints</b> and <b>51 face joints</b>.

Results

Overlayed Examples

Comparing NTU-X to other popular action datasets.

Download NTU-X dataset

<i><a href="https://www.ntu.edu.sg/rose">ROSE Lab</a>, creators of NTU-RGBD dataset have refused to give us permission to release the dataset. NTU-X addresses a fundamental raw-data-level shortcoming of existing Kinect-based dataset, a shortcoming no amount of novel deep models can somehow undo. We had hoped to release the dataset for the benefit of the community once we receive consent. Unfortunately, due to reasons best known to them, keeping the dataset out of community's reach was the decision they arrived at. Apologies to those who were awaiting the release. We hope you understand. </i>

Pretained Models

Few experiments are performed to benchmark this new dataset using the top performing models of the original NTU RGB+D dataset. Details about this models can be found at Models

The classes included in NTU-X

NTU-X contains same classes as NTU RGB+D dataset. The action labels are mentioned below:

FAQs

<b>1. How is the NTU-X dataset created?</b>

It is collected by estimating 3D SMPL-X pose outputs from the RGB frames of the NTU-60 RGB videos. We use both SMPL-X and Expose to perform these estimations.

<b>2. How the pose extractor (SMPLx/ExPose) is decided for each class?</b>

We use a semi-automatic approach to estimate the 3D pose for the videos of each class. Keeping the intra-view and intra-subject variance of the NTU dataset in mind, we sample random videos covering each view perclass of NTU and estimate the SMPL-X, ExPose outputs. The estimated skeleton is then backprojected to its corresponding RGB frame and the accuracy of the alignment is used to select between SMPL-X and Expose.

<b>3. Which class IDs have ExPose used as pose extractor and which class IDs have SMPLx used as pose extractor? </b>

Empirically, we observe that ExPose,SMPL-X perform equally well for single-person actions but SMPL-X, though slow, provides better pose estimates for multi-person action class sequences. The classes selected for SMPL-X and Expose are as follows:

Expose

• A1. drink water
• A2. eat meal/snack
• A3. brushing teeth.
• A4. brushing hair.
• A8. sitting down.
• A9. standing up (from sitting position).
• A12. writing.
• A14. wear jacket.
• A15. take off jacket.
• A16. wear a shoe.
• A17. take off a shoe.
• A18. wear on glasses.
• A19. take off glasses.
• A20. put on a hat/cap.
• A21. take off a hat/cap.
• A24. kicking something.
• A25. reach into pocket.
• A26. hopping (one foot jumping).
• A29. playing with phone/tablet.
• A30. typing on a keyboard.
• A33. check time (from watch).
• A34. rub two hands together.
• A36. shake head.
• A37. wipe face.
• A38. salute.
• A41. sneeze/cough.
• A42. staggering.
• A43. falling.
• A44. touch head (headache).
• A46. touch back (backache).
• A47. touch neck (neckache).
• A48. nausea or vomiting condition.
• A49. use a fan (with hand or paper)/feeling warm.
• A61. put on headphone.
• A62. take off headphone.
• A63. shoot at the basket.
• A64. bounce ball.
• A65. tennis bat swing.
• A68. flick hair.
• A71. make ok sign.
• A72. make victory sign.
• A73. staple book.
• A75. cutting nails.
• A76. cutting paper (using scissors).
• A78. open bottle.
• A79. sniff (smell).
• A80. squat down.
• A82. fold paper.
• A83. ball up paper.
• A84. play magic cube.
• A85. apply cream on face.
• A86. apply cream on hand back.
• A87. put on bag.
• A88. take off bag.
• A89. put something into a bag.
• A90. take something out of a bag.
• A91. open a box.
• A92. move heavy objects.
• A93. shake fist.
• A94. throw up cap/hat.
• A95. hands up (both hands).
• A96. cross arms.
• A97. arm circles.
• A98. arm swings.
• A99. running on the spot.
• A100. butt kicks (kick backward).
• A101. cross toe touch.
• A103. yawn.
• A104. stretch oneself.
• A105. blow nose.
• A106. hit other person with something.
• A107. wield knife towards other person.
• A108. knock over other person (hit with body).
• A111. step on foot.
• A112. high-five.
• A114. carry something with other person.
• A119. support somebody with hand.

SMPL-X

• A5. drop.
• A6. pickup.
• A7. throw.
• A10. clapping.
• A11. reading.
• A13. tear up paper.
• A22. cheer up.
• A23. hand waving.
• A27. jump up.
• A28. make a phone call/answer phone.
• A31. pointing to something with finger.
• A32. taking a selfie.
• A35. nod head/bow.
• A39. put the palms together.
• A40. cross hands in front (say stop).
• A45. touch chest (stomachache/heart pain).
• A50. punching/slapping other person.
• A51. kicking other person.
• A52. pushing other person.
• A53. pat on back of other person.
• A54. point finger at the other person.
• A55. hugging other person.
• A56. giving something to other person.
• A57. touch other person's pocket.
• A58. handshaking.
• A59. walking towards each other.
• A60. walking apart from each other.
• A66. juggling table tennis balls.
• A67. hush (quite).
• A69. thumb up.
• A70. thumb down.
• A74. counting money.
• A77. snapping fingers.
• A81. toss a coin.
• A102. side kick.
• A109. grab other person’s stuff.
• A110. shoot at other person with a gun.
• A113. cheers and drink.
• A115. take a photo of other person.
• A116. follow other person.
• A117. whisper in other person’s ear.
• A118. exchange things with other person.
• A120. finger-guessing game (playing rock-paper-scissors).