Awesome
This project is Pose Estimation on iOS with TensorFlow Lite.<br>If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.<br>
| 2D pose estimation in real-time | 3D pose estimation |
|---|---|
| <img src="https://user-images.githubusercontent.com/37643248/77227994-99ba2a80-6bc7-11ea-9b08-9bb57723bc42.gif" width=240px> | <img src="https://user-images.githubusercontent.com/37643248/110994933-e68ca780-83bc-11eb-8331-d827e19d2d36.gif" width=240px> |
Features
- Support 2D pose estimaiton TFLite models
- Real-time demo with Metal framwork
- Photo album demo
- Support 3D pose estimation TFLite model
- Real-time demo with Metal framwork (but realtime model is not ready yet)
- Real-time pose matching demo
- Photo album demo
- Render the result keypoints of 2D pose estimation in 2D demo page
- Render the result keypoints of 3D pose estimation with SceneKit
- Render the heatmaps of 2D pose estimation output
- Part Confidence Maps for typical heatmap based models
- Part Affinity Fields for OpenPose (2D multi-person)
- Implemented pose-matching with cosine similiarity in 3D pose demo
- Implemented to fix the shoulders' keypoints in 3D pose estimation to pre-process for pose-matching
Models
<details><summary>Source Images</summary> <p>| Name | gif | img-0 | img-1 | img-2 |
|---|---|---|---|---|
| Source | - | <img src="https://user-images.githubusercontent.com/37643248/81012180-15301c80-8e94-11ea-83ec-bd45a690efb3.jpg" width=200px> | <img src="https://user-images.githubusercontent.com/37643248/81010350-1ca1f680-8e91-11ea-81fd-0ff4c78b8641.png" width=200px> | <img src="https://user-images.githubusercontent.com/37643248/81009122-0135ec00-8e8f-11ea-8a3a-e62929f19d8e.png" width=200px> |
Joint Samples
| Model Names | gif | img-0 | img-1 | img-2 |
|---|---|---|---|---|
| PoseNet | - |  |  |  |
| PEFM CPM | - |  |  |  |
| PEFM Hourglass | - |  |  |  |
| OpenPose (multi-person) | - |  |  |  |
Heatmap-ConfidenceMap Samples
| Model Names | gif | img-0 | img-1 | img-2 |
|---|---|---|---|---|
| PoseNet | - | - | - | - |
| PEFM CPM | - |  |  |  |
| PEFM Hourglass | - |  |  |  |
| OpenPose (multi-person) | - |  |  |  |
Heatmap-PAF Samples
| Model Names | gif | img-0 | img-1 | img-2 |
|---|---|---|---|---|
| OpenPose (PAF x) | - |  |  |  |
| OpenPose (PAF y) | - |  |  |  |
Meta Data
2D
| ✅ vs ☑️ | Name | Size | Inference<br>Time<br>on iPhone11Pro | Post-process<br>Time<br>on iPhone11Pro | PCKh-0.5 | multi person <br>vs<br> single person | Model Source | Paper | tflite<br>download |
|---|---|---|---|---|---|---|---|---|---|
| ✅ | PoseNet | 13.3 MB | - | - | - | single | tensorflow/tensorflow | PersonLab | |
| ✅ | PEFM CPM | 2.4 MB | - | - | - | single | edvardHua/PoseEstimationForMobile | Convolutional Pose Machines | |
| ✅ | PEFM Hourglass v1 | 1.8 MB | - | - | - | single | edvardHua/PoseEstimationForMobile | Stacked Hourglass Networks | |
| ✅ | PEFM Hourglass v2 | 1.7 MB | - | - | - | single | edvardHua/PoseEstimationForMobile | Stacked Hourglass Networks | |
| ✅ | OpenPose | 7.8 MB | - | - | - | multi | ildoonet/tf-pose-estimation | OpenPose | |
| ☑️ | AlphaPose | - | - | - | - | single | osmr/imgclsmob | RMPE | |
| ☑️ | SelecSLS | - | - | - | - | single | osmr/imgclsmob | - | |
| ☑️ | IBPPose | - | - | - | - | single | osmr/imgclsmob | - | |
| ☑️ | Lightweight OpenPose | - | - | - | - | single | osmr/imgclsmob | OpenPose |
3D
| ✅ vs ☑️ | Name | Size | Inference<br>Time<br>on iPhone11Pro | Post-process<br>Time<br>on iPhone11Pro | (metric) | Model Source | Paper | tflite<br>download |
|---|---|---|---|---|---|---|---|---|
| ✅ | Baseline3DPose | 137.1 MB | 347 ms | 79 ms | - | mks0601/3DMPPE_POSENET_RELEASE | Baseline3D | download |
| ✅ | LiteBaseline3DPose | 16.6 MB | 116 ms<br>(cpu only) | 19 ms<br>(cpu only) | SangbumChoi/MobileHumanPose | MHP | download |
Requirements
- Xcode 11.3+
- iOS 11.0+
- Swift 5
- CocoaPods
gem install cocoapods
Build & Run
- Install dependencies with cocoapods
cd ~/{PROJECT_PATH}
pod install
- Open the
PoseEstimation-TFLiteSwift.xcworkspaceand run the project
See also
- motlabs/awesome-ml-demos-with-ios
- TensorFlow Lite or Tensorflow models provided by:
- CPM and Hourglass model provided by edvardHua/PoseEstimationForMobile
- PoseNet model provided by tensorflow/examples
- OpenPose model provided by ildoonet/tf-pose-estimation
- Various model provided by osmr/imgclsmob
- 3DMPPE PoseNet model provided by mks0601/3DMPPE_POSENET_RELEASE
- Pose estimation with Core ML - tucan9389/PoseEstimation-CoreML
License
This repository is licensed under Apache 2.0. Full license text is available in LICENSE.