Awesome
<div align="center">Awesome Object-Centric Robotic Manipulation
</div> <p align="center"> <img src="docs/overview.png" alt="overview" width="90%" /> </p>This is the repository of A Survey of Embodied Learning for Object-Centric Robotic Manipulation, a comprehensive review of latest advancements in object-centric robotic manipulation including embodied perceptual learning, embodied policy learning, and embodied task-oriented learning. For details, please refer to:
<div align="center" style="font-size: 22px"> <b><a href="https://arxiv.org/abs/2408.11537">A Survey of Embodied Learning for Object-Centric Robotic Manipulation</a></b> </div> <div style="font-size: 18px; font-style: italic;">Any contributions to this list are most welcome! Please feel free to open issues or create pull requests as follows: </div>- Fork the project into your own repository.
- Add the Title, Paper link, Venue, Code/Project link in
README.md
using the following format:|[Title](Paper Link)|Venue|[Code/Project](Code/Project link)|
- Submit the pull request to this branch.
🔥 News
Last update on 2024/08/28
:hammer: Abstract
Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI. It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently. Unlike data-driven machine learning methods, embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback, making it especially suitable for robotic manipulation. In this paper, we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches: 1) Embodied perceptual learning, which aims to predict object pose and affordance through various data representations; 2) Embodied policy learning, which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning; 3) Embodied task-oriented learning, designed to optimize the robot's performance based on the characteristics of different tasks in object grasping and manipulation. In addition, we offer an overview and discussion of public datasets, evaluation metrics, representative applications, current challenges, and potential future research directions.
:pencil: Citation
@article{zheng2024ocrm,
author= {Zheng, Ying and Yao, Lei and Su, Yuejiao and Zhang, Yi and Wang, Yi and Zhao, Sicheng and Zhang, Yiyi and Chau, Lap-Pui},
title= {A Survey of Embodied Learning for Object-Centric Robotic Manipulation},
journal={arXiv preprint arXiv:2408.11537},
year={2024}
}
- Awesome Object-Centric Robotic Manipulation
Embodied Perceptual Learning
Data Representation
Image-Based Representation
3D-Aware Representation
Depth-based representations
Paper | Venue | Code/Project |
---|---|---|
RGB-D object detection and semantic segmentation for autonomous manipulation in clutter | IJRR 2018 | - |
Point cloud-based representations
Paper | Venue | Code/Project |
---|---|---|
Shape completion enabled robotic grasping | IROS 2017 | - |
PointNetGPD: Detecting Grasp Configurations from Point Sets | ICRA 2019 | Project |
3D Implicit Transporter for Temporally Consistent Keypoint Discovery | ICCV 2023 | Code |
Other representations
Tactile-Based Representation
Object Pose Estimation
Instance-Level Object Pose Estimation
Category-Level Object Pose Estimation
Novel Object Pose Estimation
Affordance Learning
Affordance Learning by Supervised Learning
Affordance Learning From Interaction
Embodied Policy Learning
Policy Representation
Explicit Policy
Implicit Policy
Paper | Venue | Code/Project |
---|---|---|
Regularizing Model-Based Planning with Energy-Based Models | CoRL 2019 | - |
Implicit Behavioral Cloning | CoRL 2021 | - |
Implicit Distributional Reinforcement Learning | NeurIPS 2020 | Code |
Energy-Based Imitation Learning | AAMAS 2021 | Code |
Diffusion Policy
Policy Learning
Reinforcement Learning
Imitation Learning
Other Methods
Embodied Task-Oriented Learning
Object Grasping
Single-Object Grasping
Transparent Object Grasping.
Grasping in Clutter.
Dynamic Object Grasping.
Multi-Object Grasping
Object Manipulation
Non-Dexterous Manipulation
Dexterous Manipulation
Tool Manipulation.
Datasets
Datasets for Object Grasping
Paper | Name | Venue | Code/Project |
---|---|---|---|
Efficient grasping from rgbd images: Learning using a new rectangle representation | Cornell | ICRA 2011 | - |
Real-World Multiobject, Multigrasp Detection | Multi-Object | RAL 2018 | Code |
Jacquard: A Large Scale Dataset for Robotic Grasp Detection | Jacquard | IROS 2018 | Project |
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations | VR-Grasping-101 | ICRA 2018 | Project |
ACRONYM: A Large-Scale Grasp Dataset Based on Simulation | ACRONYM | ICRA 2021 | Project |
Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation | EGAD | RAL 2020 | Project |
Graspnet-1billion: A large-scale benchmark for general object grasping | GraspNet-1Billion | CVPR 2020 | Project |
Grasp-Anything: Large-scale Grasp Dataset from Foundation Models | Grasp-Anything | ICRA 2024 | Project |
Datasets for Object Manipulation
Applications
Industrial Robots
Agricultural Robots
Paper | Venue | Code/Project |
---|---|---|
Technological revolutions in smart farming: Current trends, challenges & future directions | Comput Electron Agr 2022 | - |
Robotics in agriculture: Advanced technologies in livestock farming and crop cultivation | E3S Web Conf. 2024 | - |
State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A review | Comput Electron Agr 2020 | - |
Current status and future challenges in implementing and upscaling vertical farming systems | Nature Food 2021 | - |
Intelligent robots for fruit harvesting: Recent developments and future challenges | Precis Agric 2022 | - |
Domestic Robots
Surgical Robots
Other Applications
Paper | Venue | Code/Project |
---|---|---|
Reinforcement learning in dual-arm trajectory planning for a free-floating space robot | Aerosp Sci Technol 2020 | Project |
Robust Adaptive Learning Control of Space Robot for Target Capturing Using Neural Network | TNNLS 2023 | - |
RLBench: The Robot Learning Benchmark & Learning Environment | RAL 2020 | Project |
Challenges and Future Directions
Sim-to-Real Generalization
Multimodal Embodied LLMs
Human-Robot Collaboration
Paper | Venue | Code/Project |
---|---|---|
Progress and Prospects of the Human-Robot Collaboration | Autonomous Robots 2018 | - |
A Learning Based Hierarchical Control Framework for Human–Robot Collaboration | TASE 2022 | - |
Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration | CoRL 2022 | Project |
Model Compression and Robot Acceleration
Paper | Venue | Code/Project |
---|---|---|
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control | CoRL 2023 | Project |
Model Interpretability and Application Safety
:books: License
This repository is released under the MIT license.