Home

Awesome

<div align="center">

Awesome Object-Centric Robotic Manipulation

Arxiv Awesome license Maintenance PR's Welcome

</div> <p align="center"> <img src="docs/overview.png" alt="overview" width="90%" /> </p>

This is the repository of A Survey of Embodied Learning for Object-Centric Robotic Manipulation, a comprehensive review of latest advancements in object-centric robotic manipulation including embodied perceptual learning, embodied policy learning, and embodied task-oriented learning. For details, please refer to:

<div align="center" style="font-size: 22px"> <b><a href="https://arxiv.org/abs/2408.11537">A Survey of Embodied Learning for Object-Centric Robotic Manipulation</a></b> </div> <div style="font-size: 18px; font-style: italic;">Any contributions to this list are most welcome! Please feel free to open issues or create pull requests as follows: </div>

🔥 News

Last update on 2024/08/28

:hammer: Abstract

Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI. It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently. Unlike data-driven machine learning methods, embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback, making it especially suitable for robotic manipulation. In this paper, we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches: 1) Embodied perceptual learning, which aims to predict object pose and affordance through various data representations; 2) Embodied policy learning, which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning; 3) Embodied task-oriented learning, designed to optimize the robot's performance based on the characteristics of different tasks in object grasping and manipulation. In addition, we offer an overview and discussion of public datasets, evaluation metrics, representative applications, current challenges, and potential future research directions.

:pencil: Citation

@article{zheng2024ocrm,
  author= {Zheng, Ying and Yao, Lei and Su, Yuejiao and Zhang, Yi and Wang, Yi and Zhao, Sicheng and Zhang, Yiyi and Chau, Lap-Pui},
  title= {A Survey of Embodied Learning for Object-Centric Robotic Manipulation},
  journal={arXiv preprint arXiv:2408.11537},
  year={2024}
}

Embodied Perceptual Learning

Data Representation

Image-Based Representation

PaperVenueCode/Project
Real-time seamless single shot 6d object pose predictionCVPR 2018-
MonoGraspNet: 6-DoF Grasping with a Single RGB ImageICRA 2023-
RGBGrasp: Image-based Object Grasping by Capturing Multiple Views during Robot Arm Movement with Neural Radiance FieldsRAL 2024Project
RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose EstimationICRA 2024Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent ObjectsCoRL 2022Project
Clear-Splatting: Learning Residual Gaussian Splats for Transparent Object ManipulationICRAW 2024-

3D-Aware Representation

Depth-based representations

PaperVenueCode/Project
RGB-D object detection and semantic segmentation for autonomous manipulation in clutterIJRR 2018-

Point cloud-based representations

PaperVenueCode/Project
Shape completion enabled robotic graspingIROS 2017-
PointNetGPD: Detecting Grasp Configurations from Point SetsICRA 2019Project
3D Implicit Transporter for Temporally Consistent Keypoint DiscoveryICCV 2023Code

Other representations

PaperVenueCode/Project
RVT: Robotic View Transformer for 3D Object ManipulationCoRL 2023Project
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature FieldsCoRL 2023Project
ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic ManipulationECCV 2024Project
Object-Aware Gaussian Splatting for Robotic ManipulationICRAW 2024Project

Tactile-Based Representation

PaperVenueCode/Project
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and ForceSensors 2017-
Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulationRAL 2020Project
Allsight: A low-cost and high-resolution round tactile sensor with zero-shot learning capabilityRAL 2023Code
Learning to grasp without seeingISER 2018Project
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich TasksTRO 2020Project
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?CoRL 2017-
Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic PlayCoRL 2023Project
More Than a Feeling: Learning to Grasp and Regrasp using Vision and TouchRAL 2018-
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile RepresentationsCoRL 2022Project
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile SensorsCoRL 2022Code
Efficient Tactile Simulation with Differentiability for Robotic ManipulationCoRL 2023Project
Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data GenerationCoRL 2023-
Tactile-Augmented Radiance FieldsCVPR 2024Project

Object Pose Estimation

Instance-Level Object Pose Estimation

PaperVenueCode/Project
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes IntroductionRSS 2018Project
CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose EstimationICCV 2019-
6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose EstimationCVPR 2024-
Match Normalization: Learning-Based Point Cloud Registration for 6D Object Pose Estimation in the Real WorldTPAMI 2024-
Deep fusion transformer network with weighted vector-wise keypoints voting for robust 6d object pose estimationICCV 2023Code

Category-Level Object Pose Estimation

PaperVenueCode/Project
Normalized object coordinate space for category-level 6d object pose and size estimationCVPR 2019Project
Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimationICCV 2021Project
Rigidity-aware detection for 6d object pose estimationCVPR 2023-
Revisiting Fully Convolutional Geometric Features for Object 6D Pose EstimationICCV 2023Code
Category-Level Articulated Object Pose EstimationCVPR 2020Project
Towards Real-World Category-level Articulation Pose EstimationTIP 2022-
CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size EstimationICRA 2022Project
ShAPO: Implicit Representations for Multi Object Shape Appearance and Pose OptimizationECCV 2022Project
FSD: Fast Self-Supervised Single RGB-D to Categorical 3D ObjectsICRA 2024Project
DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose EstimationIROS 2024Project

Novel Object Pose Estimation

PaperVenueCode/Project
Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB ImagesECCV 2022Project
GigaPose: Fast and Robust Novel Object Pose Estimation via One CorrespondenceCVPR 2024Code
OnePose: One-Shot Object Pose Estimation without CAD ModelsCVPR 2022Project
Zero-Shot Category-Level Object Pose EstimationECCV 2022Code
SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose EstimationCVPR 2024Code
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel ObjectsCVPR 2024Project

Affordance Learning

Affordance Learning by Supervised Learning

PaperVenueCode/Project
Affordancenet: An end-to-end deep learning approach for object affordance detectionICRA 2018Code
Grounded Human-Object Interaction Hotspots From VideoICCV 2019Porject
Affordances from Human Videos as a Versatile Representation for RoboticsCVPR 2023Project
Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot ManipulationECCV 2024Porject
RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic ManipulationarXiv 2024Project
Open-Vocabulary Affordance Detection in 3D Point CloudsIROS 2023Project
Learning 2D Invariant Affordance Knowledge for 3D Affordance GroundingarXiv 2024Code

Affordance Learning From Interaction

PaperVenueCode/Project
Where2Act: From Pixels to Actions for Articulated 3D ObjectsICCV 2021Project
AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot InteractionsECCV 2022Porject
DualAfford: Learning Collaborative Visual Affordance for Dual-gripper ManipulationICLR 2023Project
Self-Supervised Learning of Action Affordances as Interaction ModesICRA 2023Porject
Information-driven Affordance Discovery for Efficient Robotic ManipulationarXiv 2024Project
Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated ObjectsNeurIPS 2023Project
Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud NoiseICRA 2024-
Learning environment-aware affordance for 3d articulated object manipulation under occlusionsNeurIPS 2023Code
End-to-End Affordance Learning for Robotic ManipulationICRA 2023Project

Embodied Policy Learning

Policy Representation

Explicit Policy

PaperVenueCode/Project
Robotic grasping using deep reinforcement learningCASE 2020-
Scalable Deep Reinforcement Learning for Vision-Based Robotic ManipulationCoRL 2018Porject
Rapid Motor Adaptation for Robotic Manipulator ArmsCVPR 2024-
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical RobotIROS 2023Code
Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasksICRA 2022Project

Implicit Policy

PaperVenueCode/Project
Regularizing Model-Based Planning with Energy-Based ModelsCoRL 2019-
Implicit Behavioral CloningCoRL 2021-
Implicit Distributional Reinforcement LearningNeurIPS 2020Code
Energy-Based Imitation LearningAAMAS 2021Code

Diffusion Policy

PaperVenueCode/Project
Diffusion policies as an expressive policy class for offline reinforcement learningICLR 2023Code
Diffusion Policy: Visuomotor Policy Learning via Action DiffusionRSS 2023Project
Is conditional generative modeling all you need for decision-making?ICLR 2023-
Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic ManipulationCVPR 2024Project
Dexterous Functional Pre-Grasp Manipulation with Diffusion PolicyarXiv 2024Project
Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal GoalsICRAW 2024-
Language-Guided Object-Centric Diffusion Policy for Collision-Aware Robotic ManipulationarXiv 2024-
Goal-Conditioned Imitation Learning using Score-based Diffusion PoliciesRSS 2023Project

Policy Learning

Reinforcement Learning

PaperVenueCode/Project
Learning dexterous in-hand manipulationIJRR 2020-
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical RobotIROS 2023Code
Rapid Motor Adaptation for Robotic Manipulator ArmsCVPR 2024-
SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and RenderingRSS 2023Project
What Matters in Learning from Offline Human Demonstrations for Robot ManipulationCoRL 2021Project
Demonstration-Guided reinforcement learning with efficient exploration for task automation of surgical robotICRA 2023Code
Text2Reward: Reward Shaping with Language Models for Reinforcement LearningICLR 2024Project
Eureka: Human-Level Reward Design via Coding Large Language ModelsICLR 2024Project

Imitation Learning

PaperVenueCode/Project
Behavioral Cloning from ObservationIJCAI 2018-
Maximum Entropy Inverse Reinforcement LearningAAAI 2008-
SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained NetworksICRA 2024Project
Transformer-based deep imitation learning for dual-arm robot manipulationIROS 2021Project
Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareRSS 2023Project
MimicGen: A Data Generation System for Scalable Robot Learning using Human DemonstrationsCoRL 2023Project
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain DatasetsRSS 2022Project
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention TransformersRSS 2024Project
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action RepresentationsICRA 2024Project
Learning Manipulation by Predicting InteractionRSS 2024Project
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for ManipulationICRA 2022Project
Local Neural Descriptor Fields: Locally Conditioned Object Representations for ManipulationICRA 2023Project
Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation LearningICLR 2023Code
Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic ManipulationCVPR 2024Project
EDGI: Equivariant Diffusion for Planning with Embodied AgentsNeurIPS 2023-
SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusionICRA 2023Project

Other Methods

PaperVenueCode/Project
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned PolicyCVPR 2023Project
UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist LearningICCV 2023-
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language PlanningarXiv 2023Project
Grounding Multimodal Large Language Models in ActionsarXiv 2024-
OpenVLA: An Open-Source Vision-Language-Action ModelarXiv 2024Project
3D-VLA: A 3D Vision-Language-Action Generative World ModelICML 2024Code
RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language ModelsICML 2024Project

Embodied Task-Oriented Learning

Object Grasping

Single-Object Grasping

PaperVenueCode/Project
Deep Learning for Detecting Robotic GraspsIJRR 2015-
Real-Time Grasp Detection Using Convolutional Neural NetworksICRA 2015-
RGB-D Object Recognition and Grasp Detection using Hierarchical Cascaded ForestsTRO 2017-
FANet: Fast and Accurate Robotic Grasp Detection Based on KeypointsTASE 2023-
Anygrasp: Robust and efficient grasp perception in spatial and temporal domainsTRO 2023-
Dynamic grasp and trajectory planning for moving objectsAutonomous Robots 2019-
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis ApproachRSS 2018Code
DITTO: Demonstration Imitation by Trajectory TransformationAutonomous Robots 2019Project
D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object InteractionsCVPR 2022Project
GraspXL: Generating Grasping Motions for Diverse Objects at ScaleECCV 2024Project

Transparent Object Grasping.

PaperVenueCode/Project
Transcg: A large-scale real-world dataset for transparent object depth completion and a grasping baselineRAL 2022Project
Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent ObjectsCoRL 2021Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent ObjectsCoRL 2022Project
Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerfICRA 2023Project
NFL: Normal Field Learning for 6-DoF Grasping of Transparent ObjectsRAL 2023Project
TRansPose: Large-scale multispectral dataset for transparent objectIJRR 2024Project
TODE-Trans: Transparent Object Depth Estimation with TransformerICRA 2023Code

Grasping in Clutter.

PaperVenueCode/Project
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered ScenesICRA 2021Project
CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from SimulationICRA 2022Project
6-DOF Grasping for Target-driven Object Manipulation in ClutterICRA 2020-
DDGC: Generative Deep Dexterous Grasping in ClutterRAL 2021Project
Graspness Discovery in Clutters for Fast and Accurate Grasp DetectionICCV 2021-
Discriminative Active Learning for Robotic Grasping in Cluttered SceneRAL 2023-
Efficient learning of goal-oriented push-grasping synergy in clutterRAL 2021Code
Robust object grasping in clutter via singulationICRA 2019-
A deep learning approach to grasping the invisibleRAL 2020Project
A joint modeling of vision-language-action for target-oriented grasping in clutterICRA 2023Code

Dynamic Object Grasping.

PaperVenueCode/Project
Predicting Human Intentions in Human–Robot Hand-Over Tasks Through Multimodal LearningTASE 2021-
Reactive Human-to-Robot Handovers of Arbitrary ObjectsICRA 2021Project
Flexible Handover with Real-Time Robust Dynamic Grasp Trajectory GenerationIROS 2023-
Learning Human-to-Robot Handovers from Point CloudsCVPR 2023Project
GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation Demonstration and ImitationCVPR 2024Project
SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot HandoversICRA 2024Project
Dynamic Grasping with Reachability and Motion AwarenessIROS 2021Project
Target-referenced Reactive Grasping for Dynamic ObjectsCVPR 2023Project
ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation3DV 2024Project

Multi-Object Grasping

PaperVenueCode/Project
Multi-Object Grasping in the PlaneISRR 2022Project
Multi-Object Grasping - Types and TaxonomyICRA 2022-
Learning to Efficiently Plan Robust Frictional Multi-Object GraspsIROS 2023-
Multi-Object Grasping—Experience Forest for Robotic Finger Movement StrategiesRAL 2024-
Push-MOG: Efficient Pushing to Consolidate Polygonal Objects for Multi-Object GraspingCASE 2023Project
Exploiting Kinematic Redundancy for Robotic Grasping of Multiple ObjectsTRO 2023-
Grasp multiple objects with one handRAL 2024Project

Object Manipulation

Non-Dexterous Manipulation

PaperVenueCode/Project
Pick and place without geometric object modelsICRA 2018-
Self-supervised Learning for Precise Pick-and-place without Object ModelRAL 2020Project
SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene GraphsICRA 2024Project
Form2fit: Learning shape priors for generalizable assembly from disassemblyICRA 2020Project
Mesh-based Dynamics with Occlusion Reasoning for Cloth ManipulationRSS 2022Project
Learning where to trust unreliable models in an unstructured world for deformable object manipulationScience Robotics 2021-
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable PartsCVPR 2023Project
UniDoorManip: Learning Universal Door Manipulation Policy Over Large-scale and Diverse Door Manipulation EnvironmentsarXiv 2024Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud ObservationsCVPR 2023Project
Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic ManipulationRAL 2022Project
Self-Corrected Multimodal Large Language Model for End-to-End Robot ManipulationarXiv 2024Project

Dexterous Manipulation

PaperVenueCode/Project
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous ManipulationCoRL 2023Project
MyoDex: A Generalizable Prior for Dexterous ManipulationICML 2023Project
Deep Dynamics Models for Learning Dexterous ManipulationCoRL 2020Project
Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous ManipulationICRA 2023Project
Visual Dexterity: In-hand Dexterous Manipulation from DepthICMLW 2023-
Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physicsICLR 2023Project

Tool Manipulation.

PaperVenueCode/Project
Tactile Tool ManipulationICRA 2023-
Understanding Tools: Task-Oriented Object Modeling, Learning and RecognitionCVPR 2015-
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep LearningRAL 2021-
KETO: Learning Keypoint Representations for Tool ManipulationICRA 2020-
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-SupervisionIJRR 2020Project
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with ToolsCoRL 2023Project
Leveraging Language for Accelerated Learning of Tool ManipulationICLR 2022Project
Creative robot tool use with large language modelsarXiv 2023Project

Datasets

Datasets for Object Grasping

PaperNameVenueCode/Project
Efficient grasping from rgbd images: Learning using a new rectangle representationCornellICRA 2011-
Real-World Multiobject, Multigrasp DetectionMulti-ObjectRAL 2018Code
Jacquard: A Large Scale Dataset for Robotic Grasp DetectionJacquardIROS 2018Project
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D RepresentationsVR-Grasping-101ICRA 2018Project
ACRONYM: A Large-Scale Grasp Dataset Based on SimulationACRONYMICRA 2021Project
Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulationEGADRAL 2020Project
Graspnet-1billion: A large-scale benchmark for general object graspingGraspNet-1BillionCVPR 2020Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation ModelsGrasp-AnythingICRA 2024Project

Datasets for Object Manipulation

PaperNameVenueCode/Project
Yale-CMU-Berkeley dataset for robotic manipulation researchYCBIJRR 2017Project
AKB-48: A Real-World Articulated Object Knowledge BaseMulti-ObjectCVPR 2022Project
SAPIEN: A SimulAted Part-based Interactive ENvironmentPartNet-MobilityCVPR 2020Project
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable PartsGAPartNetCVPR 2023Project
Maniskill2: A unified benchmark for generalizable manipulation skillsManiSkill2ICLR 2023Project
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D ScenesARNOLDICCV 2023Project
Bi-DexHands: Towards Human-Level Bimanual Dexterous ManipulationBi-DexHandsTPAMI 2023Code
DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated ObjectsDexArtCVPR 2023Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation ModelsGrasp-AnythingICRA 2024Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud ObservationsPartManipCVPR 2023Project
BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic SimulationBEHAVIOR-1KCoRL 2022Project

Applications

Industrial Robots

PaperVenueCode/Project
Robot learning of industrial assembly task via human demonstrationsAutonomous Robots 2019-
Applying a 6 DoF robotic arm and digital twin to automate fan-blade reconditioning for aerospace maintenance, repair, and overhaulSensors 2020-
Advanced predictive maintenance with machine learning failure estimation in industrial packaging robotsDAS 2020-
Compound fault diagnosis for industrial robots based on dual-transformer networksManu. Sys. 2023-

Agricultural Robots

PaperVenueCode/Project
Technological revolutions in smart farming: Current trends, challenges & future directionsComput Electron Agr 2022-
Robotics in agriculture: Advanced technologies in livestock farming and crop cultivationE3S Web Conf. 2024-
State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A reviewComput Electron Agr 2020-
Current status and future challenges in implementing and upscaling vertical farming systemsNature Food 2021-
Intelligent robots for fruit harvesting: Recent developments and future challengesPrecis Agric 2022-

Domestic Robots

PaperVenueCode/Project
Learning arbitrary-goal fabric folding with one hour of real robot experienceCoRL 2021Project
RCareWorld: A Human-centric Simulation World for Caregiving RobotsIROS 2022Project
Improving Robotic Cooking Using Batch Bayesian OptimizationRAL 2022-
Robot Cooking With Stir-Fry: Bimanual Non-Prehensile Manipulation of Semi-Fluid ObjectsRAL 2022-
Efficient Task/Motion Planning for a Dual-arm Robot from Language Instructions and Cooking ImagesIROS 2022-
Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box OptimizationRAL 2024-

Surgical Robots

PaperVenueCode/Project
Autonomous tissue manipulation via surgical robot using learning based model predictive controlICRA 2019Project
Super deep: A surgical perception framework for robotic tissue manipulation using deep learning for feature extractionICRA 2021-
Transition state clustering: Unsupervised surgical trajectory segmentation for robot learningIJRR 2017Project
SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot LearningIROS 2021Code
Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg TransferRAL 2023-
Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot LearningRAL 2023Project
AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic HysterectomyMICCAI 2022Project
ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented DexterityICRA 2024Project

Other Applications

PaperVenueCode/Project
Reinforcement learning in dual-arm trajectory planning for a free-floating space robotAerosp Sci Technol 2020Project
Robust Adaptive Learning Control of Space Robot for Target Capturing Using Neural NetworkTNNLS 2023-
RLBench: The Robot Learning Benchmark & Learning EnvironmentRAL 2020Project

Challenges and Future Directions

Sim-to-Real Generalization

PaperVenueCode/Project
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic GraspingICRA 2018Project
Sim-to-Real Transfer of Robotic Control with Dynamics RandomizationICRA 2018-
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real WorldIROS 2017Project
On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real TransferRAL 2024-
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online CorrectionarXiv 2024Project
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust ManipulationCVPR 2024Project
Generalizing 6-DoF Grasp Detection via Domain Prior KnowledgeRSS 2024Project
Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin PickingECCV 2022Project
Kalman Filter-Based One-Shot Sim-to-Real Transfer LearningRAL 2023-

Multimodal Embodied LLMs

PaperVenueCode/Project
Embodied Task Planning with Large Language ModelsarXiv 2023Project
PaLM-E: An Embodied Multimodal Language ModelICML 2023Project
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic ManipulationCVPR 2024Project
Reasoning Tuning Grasp: Adapting Multi-Modal Large Language Models for Robotic GraspingCoRLW 2023Project
ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language ModelsIROS 2024Code
A3VLM: Actionable Articulation-Aware Vision Language Modearxiv 2024Code

Human-Robot Collaboration

PaperVenueCode/Project
Progress and Prospects of the Human-Robot CollaborationAutonomous Robots 2018-
A Learning Based Hierarchical Control Framework for Human–Robot CollaborationTASE 2022-
Co-GAIL: Learning Diverse Strategies for Human-Robot CollaborationCoRL 2022Project

Model Compression and Robot Acceleration

PaperVenueCode/Project
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlCoRL 2023Project

Model Interpretability and Application Safety

PaperVenueCode/Project
Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic ManipulationIROS 2019Project
Efficient and Interpretable Robot Manipulation with Graph Neural NetworksRAL 2022-
Interpretable Robotic Manipulation from LanguagearXiv 2024Code
Safety-Gymnasium: A Unified Safe Reinforcement Learning BenchmarkNeurIPS 2023Project
Physical Adversarial Attack on a Robotic ArmRAL 2022-
SMOF - A Safety MOnitoring Framework for Autonomous SystemsTSMC 2016-

:books: License

This repository is released under the MIT license.