Awesome

Insight: A Haptic Sensor Powered by Vision and Machine Learning

*This is the only official content release regarding the design of Insight.*

This project shows the principle desing of Insight. It is a soft thumb-sized haptic sensor that can enbable robots feel directional force distributions all over a 3D conical surface.

The paper is titled "A soft thumb-sized vision-based sensor with accurate all-round force perception", published at Nature Machine Intelligence, 2022.

This work was done by Huanbo Sun, Katherine J. Kuchenbecker, Georg Martius at the Max Planck Institute for Intelligent Systems.

If you make used of our work, please use the citation information below.

Abstract

Vision-based haptic sensors have emerged as a promising approach to robotic touch due to affordable high-resolution cameras and successful computer vision techniques; however, their physical design and the information they provide do not yet meet the requirements of real applications. We present a robust, soft, low-cost, vision-based, thumb-sized three-dimensional haptic sensor named Insight, which continually provides a directional force-distribution map over its entire conical sensing surface. Constructed around an internal monocular camera, the sensor has only a single layer of elastomer over-moulded on a stiff frame to guarantee sensitivity, robustness and soft contact. Furthermore, Insight uniquely combines photometric stereo and structured light using a collimator to detect the three-dimensional deformation of its easily replaceable flexible outer shell. The force information is inferred by a deep neural network that maps images to the spatial distribution of three-dimensional contact force (normal and shear). Insight has an overall spatial resolution of 0.4 mm, a force magnitude accuracy of around 0.03 N and a force direction accuracy of around five degrees over a range of 0.03–2 N for numerous distinct contacts with varying contact area. The presented hardware and software design concepts can be transferred to a wide variety of robot parts.

Working Mechanism

Hardware Design

Insight is a vision-based haptic sensor that uses a camera and a structured lighting system to detect external contact happened on a soft hollow elastomer.

Software Design

Insight constantly records images from inside using a camera. Feeding these images and a reference image into a trained machine learning model makes it possible to estimate the force distribution all over a surface. Each pixel in the force prediction map has three values that indicate the force strength in three directions (x, y, and z). In a force visualization on the right, each point corresponding to each pixel in the force map shows the force distribution of the contact in both normal and shear directions.

Key Components

Four key design components make Insight work properly: the mechanical design of its shell, the tailored imaging system inside, automatic data collection, and cutting-edge deep learning are used behind the scenes to create a cheap, robust, and high-fidelity device.

Mechanics

Our hybrid structure of a soft elastomer shell enclosing a stiff metal skeleton ensures high sensitivity and robustness. The soft elastomer is for detecting light contact with high sensitivity. The metal skeleton is for holding the elastomer’s global shape and sustaining high-impact forces.

Elastomer

3D Printer: Formlabs Form 3
3D Printing Material: Tough Resin FLTOTL05. Note: The material is updated.
Mold Design:
- Mold 1
- Mold 2
- Mold 3
Elastomer Material:
Vacuum Chamber VP1100, 5 Pa. Note: The Pump is updated.

Metal Skeleton

3D Printer: ExOne X1 25 Pro
3D Printing Material: AlSi10Mg-0403 Alluminum Alloy Note: The material is updated.
Geometry Design: Skeleton
Printing Service: Shapeways

Connector and Supporter

3D Printer: ExOne X1 25 Pro
3D Printing Material: AlSi10Mg-0403 Alluminum Alloy Note: The material is updated.
Geometry Design:
- Connector
- Supportor
Printing Service: Shapeways

Imaging System

We use a LED ring and a collimator to create structured light. The structured light forms a light cone inside the enclosed volume. Each voxel inside this volume has a specific color code with 3D information. External contact moves the elastomer from one color zone to another. Together with shading effects the camera can detect 3D deformations of the surface from a single 2D image. As shown on the right figure, mimicking human fingerprint, we also add ridges to the surface to help the system recognize the directions of the surface extension.

Camera

Maker Hawk Raspberry Pi Camera Module 8 MP (Raspberry Pi camera V.2.0)

LED Ring

Neopixel Ring with eight pieces of WS2812 5050

Collimator

3D Printer: Formlabs Form 3
Material: Standard Black
Geometry Design: Collimator

DAQ

Testbed

We design a test bed to stimulate the sensor in a controlled way, with both normal and shear forces, over the entire 3D sensing surface.

Linear Guide for 3 DOF Translation

Motors for 2 DOF Rotation

Force/Torque Sensor

Software

Data Collection

Communication

Coordinate Transfer

Data Collection

Data Processing

Direct Single Contact Prediction

Force Map Predcition

Mapping/Visualization

Citation

Please use the following citation if you make use of our work:

BibTex

@article{Insight,
  title = {A soft thumb-sized vision-based sensor with accurate all-round force perception},
  author = {Sun, Huanbo and Kuchenbecker, Katherine J. and Martius, Georg},
  journal = {Nature Machine Intelligence},
  volume = {4},
  organization = {Max Planck Institute for Intelligent Systems},
  month = feb,
  year = {2022},
  doi = {10.1038/s42256-021-00439-3},
  url = {https://rdcu.be/cHCl9},
  month_numeric = {2}
  }

RIS

TY  - JOUR
AU  - Sun, Huanbo
AU  - Kuchenbecker, Katherine J.
AU  - Martius, Georg
PY  - 2022
DA  - 2022/02/01
TI  - A soft thumb-sized vision-based sensor with accurate all-round force perception
JO  - Nature Machine Intelligence
SP  - 135
EP  - 145
VL  - 4
IS  - 2
SN  - 2522-5839
UR  - https://doi.org/10.1038/s42256-021-00439-3
DO  - 10.1038/s42256-021-00439-3
ID  - Sun2022