Awesome
DeepFisheye Realtime
This code is for running pretrained model of DeepFisheyeNet. To find out more about DeepFisheye, please visit the project page.
How to use
Install prerequisites
1. Install Spinnaker
The file can be downloaded from FLIR.
2. Install PySpin
PySpin is Python API of Spinnaker. It is possible to access an FLIR camera with this library. Install instructions and the library file are in the Spinnaker.
3. Install pip packages
$ pip install -r requirements.txt
4. Download the pretrained weight file
Download the file from Google Drive and put the file in the root directory.
5. Run
$ python demo.py
Apparatus
DeepFisheye is tested with the following apparatus.
- FLIR Chameleon3 (FLIR CM3-U3-13Y3C-CS)
- Fisheye lens (M12) (from AliExpress, No model ID)
- FLIR S Mount (M12) Lens Holder
It is possible to use different cameras. However, in that case, there are some issues to take care about.
Motion blur
Motion blur makes harder for the network to recognize hands. We tried chip USB cameras, but they had too much motion blur. It is why we used the machine vision camera.
Camera calibration
If the same lens (and the same lens mount) is used, then our camera parameters in setting.yaml
would work fine. However, different apparatus is used, then camera calibration has to be done.
The calibration can be done with OCamCalib. It is really great tool. Here is a rough process of camera calibration.
OCamCalib calibration
- Take 8 to 10 images with the camera.
- Crop the images to square images. Do not randomly cut the images. Carefully chose center and width of the cropping. And set
crop_width_center
,crop_height_center
,width
, andheight
insetting.yaml
. This code cuts images from the camera with these parameters. - Get camera parameters from OCamCalib.
- Put the camera parameters in
setting.yaml
(z_calib
andaffine_calib
).
Tips
Hand size
The outputs of DeepFisheyeNet work is not in correct scale. Therefore, joint_scale
should be set correctly in setting.yaml
. The length between a user's wrist and Metacarpophalangeal joint of middle finger (hand size) in milli meter should be put in joint_scale
. If you don't know your hand size, 70
is a good value to start. Try adjusting this value.
Filter
The outputs of DeepFisheyeNet cannot be directly used for interaction application, because they are too noisy. Therefore, we added One Euro filters for the demo application (not in experiment).
However, we did not include filters in this code. But, we share the parameters that we used.
# One Euro filter params
min_cutoff = 1
beta = 0.05
d_cutoff = 1.0
Camera exposure
Camera exposure can be controlled in setting.yaml
. Please set exposure not to make image too bright or too dark. If exposure: -1
, it is set automatically.
Contact
If you have any questions, please contact Keunwoo Park or add an issue.