Awesome
Understanding Teacher Gaze Patterns for Robot Learning
This respository contains code for the following paper:
A. Saran*, E.S. Short, A. Thomaz, S. Niekum. Understanding Teacher Gaze Patterns for Robot Learning. Conference on Robot Learning (CoRL), November 2019.
The data for the user study associated with this paper is not permissible for release under IRB constraints. We include dummy data with our code wherever possible.
Code for Statistical Analysis of Gaze Patterns (stat_analysis/)
Pouring Task
stat_analysis/pouring/exp1.py
: Percentage of time during entire demo spent fixating on objects, gripper or other parts of workspacestat_analysis/pouring/exp2.py
: Perecentage accuarcy to predict reference frame per keyframestat_analysis/pouring/exp3.py
: Gaze Patterns between Novice and Expert Usersstat_analysis/pouring/exp4.py
: Gaze-patterns around step KF and non-step KF (KT demos only)stat_analysis/pouring/gaze_filtering_demo.py
: A demo file with dummy data for gaze fixation filtering
Placement Task
stat_analysis/placement/exp1.py
: Percentage of time during entire demo spent fixating on objects, gripper or other parts of workspacestat_analysis/placement/exp2.py
: Gaze fixations on bowl and plate for different relative placement strategy
Code for SubTask Classification (subtask_detection/)
- Non-local neural network
- Compact Generalized Non-local neural network
- Training data for these networks cannot be made available due to IRB restrictions, however we do link trained models and some dummy data for validation.
- Detailed instructions to run the code are available in
subtask_detection/README.md
. - The files which have been modified to add gaze data into the training framework are train_val.py and models/resnet.py.
Code for Reward Learning with Gaze-augmented BIRL (gaze_birl/)
-
To run Gaze-augmented BIRL for KT demos:
- instruction relative to plate:
./test_gaze_birl.py --demo_type 'KT' --exp 'plate' --use_gaze
- instruction relative to bowl:
./test_gaze_birl.py --demo_type 'KT' --exp 'bowl' --use_gaze
- instruction relative to plate:
-
To run Gaze-augmented BIRL for video demos:
- instruction relative to plate:
./test_gaze_birl.py --demo_type 'video' --exp 'plate' --use_gaze
- instruction relative to bowl:
./test_gaze_birl.py --demo_type 'video' --exp 'bowl' --use_gaze
- instruction relative to plate:
-
To run standard BIRL for KT demos:
- instruction relative to plate:
./test_gaze_birl.py --demo_type 'KT' --exp 'plate'
- instruction relative to bowl:
./test_gaze_birl.py --demo_type 'KT' --exp 'bowl'
- instruction relative to plate:
-
To run standard BIRL for video demos:
- instruction relative to plate:
./test_gaze_birl.py --demo_type 'video' --exp 'plate'
- instruction relative to bowl:
./test_gaze_birl.py --demo_type 'video' --exp 'bowl'
- instruction relative to plate:
Bibliography
If you find our work to be useful in your research, please cite:
@article{saran2019understanding,
title={Understanding Teacher Gaze Patterns for Robot Learning},
author={Saran, Akanksha and Short, Elaine Schaertl and Thomaz, Andrea and Niekum, Scott},
booktitle={Conference on Robot Learning},
year={2019},
organization={PMLR}
}