Carl Mueller

Augmented Reality for Constrained Learning from Demonstration

We recently submitted a paper to ICRA 2021 that utilizes Augmented Reality to visualize models learned through user demonstration as well as an interface to add/remove/edit constraints on those models.


ARC-LfD is an augmented reality interfaces that enables users to visualize a learned keyframe model from CC-LfD. It accomplishes this through visualizations of sampled keyframes that the robot will follow as waypoints, as well as visualizations of constraints assigned to keyframes. In the below image, you can see sequences of expected robot behavior through keyframes, as well as visualizations of applied constraints.

ar viz fig
AR visualizations of expected robot skill execution and various constraints.

ARC-LfD also enables users to edit constraints through menus that allow for constraint specific parameter editing. Users can also add and remove the application of constraints by selecting an end-effector keyframe.

ar interaction fig
Editing menus of ARC-LfD to reparameterize constraints applied to the CC-LfD model

Below is a video describing the ARC-LfD system:

Interaction Flow

The use interaction flow for the ARC-LfD:

  1. Recording of demonstration trajectories and constraints
  2. Visualization of the learned model
  3. Constraint editing/application
  4. Relearning of Constrained Keyframe Model via CC-LfD algorithm
  5. Model acceptance and resulting skill execution

The iterative aspect is represented by the looping of steps 2, 3, and 4.

ARC LfD Editing Flowchart
The iterative interaction and model relearning flow diagram of ARC-LfD.


The ARC-LfD architecture consists of two major subsystems:

  1. Augmented Reality Subsystem
  2. CC-LfD Subsystem

These two subsystems communicate via the pub/sub architecture of the Robot Operating System (ROS). Any time the user updates constraints on the mode, the AR Subsystem sends those updates to the CC-LfD subsystem which produces a new model, and sends a representation back to the AR Subsystem to update the visualizations.

ARCLfD Architecture
The iterative interaction and model relearning flow diagram of ARC-LfD.

Future Work

Future work will consist of running human-subjects studies to test the efficacy of this system both objectively: the skill success percentage compared to traditional constraint application, and subjectively: do users display a preference using this interface?


About the Cover

Sawyer hoping to avoid spilling coffee...

© 2021 Carl Mueller

This is a static website built with React and Gatsby JS. It is hosted on GitHub pages.