Implementation and analysis of human-in-the-loop RL algorithms with a wide range of modes of interaction with the human. The modalities include learning from demonstration, action advice, positive/negative feedback, binary preferences, and different mixtures of them.