Project Documentation

Install python 3.7 for Taxi's venv and 3.12 for Ollama's venv

• If you're on Mac, first run:

brew install swig.

Install from the wheel located in root with

 python -m pip install rlang-0.2.5-py3-none-any.whl.

Further instructions regarding Rlang can be found here.

To run the interpreatability plots, ensure that the following libraries are installed, specific version of shap, lime, sklearn is needed as provided in requirements.txt:

pip install -r requirements.txt

Project Structure

This project contains two main components:

1. Ollama - Contains three sequential stages (stage1.py, stage2.py, stage3.py)
2. Taxi - Contains reinforcement learning implementations (dyna_q.py, q_learning.py, r_max.py)

.
├── .venv                  # Main virtual environment
│   ├── bin
│   ├── include
│   ├── lib
│   └── share
├── .gitignore
├── pip-selfcheck.json
├── pyvenv.cfg
├── ollama                 # Ollama component
│   ├── .ollama_venv       # Ollama-specific virtual environment
│   ├── stage1.py
│   ├── stage2.py
│   └── stage3.py
└── taxi                   # Taxi component
    ├── __pycache__
    ├── dyna_q.py
    ├── grounding.py
    ├── q_learning.py
    ├── r_max.py
    ├── taxi.rlang
    ├── vocab.json
    └── rlang-0.2.5-py3-none-any.whl

Virtual Environments

This project uses multiple virtual environments:

1. Main Environment (.venv/) - Root-level virtual environment
2. Ollama Environment (ollama/.ollama_venv/) - Specific to the Ollama component

Running the Ollama Component

To run the Ollama stages, you need to activate the Ollama-specific virtual environment:

# Navigate to the project root
cd path/to/project

# Activate the Ollama virtual environment
source ollama/.ollama_venv/bin/activate  # On Linux/Mac


# Run the stages in sequence
python3 ollama/stage1.py
python3 ollama/stage2.py
python3 ollama/stage3.py

Running the Taxi Component

To run the Taxi reinforcement learning implementations, you need to activate the main virtual environment:

# Navigate to the project root
cd path/to/project

# Activate the main virtual environment
source .venv/bin/activate  # On Linux/Mac


python3 taxi/dyna_q.py
python3 taxi/q_learning.py
python3 taxi/r_max.py

Reinforcement Learning Algorithms

The Taxi component contains three reinforcement learning algorithms:

1. Dyna-Q (dyna_q.py) - A model-based reinforcement learning algorithm that uses a model of the environment to simulate experience. Uses RLang policy with some probability and uses transition functions for initialization.

2. Q-Learning (q_learning.py) - A model-free reinforcement learning algorithm that learns the value of an action in a particular state. Utilizes Rlang reward functions to initialize the Q-table.

3. R-Max (r_max.py) - An algorithm that optimistically assumes maximum reward for unknown state-action pairs. Creates the empirical reward and transition tables using Rlang knowledge.

This project provides various methods for explaining the behavior of a Q-learning agent in the Taxi-v3 environment. The code integrates multiple explainability techniques such as LIME, SHAP, Decision Tree, and other visualizations to analyze and understand the decisions made by the agent during training.

Interpretability Features

The following visualizations and analyses are generated by this project:

1. Policy Visualization

• Displays the learned policy for the Taxi-v3 environment, which shows the best action (such as South, North, Pickup, etc.) to take for each state.
• The grid visualizes the learned policy at each state (position).

2. Decision Tree Visualization

• Visualizes a Decision Tree classifier trained to predict actions based on state features (taxi_row, taxi_col, passenger_loc, destination).
• The decision tree is generated using the sklearn Decision Tree classifier and can be exported into a visual format using graphviz.

3. Feature Importance

• A bar plot showing the importance of each feature (taxi_row, taxi_col, passenger_loc, destination) in the Decision Tree model.
• This helps to understand which features are most influential in determining actions.

4. SHAP Summary Plot

• Displays SHAP (SHapley Additive exPlanations) values for a Logistic Regression model, which helps explain the impact of each feature on the action taken.
• A SHAP summary plot is generated to visualize the contribution of each feature to the predicted outcome.

5. Probability Heatmaps

• Visualizes the predicted probabilities for each state-action pair using the learned Q-table and Logistic Regression model.
• The heatmap shows the likelihood of taking each action in every state.

6. Action Trajectory

• Plots the sequence of actions chosen by the Q-learning agent during a single episode.
• This shows how the agent's behavior evolves over time.

7. State Visit Heatmap

• Visualizes a heatmap of which states are visited most frequently during a single episode.
• This is useful for understanding how often the agent visits certain states.

8. Reward Trajectory

• Plots the rewards received by the agent at each time step during a single episode.
• This helps to analyze the reward structure of the environment and how well the agent is performing.

9. LIME Explanations

• Provides LIME (Local Interpretable Model-agnostic Explanations) explanations for specific states.
• LIME generates local explanations for why the model predicts a specific action for a given state, allowing for better interpretability.

Additional Files

• taxi.rlang - Likely a domain-specific language file for defining the taxi environment
• vocab.json - Vocabulary configuration
• grounding.py - Possibly handles grounding of symbols or concepts in the environment
• rlang-0.2.5-py3-none-any.whl - A wheel file for the rlang package (version 0.2.5)