Split README into per-workflow docs/ pages

docs/cross-workflow.md

@@ -0,0 +1,73 @@
+# Cross-workflow tasks
+
+Tasks that pull signals from multiple workflows and synthesize them into a
+single artifact for the instructor.
+
+| Task | Summary |
+|---|---|
+| [`prep-class-digest`](#prep-class-digest) | Synthesize a 1-page Markdown brief on cohort gaps from quiz misses, follow-up reply themes, and media-recording transcripts; ends with 2–3 suggested in-class discussion questions. |
+
+Per-task `--help` is also available from the CLI:
+`python canvigator.py <task> --help`.
+
+← [Back to main README](../README.md)
+
+---
+
+## `prep-class-digest`
+
+*Synthesize a 1-page Markdown brief on cohort gaps and 2–3 in-class discussion questions.*
+
+Synthesizes a 1-page brief from three signal sources collected over a
+configurable lookback window (default 7 days, override with `--days N`):
+
+- **Recent quiz misses** — joined to the topic tags from
+  `*_questions_w_tags_*.csv` so misses are aggregated by concept rather than
+  by individual question.
+- **Follow-up reply themes** — failing/borderline rows in
+  `*_followup_assessments.csv` (within the window) are summarized by a local
+  Gemma 4 call into "where students went wrong" bullets, one block per
+  quiz/question.
+- **Media-recording transcripts** — every
+  `assignment<id>_recordings_*.csv` in the window is run through the existing
+  tag-classification + theme-extraction Gemma 4 pipeline (the same one used
+  by [`analyze-media-recordings`](media-recording-checkins.md#analyze-media-recordings)).
+
+The brief ends with **2–3 suggested in-class discussion questions** targeted
+at the highest-priority gaps. By default these are drafted by the local
+Gemma 4 model with a full-fidelity prompt that includes the derived theme
+bullets and evidence snippets. With `--cloud-questions` (`-q`) the step
+routes to cloud Gemini 3 instead, but only with a *redacted* prompt that
+contains tag names + integer miss counts + integer theme-cluster counts —
+no transcripts, no `criteria_evaluations`, no theme text. Student-derived
+content stays local in either configuration.
+
+Empty windows (no quiz/follow-up/recording activity in the lookback
+period) short-circuit with a friendly message and write no file. Same-day
+re-runs overwrite.
+
+**Before running:** `data/<course>/` should already be populated by upstream
+tasks: [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events)
+and [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions)
+for misses, [`assess-replies`](quiz-followup-questions.md#assess-replies) for
+follow-up themes, [`get-media-recordings`](media-recording-checkins.md#get-media-recordings)
+for transcripts.
+
+| | Files |
+|---|---|
+| **Input** | `data/<course>/<quiz>_<id>_all_subs_by_question_YYYYMMDD.csv` files in window (from [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events)) |
+| | `data/<course>/<quiz>_<id>_questions_w_tags_YYYYMMDD.csv` per quiz (from [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions)) |
+| | `data/<course>/<quiz>_<id>_followup_assessments.csv` (from [`assess-replies`](quiz-followup-questions.md#assess-replies), persistent) |
+| | `data/<course>/assignment<id>_recordings_YYYYMMDD.csv` files in window (from [`get-media-recordings`](media-recording-checkins.md#get-media-recordings)) |
+| **Output** | `data/<course>/class_digest_YYYYMMDD.md` — Markdown report (5 sections: header, quiz performance table, follow-up reply themes, media-recording themes, suggested discussion questions) |
+
+### Examples
+
+```bash
+python canvigator.py prep-class-digest                  # default 7-day window, Gemma-only
+python canvigator.py --days 14 prep-class-digest        # widen the window
+python canvigator.py --cloud-questions prep-class-digest  # opt the question step into Gemini
+```
+
+Open the generated `class_digest_*.md`, scan the priority gaps, and use the
+suggested questions to seed a 5–10 minute in-class discussion.

docs/installation.md

@@ -0,0 +1,83 @@
+# Installation and setup
+
+← [Back to main README](../README.md)
+
+Note that installation and usage requires some basic knowledge on how
+to use the command line. If necessary, there are many brief
+tutorials/lessons available to help in this area, e.g.,
+[freecodecamp.org](https://www.freecodecamp.org/news/command-line-for-beginners/).
+
+1. **Clone the repository**:
+    [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this repository to your local system. This can either be done 
+in just one place on your local system, or separately for each course that it will be used in. The latter option is our preferred method since we tend to have a separate directory for each course we teach, and prefer to keep the course data separate.
+
+2. **Generate a Canvas API Token**:
+    Naviagte to the `canvigator` directory then run configuration setup script by simply typing, `./configure.sh` at the terminal prompt.
+   ```bash
+   cd canvigator
+   ```
+   ```bash
+   ./configure.sh
+   ```
+   You will be prompted to enter:
+   - Your institution's Canvas LMS base URL (you can find this on your Canvas home page).
+   - Your Canvas token, which can be created by navigating to _'Account'_, then _'Settings'_. Towards the bottom of this window in Canvas you will see a blue button, _'+ New Access Token_'. Click on this button to copy/download the token to your local system. DO NOT TO SHARE YOUR CANVAS TOKEN w/ ANYONE (e.g. do not save it in a shared directory).
+   - Your Ollama API key (optional — only needed if you plan to use the cloud-hosted text model for the LLM-powered tasks below). Leave it blank if you plan to use only local Ollama models (or none at all). See the [Ollama setup](#ollama-setup-optional) section for how to generate one.
+
+This will prompt the creation of the _data/_ and _figures/_ subdirectories.
+
+3. **Verify Setup**:
+    Once this is complete, double check that the configuration script, _set_env.sh_ has been created, that it has the correct values for your URL and token, and that the subdirectories have been created. 
+    ```bash
+    set_env.sh
+    ```
+    ```bash
+    env
+    ```
+    This command will list all environment variables, allowing you to confirm that the necessary variables have been set correctly.
+
+4. **Verify Python Libraries**:
+    Before running the project, ensure that all required Python libraries are installed:
+    1. Install the required libraries using `pip`:
+
+    ```bash
+    pip install -r requirements.txt 
+    ```
+
+    2. Alternatively, you can install each library individually if needed.
+
+        * ![canvasapi](https://img.shields.io/badge/canvasapi-3.3.0-blue)
+        * ![matplotlib](https://img.shields.io/badge/matplotlib-3.8.4-brightgreen)
+        * ![numpy](https://img.shields.io/badge/numpy-1.26.4-yellow)
+        * ![pandas](https://img.shields.io/badge/pandas-2.2.2-orange)
+        * ![requests](https://img.shields.io/badge/requests-2.33.0-red)
+        * ![scipy](https://img.shields.io/badge/scipy-1.13.1-lightgrey)
+        * ![seaborn](https://img.shields.io/badge/seaborn-0.13.2-blueviolet)
+        * ![ollama](https://img.shields.io/badge/ollama-0.4.7-black)
+
+    If no errors are thrown, the libraries are successfully installed. The `ollama` client is only used by the LLM-assisted tasks described below — see [Ollama setup](#ollama-setup-optional).
+
+
+## Ollama setup (optional)
+
+Several tasks use a Large Language Model (LLM) via [Ollama](https://ollama.com) to draft and tag questions, generate open-ended follow-ups, transcribe student audio, and assess student replies. You can skip this section if you will not be running any of these tasks (`create-quiz` with the `[g]enerate w/ LLM` option, `get-quiz-questions --tag`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`).
+
+Canvigator uses two kinds of models:
+
+1. **A cloud-hosted text model** (default `gemini-3-flash-preview`, set via `OLLAMA_TEXT_MODEL`) for instructor-side text generation — drafting quiz questions in `create-quiz`, tagging quiz questions, and generating open-ended follow-up questions. These tasks never see student data, so a larger cloud model is a good fit.
+2. **Local models** for tasks that process student input — `gemma4:31b` (default `OLLAMA_MODEL`) for assessing text/image replies, and `gemma4:e4b` (default `OLLAMA_AUDIO_MODEL`) for transcribing student audio. Keeping these local is deliberate: student submissions should not leave your machine.
+
+**To use the cloud text model:**
+1. Sign in at [ollama.com](https://ollama.com) and create an API key from your account settings.
+2. Paste the key when prompted by `./configure.sh` (or add `export OLLAMA_API_KEY="..."` directly to `set_env.sh`).
+
+**To use the local models:**
+1. Install Ollama from [ollama.com/download](https://ollama.com/download) and start it (`ollama serve`, or use the Ollama desktop app).
+2. Pull the models you need:
+   ```bash
+   ollama pull gemma4:31b   # assessment of text/image replies (assess-replies)
+   ollama pull gemma4:e4b   # transcription of student audio (assess-replies, explain mode)
+   ```
+   Only pull the models for tasks you actually plan to run. `gemma4:31b` is a large download (~20 GB) and is only required for `assess-replies`.
+
+All model names are overridable via env vars (`OLLAMA_TEXT_MODEL`, `OLLAMA_MODEL`, `OLLAMA_AUDIO_MODEL`), and the local host can be overridden via the standard `OLLAMA_HOST` env var.

docs/media-recording-checkins.md

@@ -0,0 +1,119 @@
+# Media-recording check-ins
+
+Three tasks form an end-to-end loop for short audio check-ins where each
+student records a brief reflection (e.g. *"What was hardest on this week's
+quiz?"*), the recording is downloaded and transcribed locally, the instructor
+grades each submission, and the cohort's transcripts are then summarized
+against the quiz topics.
+
+| Task | Summary |
+|---|---|
+| [`create-media-recording-assignment`](#create-media-recording-assignment) | Create a Canvas assignment that accepts only media_recording submissions. |
+| [`get-media-recordings`](#get-media-recordings) | Fetch + transcode + transcribe each submission and either prompt for a grade or auto-award `points_possible`. |
+| [`analyze-media-recordings`](#analyze-media-recordings) | Two local-LLM passes against the transcripts: tag classification + theme extraction. |
+
+Per-task `--help` is also available from the CLI:
+`python canvigator.py <task> --help`.
+
+← [Back to main README](../README.md)
+
+## End-to-end workflow
+
+1. **`create-media-recording-assignment`** — create the assignment on Canvas.
+2. **`get-media-recordings`** — fetch and transcribe each submission, and grade.
+3. **`analyze-media-recordings`** — surface tag coverage and recurring themes
+   across the cohort.
+
+Audio handling is local-only: transcripts are produced via the local
+`OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) and the cohort analysis runs
+against the local `OLLAMA_MODEL` (default `gemma4:31b`). Student audio never
+leaves your machine.
+
+---
+
+## `create-media-recording-assignment`
+
+*Create a Canvas assignment that accepts only media_recording submissions.*
+
+Interactively creates a Canvas assignment with `submission_types=['media_recording']`.
+Prompts for the title, prompt body (becomes the HTML `description`),
+`points_possible` (default 1), an optional ISO `due_at`, and whether to publish
+immediately (default unpublished so you can review before exposing it to
+students). Logs the new assignment's ID and Canvas URL on success.
+
+| | Files |
+|---|---|
+| **Input** | _(none — interactive prompts only)_ |
+| **Output** | _(none — assignment is created directly on Canvas)_ |
+
+---
+
+## `get-media-recordings`
+
+*Fetch + transcode + transcribe each submission and either prompt for a grade or auto-award `points_possible`.*
+
+Pulls every media-recording submission for a chosen assignment, re-encodes the
+audio to 16 kHz mono PCM WAV (via ffmpeg directly on Canvas's playback URL —
+DASH manifests are handled in one pass), and transcribes each WAV locally
+using `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`). Audio files are saved to
+`data/<course>/media_recordings/assignment<id>/<student_id>_<submission_id>.wav`.
+
+By default, after each transcription the transcript is displayed and the
+instructor is prompted for the points to award:
+
+- **Enter** — award `points_possible` (full credit).
+- **A number** — award that value.
+- **`s` / `skip`** — leave the submission ungraded.
+
+The chosen value is posted via Canvas's `Submission.edit({'posted_grade': ...})`.
+
+Pass `--auto-grade` to skip the per-student review and award `points_possible`
+to every submitter automatically. Combine with `--dry-run` to preview the
+auto-graded writes without mutating Canvas; in dry-run, transcripts are still
+shown and the CSV is still written, but the grade write is suppressed.
+
+Pressing Ctrl-C mid-loop flushes a partial CSV before re-raising, so review
+progress is never lost.
+
+| | Files |
+|---|---|
+| **Input** | _(Canvas — selected interactively)_ |
+| **Output** | `data/<course>/assignment<id>_recordings_YYYYMMDD.csv` — columns: `student_id`, `student_name`, `submission_id`, `submitted_at`, `audio_path`, `transcript`, `transcribed_at`, `grade`, `graded_at` |
+| | `data/<course>/media_recordings/assignment<id>/<student_id>_<submission_id>.wav` — one WAV per submission |
+| **Canvas side-effect** | Posts `posted_grade` on each graded submission (skipped in `--dry-run` mode) |
+
+---
+
+## `analyze-media-recordings`
+
+*Two local-LLM passes against the transcripts: tag classification + theme extraction.*
+
+Loads the most recent `assignment<id>_recordings_*.csv` produced by
+`get-media-recordings`, prompts the instructor to pick a `*_questions_w_tags_*.csv`
+from the same course directory (so tag vocabulary stays grounded in actual
+quiz content), then runs two local-LLM passes against `OLLAMA_MODEL` (default
+`gemma4:31b`):
+
+1. **Tag classification** — one call per non-empty transcript, asking which of
+   the unique tags from the quiz CSV's `keywords` column the student is
+   actually discussing. Paraphrase-tolerant, so *"I got confused on the linked
+   list one"* maps to `linked lists` even when the literal substring isn't
+   present.
+2. **Theme extraction** — a single cohort-level call that surfaces 3–5
+   recurring themes across all transcripts, with the tag list as background
+   context.
+
+The output is a Markdown report with three sections: a tag-grounded table
+ranked by descending student count (the *"specialized word cloud"*), the
+LLM-extracted themes, and a roster mapping the 1-based student indices used
+in the themes section back to names.
+
+**Before running:** `get-media-recordings` must have produced the recordings
+CSV, and [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions)
+must have produced a tags CSV in the same course directory.
+
+| | Files |
+|---|---|
+| **Input** | `data/<course>/assignment<id>_recordings_YYYYMMDD.csv` (from `get-media-recordings`) |
+| | `data/<course>/<quiz>_<id>_questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) |
+| **Output** | `data/<course>/assignment<id>_analysis_YYYYMMDD.md` — Markdown report |