diff --git a/README.md b/README.md index ab96bed..487d4e0 100644 --- a/README.md +++ b/README.md @@ -23,87 +23,11 @@ plans for this project include extending it to other LMSs and creating a richer for other functionality/features (see Issues), as well as any feedback on the project, are welcomed. -### Installation +### Setup -Note that installation and usage requires some basic knowledge on how -to use the command line. If necessary, there are many brief -tutorials/lessons available to help in this area, e.g., -[freecodecamp.org](https://www.freecodecamp.org/news/command-line-for-beginners/). - -1. **Clone the repository**: - [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this repository to your local system. This can either be done -in just one place on your local system, or separately for each course that it will be used in. The latter option is our preferred method since we tend to have a separate directory for each course we teach, and prefer to keep the course data separate. - -2. **Generate a Canvas API Token**: - Naviagte to the `canvigator` directory then run configuration setup script by simply typing, `./configure.sh` at the terminal prompt. - ```bash - cd canvigator - ``` - ```bash - ./configure.sh - ``` - You will be prompted to enter: - - Your institution's Canvas LMS base URL (you can find this on your Canvas home page). - - Your Canvas token, which can be created by navigating to _'Account'_, then _'Settings'_. Towards the bottom of this window in Canvas you will see a blue button, _'+ New Access Token_'. Click on this button to copy/download the token to your local system. DO NOT TO SHARE YOUR CANVAS TOKEN w/ ANYONE (e.g. do not save it in a shared directory). - - Your Ollama API key (optional — only needed if you plan to use the cloud-hosted text model for the LLM-powered tasks below). Leave it blank if you plan to use only local Ollama models (or none at all). See the [Ollama setup](#ollama-setup-optional) section for how to generate one. - -This will prompt the creation of the _data/_ and _figures/_ subdirectories. - -3. **Verify Setup**: - Once this is complete, double check that the configuration script, _set_env.sh_ has been created, that it has the correct values for your URL and token, and that the subdirectories have been created. - ```bash - set_env.sh - ``` - ```bash - env - ``` - This command will list all environment variables, allowing you to confirm that the necessary variables have been set correctly. - -4. **Verify Python Libraries**: - Before running the project, ensure that all required Python libraries are installed: - 1. Install the required libraries using `pip`: - - ```bash - pip install -r requirements.txt - ``` - - 2. Alternatively, you can install each library individually if needed. - - * ![canvasapi](https://img.shields.io/badge/canvasapi-3.3.0-blue) - * ![matplotlib](https://img.shields.io/badge/matplotlib-3.8.4-brightgreen) - * ![numpy](https://img.shields.io/badge/numpy-1.26.4-yellow) - * ![pandas](https://img.shields.io/badge/pandas-2.2.2-orange) - * ![requests](https://img.shields.io/badge/requests-2.33.0-red) - * ![scipy](https://img.shields.io/badge/scipy-1.13.1-lightgrey) - * ![seaborn](https://img.shields.io/badge/seaborn-0.13.2-blueviolet) - * ![ollama](https://img.shields.io/badge/ollama-0.4.7-black) - - If no errors are thrown, the libraries are successfully installed. The `ollama` client is only used by the LLM-assisted tasks described below — see [Ollama setup](#ollama-setup-optional). - - -### Ollama setup (optional) - -Several tasks use a Large Language Model (LLM) via [Ollama](https://ollama.com) to draft and tag questions, generate open-ended follow-ups, transcribe student audio, and assess student replies. You can skip this section if you will not be running any of these tasks (`create-quiz` with the `[g]enerate w/ LLM` option, `get-quiz-questions --tag`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`). - -Canvigator uses two kinds of models: - -1. **A cloud-hosted text model** (default `gemini-3-flash-preview`, set via `OLLAMA_TEXT_MODEL`) for instructor-side text generation — drafting quiz questions in `create-quiz`, tagging quiz questions, and generating open-ended follow-up questions. These tasks never see student data, so a larger cloud model is a good fit. -2. **Local models** for tasks that process student input — `gemma4:31b` (default `OLLAMA_MODEL`) for assessing text/image replies, and `gemma4:e4b` (default `OLLAMA_AUDIO_MODEL`) for transcribing student audio. Keeping these local is deliberate: student submissions should not leave your machine. - -**To use the cloud text model:** -1. Sign in at [ollama.com](https://ollama.com) and create an API key from your account settings. -2. Paste the key when prompted by `./configure.sh` (or add `export OLLAMA_API_KEY="..."` directly to `set_env.sh`). - -**To use the local models:** -1. Install Ollama from [ollama.com/download](https://ollama.com/download) and start it (`ollama serve`, or use the Ollama desktop app). -2. Pull the models you need: - ```bash - ollama pull gemma4:31b # assessment of text/image replies (assess-replies) - ollama pull gemma4:e4b # transcription of student audio (assess-replies, explain mode) - ``` - Only pull the models for tasks you actually plan to run. `gemma4:31b` is a large download (~20 GB) and is only required for `assess-replies`. - -All model names are overridable via env vars (`OLLAMA_TEXT_MODEL`, `OLLAMA_MODEL`, `OLLAMA_AUDIO_MODEL`), and the local host can be overridden via the standard `OLLAMA_HOST` env var. +First-time setup (Canvas API token, Python dependencies, and the optional +Ollama configuration for LLM-assisted tasks) is documented in +**[docs/installation.md](docs/installation.md)**. ### Usage @@ -133,761 +57,28 @@ The `--crn` option selects a course by its CRN (the last 5 digits of the Canvas course code), bypassing the interactive course selection prompt. This is useful for automated/scheduled runs, e.g. `python canvigator.py --crn 12345 get-activity`. -Available tasks: `analyze-media-recordings`, `assess-replies`, `award-bonus`, `award-bonus-partner-only`, `award-bonus-retake-only`, `create-media-recording-assignment`, `create-pairs`, `create-quiz`, `delete-old-conversations`, `export-anon-data`, `generate-follow-up-questions`, `get-activity`, `get-conversations`, `get-gradebook`, `get-media-recordings`, `get-quiz-questions`, `get-quiz-submission-events`, `get-roster`, `prep-class-digest`, `send-follow-up-assessments`, `send-follow-up-question`, `send-quiz-reminder` - -All tasks begin by prompting you to select a course. Output files are written to -`data//` and `figures//`, where `` is derived from the -Canvas course code. In the file names below, `` refers to the quiz title +All tasks begin by prompting you to select a course (unless `--crn` is +supplied). Output files are written to `data//` and +`figures//`, where `` is derived from the Canvas course code. +In the file names referenced from the docs below, `` is the quiz title (lowercased, spaces replaced with underscores), `` is the Canvas quiz ID, -and `YYYYMMDD` is the current date. Figure files now follow the same pattern as -the CSV exports, with the date at the end of the filename. Event exports and -per-question histograms are only generated by `get-quiz-submission-events`. - -#### LLM-assisted follow-up workflow - -Several of the newer tasks are designed to chain together into a single -end-to-end flow. Run them in this order for a given quiz: - -1. `python canvigator.py --tag get-quiz-questions` — export the quiz's question content and add LLM topic tags (cloud model, requires `OLLAMA_API_KEY`). -2. `python canvigator.py generate-follow-up-questions` — generate 3 candidate open-ended follow-up questions per original question, with an assessment guide for each. **Review the output CSV and set `selected_question=1` on one row per question group before moving on.** -3. _(optional)_ `python canvigator.py send-quiz-reminder` — nudge students who haven't attempted the quiz or who scored below perfect. Imperfect-score students get a bulleted list of the topics (from the tags) they missed. -4. `python canvigator.py send-follow-up-question` — send the instructor-selected open-ended question (the first row in the CSV with `selected_question=1`) to each student who missed the corresponding original quiz question. Students reply via Canvas conversations with an audio recording ("explain") or a photo ("draw"). -5. `python canvigator.py assess-replies` — pull the students' replies (and attached audio/images) back from Canvas, then run them through the local `gemma4` models (transcription + assessment) to produce pass/fail + feedback. Results are merged into a single persistent `*_followup_assessments.csv` (no date suffix); rows where `sent_assessment=1` are preserved verbatim across re-runs. -6. **Edit the `feedback` column** in `*_followup_assessments.csv` for any student where the LLM's assessment needs correction. -7. `python canvigator.py send-follow-up-assessments` — post the (curated) `feedback` text back to each student as a reply on the existing follow-up conversation thread; sets `sent_assessment=1`. - -Steps 4–7 can be repeated as students keep replying: `assess-replies` picks up new messages and reassesses against the latest reply per student (skipping rows already sent), and `send-follow-up-assessments` only sends rows that haven't been sent yet. - ---- - -#### `assess-replies` — Fetch and assess student follow-up replies with a local LLM - -First fetches the latest student replies from Canvas (loading the manifest -written by `send-follow-up-question`, finding matching sent conversations by -subject, downloading image attachments and audio recordings to -`data//replies/`, and writing a dated `*_followup_replies_*.csv`). -Only replies received within the configured window after the follow-up was -sent are accepted (default 5 days; override with `--reply-window-days N`). - -Then loads the latest reply per student from that CSV and uses local LLMs via -Ollama to evaluate each one against the original question. Two model pipelines -are used depending on `question_mode`: - -- **`explain` mode**: `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) transcribes - the student's audio recording, then `OLLAMA_MODEL` (default `gemma4:31b`) - assesses the transcript against the question context. -- **`draw` mode**: `OLLAMA_MODEL` directly assesses the student's submitted - image against the question context. - -Each assessment yields a pass/fail result plus 2–3 sentences of feedback. -Re-running this task picks up any new student replies and re-assesses against -the latest response per student. - -**Prerequisite**: run `send-follow-up-question` first so the -`*_followup_sent_*.csv` manifest is on disk. Requires a running Ollama server -with both models pulled. - -| | Files | -|---|---| -| **Input** | `data//__followup_sent_YYYYMMDD.csv` (from `send-follow-up-question`) | -| | `data//__open_ended_YYYYMMDD.csv` (provides the rubric, assessment guide, and exemplars used by the grader) | -| **Output** | `data//__followup_replies_YYYYMMDD.csv` | -| | `data//replies/` — downloaded image attachments and audio recordings | -| | `data//__followup_assessments.csv` (persistent — merged across runs) | - -The replies CSV contains columns: `student_id`, `student_name`, `question_id`, -`question_mode`, `conversation_id`, `message_id`, `reply_text`, `has_attachment`, -`attachment_path`, `has_audio`, `audio_path`, `replied_at`, `latest`. The -`latest` flag marks the most recent reply per student (used for assessment). - -The assessments CSV contains columns: `student_id`, `student_name`, `question_id`, -`question_mode`, `conversation_id`, `result` (`pass` or `fail`), `confidence` -(`high` when all self-consistency runs agreed, `borderline` when they split), -`feedback`, `transcript` (for `explain` mode), `criteria_evaluations` (a JSON -blob with the per-criterion `met`/`partial`/`missing` ratings the grader produced), -`assessed_at`, `sent_assessment` (`0` until `send-follow-up-assessments` posts -the row, then `1`), `sent_at`. - -Each reply is graded with N=3 self-consistency passes and a majority vote; -`confidence=borderline` flags the rows worth instructor review before sending. -The grader is also primed with the structured rubric and exemplars from -`*_open_ended_*.csv`, plus up to 3 prior pass + 3 prior fail -`sent_assessment=1` responses for the same question (sampled from the -assessments file itself) as in-context calibration — so once you correct a few -borderline cases and send them, future re-runs anchor to your judgment. - -Re-running this task is safe: rows with `sent_assessment=1` are skipped and -preserved verbatim, so previously sent feedback is never overwritten. Rows -with `sent_assessment=0` are re-assessed in place. On the first run, an existing -dated `*_followup_assessments_YYYYMMDD.csv` is migrated forward into the -non-dated file. - -#### `send-follow-up-assessments` — Send instructor-curated feedback back to students - -Reads `*_followup_assessments.csv` and, for every row with `sent_assessment=0` -and a non-empty `feedback` value, posts the feedback text as a reply on the -existing follow-up conversation thread (`Conversation.add_message`). On -success, sets `sent_assessment=1` and stamps `sent_at`, then rewrites the file -in place. The `--dry-run` flag previews what would be sent without touching -Canvas (and leaves `sent_assessment=0`). - -The expected workflow is to **edit the `feedback` column** between -`assess-replies` and `send-follow-up-assessments` for any rows where the LLM -assessment needs correction; the curated text is what the student sees. - -**Prerequisite**: run `assess-replies` first. - -| | Files | -|---|---| -| **Input** | `data//__followup_assessments.csv` (from `assess-replies`, edited by instructor) | -| | `data//__followup_sent_*.csv` (fallback for `conversation_id` lookup) | -| **Output** | The same `*_followup_assessments.csv`, with `sent_assessment` and `sent_at` updated | - ---- - -#### `award-bonus` — Award both partner and retake bonus points - -Detects partner groups and qualifying retakers, then awards both bonus types -as combined fudge points on their Canvas submissions. Each bonus defaults to -15% of quiz points; a student earning both receives 30% total. - -**Partner bonus:** Students whose first attempts show a high fraction of -matching scores (default ≥ 80%) and closely timed answers (default within -5 seconds on ≥ 80% of questions) are grouped as partners. Groups larger than -3 are flagged for manual review. - -**Retake bonus:** Students who retook the quiz at least 3 times with at least -1 day (24 hours) between qualifying attempts. - -**Before running:** the `get-quiz-submission-events` task must have been run first for the same -quiz so that the submission CSVs exist. - -| | Files | -|---|---| -| **Input** | `data//__all_subs_and_events_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| | `data//__all_subs_by_question_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| **Output** | `data//__detected_partners_YYYYMMDD.csv` — detected partner groups | -| | `data//__retake_qualified_YYYYMMDD.csv` — students qualifying for retake bonus | -| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with partner_bonus, retake_bonus, and combined bonus columns | -| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment describing the bonus breakdown (skipped in `--dry-run` mode) | - -Use `--dry-run` to preview which students would receive bonus points without -modifying anything in Canvas. In dry-run mode the scores CSV is named -`*_scores_w_bonus_dryrun_*.csv`. - -**Typical workflow:** -1. Run `python canvigator.py get-quiz-submission-events` to export submission data for all quizzes. -2. Run `python canvigator.py [--dry-run] award-bonus` and select the course, quiz, and date when prompted. -3. Review the `*_detected_partners_*.csv` and `*_retake_qualified_*.csv` to verify the results. - ---- - -#### `award-bonus-partner-only` — Award only the partner bonus - -Same as `award-bonus` but only detects and awards the partner bonus (no retake -bonus). Useful when you want to reward collaborative work without the retake -incentive. - -| | Files | -|---|---| -| **Input** | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| | `data//__all_subs_by_question_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| | `data//__all_subs_and_events_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| **Output** | `data//__detected_partners_YYYYMMDD.csv` — detected partner groups | -| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with bonus column | -| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment (skipped in `--dry-run` mode) | - ---- - -#### `award-bonus-retake-only` — Award only the retake bonus - -Same as `award-bonus` but only detects and awards the retake bonus (no partner -detection). Useful when you want to incentivize retakes independently. - -| | Files | -|---|---| -| **Input** | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | -| **Output** | `data//__retake_qualified_YYYYMMDD.csv` — students qualifying for retake bonus | -| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with bonus column | -| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment (skipped in `--dry-run` mode) | - ---- - -#### `create-pairs` — Create student pairings from quiz scores - -Uses an independent (pre-class) quiz to pair students for a collaborative -in-class quiz. Computes pairwise Euclidean distance between student -answer-score vectors and applies a greedy pairing algorithm. This task writes -the raw quiz report plus pairing-specific figures, but does not export -submission events or histograms. - -**Before running:** create a `present_*.csv` file in `data//` marking -which students are physically present. See `data/present_example.csv` for the -required format (columns: `name`, `id`, `present` where 1 = present). - -| | Files | -|---|---| -| **Input** | `data//present_*.csv` (user-created, selected via prompt) | -| **Output — data** | `data//__student_analysis_YYYYMMDD.csv` — raw quiz report downloaded from Canvas | -| | `data//pairings_based_on___YYYYMMDD.csv` — student pairings (person3/id3 columns omitted when no triples exist) | -| **Output — figures** | `figures//__dist_euclid_YYYYMMDD.png` — distance matrix heatmap (present students) | -| | `figures//__compare_pairing_methods_YYYYMMDD.png` — comparison of all four pairing methods | - -**Typical workflow:** -1. Mark which students are present in your `present_*.csv` file. -2. Run `python canvigator.py create-pairs` and select the course, quiz, and presence CSV when prompted. -3. Open the generated `pairings_based_on_*.csv` and share pairings with the class. - ---- - -#### `create-quiz` — Create an unpublished quiz on Canvas, with placeholder or LLM-generated questions - -Interactively creates a new unpublished quiz on Canvas. Prompts the user for a -quiz title, then for each question prompts: - -``` -QN — [p]laceholder, [g]enerate w/ LLM, [e]nd quiz: -``` - -- **`p`** (placeholder) — asks for a one-line description and creates a - multiple-choice placeholder with 1 point possible (and no answer choices). -- **`g`** (generate w/ LLM) — asks the user for a natural-language seed prompt - (e.g. *"a question about combinatorics asking how many ways 11 soccer players - can be selected from a team of 23"*) and routes it through the cloud text - model (default `gemini-3-flash-preview`, requires `OLLAMA_API_KEY` — see - [Ollama setup](#ollama-setup-optional)). The LLM returns a complete - Canvas-shaped question of one of seven auto-gradable types: multiple-choice, - multiple-answers, true/false, fill-in-multiple-blanks, multiple-dropdowns, - matching, or calculated. The proposed question is rendered inline (type, - name, text, answer choices with `*` marking the correct one, plus per-type - extras like match pairs or formula variables) and the user is prompted - `[y]/[r]/[s]` — accept, regenerate, or skip. Each `[r]egenerate` accumulates - the rejected draft and passes the full list back to the LLM as anti-context - (with sampling temperature bumped from 0.4 to 0.8) so successive drafts - diverge in scenario / values / question_type instead of recycling the same - angle. -- **`e`** or empty input — finalizes the quiz with the questions added so far. - -`points_possible` is forced to `1` for both placeholder and LLM-generated -questions; the instructor can refine wording, distractors, and points later in -the Canvas UI. The quiz is created with default settings: -`quiz_type='assignment'`, `time_limit=30`, `one_question_at_a_time=True`, -`cant_go_back=True`, `shuffle_answers=True`. - -The cloud text model is lazy-loaded only on the first `g` choice, so a -pure-placeholder run still works without `OLLAMA_API_KEY` set. - -| | Files | -|---|---| -| **Input** | _(none — interactive prompts only)_ | -| **Output** | _(none — quiz is created directly on Canvas, unpublished)_ | - ---- - -#### `export-anon-data` — Export anonymized course data - -Anonymizes all CSV files in a course data directory by replacing student IDs -with hashed anonymous IDs (SHA-256 mod 10^10) and removing identifying columns -(`name`, `sis_id`). This task works entirely with local files and does not -require Canvas API credentials. Course selection is done from existing `data/` -subdirectories (by `--crn` or interactively). - -> **Note:** Anonymization alone does not make data shareable. Do not publish/share -> data without participant consent, IRB approval, and FERPA compliance. - -| | Files | -|---|---| -| **Input** | All CSV files in `data//` | -| **Output** | `data//anon_mapping_YYYYMMDD.csv` — mapping of original IDs to anonymous IDs | -| | `data//anonymized/` — directory containing anonymized copies of all CSVs | -| | `data//anonymized_YYYYMMDD.zip` — zip archive of the anonymized directory | - ---- - -#### `generate-follow-up-questions` — Generate open-ended questions from a tagged quiz - -Reads the tagged questions CSV for a selected quiz and uses a cloud-hosted LLM -(via Ollama's hosted endpoint, default `gemini-3-flash-preview`) to produce -open-ended follow-up candidates. For each original quiz question the task runs -five steps: - -1. **Classify** — The LLM decides whether an oral explanation ("explain") or a - hand-drawn diagram ("draw") would be the better way to assess student - understanding. Inherently visual topics (e.g. data structures, memory - layouts, process flows) are classified as "draw"; verbal topics (e.g. - trade-offs, algorithm logic, definitions) as "explain". -2. **Generate 3 candidates** — Using the classification, the LLM generates 3 - self-contained candidate open-ended questions. "Explain" questions begin with - "Explain..." and target a ~1 minute oral response. "Draw" questions begin - with "Draw a diagram..." or "Draw a figure..." and target a ~2 minute - hand-drawn response. -3. **Assessment guide** — For each candidate, the LLM writes a short - `assessment_guide` describing the key concepts/elements a passing student - response must include. This guide is the human-readable summary shown in - the CSV and used as backup rubric material in `assess-replies`. -4. **Structured rubric** — For each candidate, the LLM also emits a JSON - object in the `rubric_json` column with fields `canonical_answer`, - `model_answer` (an 80–120-word A-grade exemplar response in a real - student's voice), `pass_criteria`, `acceptable_alternatives`, - `common_misconceptions`, `fatal_errors` (plus `required_visual_elements` - for draw questions). This structured rubric is what the local Gemma grader - uses during `assess-replies`; edit the JSON blob if you want to tighten the - criteria. Do not open the CSV in Excel and save — Excel will mangle the - JSON quoting. -5. **Calibration exemplars** — A separate LLM call drafts one passing and one - failing example response (for explain mode, a transcript fragment; for - draw mode, a prose description of the drawing) plus a one-sentence note - explaining what makes each one pass or fail. These land in the - `exemplar_pass`, `exemplar_pass_note`, `exemplar_fail`, - `exemplar_fail_note` columns and are passed to the grader at assessment - time as an "anchor the bar here" reference (separate from the live - instructor-approved examples that `assess-replies` accumulates). - -The output CSV is intended for instructor review. Three rows are written per -original question (one per candidate) with `selected_question=0`; **the -instructor must review the candidates offline and set `selected_question=1` -on exactly one row per question group** before running -`send-follow-up-question`. - -**Prerequisite**: run `get-quiz-questions --tag` for the same quiz first so the -`*_questions_w_tags_*.csv` is on disk. Requires `OLLAMA_API_KEY` to be set — -see [Ollama setup](#ollama-setup-optional). - -| | Files | -|---|---| -| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | -| **Output** | `data//__open_ended_YYYYMMDD.csv` | - -The output CSV contains columns: `selected_question`, `question_id`, -`position`, `question_name`, `keywords`, `question_mode` (`explain` or -`draw`), `open_ended_question`, `assessment_guide`, `rubric_json`, -`exemplar_pass`, `exemplar_pass_note`, `exemplar_fail`, `exemplar_fail_note`, -`original_question_text`. - -**Typical workflow:** -1. Run `python canvigator.py --tag get-quiz-questions` and select the quiz. -2. Run `python canvigator.py generate-follow-up-questions` and select the tagged questions CSV. -3. Open the `*_open_ended_*.csv`, review the 3 candidates per question, and set `selected_question=1` on the single row you want to use per question group (edit the question text, `question_mode`, or `assessment_guide` as desired). - ---- - -#### `get-activity` — Export student activity - -Fetches enrollment activity data and course-level summary data from Canvas, -merges them, and saves a single CSV. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output** | `data//course_activity_YYYYMMDD.csv` | - -The output CSV contains columns: `name`, `id`, `page_views`, `page_views_level`, -`participations`, `participations_level`, `missing`, `late`, `on_time`, -`total_activity_mins`, `first_activity_at`, `last_activity_at`, -`active_days_total`, `active_days_last_14`, `views_last_7d`, -`messages_to_instructor`. - -The `*_level` columns are Canvas-computed 0–3 buckets summarizing the raw counts. -`first_activity_at` / `last_activity_at` come from per-student page-view buckets -(course-scoped analytics) — more accurate than the enrollment-level field, which -Canvas often leaves stale. `messages_to_instructor` counts student-initiated -messages in the course conversation thread. - -If the Canvas course has a `start_at` date set, additional columns -`views_wk_01`, `views_wk_02`, … are appended (one per 7-day window from the -course start, capped at 16 weeks). Useful for spotting an engagement drop -mid-semester. - -If the Canvas token has admin-level permission to read user profiles / view -usage reports, four further columns are appended: `top_browser`, `top_os`, -`used_mobile_app`, `n_distinct_user_agents` — derived from per-user page-view -records for this course. Most instructor tokens lack this permission, in which -case these columns are simply omitted from the CSV. - ---- - -#### `get-quiz-submission-events` — Export quiz submissions and events - -Downloads detailed submission history and events for a user-selected quiz, -and renders per-question score histograms. Pass `--all` to iterate over every -published quiz in the course instead of prompting for one. This is the only -task that exports event history and per-question histogram figures. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output (per quiz)** | `data//__student_analysis_YYYYMMDD.csv` — raw quiz report | -| | `data//__all_submissions_YYYYMMDD.csv` — all submission attempts with scores | -| | `data//__all_subs_by_question_YYYYMMDD.csv` — per-question results for each attempt | -| | `data//__all_subs_and_events_YYYYMMDD.csv` — timestamped submission events | -| **Output — figures** | `figures//__score_histograms_YYYYMMDD.png` — per-question score histograms | -| | `figures//__timing_first_attempt_YYYYMMDD.png` — per-question first-attempt timing histograms (minutes) | -| | `figures//__blurs_first_attempt_YYYYMMDD.png` — per-question first-attempt page-blur counts | - ---- - -#### `get-gradebook` — Export course gradebook - -Fetches all published assignments and their submissions from Canvas, joins with -enrolled student data, and exports a comprehensive gradebook CSV. One row per -student per assignment. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output** | `data//gradebook_YYYYMMDD.csv` | - -The output CSV contains columns: `name`, `sortable_name`, `user_id`, -`assignment_name`, `assignment_id`, `points_possible`, `grade`, `score`. - ---- - -#### `get-roster` — Export the full course roster - -Iterates `canvas_course.get_enrollments()` with no role filter and writes a -minimal CSV of every enrolled person — students, teachers, TAs, designers, and -observers. Useful as a course-wide identity reference and as a stable input -for scripts outside Canvigator. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output** | `data//roster_YYYYMMDD.csv` | - -The output CSV contains columns: `name`, `id`, `sis_id`, `enrollment_type`, -`state`. `enrollment_type` is the Canvas type string (`StudentEnrollment`, -`TeacherEnrollment`, `TaEnrollment`, `DesignerEnrollment`, `ObserverEnrollment`); -`state` is the enrollment state (`active`, `invited`, `inactive`, `completed`, -`rejected`, `deleted`). - ---- - -#### `get-conversations` — Export Canvas conversations involving course students - -Canvas conversations are not course-scoped, so this pulls the instructor's -full inbox and sent folders via `canvas.get_conversations()` (default scope -plus `scope='sent'`, deduped by `conversation_id`) and filters each -conversation's participants list against the active-student roster for the -selected course. Primary use case: back-fill the `conversation_id` for -follow-up sends made before the per-send manifest was added, so `assess-replies` -can still fetch those threads by ID. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output** | `data//conversations_YYYYMMDD.csv` | - -The output CSV is sorted newest first and contains columns: `conversation_id`, -`subject`, `last_message_at`, `first_message_at`, `message_count`, -`workflow_state`, `student_ids` (comma-separated), `student_names` -(semicolon-separated), `n_student_participants`, `last_message`. -`first_message_at` is derived by fetching each matched conversation (one extra -API call per conversation) and taking the earliest message `created_at`; it -is also substituted into `last_message_at` when the list endpoint returns an -empty value (occasionally happens for older/archived threads), so sort order -stays meaningful. - ---- - -#### `delete-old-conversations` — Delete inbox conversations older than N months - -Sweeps the instructor's Canvas inbox, sent folder, and archived folder for -conversations whose most recent message is older than `N` months -(approximate; one month = 30 days, default `N = 6`). Each match is shown -(last_message_at, subject, participant names) before any action is taken. -In live mode the task requires an explicit `yes` confirmation at the prompt -before any deletes run; `--dry-run` previews only. - -Canvas conversations are not course-scoped, so this task skips course -selection entirely and operates on the full account inbox. Deletion uses -`Conversation.delete()` to remove the whole thread (not just individual -messages). - -Override the cutoff with `--months`/`-m`, e.g. -`python canvigator.py --months 12 delete-old-conversations` keeps the -most recent year and deletes everything older. - -| | Files | -|---|---| -| **Input** | _(none — operates directly against the Canvas inbox)_ | -| **Output** | _(none — matched conversations are listed in the terminal)_ | -| **Canvas side-effect** | Permanently deletes matched conversations (skipped in `--dry-run` mode); requires explicit `yes` confirmation in live mode | - ---- - -#### `get-quiz-questions` — Export quiz question content - -Exports quiz metadata and question content to a CSV file. Skips downloading -student submission data, so this is a quick way to get a snapshot of the quiz -structure. - -| | Files | -|---|---| -| **Input** | _(none — data comes from the Canvas API)_ | -| **Output (default)** | `data//__questions_YYYYMMDD.csv` | -| **Output (with `--tag`)** | `data//__questions_w_tags_YYYYMMDD.csv` | - -The output CSV contains columns: `quiz_id`, `assignment_id`, `question_id`, -`position`, `question_name`, `question_type`, `question_text`, `points_possible`, -`answers` (JSON-encoded). The `assignment_id` enables joining quiz data with -gradebook/assignment exports. - -Pass `--tag` to add a `keywords` column (inserted before `question_text`) with -1–3 short topical tags per question, produced by an LLM via -[Ollama](https://ollama.com). The output is written to a separate file -(`*_questions_w_tags_*.csv`) so untagged and tagged exports never overwrite -each other. By default this uses the cloud-hosted text model -`gemini-3-flash-preview` at `https://ollama.com`, which requires `OLLAMA_API_KEY` -to be set — see [Ollama setup](#ollama-setup-optional). The model can be -overridden via `OLLAMA_TEXT_MODEL`. - -Pass `--all` to skip the interactive quiz prompt and export every quiz in the -selected course — one CSV per quiz, using each quiz's own filename prefix. -`--all` and `--tag` can be used independently or together. - ---- - -#### `send-follow-up-question` — Send the instructor-selected open-ended follow-up question to students - -Reads the `*_open_ended_*.csv`, picks the first row marked -`selected_question=1` as the question to send, and sends it via a Canvas -conversation message to each student who missed the corresponding original -quiz question on their latest attempt. Each thread uses `force_new=True` so -the follow-up exchange lives in its own dedicated conversation. The -Canvas-assigned `conversation_id` is captured at send time and recorded in -the manifest so `assess-replies` can fetch each thread directly by ID. - -The subject line includes a compact course code (prefix-number-CRN, with the -section dropped), the quiz name, and the specific question number — e.g. -`CSI-3300-12345 - Quiz 1 - Q3 Follow-Up`. This makes it easy to triage -conversations across multiple classes/quizzes/questions; the thread is still -tracked internally by `conversation_id`, so the subject format has no effect -on reply collection. - -The wording of the response instructions depends on the `question_mode` of -the open-ended question: `explain` asks the student to record a short voice -response, while `draw` asks them to attach a photo of a hand-drawn diagram. - -When not in `--dry-run` mode, the instructor is prompted for each student -with a full message preview and a `[send/SKIP]` choice (default: skip). Only -messages the instructor approves are sent and recorded in the manifest. -Submissions are auto-refreshed via `getAllSubmissionsAndEvents()` on every -run so the recipient list reflects the latest attempts. - -**Prerequisite**: run `get-quiz-questions --tag` and then -`generate-follow-up-questions` for the same quiz first so both the -`*_questions_w_tags_*.csv` and `*_open_ended_*.csv` are on disk. - -| | Files | -|---|---| -| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | -| | `data//__open_ended_YYYYMMDD.csv` (from `generate-follow-up-questions`) | -| **Output** | Fresh `*_all_submissions_*.csv`, `*_all_subs_by_question_*.csv`, and `*_all_subs_and_events_*.csv` (auto-refreshed via `getAllSubmissionsAndEvents()`) | -| | `data//__followup_sent_YYYYMMDD.csv` — manifest of approved sends (for use by `assess-replies`) | -| **Canvas side-effect** | Sends a Canvas conversation message (in a dedicated thread) to each approved student (skipped in `--dry-run` mode) | - -Use `--dry-run` to preview every candidate message without sending anything -or prompting interactively. - ---- - -#### `send-quiz-reminder` — Send quiz reminder messages to students - -Sends personalized Canvas messages to students based on their quiz performance. -Students who have not yet attempted the quiz receive a reminder to make an -attempt. Students who attempted but scored below perfect receive encouragement -to retake, plus — when their most recent attempt had any missed questions — a -bulleted list of the concepts/topics those questions covered along with their -per-question score. Students with perfect scores are skipped. - -**Prerequisite**: run `get-quiz-questions --tag` for the same quiz first so the -`*_questions_w_tags_*.csv` is on disk. The reminder task will not run -`get-quiz-questions` automatically — quiz content is static, so you should -generate it once ahead of time. Submissions, on the other hand, change right -up to the moment the reminder runs, so the task automatically invokes -`getAllSubmissionsAndEvents()` on every run to pick up the latest data. - -| | Files | -|---|---| -| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | -| **Output** | Fresh `data//__all_submissions_YYYYMMDD.csv`, `..._all_subs_by_question_YYYYMMDD.csv`, and `..._all_subs_and_events_YYYYMMDD.csv` (written automatically via `getAllSubmissionsAndEvents()`) | -| | `data//__reminder_sent_YYYYMMDD.csv` — audit log of approved sends (one row per student, with `conversation_id`); a `..._reminder_sent_dryrun_*.csv` is written in `--dry-run` mode | -| **Canvas side-effect** | Sends a Canvas conversation message to each student who hasn't attempted or hasn't achieved a perfect score (skipped in `--dry-run` mode) | - -Example of the appended section for an imperfect-score student: - -``` -The questions that you missed on this most recent attempt covered the concepts/topics: -• recursion, base case, stack frames — 0.50 / 1.00 points -• big-o, sorting — 0.00 / 1.00 points -``` - -When not in `--dry-run` mode, the instructor is prompted for each student -with a full message preview and a `[send/SKIP]` choice (default: skip), so -every send is reviewed before going to Canvas. - -Use `--dry-run` to preview all messages (recipient, subject, body, and reason) -without sending anything to Canvas and without the interactive prompt. - -Pass `--all` (or `-a`) to skip the interactive quiz picker and instead -iterate every published quiz with a future `due_at`. Each student's state is -aggregated across those quizzes and the task sends **one consolidated Canvas -message per student** listing every eligible quiz they still have work to do -on (no attempt, imperfect score, page-blur, or perfect-clean), with a single -course-level manifest `course_reminder_sent[_dryrun]_YYYYMMDD.csv` instead of -the per-quiz manifest. Quizzes without a `*_questions_w_tags_*.csv` are -skipped with a warning. Combine with `--dry-run` to preview the consolidated -messages without sending. - ---- - -#### Media-recording check-in workflow - -Three tasks form an end-to-end loop for short audio check-ins where each -student records a brief reflection (e.g. *"What was hardest on this week's -quiz?"*), the recording is downloaded and transcribed locally, the instructor -grades each submission, and the cohort's transcripts are then summarized -against the quiz topics: - -1. `create-media-recording-assignment` — create the assignment on Canvas. -2. `get-media-recordings` — fetch and transcribe each submission, and grade. -3. `analyze-media-recordings` — surface tag coverage and recurring themes - across the cohort. - -Audio handling is local-only: transcripts are produced via the local -`OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) and the cohort analysis runs -against the local `OLLAMA_MODEL` (default `gemma4:31b`). Student audio never -leaves your machine. - ---- - -#### `create-media-recording-assignment` — Create a media-recording assignment on Canvas - -Interactively creates a Canvas assignment with `submission_types=['media_recording']`. -Prompts for the title, prompt body (becomes the HTML `description`), -`points_possible` (default 1), an optional ISO `due_at`, and whether to publish -immediately (default unpublished so you can review before exposing it to -students). Logs the new assignment's ID and Canvas URL on success. - -| | Files | -|---|---| -| **Input** | _(none — interactive prompts only)_ | -| **Output** | _(none — assignment is created directly on Canvas)_ | - ---- - -#### `get-media-recordings` — Fetch, transcribe, and grade student audio submissions - -Pulls every media-recording submission for a chosen assignment, re-encodes the -audio to 16 kHz mono PCM WAV (via ffmpeg directly on Canvas's playback URL — -DASH manifests are handled in one pass), and transcribes each WAV locally -using `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`). Audio files are saved to -`data//media_recordings/assignment/_.wav`. - -By default, after each transcription the transcript is displayed and the -instructor is prompted for the points to award: - -- **Enter** — award `points_possible` (full credit). -- **A number** — award that value. -- **`s` / `skip`** — leave the submission ungraded. - -The chosen value is posted via Canvas's `Submission.edit({'posted_grade': ...})`. - -Pass `--auto-grade` to skip the per-student review and award `points_possible` -to every submitter automatically. Combine with `--dry-run` to preview the -auto-graded writes without mutating Canvas; in dry-run, transcripts are still -shown and the CSV is still written, but the grade write is suppressed. - -Pressing Ctrl-C mid-loop flushes a partial CSV before re-raising, so review -progress is never lost. - -| | Files | -|---|---| -| **Input** | _(Canvas — selected interactively)_ | -| **Output** | `data//assignment_recordings_YYYYMMDD.csv` — columns: `student_id`, `student_name`, `submission_id`, `submitted_at`, `audio_path`, `transcript`, `transcribed_at`, `grade`, `graded_at` | -| | `data//media_recordings/assignment/_.wav` — one WAV per submission | -| **Canvas side-effect** | Posts `posted_grade` on each graded submission (skipped in `--dry-run` mode) | - ---- - -#### `analyze-media-recordings` — Summarize cohort transcripts against quiz tags - -Loads the most recent `assignment_recordings_*.csv` produced by -`get-media-recordings`, prompts the instructor to pick a `*_questions_w_tags_*.csv` -from the same course directory (so tag vocabulary stays grounded in actual -quiz content), then runs two local-LLM passes against `OLLAMA_MODEL` (default -`gemma4:31b`): - -1. **Tag classification** — one call per non-empty transcript, asking which of - the unique tags from the quiz CSV's `keywords` column the student is - actually discussing. Paraphrase-tolerant, so *"I got confused on the linked - list one"* maps to `linked lists` even when the literal substring isn't - present. -2. **Theme extraction** — a single cohort-level call that surfaces 3–5 - recurring themes across all transcripts, with the tag list as background - context. - -The output is a Markdown report with three sections: a tag-grounded table -ranked by descending student count (the *"specialized word cloud"*), the -LLM-extracted themes, and a roster mapping the 1-based student indices used -in the themes section back to names. - -**Prerequisites**: run `get-media-recordings` (to produce the recordings CSV) -and `get-quiz-questions --tag` (to produce the tags CSV) first. - -| | Files | -|---|---| -| **Input** | `data//assignment_recordings_YYYYMMDD.csv` (from `get-media-recordings`) | -| | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | -| **Output** | `data//assignment_analysis_YYYYMMDD.md` — Markdown report | - ---- - -#### `prep-class-digest` — Pre-class "where the cohort actually is" Markdown brief - -Synthesizes a 1-page brief from three signal sources collected over a -configurable lookback window (default 7 days, override with `--days N`): +and `YYYYMMDD` is the current date. -- **Recent quiz misses** — joined to the topic tags from - `*_questions_w_tags_*.csv` so misses are aggregated by concept rather than - by individual question. -- **Follow-up reply themes** — failing/borderline rows in - `*_followup_assessments.csv` (within the window) are summarized by a local - Gemma 4 call into "where students went wrong" bullets, one block per - quiz/question. -- **Media-recording transcripts** — every - `assignment_recordings_*.csv` in the window is run through the existing - tag-classification + theme-extraction Gemma 4 pipeline (the same one used - by `analyze-media-recordings`). -The brief ends with **2–3 suggested in-class discussion questions** targeted -at the highest-priority gaps. By default these are drafted by the local -Gemma 4 model with a full-fidelity prompt that includes the derived theme -bullets and evidence snippets. With `--cloud-questions` (`-q`) the step -routes to cloud Gemini 3 instead, but only with a *redacted* prompt that -contains tag names + integer miss counts + integer theme-cluster counts — -no transcripts, no `criteria_evaluations`, no theme text. Student-derived -content stays local in either configuration. +### Tasks by workflow -Empty windows (no quiz/follow-up/recording activity in the lookback -period) short-circuit with a friendly message and write no file. Same-day -re-runs overwrite. +Per-task documentation has been split out by workflow. Each page below covers +a connected set of tasks plus the order in which they're typically run. -| | Files | +| Workflow | Tasks | |---|---| -| **Input** | `data//__all_subs_by_question_YYYYMMDD.csv` files in window (from `get-quiz-submission-events`) | -| | `data//__questions_w_tags_YYYYMMDD.csv` per quiz (from `get-quiz-questions --tag`) | -| | `data//__followup_assessments.csv` (from `assess-replies`, persistent) | -| | `data//assignment_recordings_YYYYMMDD.csv` files in window (from `get-media-recordings`) | -| **Output** | `data//class_digest_YYYYMMDD.md` — Markdown report (5 sections: header, quiz performance table, follow-up reply themes, media-recording themes, suggested discussion questions) | +| **[Peer instruction & retake bonuses](docs/peer-instruction-and-bonus.md)** | `create-pairs`, `award-bonus`, `award-bonus-partner-only`, `award-bonus-retake-only` | +| **[Quiz follow-up questions](docs/quiz-followup-questions.md)** | `get-quiz-questions`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`, `send-follow-up-assessments` | +| **[Media-recording check-ins](docs/media-recording-checkins.md)** | `create-media-recording-assignment`, `get-media-recordings`, `analyze-media-recordings` | +| **[Cross-workflow](docs/cross-workflow.md)** | `prep-class-digest` | +| **[Miscellaneous tasks](docs/misc-tasks.md)** | `create-quiz`, `export-anon-data`, `get-activity`, `get-quiz-submission-events`, `get-conversations`, `delete-old-conversations`, `get-gradebook`, `get-roster` | -**Typical workflow:** -1. Mid-week, after a quiz has been returned and any follow-ups assessed: - ```bash - python canvigator.py prep-class-digest # default 7-day window, Gemma-only - python canvigator.py --days 14 prep-class-digest # widen the window - python canvigator.py --cloud-questions prep-class-digest # opt the question step into Gemini - ``` -2. Open the generated `class_digest_*.md`, scan the priority gaps, and use - the suggested questions to seed a 5–10 minute in-class discussion. +For a quick on-the-CLI reference, run `python canvigator.py --help` (the +grouped task list with global options) or `python canvigator.py --help` +(a focused cheat-sheet for one task: prerequisites, inputs/outputs, applicable +flags, examples, and related tasks to run before/after). diff --git a/docs/cross-workflow.md b/docs/cross-workflow.md new file mode 100644 index 0000000..b4fae69 --- /dev/null +++ b/docs/cross-workflow.md @@ -0,0 +1,73 @@ +# Cross-workflow tasks + +Tasks that pull signals from multiple workflows and synthesize them into a +single artifact for the instructor. + +| Task | Summary | +|---|---| +| [`prep-class-digest`](#prep-class-digest) | Synthesize a 1-page Markdown brief on cohort gaps from quiz misses, follow-up reply themes, and media-recording transcripts; ends with 2–3 suggested in-class discussion questions. | + +Per-task `--help` is also available from the CLI: +`python canvigator.py --help`. + +← [Back to main README](../README.md) + +--- + +## `prep-class-digest` + +*Synthesize a 1-page Markdown brief on cohort gaps and 2–3 in-class discussion questions.* + +Synthesizes a 1-page brief from three signal sources collected over a +configurable lookback window (default 7 days, override with `--days N`): + +- **Recent quiz misses** — joined to the topic tags from + `*_questions_w_tags_*.csv` so misses are aggregated by concept rather than + by individual question. +- **Follow-up reply themes** — failing/borderline rows in + `*_followup_assessments.csv` (within the window) are summarized by a local + Gemma 4 call into "where students went wrong" bullets, one block per + quiz/question. +- **Media-recording transcripts** — every + `assignment_recordings_*.csv` in the window is run through the existing + tag-classification + theme-extraction Gemma 4 pipeline (the same one used + by [`analyze-media-recordings`](media-recording-checkins.md#analyze-media-recordings)). + +The brief ends with **2–3 suggested in-class discussion questions** targeted +at the highest-priority gaps. By default these are drafted by the local +Gemma 4 model with a full-fidelity prompt that includes the derived theme +bullets and evidence snippets. With `--cloud-questions` (`-q`) the step +routes to cloud Gemini 3 instead, but only with a *redacted* prompt that +contains tag names + integer miss counts + integer theme-cluster counts — +no transcripts, no `criteria_evaluations`, no theme text. Student-derived +content stays local in either configuration. + +Empty windows (no quiz/follow-up/recording activity in the lookback +period) short-circuit with a friendly message and write no file. Same-day +re-runs overwrite. + +**Before running:** `data//` should already be populated by upstream +tasks: [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events) +and [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions) +for misses, [`assess-replies`](quiz-followup-questions.md#assess-replies) for +follow-up themes, [`get-media-recordings`](media-recording-checkins.md#get-media-recordings) +for transcripts. + +| | Files | +|---|---| +| **Input** | `data//__all_subs_by_question_YYYYMMDD.csv` files in window (from [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events)) | +| | `data//__questions_w_tags_YYYYMMDD.csv` per quiz (from [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions)) | +| | `data//__followup_assessments.csv` (from [`assess-replies`](quiz-followup-questions.md#assess-replies), persistent) | +| | `data//assignment_recordings_YYYYMMDD.csv` files in window (from [`get-media-recordings`](media-recording-checkins.md#get-media-recordings)) | +| **Output** | `data//class_digest_YYYYMMDD.md` — Markdown report (5 sections: header, quiz performance table, follow-up reply themes, media-recording themes, suggested discussion questions) | + +### Examples + +```bash +python canvigator.py prep-class-digest # default 7-day window, Gemma-only +python canvigator.py --days 14 prep-class-digest # widen the window +python canvigator.py --cloud-questions prep-class-digest # opt the question step into Gemini +``` + +Open the generated `class_digest_*.md`, scan the priority gaps, and use the +suggested questions to seed a 5–10 minute in-class discussion. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..0389444 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,83 @@ +# Installation and setup + +← [Back to main README](../README.md) + +Note that installation and usage requires some basic knowledge on how +to use the command line. If necessary, there are many brief +tutorials/lessons available to help in this area, e.g., +[freecodecamp.org](https://www.freecodecamp.org/news/command-line-for-beginners/). + +1. **Clone the repository**: + [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this repository to your local system. This can either be done +in just one place on your local system, or separately for each course that it will be used in. The latter option is our preferred method since we tend to have a separate directory for each course we teach, and prefer to keep the course data separate. + +2. **Generate a Canvas API Token**: + Naviagte to the `canvigator` directory then run configuration setup script by simply typing, `./configure.sh` at the terminal prompt. + ```bash + cd canvigator + ``` + ```bash + ./configure.sh + ``` + You will be prompted to enter: + - Your institution's Canvas LMS base URL (you can find this on your Canvas home page). + - Your Canvas token, which can be created by navigating to _'Account'_, then _'Settings'_. Towards the bottom of this window in Canvas you will see a blue button, _'+ New Access Token_'. Click on this button to copy/download the token to your local system. DO NOT TO SHARE YOUR CANVAS TOKEN w/ ANYONE (e.g. do not save it in a shared directory). + - Your Ollama API key (optional — only needed if you plan to use the cloud-hosted text model for the LLM-powered tasks below). Leave it blank if you plan to use only local Ollama models (or none at all). See the [Ollama setup](#ollama-setup-optional) section for how to generate one. + +This will prompt the creation of the _data/_ and _figures/_ subdirectories. + +3. **Verify Setup**: + Once this is complete, double check that the configuration script, _set_env.sh_ has been created, that it has the correct values for your URL and token, and that the subdirectories have been created. + ```bash + set_env.sh + ``` + ```bash + env + ``` + This command will list all environment variables, allowing you to confirm that the necessary variables have been set correctly. + +4. **Verify Python Libraries**: + Before running the project, ensure that all required Python libraries are installed: + 1. Install the required libraries using `pip`: + + ```bash + pip install -r requirements.txt + ``` + + 2. Alternatively, you can install each library individually if needed. + + * ![canvasapi](https://img.shields.io/badge/canvasapi-3.3.0-blue) + * ![matplotlib](https://img.shields.io/badge/matplotlib-3.8.4-brightgreen) + * ![numpy](https://img.shields.io/badge/numpy-1.26.4-yellow) + * ![pandas](https://img.shields.io/badge/pandas-2.2.2-orange) + * ![requests](https://img.shields.io/badge/requests-2.33.0-red) + * ![scipy](https://img.shields.io/badge/scipy-1.13.1-lightgrey) + * ![seaborn](https://img.shields.io/badge/seaborn-0.13.2-blueviolet) + * ![ollama](https://img.shields.io/badge/ollama-0.4.7-black) + + If no errors are thrown, the libraries are successfully installed. The `ollama` client is only used by the LLM-assisted tasks described below — see [Ollama setup](#ollama-setup-optional). + + +## Ollama setup (optional) + +Several tasks use a Large Language Model (LLM) via [Ollama](https://ollama.com) to draft and tag questions, generate open-ended follow-ups, transcribe student audio, and assess student replies. You can skip this section if you will not be running any of these tasks (`create-quiz` with the `[g]enerate w/ LLM` option, `get-quiz-questions --tag`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`). + +Canvigator uses two kinds of models: + +1. **A cloud-hosted text model** (default `gemini-3-flash-preview`, set via `OLLAMA_TEXT_MODEL`) for instructor-side text generation — drafting quiz questions in `create-quiz`, tagging quiz questions, and generating open-ended follow-up questions. These tasks never see student data, so a larger cloud model is a good fit. +2. **Local models** for tasks that process student input — `gemma4:31b` (default `OLLAMA_MODEL`) for assessing text/image replies, and `gemma4:e4b` (default `OLLAMA_AUDIO_MODEL`) for transcribing student audio. Keeping these local is deliberate: student submissions should not leave your machine. + +**To use the cloud text model:** +1. Sign in at [ollama.com](https://ollama.com) and create an API key from your account settings. +2. Paste the key when prompted by `./configure.sh` (or add `export OLLAMA_API_KEY="..."` directly to `set_env.sh`). + +**To use the local models:** +1. Install Ollama from [ollama.com/download](https://ollama.com/download) and start it (`ollama serve`, or use the Ollama desktop app). +2. Pull the models you need: + ```bash + ollama pull gemma4:31b # assessment of text/image replies (assess-replies) + ollama pull gemma4:e4b # transcription of student audio (assess-replies, explain mode) + ``` + Only pull the models for tasks you actually plan to run. `gemma4:31b` is a large download (~20 GB) and is only required for `assess-replies`. + +All model names are overridable via env vars (`OLLAMA_TEXT_MODEL`, `OLLAMA_MODEL`, `OLLAMA_AUDIO_MODEL`), and the local host can be overridden via the standard `OLLAMA_HOST` env var. diff --git a/docs/media-recording-checkins.md b/docs/media-recording-checkins.md new file mode 100644 index 0000000..c977d26 --- /dev/null +++ b/docs/media-recording-checkins.md @@ -0,0 +1,119 @@ +# Media-recording check-ins + +Three tasks form an end-to-end loop for short audio check-ins where each +student records a brief reflection (e.g. *"What was hardest on this week's +quiz?"*), the recording is downloaded and transcribed locally, the instructor +grades each submission, and the cohort's transcripts are then summarized +against the quiz topics. + +| Task | Summary | +|---|---| +| [`create-media-recording-assignment`](#create-media-recording-assignment) | Create a Canvas assignment that accepts only media_recording submissions. | +| [`get-media-recordings`](#get-media-recordings) | Fetch + transcode + transcribe each submission and either prompt for a grade or auto-award `points_possible`. | +| [`analyze-media-recordings`](#analyze-media-recordings) | Two local-LLM passes against the transcripts: tag classification + theme extraction. | + +Per-task `--help` is also available from the CLI: +`python canvigator.py --help`. + +← [Back to main README](../README.md) + +## End-to-end workflow + +1. **`create-media-recording-assignment`** — create the assignment on Canvas. +2. **`get-media-recordings`** — fetch and transcribe each submission, and grade. +3. **`analyze-media-recordings`** — surface tag coverage and recurring themes + across the cohort. + +Audio handling is local-only: transcripts are produced via the local +`OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) and the cohort analysis runs +against the local `OLLAMA_MODEL` (default `gemma4:31b`). Student audio never +leaves your machine. + +--- + +## `create-media-recording-assignment` + +*Create a Canvas assignment that accepts only media_recording submissions.* + +Interactively creates a Canvas assignment with `submission_types=['media_recording']`. +Prompts for the title, prompt body (becomes the HTML `description`), +`points_possible` (default 1), an optional ISO `due_at`, and whether to publish +immediately (default unpublished so you can review before exposing it to +students). Logs the new assignment's ID and Canvas URL on success. + +| | Files | +|---|---| +| **Input** | _(none — interactive prompts only)_ | +| **Output** | _(none — assignment is created directly on Canvas)_ | + +--- + +## `get-media-recordings` + +*Fetch + transcode + transcribe each submission and either prompt for a grade or auto-award `points_possible`.* + +Pulls every media-recording submission for a chosen assignment, re-encodes the +audio to 16 kHz mono PCM WAV (via ffmpeg directly on Canvas's playback URL — +DASH manifests are handled in one pass), and transcribes each WAV locally +using `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`). Audio files are saved to +`data//media_recordings/assignment/_.wav`. + +By default, after each transcription the transcript is displayed and the +instructor is prompted for the points to award: + +- **Enter** — award `points_possible` (full credit). +- **A number** — award that value. +- **`s` / `skip`** — leave the submission ungraded. + +The chosen value is posted via Canvas's `Submission.edit({'posted_grade': ...})`. + +Pass `--auto-grade` to skip the per-student review and award `points_possible` +to every submitter automatically. Combine with `--dry-run` to preview the +auto-graded writes without mutating Canvas; in dry-run, transcripts are still +shown and the CSV is still written, but the grade write is suppressed. + +Pressing Ctrl-C mid-loop flushes a partial CSV before re-raising, so review +progress is never lost. + +| | Files | +|---|---| +| **Input** | _(Canvas — selected interactively)_ | +| **Output** | `data//assignment_recordings_YYYYMMDD.csv` — columns: `student_id`, `student_name`, `submission_id`, `submitted_at`, `audio_path`, `transcript`, `transcribed_at`, `grade`, `graded_at` | +| | `data//media_recordings/assignment/_.wav` — one WAV per submission | +| **Canvas side-effect** | Posts `posted_grade` on each graded submission (skipped in `--dry-run` mode) | + +--- + +## `analyze-media-recordings` + +*Two local-LLM passes against the transcripts: tag classification + theme extraction.* + +Loads the most recent `assignment_recordings_*.csv` produced by +`get-media-recordings`, prompts the instructor to pick a `*_questions_w_tags_*.csv` +from the same course directory (so tag vocabulary stays grounded in actual +quiz content), then runs two local-LLM passes against `OLLAMA_MODEL` (default +`gemma4:31b`): + +1. **Tag classification** — one call per non-empty transcript, asking which of + the unique tags from the quiz CSV's `keywords` column the student is + actually discussing. Paraphrase-tolerant, so *"I got confused on the linked + list one"* maps to `linked lists` even when the literal substring isn't + present. +2. **Theme extraction** — a single cohort-level call that surfaces 3–5 + recurring themes across all transcripts, with the tag list as background + context. + +The output is a Markdown report with three sections: a tag-grounded table +ranked by descending student count (the *"specialized word cloud"*), the +LLM-extracted themes, and a roster mapping the 1-based student indices used +in the themes section back to names. + +**Before running:** `get-media-recordings` must have produced the recordings +CSV, and [`get-quiz-questions --tag`](quiz-followup-questions.md#get-quiz-questions) +must have produced a tags CSV in the same course directory. + +| | Files | +|---|---| +| **Input** | `data//assignment_recordings_YYYYMMDD.csv` (from `get-media-recordings`) | +| | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | +| **Output** | `data//assignment_analysis_YYYYMMDD.md` — Markdown report | diff --git a/docs/misc-tasks.md b/docs/misc-tasks.md new file mode 100644 index 0000000..436d073 --- /dev/null +++ b/docs/misc-tasks.md @@ -0,0 +1,249 @@ +# Miscellaneous tasks + +Standalone tasks that don't belong to one of the multi-step workflows. Most +are simple Canvas exports (gradebook, roster, conversations); a few are +foundational utilities used by other workflows +(`get-quiz-submission-events` is the upstream of the +[bonus tasks](peer-instruction-and-bonus.md) and +[`prep-class-digest`](cross-workflow.md)) or one-shot administrative actions +(`create-quiz`, `delete-old-conversations`, `export-anon-data`). + +| Task | Summary | +|---|---| +| [`create-quiz`](#create-quiz) | Interactively create an unpublished quiz on Canvas (placeholder or LLM-generated questions). | +| [`export-anon-data`](#export-anon-data) | Anonymize the local CSVs for a course (no Canvas API needed); produces a zip archive plus an ID mapping file. | +| [`get-activity`](#get-activity) | Export per-student page-view, participation, and weekly-activity data for the course. | +| [`get-quiz-submission-events`](#get-quiz-submission-events) | Export per-attempt submission and event data for a quiz; renders score, timing, and page-blur histograms. | +| [`get-conversations`](#get-conversations) | Export every Canvas conversation involving an active student in the course. | +| [`delete-old-conversations`](#delete-old-conversations) | Sweep + delete the instructor's old Canvas conversations (account-wide, not course-scoped). | +| [`get-gradebook`](#get-gradebook) | Export the full course gradebook (one row per student-assignment pair). | +| [`get-roster`](#get-roster) | Export the full course roster: students, teachers, TAs, designers, observers. | + +Per-task `--help` is also available from the CLI: +`python canvigator.py --help`. + +← [Back to main README](../README.md) + +--- + +## `create-quiz` + +*Interactively create an unpublished quiz on Canvas (placeholder or LLM-generated questions).* + +Interactively creates a new unpublished quiz on Canvas. Prompts the user for a +quiz title, then for each question prompts: + +``` +QN — [p]laceholder, [g]enerate w/ LLM, [e]nd quiz: +``` + +- **`p`** (placeholder) — asks for a one-line description and creates a + multiple-choice placeholder with 1 point possible (and no answer choices). +- **`g`** (generate w/ LLM) — asks the user for a natural-language seed prompt + (e.g. *"a question about combinatorics asking how many ways 11 soccer players + can be selected from a team of 23"*) and routes it through the cloud text + model (default `gemini-3-flash-preview`, requires `OLLAMA_API_KEY` — see + [Ollama setup](installation.md#ollama-setup-optional)). The LLM returns a + complete Canvas-shaped question of one of seven auto-gradable types: + multiple-choice, multiple-answers, true/false, fill-in-multiple-blanks, + multiple-dropdowns, matching, or calculated. The proposed question is + rendered inline (type, name, text, answer choices with `*` marking the + correct one, plus per-type extras like match pairs or formula variables) and + the user is prompted `[y]/[r]/[s]` — accept, regenerate, or skip. Each + `[r]egenerate` accumulates the rejected draft and passes the full list back + to the LLM as anti-context (with sampling temperature bumped from 0.4 to + 0.8) so successive drafts diverge in scenario / values / question_type + instead of recycling the same angle. +- **`e`** or empty input — finalizes the quiz with the questions added so far. + +`points_possible` is forced to `1` for both placeholder and LLM-generated +questions; the instructor can refine wording, distractors, and points later in +the Canvas UI. The quiz is created with default settings: +`quiz_type='assignment'`, `time_limit=30`, `one_question_at_a_time=True`, +`cant_go_back=True`, `shuffle_answers=True`. + +The cloud text model is lazy-loaded only on the first `g` choice, so a +pure-placeholder run still works without `OLLAMA_API_KEY` set. + +| | Files | +|---|---| +| **Input** | _(none — interactive prompts only)_ | +| **Output** | _(none — quiz is created directly on Canvas, unpublished)_ | + +--- + +## `export-anon-data` + +*Anonymize the local CSVs for a course (no Canvas API needed); produces a zip archive plus an ID mapping file.* + +Anonymizes all CSV files in a course data directory by replacing student IDs +with hashed anonymous IDs (SHA-256 mod 10^10) and removing identifying columns +(`name`, `sis_id`). This task works entirely with local files and does not +require Canvas API credentials. Course selection is done from existing `data/` +subdirectories (by `--crn` or interactively). + +> **Note:** Anonymization alone does not make data shareable. Do not publish/share +> data without participant consent, IRB approval, and FERPA compliance. + +| | Files | +|---|---| +| **Input** | All CSV files in `data//` | +| **Output** | `data//anon_mapping_YYYYMMDD.csv` — mapping of original IDs to anonymous IDs | +| | `data//anonymized/` — directory containing anonymized copies of all CSVs | +| | `data//anonymized_YYYYMMDD.zip` — zip archive of the anonymized directory | + +--- + +## `get-activity` + +*Export per-student page-view, participation, and weekly-activity data for the course.* + +Fetches enrollment activity data and course-level summary data from Canvas, +merges them, and saves a single CSV. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output** | `data//course_activity_YYYYMMDD.csv` | + +The output CSV contains columns: `name`, `id`, `page_views`, `page_views_level`, +`participations`, `participations_level`, `missing`, `late`, `on_time`, +`total_activity_mins`, `first_activity_at`, `last_activity_at`, +`active_days_total`, `active_days_last_14`, `views_last_7d`, +`messages_to_instructor`. + +The `*_level` columns are Canvas-computed 0–3 buckets summarizing the raw counts. +`first_activity_at` / `last_activity_at` come from per-student page-view buckets +(course-scoped analytics) — more accurate than the enrollment-level field, which +Canvas often leaves stale. `messages_to_instructor` counts student-initiated +messages in the course conversation thread. + +If the Canvas course has a `start_at` date set, additional columns +`views_wk_01`, `views_wk_02`, … are appended (one per 7-day window from the +course start, capped at 16 weeks). Useful for spotting an engagement drop +mid-semester. + +If the Canvas token has admin-level permission to read user profiles / view +usage reports, four further columns are appended: `top_browser`, `top_os`, +`used_mobile_app`, `n_distinct_user_agents` — derived from per-user page-view +records for this course. Most instructor tokens lack this permission, in which +case these columns are simply omitted from the CSV. + +--- + +## `get-quiz-submission-events` + +*Export per-attempt submission and event data for a quiz; renders score, timing, and page-blur histograms.* + +Downloads detailed submission history and events for a user-selected quiz, +and renders per-question score histograms. Pass `--all` to iterate over every +published quiz in the course instead of prompting for one. This is the only +task that exports event history and per-question histogram figures. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output (per quiz)** | `data//__student_analysis_YYYYMMDD.csv` — raw quiz report | +| | `data//__all_submissions_YYYYMMDD.csv` — all submission attempts with scores | +| | `data//__all_subs_by_question_YYYYMMDD.csv` — per-question results for each attempt | +| | `data//__all_subs_and_events_YYYYMMDD.csv` — timestamped submission events | +| **Output — figures** | `figures//__score_histograms_YYYYMMDD.png` — per-question score histograms | +| | `figures//__timing_first_attempt_YYYYMMDD.png` — per-question first-attempt timing histograms (minutes) | +| | `figures//__blurs_first_attempt_YYYYMMDD.png` — per-question first-attempt page-blur counts | + +--- + +## `get-conversations` + +*Export every Canvas conversation involving an active student in the course.* + +Canvas conversations are not course-scoped, so this pulls the instructor's +full inbox and sent folders via `canvas.get_conversations()` (default scope +plus `scope='sent'`, deduped by `conversation_id`) and filters each +conversation's participants list against the active-student roster for the +selected course. Primary use case: back-fill the `conversation_id` for +follow-up sends made before the per-send manifest was added, so `assess-replies` +can still fetch those threads by ID. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output** | `data//conversations_YYYYMMDD.csv` | + +The output CSV is sorted newest first and contains columns: `conversation_id`, +`subject`, `last_message_at`, `first_message_at`, `message_count`, +`workflow_state`, `student_ids` (comma-separated), `student_names` +(semicolon-separated), `n_student_participants`, `last_message`. +`first_message_at` is derived by fetching each matched conversation (one extra +API call per conversation) and taking the earliest message `created_at`; it +is also substituted into `last_message_at` when the list endpoint returns an +empty value (occasionally happens for older/archived threads), so sort order +stays meaningful. + +--- + +## `delete-old-conversations` + +*Sweep + delete the instructor's old Canvas conversations (account-wide, not course-scoped).* + +Sweeps the instructor's Canvas inbox, sent folder, and archived folder for +conversations whose most recent message is older than `N` months +(approximate; one month = 30 days, default `N = 6`). Each match is shown +(last_message_at, subject, participant names) before any action is taken. +In live mode the task requires an explicit `yes` confirmation at the prompt +before any deletes run; `--dry-run` previews only. + +Canvas conversations are not course-scoped, so this task skips course +selection entirely and operates on the full account inbox. Deletion uses +`Conversation.delete()` to remove the whole thread (not just individual +messages). + +Override the cutoff with `--months`/`-m`, e.g. +`python canvigator.py --months 12 delete-old-conversations` keeps the +most recent year and deletes everything older. + +| | Files | +|---|---| +| **Input** | _(none — operates directly against the Canvas inbox)_ | +| **Output** | _(none — matched conversations are listed in the terminal)_ | +| **Canvas side-effect** | Permanently deletes matched conversations (skipped in `--dry-run` mode); requires explicit `yes` confirmation in live mode | + +--- + +## `get-gradebook` + +*Export the full course gradebook (one row per student-assignment pair).* + +Fetches all published assignments and their submissions from Canvas, joins with +enrolled student data, and exports a comprehensive gradebook CSV. One row per +student per assignment. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output** | `data//gradebook_YYYYMMDD.csv` | + +The output CSV contains columns: `name`, `sortable_name`, `user_id`, +`assignment_name`, `assignment_id`, `points_possible`, `grade`, `score`. + +--- + +## `get-roster` + +*Export the full course roster: students, teachers, TAs, designers, observers.* + +Iterates `canvas_course.get_enrollments()` with no role filter and writes a +minimal CSV of every enrolled person — students, teachers, TAs, designers, and +observers. Useful as a course-wide identity reference and as a stable input +for scripts outside Canvigator. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output** | `data//roster_YYYYMMDD.csv` | + +The output CSV contains columns: `name`, `id`, `sis_id`, `enrollment_type`, +`state`. `enrollment_type` is the Canvas type string (`StudentEnrollment`, +`TeacherEnrollment`, `TaEnrollment`, `DesignerEnrollment`, `ObserverEnrollment`); +`state` is the enrollment state (`active`, `invited`, `inactive`, `completed`, +`rejected`, `deleted`). diff --git a/docs/peer-instruction-and-bonus.md b/docs/peer-instruction-and-bonus.md new file mode 100644 index 0000000..a4e9645 --- /dev/null +++ b/docs/peer-instruction-and-bonus.md @@ -0,0 +1,128 @@ +# Peer instruction and retake incentives + +Tasks for quizzes built around peer collaboration and retake bonuses. + +| Task | Summary | +|---|---| +| [`create-pairs`](#create-pairs) | Pair students for a collaborative in-class quiz using a prior quiz's score vectors. | +| [`award-bonus`](#award-bonus) | Award both partner and retake bonus points as fudge points on each student's best attempt. | +| [`award-bonus-partner-only`](#award-bonus-partner-only) | Same as `award-bonus` but skips retake detection. | +| [`award-bonus-retake-only`](#award-bonus-retake-only) | Same as `award-bonus` but skips partner detection. | + +Per-task `--help` is also available from the CLI: +`python canvigator.py --help`. + +← [Back to main README](../README.md) + +## End-to-end workflow + +1. _(Optional, before a collaborative in-class quiz)_ Mark attendance in + `data//present_*.csv`, then run **`create-pairs`** to pair up + present students using a prior quiz's score vectors. +2. After the quiz attempts have been collected, run + [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events) to + export the submission CSVs that the bonus tasks read from. +3. Run **`award-bonus`** (or `award-bonus-partner-only` / + `award-bonus-retake-only`) to detect partners and/or retakers and award + fudge points on each student's best attempt. Use `--dry-run` first to + preview without writing to Canvas. + +--- + +## `create-pairs` + +*Pair students for a collaborative in-class quiz using a prior quiz's score vectors.* + +Uses an independent (pre-class) quiz to pair students for a collaborative +in-class quiz. Computes pairwise Euclidean distance between student +answer-score vectors and applies a greedy pairing algorithm. This task writes +the raw quiz report plus pairing-specific figures, but does not export +submission events or histograms. + +**Before running:** create a `present_*.csv` file in `data//` marking +which students are physically present. See `data/present_example.csv` for the +required format (columns: `name`, `id`, `present` where 1 = present). + +| | Files | +|---|---| +| **Input** | `data//present_*.csv` (user-created, selected via prompt) | +| **Output — data** | `data//__student_analysis_YYYYMMDD.csv` — raw quiz report downloaded from Canvas | +| | `data//pairings_based_on___YYYYMMDD.csv` — student pairings (person3/id3 columns omitted when no triples exist) | +| **Output — figures** | `figures//__dist_euclid_YYYYMMDD.png` — distance matrix heatmap (present students) | +| | `figures//__compare_pairing_methods_YYYYMMDD.png` — comparison of all four pairing methods | + +--- + +## `award-bonus` + +*Award both partner and retake bonus points as fudge points on each student's best attempt.* + +Detects partner groups and qualifying retakers, then awards both bonus types +as combined fudge points on their Canvas submissions. Each bonus defaults to +15% of quiz points; a student earning both receives 30% total. + +**Partner bonus:** Students whose first attempts show a high fraction of +matching scores (default ≥ 80%) and closely timed answers (default within +5 seconds on ≥ 80% of questions) are grouped as partners. Groups larger than +3 are flagged for manual review. + +**Retake bonus:** Students who retook the quiz at least 3 times with at least +1 day (24 hours) between qualifying attempts. + +**Before running:** [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events) +must have been run first for the same quiz so that the submission CSVs exist. + +| | Files | +|---|---| +| **Input** | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| | `data//__all_subs_by_question_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| | `data//__all_subs_and_events_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| **Output** | `data//__detected_partners_YYYYMMDD.csv` — detected partner groups | +| | `data//__retake_qualified_YYYYMMDD.csv` — students qualifying for retake bonus | +| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with partner_bonus, retake_bonus, and combined bonus columns | +| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment describing the bonus breakdown (skipped in `--dry-run` mode) | + +Use `--dry-run` to preview which students would receive bonus points without +modifying anything in Canvas. In dry-run mode the scores CSV is named +`*_scores_w_bonus_dryrun_*.csv`. + +--- + +## `award-bonus-partner-only` + +*Same as `award-bonus` but skips retake detection.* + +Same as `award-bonus` but only detects and awards the partner bonus (no retake +bonus). Useful when you want to reward collaborative work without the retake +incentive. + +**Before running:** [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events) +must have been run first for the same quiz. + +| | Files | +|---|---| +| **Input** | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| | `data//__all_subs_by_question_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| | `data//__all_subs_and_events_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| **Output** | `data//__detected_partners_YYYYMMDD.csv` — detected partner groups | +| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with bonus column | +| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment (skipped in `--dry-run` mode) | + +--- + +## `award-bonus-retake-only` + +*Same as `award-bonus` but skips partner detection.* + +Same as `award-bonus` but only detects and awards the retake bonus (no partner +detection). Useful when you want to incentivize retakes independently. + +**Before running:** [`get-quiz-submission-events`](misc-tasks.md#get-quiz-submission-events) +must have been run first for the same quiz. + +| | Files | +|---|---| +| **Input** | `data//__all_submissions_YYYYMMDD.csv` (from `get-quiz-submission-events`) | +| **Output** | `data//__retake_qualified_YYYYMMDD.csv` — students qualifying for retake bonus | +| | `data//__scores_w_bonus_YYYYMMDD.csv` — scores with bonus column | +| **Canvas side-effect** | Sets `fudge_points` on the student's highest-scoring attempt and leaves a submission comment (skipped in `--dry-run` mode) | diff --git a/docs/quiz-followup-questions.md b/docs/quiz-followup-questions.md new file mode 100644 index 0000000..5eb4bc1 --- /dev/null +++ b/docs/quiz-followup-questions.md @@ -0,0 +1,357 @@ +# Quiz follow-up questions + +A connected set of tasks that turn a Canvas quiz into a personalized, +LLM-graded follow-up loop: tag the questions by topic, generate open-ended +follow-ups, nudge students who haven't aced the quiz, send the follow-up to +students who missed a specific question, then assess (and finally send back) +the replies. + +| Task | Summary | +|---|---| +| [`get-quiz-questions`](#get-quiz-questions) | Export quiz metadata + question content; with `--tag` adds LLM-generated topic tags. | +| [`generate-follow-up-questions`](#generate-follow-up-questions) | Per original question, draft 3 candidate open-ended follow-ups + rubric + exemplars; instructor curates. | +| [`send-quiz-reminder`](#send-quiz-reminder) | Personalized Canvas reminders to students who haven't attempted or are below perfect. | +| [`send-follow-up-question`](#send-follow-up-question) | Send the curated follow-up via Canvas conversation to students who missed the corresponding question. | +| [`assess-replies`](#assess-replies) | Fetch student replies (audio/image) from Canvas and grade them with the local LLM. | +| [`send-follow-up-assessments`](#send-follow-up-assessments) | Post curated feedback back as a reply on the existing follow-up thread. | + +Per-task `--help` is also available from the CLI: +`python canvigator.py --help`. + +← [Back to main README](../README.md) + +## End-to-end workflow + +Several of the newer tasks are designed to chain together into a single +end-to-end flow. Run them in this order for a given quiz: + +1. `python canvigator.py --tag get-quiz-questions` — export the quiz's question content and add LLM topic tags (cloud model, requires `OLLAMA_API_KEY`). +2. `python canvigator.py generate-follow-up-questions` — generate 3 candidate open-ended follow-up questions per original question, with an assessment guide for each. **Review the output CSV and set `selected_question=1` on one row per question group before moving on.** +3. _(optional)_ `python canvigator.py send-quiz-reminder` — nudge students who haven't attempted the quiz or who scored below perfect. Imperfect-score students get a bulleted list of the topics (from the tags) they missed. +4. `python canvigator.py send-follow-up-question` — send the instructor-selected open-ended question (the first row in the CSV with `selected_question=1`) to each student who missed the corresponding original quiz question. Students reply via Canvas conversations with an audio recording ("explain") or a photo ("draw"). +5. `python canvigator.py assess-replies` — pull the students' replies (and attached audio/images) back from Canvas, then run them through the local `gemma4` models (transcription + assessment) to produce pass/fail + feedback. Results are merged into a single persistent `*_followup_assessments.csv` (no date suffix); rows where `sent_assessment=1` are preserved verbatim across re-runs. +6. **Edit the `feedback` column** in `*_followup_assessments.csv` for any student where the LLM's assessment needs correction. +7. `python canvigator.py send-follow-up-assessments` — post the (curated) `feedback` text back to each student as a reply on the existing follow-up conversation thread; sets `sent_assessment=1`. + +Steps 4–7 can be repeated as students keep replying: `assess-replies` picks up new messages and reassesses against the latest reply per student (skipping rows already sent), and `send-follow-up-assessments` only sends rows that haven't been sent yet. + +### Data flow + +``` +get-quiz-questions --tag + │ + └─→ *_questions_w_tags_*.csv ─────────────────────────────┐ + │ (read by) +generate-follow-up-questions ◄─ reads *_questions_w_tags_* │ + │ │ + └─→ *_open_ended_*.csv ──┐ │ + (instructor sets │ │ + selected_question=1) │ │ + │ │ +send-follow-up-question ◄────┼─────────────────────────────────┘ + │ (recipients = students who missed the original question on + │ their latest attempt; auto-refreshes submission CSVs) + │ + └─→ *_followup_sent_*.csv ──┐ (manifest of approved sends with + │ conversation_id per row) + │ +assess-replies ◄────────────────┘ + │ (fetches Canvas replies; reads *_open_ended_*.csv for + │ the rubric, assessment guide, and exemplars) + │ + ├─→ *_followup_replies_*.csv (refreshed every run) + └─→ *_followup_assessments.csv (persistent across runs; + instructor edits `feedback` column) + │ +send-follow-up-assessments ◄────┘ + │ + └─→ updates *_followup_assessments.csv (sent_assessment=1, sent_at) +``` + +`send-quiz-reminder` is an optional side-branch off this chain — it reads the +same `*_questions_w_tags_*.csv`, auto-refreshes submission CSVs, and writes a +`*_reminder_sent_*.csv` manifest, but doesn't feed any of the downstream tasks. + +--- + +## `get-quiz-questions` + +*Export quiz metadata + question content; with `--tag` adds LLM-generated topic tags.* + +Exports quiz metadata and question content to a CSV file. Skips downloading +student submission data, so this is a quick way to get a snapshot of the quiz +structure. + +| | Files | +|---|---| +| **Input** | _(none — data comes from the Canvas API)_ | +| **Output (default)** | `data//__questions_YYYYMMDD.csv` | +| **Output (with `--tag`)** | `data//__questions_w_tags_YYYYMMDD.csv` | + +The output CSV contains columns: `quiz_id`, `assignment_id`, `question_id`, +`position`, `question_name`, `question_type`, `question_text`, `points_possible`, +`answers` (JSON-encoded). The `assignment_id` enables joining quiz data with +gradebook/assignment exports. + +Pass `--tag` to add a `keywords` column (inserted before `question_text`) with +1–3 short topical tags per question, produced by an LLM via +[Ollama](https://ollama.com). The output is written to a separate file +(`*_questions_w_tags_*.csv`) so untagged and tagged exports never overwrite +each other. By default this uses the cloud-hosted text model +`gemini-3-flash-preview` at `https://ollama.com`, which requires `OLLAMA_API_KEY` +to be set — see [Ollama setup](installation.md#ollama-setup-optional). The +model can be overridden via `OLLAMA_TEXT_MODEL`. + +Pass `--all` to skip the interactive quiz prompt and export every quiz in the +selected course — one CSV per quiz, using each quiz's own filename prefix. +`--all` and `--tag` can be used independently or together. + +--- + +## `generate-follow-up-questions` + +*Per original question, draft 3 candidate open-ended follow-ups + rubric + exemplars; instructor curates.* + +Reads the tagged questions CSV for a selected quiz and uses a cloud-hosted LLM +(via Ollama's hosted endpoint, default `gemini-3-flash-preview`) to produce +open-ended follow-up candidates. For each original quiz question the task runs +five steps: + +1. **Classify** — The LLM decides whether an oral explanation ("explain") or a + hand-drawn diagram ("draw") would be the better way to assess student + understanding. Inherently visual topics (e.g. data structures, memory + layouts, process flows) are classified as "draw"; verbal topics (e.g. + trade-offs, algorithm logic, definitions) as "explain". +2. **Generate 3 candidates** — Using the classification, the LLM generates 3 + self-contained candidate open-ended questions. "Explain" questions begin with + "Explain..." and target a ~1 minute oral response. "Draw" questions begin + with "Draw a diagram..." or "Draw a figure..." and target a ~2 minute + hand-drawn response. +3. **Assessment guide** — For each candidate, the LLM writes a short + `assessment_guide` describing the key concepts/elements a passing student + response must include. This guide is the human-readable summary shown in + the CSV and used as backup rubric material in `assess-replies`. +4. **Structured rubric** — For each candidate, the LLM also emits a JSON + object in the `rubric_json` column with fields `canonical_answer`, + `model_answer` (an 80–120-word A-grade exemplar response in a real + student's voice), `pass_criteria`, `acceptable_alternatives`, + `common_misconceptions`, `fatal_errors` (plus `required_visual_elements` + for draw questions). This structured rubric is what the local Gemma grader + uses during `assess-replies`; edit the JSON blob if you want to tighten the + criteria. Do not open the CSV in Excel and save — Excel will mangle the + JSON quoting. +5. **Calibration exemplars** — A separate LLM call drafts one passing and one + failing example response (for explain mode, a transcript fragment; for + draw mode, a prose description of the drawing) plus a one-sentence note + explaining what makes each one pass or fail. These land in the + `exemplar_pass`, `exemplar_pass_note`, `exemplar_fail`, + `exemplar_fail_note` columns and are passed to the grader at assessment + time as an "anchor the bar here" reference (separate from the live + instructor-approved examples that `assess-replies` accumulates). + +The output CSV is intended for instructor review. Three rows are written per +original question (one per candidate) with `selected_question=0`; **the +instructor must review the candidates offline and set `selected_question=1` +on exactly one row per question group** before running +`send-follow-up-question`. + +**Before running:** `get-quiz-questions --tag` must have produced +`*_questions_w_tags_*.csv` for the same quiz. Requires `OLLAMA_API_KEY` to be +set — see [Ollama setup](installation.md#ollama-setup-optional). + +| | Files | +|---|---| +| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | +| **Output** | `data//__open_ended_YYYYMMDD.csv` | + +The output CSV contains columns: `selected_question`, `question_id`, +`position`, `question_name`, `keywords`, `question_mode` (`explain` or +`draw`), `open_ended_question`, `assessment_guide`, `rubric_json`, +`exemplar_pass`, `exemplar_pass_note`, `exemplar_fail`, `exemplar_fail_note`, +`original_question_text`. + +--- + +## `send-quiz-reminder` + +*Personalized Canvas reminders to students who haven't attempted or are below perfect.* + +Sends personalized Canvas messages to students based on their quiz performance. +Students who have not yet attempted the quiz receive a reminder to make an +attempt. Students who attempted but scored below perfect receive encouragement +to retake, plus — when their most recent attempt had any missed questions — a +bulleted list of the concepts/topics those questions covered along with their +per-question score. Students with perfect scores are skipped. + +When not in `--dry-run` mode, the instructor is prompted for each student +with a full message preview and a `[send/SKIP]` choice (default: skip), so +every send is reviewed before going to Canvas. + +**Before running:** `get-quiz-questions --tag` must have produced +`*_questions_w_tags_*.csv` for the quiz. The reminder task will not run +`get-quiz-questions` automatically — quiz content is static, so you should +generate it once ahead of time. Submissions, on the other hand, change right +up to the moment the reminder runs, so the task automatically invokes +`getAllSubmissionsAndEvents()` on every run to pick up the latest data. + +| | Files | +|---|---| +| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | +| **Output** | Fresh `data//__all_submissions_YYYYMMDD.csv`, `..._all_subs_by_question_YYYYMMDD.csv`, and `..._all_subs_and_events_YYYYMMDD.csv` (written automatically via `getAllSubmissionsAndEvents()`) | +| | `data//__reminder_sent_YYYYMMDD.csv` — audit log of approved sends (one row per student, with `conversation_id`); a `..._reminder_sent_dryrun_*.csv` is written in `--dry-run` mode | +| **Canvas side-effect** | Sends a Canvas conversation message to each student who hasn't attempted or hasn't achieved a perfect score (skipped in `--dry-run` mode) | + +Example of the appended section for an imperfect-score student: + +``` +The questions that you missed on this most recent attempt covered the concepts/topics: +• recursion, base case, stack frames — 0.50 / 1.00 points +• big-o, sorting — 0.00 / 1.00 points +``` + +Use `--dry-run` to preview all messages (recipient, subject, body, and reason) +without sending anything to Canvas and without the interactive prompt. + +Pass `--all` (or `-a`) to skip the interactive quiz picker and instead +iterate every published quiz with a future `due_at`. Each student's state is +aggregated across those quizzes and the task sends **one consolidated Canvas +message per student** listing every eligible quiz they still have work to do +on (no attempt, imperfect score, page-blur, or perfect-clean), with a single +course-level manifest `course_reminder_sent[_dryrun]_YYYYMMDD.csv` instead of +the per-quiz manifest. Quizzes without a `*_questions_w_tags_*.csv` are +skipped with a warning. Combine with `--dry-run` to preview the consolidated +messages without sending. + +--- + +## `send-follow-up-question` + +*Send the curated follow-up via Canvas conversation to students who missed the corresponding question.* + +Reads the `*_open_ended_*.csv`, picks the first row marked +`selected_question=1` as the question to send, and sends it via a Canvas +conversation message to each student who missed the corresponding original +quiz question on their latest attempt. Each thread uses `force_new=True` so +the follow-up exchange lives in its own dedicated conversation. The +Canvas-assigned `conversation_id` is captured at send time and recorded in +the manifest so `assess-replies` can fetch each thread directly by ID. + +The subject line includes a compact course code (prefix-number-CRN, with the +section dropped), the quiz name, and the specific question number — e.g. +`CSI-3300-12345 - Quiz 1 - Q3 Follow-Up`. This makes it easy to triage +conversations across multiple classes/quizzes/questions; the thread is still +tracked internally by `conversation_id`, so the subject format has no effect +on reply collection. + +The wording of the response instructions depends on the `question_mode` of +the open-ended question: `explain` asks the student to record a short voice +response, while `draw` asks them to attach a photo of a hand-drawn diagram. + +When not in `--dry-run` mode, the instructor is prompted for each student +with a full message preview and a `[send/SKIP]` choice (default: skip). Only +messages the instructor approves are sent and recorded in the manifest. +Submissions are auto-refreshed via `getAllSubmissionsAndEvents()` on every +run so the recipient list reflects the latest attempts. + +**Before running:** `get-quiz-questions --tag` and then +`generate-follow-up-questions` must have been run for the same quiz so both +the `*_questions_w_tags_*.csv` and `*_open_ended_*.csv` are on disk. + +| | Files | +|---|---| +| **Input** | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | +| | `data//__open_ended_YYYYMMDD.csv` (from `generate-follow-up-questions`) | +| **Output** | Fresh `*_all_submissions_*.csv`, `*_all_subs_by_question_*.csv`, and `*_all_subs_and_events_*.csv` (auto-refreshed via `getAllSubmissionsAndEvents()`) | +| | `data//__followup_sent_YYYYMMDD.csv` — manifest of approved sends (for use by `assess-replies`) | +| **Canvas side-effect** | Sends a Canvas conversation message (in a dedicated thread) to each approved student (skipped in `--dry-run` mode) | + +Use `--dry-run` to preview every candidate message without sending anything +or prompting interactively. + +--- + +## `assess-replies` + +*Fetch student replies (audio/image) from Canvas and grade them with the local LLM.* + +First fetches the latest student replies from Canvas (loading the manifest +written by `send-follow-up-question`, finding matching sent conversations by +subject, downloading image attachments and audio recordings to +`data//replies/`, and writing a dated `*_followup_replies_*.csv`). +Only replies received within the configured window after the follow-up was +sent are accepted (default 5 days; override with `--reply-window-days N`). + +Then loads the latest reply per student from that CSV and uses local LLMs via +Ollama to evaluate each one against the original question. Two model pipelines +are used depending on `question_mode`: + +- **`explain` mode**: `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) transcribes + the student's audio recording, then `OLLAMA_MODEL` (default `gemma4:31b`) + assesses the transcript against the question context. +- **`draw` mode**: `OLLAMA_MODEL` directly assesses the student's submitted + image against the question context. + +Each reply is graded with N=3 self-consistency passes and a majority vote; +`confidence=borderline` flags the rows worth instructor review before sending. +The grader is also primed with the structured rubric and exemplars from +`*_open_ended_*.csv`, plus up to 3 prior pass + 3 prior fail +`sent_assessment=1` responses for the same question (sampled from the +assessments file itself) as in-context calibration — so once you correct a few +borderline cases and send them, future re-runs anchor to your judgment. + +Re-running this task is safe: rows with `sent_assessment=1` are skipped and +preserved verbatim, so previously sent feedback is never overwritten. Rows +with `sent_assessment=0` are re-assessed in place. On the first run, an +existing dated `*_followup_assessments_YYYYMMDD.csv` is migrated forward into +the non-dated file. + +**Before running:** `send-follow-up-question` must have produced +`*_followup_sent_*.csv`. Requires a running Ollama server with both models +pulled. + +| | Files | +|---|---| +| **Input** | `data//__followup_sent_YYYYMMDD.csv` (from `send-follow-up-question`) | +| | `data//__open_ended_YYYYMMDD.csv` (provides the rubric, assessment guide, and exemplars used by the grader) | +| **Output** | `data//__followup_replies_YYYYMMDD.csv` | +| | `data//replies/` — downloaded image attachments and audio recordings | +| | `data//__followup_assessments.csv` (persistent — merged across runs) | + +The replies CSV contains columns: `student_id`, `student_name`, `question_id`, +`question_mode`, `conversation_id`, `message_id`, `reply_text`, `has_attachment`, +`attachment_path`, `has_audio`, `audio_path`, `replied_at`, `latest`. The +`latest` flag marks the most recent reply per student (used for assessment). + +The assessments CSV contains columns: `student_id`, `student_name`, `question_id`, +`question_mode`, `conversation_id`, `result` (`pass` or `fail`), `confidence` +(`high` when all self-consistency runs agreed, `borderline` when they split), +`feedback`, `transcript` (for `explain` mode), `criteria_evaluations` (a JSON +blob with the per-criterion `met`/`partial`/`missing` ratings the grader produced), +`assessed_at`, `sent_assessment` (`0` until `send-follow-up-assessments` posts +the row, then `1`), `sent_at`. + +--- + +## `send-follow-up-assessments` + +*Post curated feedback back as a reply on the existing follow-up thread.* + +Reads `*_followup_assessments.csv` and, for every row with `sent_assessment=0` +and a non-empty `feedback` value, posts the feedback text as a reply on the +existing follow-up conversation thread (`Conversation.add_message`). On +success, sets `sent_assessment=1` and stamps `sent_at`, then rewrites the file +in place. The `--dry-run` flag previews what would be sent without touching +Canvas (and leaves `sent_assessment=0`). + +The expected workflow is to **edit the `feedback` column** between +`assess-replies` and `send-follow-up-assessments` for any rows where the LLM +assessment needs correction; the curated text is what the student sees. + +**Before running:** `assess-replies` must have produced +`*_followup_assessments.csv`. + +| | Files | +|---|---| +| **Input** | `data//__followup_assessments.csv` (from `assess-replies`, edited by instructor) | +| | `data//__followup_sent_*.csv` (fallback for `conversation_id` lookup) | +| **Output** | The same `*_followup_assessments.csv`, with `sent_assessment` and `sent_at` updated |