diff --git a/README.md b/README.md index 255021f..0c4d757 100644 --- a/README.md +++ b/README.md @@ -83,11 +83,11 @@ This will prompt the creation of the _data/_ and _figures/_ subdirectories. ### Ollama setup (optional) -Several tasks use a Large Language Model (LLM) via [Ollama](https://ollama.com) to tag questions, generate open-ended follow-ups, transcribe student audio, and assess student replies. You can skip this section if you will not be running any of these tasks (`get-quiz-questions --tag`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`). +Several tasks use a Large Language Model (LLM) via [Ollama](https://ollama.com) to draft and tag questions, generate open-ended follow-ups, transcribe student audio, and assess student replies. You can skip this section if you will not be running any of these tasks (`create-quiz` with the `[g]enerate w/ LLM` option, `get-quiz-questions --tag`, `generate-follow-up-questions`, `send-quiz-reminder`, `send-follow-up-question`, `assess-replies`). Canvigator uses two kinds of models: -1. **A cloud-hosted text model** (default `gemini-3-flash-preview`, set via `OLLAMA_TEXT_MODEL`) for instructor-side text generation — tagging quiz questions and generating open-ended questions. These tasks never see student data, so a larger cloud model is a good fit. +1. **A cloud-hosted text model** (default `gemini-3-flash-preview`, set via `OLLAMA_TEXT_MODEL`) for instructor-side text generation — drafting quiz questions in `create-quiz`, tagging quiz questions, and generating open-ended follow-up questions. These tasks never see student data, so a larger cloud model is a good fit. 2. **Local models** for tasks that process student input — `gemma4:31b` (default `OLLAMA_MODEL`) for assessing text/image replies, and `gemma4:e4b` (default `OLLAMA_AUDIO_MODEL`) for transcribing student audio. Keeping these local is deliberate: student submissions should not leave your machine. **To use the cloud text model:** @@ -112,17 +112,18 @@ All model names are overridable via env vars (`OLLAMA_TEXT_MODEL`, `OLLAMA_MODEL source set_env.sh # set Canvas (and optional Ollama) environment variables (once per terminal session) python canvigator.py # run a task (prompts for course selection) python canvigator.py --crn # select course by CRN (last 5 digits of course code) -python canvigator.py --dry-run # preview changes without modifying Canvas (bonus, reminder, follow-up, and delete-old-conversations tasks) +python canvigator.py --dry-run # preview changes without modifying Canvas (bonus, reminder, follow-up, delete-old-conversations, get-media-recordings) python canvigator.py --tag get-quiz-questions # add LLM-generated topic tags to the quiz questions export python canvigator.py --reply-window-days N # set the days-after-send window for assess-replies (default: 5) python canvigator.py --months N delete-old-conversations # cutoff age in months for delete-old-conversations (default: 6) +python canvigator.py --auto-grade get-media-recordings # skip per-student review and auto-grade every submission at full credit ``` The `--crn` option selects a course by its CRN (the last 5 digits of the Canvas course code), bypassing the interactive course selection prompt. This is useful for automated/scheduled runs, e.g. `python canvigator.py --crn 12345 get-activity`. -Available tasks: `assess-replies`, `award-bonus`, `award-bonus-partner-only`, `award-bonus-retake-only`, `create-pairs`, `create-quiz`, `delete-old-conversations`, `export-anon-data`, `generate-follow-up-questions`, `get-activity`, `get-conversations`, `get-gradebook`, `get-quiz-questions`, `get-quiz-submission-events`, `get-roster`, `send-follow-up-assessments`, `send-follow-up-question`, `send-quiz-reminder` +Available tasks: `analyze-media-recordings`, `assess-replies`, `award-bonus`, `award-bonus-partner-only`, `award-bonus-retake-only`, `create-media-recording-assignment`, `create-pairs`, `create-quiz`, `delete-old-conversations`, `export-anon-data`, `generate-follow-up-questions`, `get-activity`, `get-conversations`, `get-gradebook`, `get-media-recordings`, `get-quiz-questions`, `get-quiz-submission-events`, `get-roster`, `send-follow-up-assessments`, `send-follow-up-question`, `send-quiz-reminder` All tasks begin by prompting you to select a course. Output files are written to `data//` and `figures//`, where `` is derived from the @@ -318,13 +319,39 @@ required format (columns: `name`, `id`, `present` where 1 = present). --- -#### `create-quiz` — Create an unpublished placeholder quiz on Canvas +#### `create-quiz` — Create an unpublished quiz on Canvas, with placeholder or LLM-generated questions Interactively creates a new unpublished quiz on Canvas. Prompts the user for a -quiz title, then iteratively prompts for question descriptions. Each question is -added as a multiple-choice placeholder with 1 point possible. The quiz is -created with default settings: `quiz_type='assignment'`, `time_limit=30`, -`one_question_at_a_time=True`, `cant_go_back=True`, `shuffle_answers=True`. +quiz title, then for each question prompts: + +``` +QN — [p]laceholder, [g]enerate w/ LLM, [e]nd quiz: +``` + +- **`p`** (placeholder) — asks for a one-line description and creates a + multiple-choice placeholder with 1 point possible (and no answer choices). +- **`g`** (generate w/ LLM) — asks the user for a natural-language seed prompt + (e.g. *"a question about combinatorics asking how many ways 11 soccer players + can be selected from a team of 23"*) and routes it through the cloud text + model (default `gemini-3-flash-preview`, requires `OLLAMA_API_KEY` — see + [Ollama setup](#ollama-setup-optional)). The LLM returns a complete + Canvas-shaped question of one of seven auto-gradable types: multiple-choice, + multiple-answers, true/false, fill-in-multiple-blanks, multiple-dropdowns, + matching, or calculated. The proposed question is rendered inline (type, + name, text, answer choices with `*` marking the correct one, plus per-type + extras like match pairs or formula variables) and the user is prompted + `[y]/[r]/[s]` — accept, regenerate (re-call the LLM with the same seed), or + skip. +- **`e`** or empty input — finalizes the quiz with the questions added so far. + +`points_possible` is forced to `1` for both placeholder and LLM-generated +questions; the instructor can refine wording, distractors, and points later in +the Canvas UI. The quiz is created with default settings: +`quiz_type='assignment'`, `time_limit=30`, `one_question_at_a_time=True`, +`cant_go_back=True`, `shuffle_answers=True`. + +The cloud text model is lazy-loaded only on the first `g` choice, so a +pure-placeholder run still works without `OLLAMA_API_KEY` set. | | Files | |---|---| @@ -659,3 +686,105 @@ every send is reviewed before going to Canvas. Use `--dry-run` to preview all messages (recipient, subject, body, and reason) without sending anything to Canvas and without the interactive prompt. + +--- + +#### Media-recording check-in workflow + +Three tasks form an end-to-end loop for short audio check-ins where each +student records a brief reflection (e.g. *"What was hardest on this week's +quiz?"*), the recording is downloaded and transcribed locally, the instructor +grades each submission, and the cohort's transcripts are then summarized +against the quiz topics: + +1. `create-media-recording-assignment` — create the assignment on Canvas. +2. `get-media-recordings` — fetch and transcribe each submission, and grade. +3. `analyze-media-recordings` — surface tag coverage and recurring themes + across the cohort. + +Audio handling is local-only: transcripts are produced via the local +`OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`) and the cohort analysis runs +against the local `OLLAMA_MODEL` (default `gemma4:31b`). Student audio never +leaves your machine. + +--- + +#### `create-media-recording-assignment` — Create a media-recording assignment on Canvas + +Interactively creates a Canvas assignment with `submission_types=['media_recording']`. +Prompts for the title, prompt body (becomes the HTML `description`), +`points_possible` (default 1), an optional ISO `due_at`, and whether to publish +immediately (default unpublished so you can review before exposing it to +students). Logs the new assignment's ID and Canvas URL on success. + +| | Files | +|---|---| +| **Input** | _(none — interactive prompts only)_ | +| **Output** | _(none — assignment is created directly on Canvas)_ | + +--- + +#### `get-media-recordings` — Fetch, transcribe, and grade student audio submissions + +Pulls every media-recording submission for a chosen assignment, re-encodes the +audio to 16 kHz mono PCM WAV (via ffmpeg directly on Canvas's playback URL — +DASH manifests are handled in one pass), and transcribes each WAV locally +using `OLLAMA_AUDIO_MODEL` (default `gemma4:e4b`). Audio files are saved to +`data//media_recordings/assignment/_.wav`. + +By default, after each transcription the transcript is displayed and the +instructor is prompted for the points to award: + +- **Enter** — award `points_possible` (full credit). +- **A number** — award that value. +- **`s` / `skip`** — leave the submission ungraded. + +The chosen value is posted via Canvas's `Submission.edit({'posted_grade': ...})`. + +Pass `--auto-grade` to skip the per-student review and award `points_possible` +to every submitter automatically. Combine with `--dry-run` to preview the +auto-graded writes without mutating Canvas; in dry-run, transcripts are still +shown and the CSV is still written, but the grade write is suppressed. + +Pressing Ctrl-C mid-loop flushes a partial CSV before re-raising, so review +progress is never lost. + +| | Files | +|---|---| +| **Input** | _(Canvas — selected interactively)_ | +| **Output** | `data//assignment_recordings_YYYYMMDD.csv` — columns: `student_id`, `student_name`, `submission_id`, `submitted_at`, `audio_path`, `transcript`, `transcribed_at`, `grade`, `graded_at` | +| | `data//media_recordings/assignment/_.wav` — one WAV per submission | +| **Canvas side-effect** | Posts `posted_grade` on each graded submission (skipped in `--dry-run` mode) | + +--- + +#### `analyze-media-recordings` — Summarize cohort transcripts against quiz tags + +Loads the most recent `assignment_recordings_*.csv` produced by +`get-media-recordings`, prompts the instructor to pick a `*_questions_w_tags_*.csv` +from the same course directory (so tag vocabulary stays grounded in actual +quiz content), then runs two local-LLM passes against `OLLAMA_MODEL` (default +`gemma4:31b`): + +1. **Tag classification** — one call per non-empty transcript, asking which of + the unique tags from the quiz CSV's `keywords` column the student is + actually discussing. Paraphrase-tolerant, so *"I got confused on the linked + list one"* maps to `linked lists` even when the literal substring isn't + present. +2. **Theme extraction** — a single cohort-level call that surfaces 3–5 + recurring themes across all transcripts, with the tag list as background + context. + +The output is a Markdown report with three sections: a tag-grounded table +ranked by descending student count (the *"specialized word cloud"*), the +LLM-extracted themes, and a roster mapping the 1-based student indices used +in the themes section back to names. + +**Prerequisites**: run `get-media-recordings` (to produce the recordings CSV) +and `get-quiz-questions --tag` (to produce the tags CSV) first. + +| | Files | +|---|---| +| **Input** | `data//assignment_recordings_YYYYMMDD.csv` (from `get-media-recordings`) | +| | `data//__questions_w_tags_YYYYMMDD.csv` (from `get-quiz-questions --tag`) | +| **Output** | `data//assignment_analysis_YYYYMMDD.md` — Markdown report | diff --git a/canvigator_course.py b/canvigator_course.py index da9db2b..74030a6 100644 --- a/canvigator_course.py +++ b/canvigator_course.py @@ -154,7 +154,7 @@ def sendAllQuizReminders(self, dry_run=False): _saveCourseReminderManifest(sent_messages, student_state, subject_str, dry_run, self.config.data_path) def createQuiz(self): - """Create a new placeholder quiz with stub questions on Canvas.""" + """Create a new quiz on Canvas, with each question chosen as a placeholder or LLM-generated draft.""" title = input("Enter quiz title: ").strip() if not title: print("Quiz title cannot be empty.") @@ -175,24 +175,53 @@ def createQuiz(self): logger.info(f"Created quiz: '{quiz.title}' (id={quiz.id})") position = 1 - print("\nAdd placeholder questions (press Enter on an empty line to finish):") + print("\nAdd questions (choose [e] or press Enter at the prompt to finish):") + cl = None # lazy-imported canvigator_llm module + llm_client = None + llm_model = None while True: - description = input(f" Q{position} description: ").strip() - if not description: + choice = input(f" Q{position} — [p]laceholder, [g]enerate w/ LLM, [e]nd quiz: ").strip().lower() + if not choice or choice == 'e': break - quiz.create_question(question={ - 'question_name': f"Question {position}", - 'question_text': description, - 'question_type': 'multiple_choice_question', - 'points_possible': 1, - 'position': position, - }) - print(f" Added Q{position}: {description}") - logger.info(f" Added question {position}: {description}") + if choice not in ('p', 'g'): + print(" Please enter 'p', 'g', or 'e' (or Enter to end).") + continue + + if choice == 'p': + description = input(f" Q{position} description: ").strip() + if not description: + continue + question_dict = { + 'question_name': f"Question {position}", + 'question_text': description, + 'question_type': 'multiple_choice_question', + 'points_possible': 1, + 'position': position, + } + else: + seed = input(f" Q{position} prompt for LLM: ").strip() + if not seed: + continue + if cl is None: + import canvigator_llm as cl + try: + llm_client = cl._make_client(cloud=True) + except RuntimeError as e: + print(f" ERROR: {e}") + cl = None + continue + llm_model = cl.DEFAULT_TEXT_MODEL + question_dict = _reviewGeneratedQuestion(cl, llm_client, llm_model, seed, position) + if question_dict is None: + continue + + quiz.create_question(question=question_dict) + print(f" Added Q{position} ({question_dict['question_type']}): {question_dict['question_name']}") + logger.info(f" Added question {position} ({question_dict['question_type']}): {question_dict['question_name']}") position += 1 n_questions = position - 1 - print(f"\nQuiz '{title}' created with {n_questions} placeholder question{'s' if n_questions != 1 else ''}.") + print(f"\nQuiz '{title}' created with {n_questions} question{'s' if n_questions != 1 else ''}.") logger.info(f"Quiz '{title}' complete: {n_questions} questions") def exportRoster(self, data_path): @@ -1113,3 +1142,81 @@ def exportAnonymizedData(data_path): n_total = len(csv_files) print(f"Exported {n_total} file{'s' if n_total != 1 else ''} ({n_anonymized} anonymized) to {anon_dir.name}/") logger.info(f"Exported {n_total} anonymized files to {zip_path}.zip") + + +def _renderQuestionPreview(question_dict): + """Print a compact human-readable preview of an LLM-generated question dict.""" + qtype = question_dict.get('question_type', '?') + name = question_dict.get('question_name', '') + text = question_dict.get('question_text', '') + print(f" type: {qtype}") + print(f" name: {name}") + print(f" text: {text}") + + answers = question_dict.get('answers') or [] + if qtype == 'matching_question': + for a in answers: + left = a.get('answer_match_left', '') if isinstance(a, dict) else '' + right = a.get('answer_match_right', '') if isinstance(a, dict) else '' + print(f" match: {left} -> {right}") + distractors = question_dict.get('matching_answer_incorrect_matches') + if distractors: + print(f" distractors: {distractors!r}") + elif qtype == 'calculated_question': + for v in question_dict.get('variables') or []: + print(f" var: {v}") + for f in question_dict.get('formulas') or []: + print(f" formula: {f}") + dp = question_dict.get('formula_decimal_places') + if dp is not None: + print(f" decimals: {dp}") + else: + for a in answers: + if not isinstance(a, dict): + continue + mark = '*' if a.get('answer_weight') == 100 else ' ' + blank = a.get('blank_id') + label = f"[{blank}] " if blank else "" + print(f" {mark} {label}{a.get('answer_text', '')}") + + +def _reviewGeneratedQuestion(cl, client, model, seed, position): + """Generate a question via the LLM, preview it, and prompt the instructor [y]/[r]/[s]. + + Rejected drafts are accumulated and passed back to the LLM on each + [r]egenerate so successive drafts diverge instead of recycling the same + angle. Returns the finalized Canvas question dict (with points_possible=1 + and position stamped) on accept, or None on skip / generation failure. + """ + rejected = [] + while True: + label = f" (regenerate, {len(rejected)} prior)" if rejected else "" + print(f" Generating Q{position} via {model}{label}...") + question_dict = cl.generate_quiz_question( + seed, client=client, model=model, prior_drafts=rejected or None, + ) + if question_dict is None: + print(" ERROR: generation failed or response was invalid. Skipping.") + return None + + print(f"\n --- Q{position} preview ---") + _renderQuestionPreview(question_dict) + print(" -------------------") + + while True: + choice = input(" Accept this question? [y]es / [r]egenerate / [s]kip: ").strip().lower() + if choice in ('y', 'r', 's'): + break + print(" Please enter 'y', 'r', or 's'.") + + if choice == 's': + return None + if choice == 'r': + rejected.append(question_dict) + continue + # 'y' — finalize + if not question_dict.get('question_name'): + question_dict['question_name'] = f"Question {position}" + question_dict['points_possible'] = 1 + question_dict['position'] = position + return question_dict diff --git a/canvigator_llm.py b/canvigator_llm.py index 3dfa3c2..4867175 100644 --- a/canvigator_llm.py +++ b/canvigator_llm.py @@ -1555,3 +1555,220 @@ def extract_recording_themes(transcripts, tags, client, model): except Exception as e: logger.warning(f"Recording themes extraction failed: {e}") return f"_(theme extraction failed: {e})_" + + +# Auto-gradable Canvas question types we let the LLM emit. Open-ended types +# (essay_question, file_upload_question, short_answer_question, etc.) are +# excluded since they require manual grading. +QUIZ_QUESTION_TYPES = ( + "calculated_question", + "fill_in_multiple_blanks_question", + "matching_question", + "multiple_answers_question", + "multiple_choice_question", + "multiple_dropdowns_question", + "true_false_question", +) + +_QUIZ_QUESTION_SYSTEM_PROMPT = ( + "You are an expert university instructor drafting auto-gradable Canvas quiz questions. " + "Given an instructor's natural-language prompt, produce ONE complete quiz question as a " + "single JSON object — no prose, no markdown fences, no commentary.\n\n" + "Pick the question_type that best fits the prompt from this exact list:\n" + " - multiple_choice_question (one correct answer among 4)\n" + " - multiple_answers_question (two or more correct answers among 4-6)\n" + " - true_false_question (single correct answer: True or False)\n" + " - fill_in_multiple_blanks_question (question_text contains [blank_id] tokens)\n" + " - multiple_dropdowns_question (question_text contains [dropdown_id] tokens)\n" + " - matching_question (left/right pairs)\n" + " - calculated_question (numeric formula with variables)\n\n" + "JSON schema (only include fields relevant to the chosen type):\n" + "{\n" + " \"question_type\": \"\",\n" + " \"question_name\": \"\",\n" + " \"question_text\": \"\",\n" + " \"answers\": [ ... ] # see per-type rules below\n" + "}\n\n" + "Per-type answers schema:\n" + " multiple_choice_question / multiple_answers_question / true_false_question:\n" + " [{\"answer_text\": \"...\", \"answer_weight\": 100|0}, ...] " + "(weight 100 = correct, 0 = incorrect)\n" + " fill_in_multiple_blanks_question:\n" + " [{\"blank_id\": \"name\", \"answer_text\": \"...\", \"answer_weight\": 100}, ...] " + "and question_text MUST contain matching [name] tokens\n" + " multiple_dropdowns_question:\n" + " [{\"blank_id\": \"name\", \"answer_text\": \"...\", \"answer_weight\": 100|0}, ...] " + "and question_text MUST contain matching [name] tokens\n" + " matching_question:\n" + " [{\"answer_match_left\": \"...\", \"answer_match_right\": \"...\"}, ...] " + "and you may also include \"matching_answer_incorrect_matches\": " + "\"distractor1\\ndistractor2\" (newline-separated string)\n" + " calculated_question:\n" + " [] (empty list); also include \"variables\": [{\"name\": \"x\", \"min\": 1, \"max\": 10, " + "\"scale\": 0}, ...] and \"formulas\": [{\"formula\": \"x*2\"}], plus " + "\"formula_decimal_places\": 0\n\n" + "Rules:\n" + " - Output ONLY the JSON object — no ```json fences, no leading/trailing text.\n" + " - Do NOT include points_possible (the caller fixes points to 1).\n" + " - Write content suitable for a college-level course.\n" + " - Prefer correctness and completeness over polish — the instructor will refine in Canvas.\n" + " - For multiple_choice_question, supply exactly 4 answers with exactly 1 weighted 100.\n" + " - For multiple_answers_question, supply 4-6 answers with 2 or more weighted 100.\n" + " - For true_false_question, supply exactly 2 answers (\"True\" weighted 100 and \"False\" " + "weighted 0, or vice-versa).\n" +) + + +def _summarize_draft_for_prompt(draft): + """Render a previously-generated draft as a compact summary for the prompt.""" + if not isinstance(draft, dict): + return "" + qtype = draft.get("question_type", "?") + qtext = draft.get("question_text", "") + parts = [f" Type: {qtype}", f" Question: {qtext}"] + answers = draft.get("answers") or [] + if qtype == "matching_question": + for a in answers: + if isinstance(a, dict): + left = a.get("answer_match_left", "") + right = a.get("answer_match_right", "") + parts.append(f" Match: {left} -> {right}") + elif qtype == "calculated_question": + for v in draft.get("variables") or []: + parts.append(f" Variable: {v}") + for f in draft.get("formulas") or []: + parts.append(f" Formula: {f}") + else: + for a in answers: + if not isinstance(a, dict): + continue + mark = "*" if a.get("answer_weight") == 100 else "-" + blank = a.get("blank_id") + label = f"[{blank}] " if blank else "" + parts.append(f" {mark} {label}{a.get('answer_text', '')}") + return "\n".join(parts) + + +def _build_quiz_question_prompt(user_seed, prior_drafts=None): + """Wrap the instructor's natural-language seed into the user message. + + If ``prior_drafts`` is non-empty, append a section listing the rejected + drafts and instructing the model to produce a substantively different + question — used by the [r]egenerate flow so successive drafts diverge. + """ + seed = (user_seed or "").strip() + parts = [f"Instructor prompt: {seed}"] + if prior_drafts: + parts.append("") + parts.append( + "The instructor REJECTED the following draft(s) on this topic. " + "Produce a SUBSTANTIVELY DIFFERENT new draft — different scenario, " + "different numbers/values, different angle, different specific examples, " + "and ideally a different question_type if the topic supports it. " + "Do NOT paraphrase, trivially modify, or repeat any of these:" + ) + for i, draft in enumerate(prior_drafts, start=1): + summary = _summarize_draft_for_prompt(draft) + if summary: + parts.append(f"\nRejected draft {i}:\n{summary}") + parts.append("") + parts.append("Produce the JSON object now.") + return "\n".join(parts) + + +def _parse_quiz_question(content): + """Parse the LLM's JSON-only response into a Canvas-shaped question dict, or None. + + Tolerates accidental ```json fences and prose around the object. Validates + that question_type is in the allowlist and that question_text is non-empty. + Strips any points_possible the model emitted (the caller forces 1). + """ + if not content: + return None + + text = content.strip() + text = re.sub(r'^```(?:json)?\s*', '', text, flags=re.IGNORECASE) + text = re.sub(r'\s*```\s*$', '', text) + brace_start = text.find('{') + brace_end = text.rfind('}') + if brace_start != -1 and brace_end != -1 and brace_end > brace_start: + text = text[brace_start:brace_end + 1] + + try: + data = json.loads(text) + except (ValueError, TypeError): + logger.warning(f"Quiz-question JSON parse failed; raw response starts with: {content[:200]!r}") + return None + if not isinstance(data, dict): + return None + + qtype = data.get("question_type") + if qtype not in QUIZ_QUESTION_TYPES: + logger.warning(f"Quiz-question rejected: unknown question_type {qtype!r}") + return None + + qtext = data.get("question_text") + if not isinstance(qtext, str) or not qtext.strip(): + logger.warning("Quiz-question rejected: missing or empty question_text") + return None + + name = data.get("question_name") + if not isinstance(name, str) or not name.strip(): + name = "Generated question" + + answers = data.get("answers") + if answers is None: + answers = [] + if not isinstance(answers, list): + logger.warning(f"Quiz-question rejected: answers is not a list ({type(answers).__name__})") + return None + + out = dict(data) + out["question_type"] = qtype + out["question_name"] = name.strip() + out["question_text"] = qtext.strip() + out["answers"] = answers + out.pop("points_possible", None) + return out + + +def generate_quiz_question(user_seed, client=None, model=None, prior_drafts=None): + """Cloud-LLM call: turn an instructor's natural-language seed into a Canvas question dict. + + When ``prior_drafts`` is provided (a list of previously rejected question + dicts), the prompt asks for a substantively different draft and the + sampling temperature is bumped from 0.4 to 0.8 to encourage variety + across the [r]egenerate loop. + + Returns the validated dict (without points_possible/position — the caller + stamps those) or None if the model failed or produced an invalid payload. + """ + if not user_seed or not user_seed.strip(): + return None + try: + import ollama # noqa: F401 + except ImportError as e: + raise RuntimeError( + "The 'ollama' package is required for question generation. Install with: pip install ollama" + ) from e + + if client is None: + client = _make_client(cloud=True) + model = model or DEFAULT_TEXT_MODEL + + prompt = _build_quiz_question_prompt(user_seed, prior_drafts=prior_drafts) + temperature = 0.8 if prior_drafts else 0.4 + try: + resp = _chat_with_retry( + client, + model=model, + messages=[ + {"role": "system", "content": _QUIZ_QUESTION_SYSTEM_PROMPT}, + {"role": "user", "content": prompt}, + ], + options={"temperature": temperature}, + ) + return _parse_quiz_question(resp["message"]["content"]) + except Exception as e: + logger.warning(f"LLM quiz-question generation failed: {e}") + return None diff --git a/test_canvigator.py b/test_canvigator.py index 51b1565..7385eb5 100644 --- a/test_canvigator.py +++ b/test_canvigator.py @@ -2770,4 +2770,189 @@ def test_build_themes_prompt_numbers_students(self): prompt = _build_themes_prompt(['first', 'second'], ['graphs']) assert 'Student 1: first' in prompt assert 'Student 2: second' in prompt - assert 'graphs' in prompt + + +# --------------------------------------------------------------------------- +# canvigator_llm quiz-question generation helpers +# --------------------------------------------------------------------------- + +class TestQuizQuestionHelpers: + """Tests for _build_quiz_question_prompt and _parse_quiz_question.""" + + def test_build_quiz_question_prompt_includes_seed(self): + """The instructor's seed appears verbatim in the user prompt.""" + from canvigator_llm import _build_quiz_question_prompt + seed = "a question on Big-O for binary search" + prompt = _build_quiz_question_prompt(seed) + assert seed in prompt + assert "JSON" in prompt + + def test_build_quiz_question_prompt_handles_empty(self): + """Empty/None seed still returns a string (the wrapper text).""" + from canvigator_llm import _build_quiz_question_prompt + assert isinstance(_build_quiz_question_prompt(""), str) + assert isinstance(_build_quiz_question_prompt(None), str) + + def test_parse_quiz_question_accepts_multiple_choice(self): + """A well-formed multiple_choice_question payload round-trips and points_possible is stripped.""" + import json as _json + from canvigator_llm import _parse_quiz_question + payload = _json.dumps({ + "question_type": "multiple_choice_question", + "question_name": "Binary search complexity", + "question_text": "What is the time complexity of binary search?", + "points_possible": 5, + "answers": [ + {"answer_text": "O(log n)", "answer_weight": 100}, + {"answer_text": "O(n)", "answer_weight": 0}, + {"answer_text": "O(n log n)", "answer_weight": 0}, + {"answer_text": "O(1)", "answer_weight": 0}, + ], + }) + result = _parse_quiz_question(payload) + assert result is not None + assert result["question_type"] == "multiple_choice_question" + assert result["question_name"] == "Binary search complexity" + assert "points_possible" not in result + assert len(result["answers"]) == 4 + + def test_parse_quiz_question_handles_markdown_fence(self): + """JSON wrapped in ```json fences is tolerated.""" + from canvigator_llm import _parse_quiz_question + wrapped = ( + "```json\n" + '{"question_type": "true_false_question", "question_name": "T/F",' + ' "question_text": "Quicksort is stable.",' + ' "answers": [{"answer_text": "True", "answer_weight": 0},' + ' {"answer_text": "False", "answer_weight": 100}]}' + "\n```" + ) + result = _parse_quiz_question(wrapped) + assert result is not None + assert result["question_type"] == "true_false_question" + assert len(result["answers"]) == 2 + + def test_parse_quiz_question_rejects_unknown_type(self): + """Unknown question_type returns None.""" + import json as _json + from canvigator_llm import _parse_quiz_question + payload = _json.dumps({ + "question_type": "essay_question", + "question_name": "Discuss", + "question_text": "Discuss recursion.", + "answers": [], + }) + assert _parse_quiz_question(payload) is None + + def test_parse_quiz_question_rejects_missing_text(self): + """Missing or empty question_text returns None.""" + import json as _json + from canvigator_llm import _parse_quiz_question + payload = _json.dumps({ + "question_type": "multiple_choice_question", + "question_name": "X", + "answers": [], + }) + assert _parse_quiz_question(payload) is None + + def test_parse_quiz_question_allows_empty_answers_for_calculated(self): + """A calculated_question with formulas/variables and an empty answers list is accepted.""" + import json as _json + from canvigator_llm import _parse_quiz_question + payload = _json.dumps({ + "question_type": "calculated_question", + "question_name": "Doubling", + "question_text": "What is [x] doubled?", + "answers": [], + "variables": [{"name": "x", "min": 1, "max": 10, "scale": 0}], + "formulas": [{"formula": "x*2"}], + "formula_decimal_places": 0, + }) + result = _parse_quiz_question(payload) + assert result is not None + assert result["question_type"] == "calculated_question" + assert result["formulas"] == [{"formula": "x*2"}] + + def test_parse_quiz_question_defaults_missing_name(self): + """A payload without question_name gets a fallback name.""" + import json as _json + from canvigator_llm import _parse_quiz_question + payload = _json.dumps({ + "question_type": "multiple_choice_question", + "question_text": "Pick one.", + "answers": [{"answer_text": "A", "answer_weight": 100}], + }) + result = _parse_quiz_question(payload) + assert result is not None + assert result["question_name"] == "Generated question" + + def test_parse_quiz_question_handles_malformed_json(self): + """Invalid JSON returns None rather than raising.""" + from canvigator_llm import _parse_quiz_question + assert _parse_quiz_question("not json at all") is None + assert _parse_quiz_question("") is None + assert _parse_quiz_question(None) is None + + def test_summarize_draft_renders_multiple_choice(self): + """A multiple-choice draft summary marks the correct answer with '*' and lists distractors with '-'.""" + from canvigator_llm import _summarize_draft_for_prompt + draft = { + "question_type": "multiple_choice_question", + "question_text": "What is 2+2?", + "answers": [ + {"answer_text": "3", "answer_weight": 0}, + {"answer_text": "4", "answer_weight": 100}, + {"answer_text": "5", "answer_weight": 0}, + ], + } + summary = _summarize_draft_for_prompt(draft) + assert "What is 2+2?" in summary + assert "* 4" in summary + assert "- 3" in summary + assert "- 5" in summary + + def test_summarize_draft_renders_matching(self): + """A matching-question draft summary lists 'left -> right' pairs.""" + from canvigator_llm import _summarize_draft_for_prompt + draft = { + "question_type": "matching_question", + "question_text": "Match terms.", + "answers": [ + {"answer_match_left": "Tree", "answer_match_right": "Hierarchical"}, + {"answer_match_left": "Graph", "answer_match_right": "Network"}, + ], + } + summary = _summarize_draft_for_prompt(draft) + assert "Tree -> Hierarchical" in summary + assert "Graph -> Network" in summary + + def test_summarize_draft_handles_non_dict(self): + """Non-dict input returns an empty string rather than raising.""" + from canvigator_llm import _summarize_draft_for_prompt + assert _summarize_draft_for_prompt(None) == "" + assert _summarize_draft_for_prompt("not a dict") == "" + + def test_build_quiz_question_prompt_no_prior_drafts(self): + """Without prior_drafts the prompt has no rejected-drafts section.""" + from canvigator_llm import _build_quiz_question_prompt + prompt = _build_quiz_question_prompt("a Big-O question", prior_drafts=None) + assert "REJECTED" not in prompt + assert "SUBSTANTIVELY DIFFERENT" not in prompt + assert "a Big-O question" in prompt + + def test_build_quiz_question_prompt_with_prior_drafts(self): + """With prior_drafts the prompt instructs the model to diverge and includes draft summaries.""" + from canvigator_llm import _build_quiz_question_prompt + prior = [{ + "question_type": "multiple_choice_question", + "question_text": "What is the time complexity of binary search?", + "answers": [ + {"answer_text": "O(log n)", "answer_weight": 100}, + {"answer_text": "O(n)", "answer_weight": 0}, + ], + }] + prompt = _build_quiz_question_prompt("a Big-O question", prior_drafts=prior) + assert "REJECTED" in prompt + assert "SUBSTANTIVELY DIFFERENT" in prompt + assert "What is the time complexity of binary search?" in prompt + assert "Rejected draft 1" in prompt