Skip to content

prep-feature: ask_user bundles multiple questions but options only cover one #18

@samkeen

Description

@samkeen

What happened

During a tilth prep-feature interview, the seeder asked two questions in a single ask_user call but the option list only enumerated answers for the first one:

? What should happen on success — silent exit (0), or a confirmation message like "Added: remember the milk"? And should empty
todo strings be rejected with an error?
  1) Silent success (exit 0, no output)
  2) Confirmation message to stdout
  3) Confirmation to stderr
  0) Other (I'll specify)
>

The three options answer "what should happen on success" — none of them speaks to "should empty strings be rejected." The user is forced to either:

  1. Pick an option and silently drop the second question (the seeder then bakes in whatever default it imagined for empty-string handling).
  2. Hit 0) Other and free-form an answer to both questions, which defeats the purpose of the multi-choice anchoring.

Either way, the load-bearing decision about empty-string handling either gets buried as an open_questions entry or quietly assumed.

Why this is happening

tilth/seed/prompts.md currently has:

One question per turn; follow up freely when an answer surfaces a new question or contradicts the codebase.

That's correct but easy for the model to interpret loosely — "one topic per turn" rather than "one answer-slot per turn." Compound questions joined by "and" / "or" still feel like one topic to the model, especially when the second clause is a short clarifier.

The §3 ("Anchored interview") guidance for ask_user with options says options "must be concrete and grounded in actual code" but doesn't address the compound-question failure mode.

Suggested fix

Sharpen the rule in tilth/seed/prompts.md §3. Two candidate phrasings:

  • Strict: "One question, one answer slot. If the user's answer can't be expressed by picking exactly one option (or one free-form sentence), the question is two questions — split it across turns."
  • Mechanical: "Before sending ask_user, re-read your question text and count how many distinct decisions the user has to make to fully answer it. If the count is greater than 1, split into separate ask_user calls. Never ask 'X and Y?' with options that only enumerate answers to X."

Probably worth adding a worked counter-example (the snippet above is a good one) to anchor what the failure looks like.

Adjacent thought (not in scope for this fix)

The frontend has no enforcement for this — it accepts whatever the model passes. We could harden it by lint-checking the question text (e.g. reject if it contains " and " plus a question mark in the second clause), but prompt-side guidance is the right starting point because the harder rule (don't bury load-bearing decisions in open_questions) is also prompt-side.

Related

  • The open_questions discipline in §5 — load-bearing decisions must be asked, not logged. A compound question that drops half its content into the seeder's mental model is the quiet version of the same bug open_questions exists to prevent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions