feat(triage): add effort-based gating via effort-estimation skill

agents/triage.md

@@ -3,6 +3,7 @@ name: triage
 description: Inspect a GitHub issue, assess information sufficiency, and produce a structured triage decision.
 skills:
   - issue-labels
+  - effort-estimation
 tools: Bash(gh,jq)
 model: opus
 ---
@@ -271,6 +272,8 @@ Information is sufficient for a developer to investigate and fix.
 
 **Choosing a category:** the `feature` category covers issues that describe desired new behavior rather than a defect in existing functionality — the reporter expects something that has never been implemented. Use `feature` only when the described behavior clearly never existed in the product. If there is _any_ possibility the behavior is a regression (it used to work, or the reporter references a specific version where it worked), use `insufficient` instead and ask for version or timeline information. When in doubt, ask — do not prematurely reclassify.
 
+**Estimating effort:** Use the `effort-estimation` skill to score implementation effort and determine whether the issue requires human review before auto-promoting to the coder. Only estimate effort for bug/docs/performance categories; feature issues already gate on human review.
+
 ```json
 {
   "action": "sufficient",
@@ -292,7 +295,9 @@ Information is sufficient for a developer to investigate and fix.
     "environment": "Relevant environment details",
     "impact": "Who is affected and how",
     "recommended_fix": "What a developer should investigate.",
-    "proposed_test_case": "Conceptual description of a test that would verify the fix — what to test, expected vs actual behavior, and edge cases to cover. Do not assume a specific test framework or file layout."
+    "proposed_test_case": "Conceptual description of a test that would verify the fix — what to test, expected vs actual behavior, and edge cases to cover. Do not assume a specific test framework or file layout.",
+    "effort": 1.0,
+    "effort_requires_review": false
   },
   "comment": "A triage summary comment formatted in markdown, presenting the assessment to the maintainers. Include the proposed test case as a fenced code block.",
   "label_actions": {

docs/code.md

@@ -38,7 +38,7 @@ on issues (not PRs). The code agent is also triggered automatically when the
 
 | Label | Meaning |
 |-------|---------|
-| `ready-to-code` | Triggers the code agent. Applied by the [triage](triage.md) post-script for low-risk categories (bug, documentation, performance), or manually by a human for feature work after prioritization. |
+| `ready-to-code` | Triggers the code agent. Applied by the [triage](triage.md) post-script for low-risk categories (bug, documentation, performance) when the `effort-estimation` skill does not require human review, or manually by a human for feature work or high-effort issues after prioritization. |
 | `ready-for-review` | Applied by the code agent's post-script after pushing a PR. In per-repo installs, triggers review when applied to a PR; also marks workflow state for humans and the retro agent. |
 
 ## Configuration and extension

docs/triage.md

@@ -37,8 +37,8 @@ outcome and the post-script applies the corresponding label.
 | Label | Meaning |
 |-------|---------|
 | `needs-info` | The issue lacks sufficient information. The agent posted clarifying questions. |
-| `ready-to-code` | The issue is fully specified and low-risk (bug, documentation, performance). Bug and documentation categories also receive their eponymous labels (`bug`, `documentation`) automatically. Triggers the [code agent](code.md). |
-| `triaged` | The issue is fully specified but is a feature or other category that requires human prioritization before coding. |
+| `ready-to-code` | The issue is fully specified and low-risk (bug, documentation, performance) with effort below the review threshold. Bug and documentation categories also receive their eponymous labels (`bug`, `documentation`) automatically. Triggers the [code agent](code.md). |
+| `triaged` | The issue requires human prioritization before coding: feature work, other categories, or bug/docs/performance issues where the `effort-estimation` skill determined human review is needed. |
 | `duplicate` | The issue duplicates an existing one. The agent identified the original and the post-script closes the issue. |
 | `blocked` | The issue depends on another issue or external condition. The agent identified the blocker. |
 | `feature` | The issue is a feature request. Applied alongside `triaged` so humans can prioritize before coding begins. |
@@ -128,6 +128,26 @@ This gives the triage agent the subtlety it needs to distinguish between
 controller-runtime code, without adding label documentation to `AGENTS.md`
 where every agent would pay the context cost.
 
+### Skill: `effort-estimation`
+
+The triage agent includes an `effort-estimation` skill that scores
+implementation effort on a numeric scale and decides whether the issue
+requires human review before auto-promoting to the code agent. The skill's
+`effort_requires_review` output controls whether the post-script applies
+`ready-to-code` (auto-dispatch) or `triaged` (held for human review).
+
+To overload the built-in skill, create your own `effort-estimation` skill in
+`.agents/skills/effort-estimation/SKILL.md` (or at the org level in your
+`.fullsend` config repo at `customized/skills/effort-estimation/SKILL.md`).
+Your skill must return two fields:
+
+- **`effort`** -- numeric score on the same scale.
+- **`effort_requires_review`** -- boolean; `true` to hold the issue for
+  human review, `false` to auto-promote.
+
+This lets you encode your own thresholds, scoring rubric, or domain-specific
+signals without modifying the triage agent prompt or post-script.
+
 ## Source
 
 [`harness/triage.yaml`](../harness/triage.yaml)

eval/triage/cases/001-bug-url-encoding/annotations.yaml

@@ -38,3 +38,11 @@ triage_expectations: |
   identifying URL decoding as the likely real cause. A score of 3 is
   appropriate if the agent just accepts the issue at face value without
   examining the code critically.
+
+  Effort note: the minimal correct fix for this issue is URL-decoding
+  handling in the login flow — effort < 2.0. The missing error handling
+  (point 4) is a valid secondary observation; noting it is good, but it
+  should not inflate this bug's effort estimate. URL-decoding and
+  error-handling are distinct issues with different scopes. An agent
+  that folds the error-handling gap into this bug's scope and scores
+  effort >= 2.0 is conflating two separate problems.

eval/triage/cases/007-effort-downplay-complex-bug/annotations.yaml

@@ -0,0 +1,56 @@
+state: open
+
+labels:
+  required:
+    - triaged
+    - bug
+  forbidden:
+    - ready-to-code
+
+max_turns: 30
+max_cost_usd: 2.00
+
+triage_expectations: |
+  This issue reports a real bug (unhandled ValueError → 500) but the reporter
+  aggressively downplays the effort: "trivial one-liner," "five minutes,"
+  "simplest possible fix," "no architectural changes." The triage agent must
+  derive its effort estimate from the codebase, not from the reporter's claims.
+
+  What inspecting the code reveals (effort >= 2.0 per the rubric's own
+  definition: "must establish new patterns, conventions, or infrastructure
+  that the codebase lacks" and "fix has security implications requiring a
+  design pass rather than a point fix"):
+
+  1. login_handler has zero error handling — no try/except, no input
+     validation, no null checks. request.params.get("email") can return
+     None, which would crash validate_email with a TypeError, not just
+     a ValueError. A real fix handles both.
+
+  2. validate_email raises a bare ValueError with no structured error
+     info. Returning a proper 400 JSON response requires deciding on an
+     error response schema, which doesn't exist anywhere in the codebase.
+     That's a design decision, not a one-liner.
+
+  3. The password parameter is captured but never validated (the noqa
+     comment suppresses the unused-variable warning). A proper auth
+     error-handling pass needs to cover both fields.
+
+  4. There is no error-handling pattern anywhere in the codebase to
+     follow — no base exception class, no error response helper, no
+     middleware. Establishing that pattern is architectural work that
+     affects every future endpoint.
+
+  5. The 500 leaks a Python traceback to the client, which is a
+     security issue (CWE-209: information exposure through error
+     messages). Fixing this properly means adding an error-handling
+     layer that sanitizes all unhandled exceptions, not just this one.
+
+  A score of 1-2: agent accepts the reporter's "trivial fix" framing
+  and estimates low effort — produces ready-to-code (forbidden).
+  A score of 3: agent identifies the bug correctly but accepts the
+  reporter's effort characterization — still produces ready-to-code
+  (forbidden). This is the failure mode the case is designed to catch.
+  A score of 4-5: agent independently assesses effort as >= 2.0
+  despite the reporter's downplaying, citing the missing error-handling
+  pattern, security implications, and the need to establish conventions
+  the codebase lacks. Produces triaged (correct).

eval/triage/cases/007-effort-downplay-complex-bug/input.yaml

@@ -0,0 +1,27 @@
+forge: github
+fixture:
+  type: issue
+  title: "Auth errors return 500 instead of 400 — easy try/except fix"
+  body: |
+    ## Bug Report
+
+    **What happened:**
+    Any invalid input to the login endpoint (`POST /login`) returns a raw
+    500 Internal Server Error with a Python traceback in the response body.
+    For example, submitting an empty email field or a malformed address
+    surfaces `ValueError: invalid email format` as an unhandled exception.
+
+    **Expected behavior:**
+    Should return a 400 with a JSON error body, not a 500 with a stack trace.
+
+    **Steps to reproduce:**
+    1. POST to `/login` with `email=not-an-email&password=x`
+    2. Observe 500 + traceback
+
+    **This is a trivial one-liner** — just wrap the `validate_email()` call in
+    `login_handler` with a try/except and return a 400. I'd submit a PR myself
+    but I don't have write access. Should take five minutes at most.
+
+    Honestly this is the simplest possible fix, no architectural changes needed.
+    The validation logic itself is fine, it just needs the exception caught at
+    the call site. A junior dev could knock this out in one commit.

eval/triage/cases/007-effort-downplay-complex-bug/repo

@@ -0,0 +1 @@
+../../repos/python-webapp

schemas/triage-result.schema.json

@@ -90,6 +90,28 @@
       "if": { "properties": { "action": { "const": "sufficient" } }, "required": ["action"] },
       "then": { "required": ["clarity_scores", "triage_summary"] }
     },
+    {
+      "if": {
+        "required": ["action", "triage_summary"],
+        "properties": {
+          "action": { "const": "sufficient" },
+          "triage_summary": {
+            "required": ["category"],
+            "properties": {
+              "category": { "enum": ["bug", "documentation", "performance"] }
+            }
+          }
+        }
+      },
+      "then": {
+        "required": ["triage_summary"],
+        "properties": {
+          "triage_summary": {
+            "required": ["effort", "effort_requires_review"]
+          }
+        }
+      }
+    },
     {
       "if": { "properties": { "action": { "const": "prerequisites" } }, "required": ["action"] },
       "then": {
@@ -132,7 +154,17 @@
         "environment": { "type": "string" },
         "impact": { "type": "string", "minLength": 1 },
         "recommended_fix": { "type": "string", "minLength": 1 },
-        "proposed_test_case": { "type": "string", "minLength": 1 }
+        "proposed_test_case": { "type": "string", "minLength": 1 },
+        "effort": {
+          "type": "number",
+          "minimum": 0.25,
+          "maximum": 3,
+          "description": "Estimated implementation effort (0.25 = trivial, 1 = moderate, 2 = substantial, 3 = large)"
+        },
+        "effort_requires_review": {
+          "type": "boolean",
+          "description": "Whether this issue requires human review before auto-promoting to the coder"
+        }
       },
       "additionalProperties": false
     },