What happened
On PR #2098, the review bot flagged removal of subagent_type: Explore as a medium-severity privilege escalation 5 times across 5 separate review runs (Jun 9–Jun 17). The PR title and description explicitly stated that subagent_type was a hallucinated field with no functional effect — confirmed by the Claude Code sub-agent docs. The bot assumed the existing configuration was correct and functional, treating its removal as a security regression without verifying the field actually existed in the Claude Code API. The PR author responded explaining the field was non-functional, but subsequent review runs re-raised the same finding.
What could go better
The security sub-agent should consider the possibility that removed configuration fields were never functional (dead code, hallucinated fields, cargo-culted settings). When a PR explicitly states a field is non-functional and cites external documentation, the sub-agent should weigh that evidence rather than defaulting to 'removal of X = privilege escalation'. This is a high-confidence finding — the same pattern would apply any time someone removes a no-op security annotation, a disabled feature flag, or a misconfigured access control that never enforced anything. The risk of the current behavior is that repeated false positives on the same PR erode trust in the review agent's security findings.
Proposed change
Add guidance to skills/pr-review/sub-agents/security.md in the fullsend-ai/agents repo instructing the sub-agent to: (1) when flagging removal of a configuration field as privilege escalation, verify that the field has a documented functional effect in the relevant platform/tool — if the PR claims the field is non-functional, check the cited documentation; (2) if functionality cannot be confirmed, downgrade the finding to info-level with a note that the field's effect could not be independently verified, rather than raising it as medium/high.
Validation criteria
On a future PR that removes a configuration field claimed to be non-functional, the security sub-agent should either not flag it as privilege escalation, or flag it at info-level with qualification. The 5x repetition of the same false-positive finding should not recur.
Generated by retro agent from fullsend-ai/fullsend#2098
What happened
On PR #2098, the review bot flagged removal of
subagent_type: Exploreas a medium-severity privilege escalation 5 times across 5 separate review runs (Jun 9–Jun 17). The PR title and description explicitly stated thatsubagent_typewas a hallucinated field with no functional effect — confirmed by the Claude Code sub-agent docs. The bot assumed the existing configuration was correct and functional, treating its removal as a security regression without verifying the field actually existed in the Claude Code API. The PR author responded explaining the field was non-functional, but subsequent review runs re-raised the same finding.What could go better
The security sub-agent should consider the possibility that removed configuration fields were never functional (dead code, hallucinated fields, cargo-culted settings). When a PR explicitly states a field is non-functional and cites external documentation, the sub-agent should weigh that evidence rather than defaulting to 'removal of X = privilege escalation'. This is a high-confidence finding — the same pattern would apply any time someone removes a no-op security annotation, a disabled feature flag, or a misconfigured access control that never enforced anything. The risk of the current behavior is that repeated false positives on the same PR erode trust in the review agent's security findings.
Proposed change
Add guidance to
skills/pr-review/sub-agents/security.mdin thefullsend-ai/agentsrepo instructing the sub-agent to: (1) when flagging removal of a configuration field as privilege escalation, verify that the field has a documented functional effect in the relevant platform/tool — if the PR claims the field is non-functional, check the cited documentation; (2) if functionality cannot be confirmed, downgrade the finding to info-level with a note that the field's effect could not be independently verified, rather than raising it as medium/high.Validation criteria
On a future PR that removes a configuration field claimed to be non-functional, the security sub-agent should either not flag it as privilege escalation, or flag it at info-level with qualification. The 5x repetition of the same false-positive finding should not recur.
Generated by retro agent from fullsend-ai/fullsend#2098