Build need-driven project recommendations with Devin final review

[sarthakagrawal927]

Objective

For each high-priority connected project, identify its 5–10 most important
needs, search Starboard's complete eligible repository catalog separately for
every need, generate grounded recommendations, and use a bounded Devin pass to
validate and finalize the result.

The output should be a project intelligence report organized around concrete
needs—not one generic list of similar repositories.

Do not invent needs to fill a quota. Target 5–10, but return fewer when the
available project evidence cannot support five distinct needs.

Product and orchestration boundary

Starboard owns the reusable intelligence system:

• public project context and fingerprints
• repository capability cards
• project needs and evidence
• full-catalog retrieval and deterministic ranking
• draft and reviewed report persistence
• UI, API, and read-only ChatGPT MCP access
• a provider-neutral external-review ingestion contract

Fleet automation owns:

• priority-project selection
• schedules and rerun policy
• Devin credentials, session creation, polling, and spend limits
• verification and submission of structured review results to Starboard

Starboard must remain useful without Devin. Devin is an optional final reviewer,
not a required production dependency or the system of record.

Optimized execution model

1. Precompute repository capability cards

Create one reusable, evidence-backed capability card per catalog repository.
It should summarize purpose, capabilities, language, tools, adoption type,
maintenance signals, embedding references, and provenance.

Compute or refresh this card when a repository is first cataloged or its
approved source fingerprint changes. Do not repeat repository-level analysis
for every priority project.

2. Fingerprint the project and cache its need map

Build a stable project fingerprint from approved evidence such as repository
metadata, README, manifests, detected tools, AI metadata, and public roadmap
signals.

When the fingerprint is unchanged, reuse the existing need map. Each need must
have a stable identifier, priority, current state, desired outcome, constraints,
supporting evidence, and one or more search intents.

Merge overlapping needs and reject unsupported needs instead of fabricating a
fixed count.

3. Search the complete catalog independently per need

For each need:

• generate several focused semantic and lexical search intents
• query the complete eligible Vectorize/FTS catalog
• add structured topic, language, tool, and maintenance signals
• deduplicate candidates while retaining every matching need and evidence path
• apply compatibility, evidence-strength, maintenance, and diversity scoring
• return an adaptive number of credible candidates rather than a fixed global
  shortlist

Vectorize/FTS should perform full-corpus retrieval. Agents must not receive all
12k repository records or source trees.

4. Produce deterministic draft recommendations

Classify each candidate as:

• adopt or integrate
• reference implementation
• architectural pattern to study
• competing product to monitor
• unsuitable / negative example

Include relevance evidence, constraints, risks, maintenance signals,
confidence, and provenance. Permit bounded additional evidence fetching only
for the strongest or most uncertain candidates.

5. Run one bounded Devin review per changed project

Do not create one Devin session per need. Fleet should submit one compact
project package containing:

• project context and fingerprint
• all validated needs
• deterministic recommendations grouped by need
• approximately 25–50 total evidence cards after deduplication and diversity
  selection, with an explicit hard bound
• a strict structured-output schema

Devin should reject unsupported needs, remove weak candidates, compare viable
alternatives, explain tradeoffs, identify evidence gaps, and return final
judgments grouped by need.

6. Make reruns incremental

Start a new external review only when at least one of these is true:

• the project fingerprint changed
• the need map changed
• a newly cataloged repository crosses a stored recommendation threshold
• the prior review was incomplete or invalid
• an operator explicitly requests a rerun

A weekly catalog run should evaluate additions against persisted need signatures
and existing candidate thresholds. It should not rebuild every project report or
invoke Devin simply because the catalog watermark changed.

7. Reuse common need retrieval

Cache reusable candidate pools by normalized need signature, project constraints,
retrieval version, and catalog generation. Projects with similar needs may reuse
retrieval work, while final scoring and judgment remain project-specific.

Functional requirements

• Support an explicitly configured set of high-priority projects.
• Target 5–10 evidence-backed, non-duplicative needs per project, permitting
  fewer when evidence is insufficient.
• Allow different query and candidate counts for different needs within hard
  project-level bounds.
• Record project fingerprints, need signatures, catalog generation, retrieval
  version, model/provider version, and evidence provenance.
• Skip unchanged stages independently: capability-card generation, need
  extraction, retrieval, deterministic ranking, and external review.
• Make retries idempotent and prevent duplicate Devin sessions or duplicate
  review ingestion.
• Expose processing state: pending, retrieving, awaiting review, complete,
  degraded, or failed.
• Keep deterministic Starboard recommendations available when Devin is
  unavailable.
• Never replace the latest successful reviewed report with incomplete,
  malformed, or over-budget output.
• Keep all external-repository operations read-only.
• Let the UI/API and ChatGPT MCP read persisted results without triggering
  external-agent spend.

Suggested persisted state

Store enough structured state to represent:

• repository capability card and source fingerprint
• project analysis run and input fingerprint
• project need, priority, constraints, evidence, and normalized signature
• search intents and retrieval version per need
• candidate repository, matching evidence, preliminary score, and classification
• catalog generation and candidate threshold used for incremental evaluation
• external review request idempotency key and status
• reviewer verdict, confidence, rationale, rejected candidates, provider, model,
  usage, and timestamps
• latest successful draft and reviewed report versions

The exact schema should be decided during design.

Safety and cost controls

• Never clone or deeply inspect all catalog repositories through Devin.
• Bound needs per project, queries per need, candidates per query, evidence
  fetches, total evidence cards, and external-review usage.
• Set an explicit Devin session usage ceiling and report usage per project/run.
• Never provide production credentials, private repository contents, or
  user-private data to Devin.
• Run external review only for explicitly configured priority projects.
• Prefer cached evidence and incremental evaluation.
• Fail closed without losing the latest successful report.
• Keep Devin-specific credentials and lifecycle code outside Starboard.

Delivery phases

• Define the need taxonomy, evidence rules, capability-card schema, project
  fingerprint, normalized need signature, catalog generation, and report schema.
• Implement reusable repository capability cards and source-fingerprint
  invalidation.
• Implement project need extraction with caching and bounded AI/heuristic
  fallback.
• Implement adaptive per-need full-catalog retrieval, candidate
  deduplication, diversity, and shared need-signature caching.
• Implement deterministic evidence-based scoring, classifications, and
  versioned draft reports.
• Implement incremental evaluation of newly cataloged repositories against
  persisted project needs and candidate thresholds.
• Add a provider-neutral external-review request/result contract and
  idempotent reviewed-report ingestion in Starboard.
• Add Fleet-owned bounded Devin orchestration with structured output,
  timeout, polling, spend limits, and sanitized evidence packages.
• Preserve the latest successful reviewed report during degraded runs.
• Expose reports through authenticated project UI/API and the read-only
  ChatGPT MCP app where appropriate.
• Add operator controls for priority-project selection, manual reruns,
  status, cache decisions, and spend visibility.
• Add unit, integration, budget, cache-invalidation, idempotency,
  failure-mode, and recommendation-quality tests.
• Document operations, privacy boundaries, cost assumptions, provider
  replacement, and rollback/disable controls.
• Pilot on a small set of high-priority Fleet projects, measure retrieval
  quality and Devin usage, and tune bounds before expansion.

Acceptance criteria

• A priority project receives up to 5–10 evidence-backed, non-duplicative needs;
  insufficient evidence produces fewer needs rather than invented ones.
• Every need is searched independently across the complete eligible catalog.
• Recommendations are grouped by need and include evidence, classification,
  confidence, provenance, and source links.
• Repository capability analysis is reused across projects.
• Unchanged project fingerprints reuse their need maps.
• New catalog additions are evaluated incrementally without rebuilding all
  project reports.
• One bounded Devin session reviews the combined changed-project package and
  returns schema-valid judgments.
• Unchanged projects do not consume another Devin session.
• Devin failure or budget exhaustion preserves deterministic recommendations
  and the latest successful reviewed report.
• Starboard contains no Devin credentials and does not require Devin to serve
  project intelligence.
• UI/API and ChatGPT MCP reads never trigger external-agent spend.
• Automated tests demonstrate full-catalog retrieval, cache correctness,
  bounded costs, idempotency, and failure-safe behavior.

Non-goals

• Sending all repository source code or all 12k repository records to Devin.
• Starting a separate Devin session for every project need.
• Automatically adopting dependencies or modifying projects.
• Scanning private repositories without a separate privacy and authorization
  design.
• Adding Fleet-specific project portfolio management to Starboard.
• Hard-coding Devin as Starboard's only possible reviewer.
• Replacing Starboard's deterministic retrieval with an opaque agent-only
  recommendation flow.