Batch LLM calls during node summary generation to reduce API round-trips

[arozumenko]

Problem

FilesystemRepositoryIndexer.populate_summaries() calls llm.invoke() once per node inside the batch loop — 217 nodes → ~217 sequential LLM calls → ~7 minutes on Bedrock (Haiku, ~2s/call).

Solution

Option A — Single prompt per batch (recommended)

Send all N symbols in one LLM call per batch:

Summarise each of the following code symbols in 2-3 sentences. Focus on role, key behaviour, and what callers need to know. Return a numbered list (1. 2. 3. ...) matching the input order exactly.

1. class JobQueue:
   """AsyncIO job queue..."""
   def __init__(self, max_size=1000):
       ...

2. class Worker:
   ...

Parse the numbered response back to individual summaries. Cost: ~11 calls for 217 nodes (batch_size=20) instead of 217.

Failure handling: if parsing fails or count mismatches, fall back to per-node calls for that batch only.

Option B — Parallel async (alternative)

Keep one call per symbol but fire concurrently via asyncio.gather() with asyncio.to_thread(llm.invoke, ...). Same token cost, lower wall-clock. Respects LLM pool when #294 lands.

Implementation

In FilesystemRepositoryIndexer.populate_summaries / _summary_fn:

def _summary_fn(texts: list[str]) -> list[str]:
    # Build one prompt for the whole batch
    numbered = "\n\n".join(
        f"{i+1}. {text[:1500]}"
        for i, text in enumerate(texts)
    )
    prompt = (
        "Summarise each code symbol below in 2-3 sentences "
        "(role, key behaviour, what callers need to know). "
        f"Return exactly {len(texts)} numbered items matching input order.\n\n"
        f"{numbered}"
    )
    try:
        response = llm.invoke(prompt)
        raw = response.content if hasattr(response, "content") else str(response)
        summaries = _parse_numbered_list(raw, expected=len(texts))
        if len(summaries) == len(texts):
            return [s[:400] or None for s in summaries]
    except Exception as exc:
        logger.warning("Batch summary failed: %s — falling back to per-node", exc)
    # Fallback: per-node calls
    return [_single_summary(llm, t) for t in texts]

_parse_numbered_list extracts 1. ..., 2. ... patterns, strips them, returns list.

Expected impact

• 217 nodes, batch_size=20 → ~11 LLM calls instead of 217
• ~20× reduction in indexing time for the summary phase
• Same output quality (Haiku handles 20 short snippets in one 200K context easily)

Acceptance

• Node summaries populated correctly for all architectural nodes
• Log shows ~11 calls instead of 217 for the test repo
• Fallback fires gracefully when parse fails
• batch_size remains configurable