[Detail Bug] Retrieval: Single-document results bypass reranker relevance gating and always produce a citation

[detail-app]

Detail Bug Report

https://app.detail.dev/org_befd6425-a158-4e24-9d4d-1e5c08769515/bugs/bug_e1962849-683e-4558-bbee-89623d58729d

Introduced in #141 by @WilliamAGH on Jul 28, 2026

Summary

• Context: The RerankerService is the retrieval pipeline's relevance gate, responsible for selecting only documents the LLM judges relevant to the query.
• Bug: When documents.size() == 1, the service returns immediately without calling the LLM, bypassing the relevance gate entirely.
• Actual vs. expected: A single off-topic document is always returned as relevant; the expected behavior is that the LLM judges the single document and may return an empty selection if it's irrelevant.
• Impact: For non-Java-API queries (majority path), an irrelevant single retrieved document can survive into prompt context and be emitted as a citation, even if the assistant text correctly declines the off-topic query.

Code with Bug

// RerankerService.java:126-128
public List<Document> rerank(String query, List<Document> documents, int returnK, long stageDeadlineNanos) {
    requireRemainingStageBudget(stageDeadlineNanos);
    if (documents.size() <= 1) {
        return documents; // <-- BUG 🔴 bypasses LLM relevance selection when exactly 1 doc
    }
    // ... rest of method calls LLM for relevance judgment
}

Explanation

The reranker contract/prompt was changed to select only relevant documents and allow returning an empty selection ({"order":[]}) when nothing matches. The guard documents.size() <= 1 is a stale condition from the prior “reorder all documents” contract; it incorrectly assumes a single document should always be returned. Under the new contract, the single document must still be judged for relevance and potentially dropped.

A failing test (added during investigation) demonstrates the bug: when the mocked LLM would return an empty order for a single irrelevant document, rerank() still returns the document due to the early return.

Codebase Inconsistency

Retrieval has two paths:

• Explicit Java API member queries may use CitationCandidateRanker.
• All other queries go through RerankerService.rerank().

CitationCandidateRanker.selectPromptContextForCitationQuery() contains a pass-through fallback (no filtering) for most natural-language queries, so the reranker is the sole relevance gate in those cases.

// CitationCandidateRanker.java:59-68
static List<Document> selectPromptContextForCitationQuery(String citationQuery, List<Document> promptDocuments) {
    return JavaApiMethodSelector.uniqueExplicitJavaApiMemberFromQuery(citationQuery)
            .filter(selector -> selector.exactOverloadAnchor().isPresent())
            .map(selector -> exactOverloadCandidates(selector, promptDocuments))
            .orElseGet(() -> JavaApiMethodSelector.uniqueExplicitJavaApiMemberFromQuery(citationQuery)
                    .map(selector -> memberFamilyCandidates(selector, promptDocuments))
                    .orElseGet(() -> List.copyOf(promptDocuments))); // <-- BUG 🔴 pass-through fallback (no relevance filter)
}

Recommended Fix

// RerankerService.java:126-128
// BEFORE (buggy):
if (documents.size() <= 1) {
    return documents;
}

// AFTER (fixed):
if (documents.isEmpty()) {
    return documents;
}

Add tests for single-document behavior:

• single relevant doc: LLM returns {"order":[0]} → doc is kept
• single irrelevant doc: LLM returns {"order":[]} → empty list returned

History

This bug was introduced in commit 2fed38c. The commit changed the reranker's contract from "reorder all documents" to "select only relevant documents," updating the Javadoc, prompt (to support {"order":[]}), and parser (to accept strict subsets). However, the early return guard if (documents.size() <= 1) was carried forward unchanged from the original implementation (commit 6f249e0), where it was correct under the "reorder all" contract but became incorrect under the new "select relevant" contract.