fix: tier-based acronym ranking, real scores on rescued documents - #15
Conversation
Acronym queries returned their most relevant documents last. For q=DIET,
"DIET Stakeholder Identity Map" (source_id 219) came back at rank 77 with
score 0.2250 and every field_score null, below documents that merely
mention education. release-2.1.0's whole first page contained no DIET
document at all.
Two causes, both fixed here.
1. Documents that failed filter_score were re-added by
_fetch_field_match_docs with a synthetic FLOOR_SCORE (0.15) and blanked
field_scores, discarding the score the pipeline had already computed for
them. _process_and_filter_results now snapshots each source's best entry
BEFORE the threshold runs (prefilter_scores_out) so those documents are
restored with their real weighted_score, field_scores and tier. Sources
genuinely absent from the candidate pool still take the floor path.
2. The is_acronym_query dense/sparse weight-flip is removed. It applied a
second scoring formula to acronym queries only, and aimed at the wrong
signal: the documents it was meant to lift had keyword_score 0.0, so it
never fired for them. What it did lift was BM25 noise -- the sparse query
is an OR-bag of the acronym plus every expansion word, so documents
containing "Education"/"Training" scored on it. Removing it drops 12 such
results from q=DIET (Orthography, FC, Cohort, Washback Effect ...).
Replaced by lexical tiering: 4 = acronym in title, 3 = in summary, 2 =
expansion in title, 1 = expansion in summary, 0 = none. Candidates are
sorted by (tier, weighted_score), and the tier is baked into the exposed
score as (tier + score) / 5 so a caller re-sorting on score alone still
reproduces tier order -- the commons media API does exactly that.
Expansions are matched on content words (_phrase_in_text), not verbatim.
Matching them literally made tiers 2 and 1 unreachable in practice: on
q=DIET all 74 results landed in tier 4 or 0, and "Strengthening of District
Institutes of Education and Training" scored 0 purely because of the
plural. All content words are required, so near-misses sharing only common
words ("District Primary Education Programme") are still not promoted. The
acronym itself keeps exact whole-word matching so DIET never matches
"dietary".
Also batches the dense query embeddings into one model call -- the second
embedding costs +5ms instead of +23ms -- and adds PTM, MIP, LFA, NCF and
SDG to the seed CSV. MIP carries both "Micro Improvement Project" (the
corpus's own definition, 214 uses) and "Micro Improvement Plan"; the first
expansion drives the substituted embedding, and tiering checks all of them.
Verified against a live release-2.1.0: non-acronym queries return identical
result sets, totals and scores, with no document changed except ones
previously floor-injected. 219 moves to rank 6 with real field_scores.
Costs +13-18% latency on acronym queries.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| if not words: | ||
| return False | ||
| return all( | ||
| any(word == term or word.startswith(term) or term.startswith(word) |
There was a problem hiding this comment.
@borkarsaish65 check this if acronym expansion is "Institute of Management Studies" , the words which contains will be ["institute", "management", "studies"], if the document is "Man in Network Security" , so this document will get true right ?
| # filtering upstream, which needs the real weighted_score, not | ||
| # the tier-compressed one. | ||
| for r in top_results: | ||
| r["weighted_score"] = (r.get("tier", 0) + r["weighted_score"]) / 5.0 |
There was a problem hiding this comment.
@borkarsaish65 if the doc1 semantic weighted_score is less but if the acronym tire is top, for other doc2 semantic weighted_score is higher but tier is low in this case which should come top ?
Can you check this With other use case also ?
2 participants
Acronym queries returned their most relevant documents last. For q=DIET, "DIET Stakeholder Identity Map" (source_id 219) came back at rank 77 with score 0.2250 and every field_score null, below documents that merely mention education. release-2.1.0's whole first page contained no DIET document at all.
Two causes, both fixed here.
Documents that failed filter_score were re-added by _fetch_field_match_docs with a synthetic FLOOR_SCORE (0.15) and blanked field_scores, discarding the score the pipeline had already computed for them. _process_and_filter_results now snapshots each source's best entry BEFORE the threshold runs (prefilter_scores_out) so those documents are restored with their real weighted_score, field_scores and tier. Sources genuinely absent from the candidate pool still take the floor path.
The is_acronym_query dense/sparse weight-flip is removed. It applied a second scoring formula to acronym queries only, and aimed at the wrong signal: the documents it was meant to lift had keyword_score 0.0, so it never fired for them. What it did lift was BM25 noise -- the sparse query is an OR-bag of the acronym plus every expansion word, so documents containing "Education"/"Training" scored on it. Removing it drops 12 such results from q=DIET (Orthography, FC, Cohort, Washback Effect ...).
Replaced by lexical tiering: 4 = acronym in title, 3 = in summary, 2 = expansion in title, 1 = expansion in summary, 0 = none. Candidates are sorted by (tier, weighted_score), and the tier is baked into the exposed score as (tier + score) / 5 so a caller re-sorting on score alone still reproduces tier order -- the commons media API does exactly that.
Expansions are matched on content words (_phrase_in_text), not verbatim. Matching them literally made tiers 2 and 1 unreachable in practice: on q=DIET all 74 results landed in tier 4 or 0, and "Strengthening of District Institutes of Education and Training" scored 0 purely because of the plural. All content words are required, so near-misses sharing only common words ("District Primary Education Programme") are still not promoted. The acronym itself keeps exact whole-word matching so DIET never matches "dietary".
Also batches the dense query embeddings into one model call -- the second embedding costs +5ms instead of +23ms -- and adds PTM, MIP, LFA, NCF and SDG to the seed CSV. MIP carries both "Micro Improvement Project" (the corpus's own definition, 214 uses) and "Micro Improvement Plan"; the first expansion drives the substituted embedding, and tiering checks all of them.
Verified against a live release-2.1.0: non-acronym queries return identical result sets, totals and scores, with no document changed except ones previously floor-injected. 219 moves to rank 6 with real field_scores. Costs +13-18% latency on acronym queries.