The trap I keep seeing is that once a reranker goes in, recall becomes the binding constraint and nobody notices — if the right passage never made the candidate set, no amount of reordering saves you, yet the pipeline still looks healthier because the top slot improved. It survives review because most eval suites score the final ranking rather than candidate coverage, so the failure is invisible in exactly the metric people watch. Worth measuring retriever recall@k separately before crediting the reranker for anything.
Exactly, the reranker gets credit for a ceiling the retriever set. if recall@k already dropped the right passage, reordering just polishes a candidate set that never held the answer. scoring the final ranking hides it completely, which is why recall@k has to be its own line before the reranker earns any praise. sharp catch.
Running the default 100 here too, and no, never swept it—mostly because the eval never flagged retrieval as the bottleneck, which after reading this probably just means I wasn't measuring recall at fetch depth separately. The bge-reranker token limit thing actually explains a weird ranking drop I saw months back that I never traced to chunk size. Testing 50 vs 150 this week instead of guessing.
That's exactly the trap, the eval can't flag a bottleneck it never measures, so recall at fetch depth stays invisible until you break it out. the bge token limit biting on long chunks is a common one, chunk size and reranker window have to be tuned together. curious what 50 vs 150 does for you, report back.
The trap I keep seeing is that once a reranker goes in, recall becomes the binding constraint and nobody notices — if the right passage never made the candidate set, no amount of reordering saves you, yet the pipeline still looks healthier because the top slot improved. It survives review because most eval suites score the final ranking rather than candidate coverage, so the failure is invisible in exactly the metric people watch. Worth measuring retriever recall@k separately before crediting the reranker for anything.
Exactly, the reranker gets credit for a ceiling the retriever set. if recall@k already dropped the right passage, reordering just polishes a candidate set that never held the answer. scoring the final ranking hides it completely, which is why recall@k has to be its own line before the reranker earns any praise. sharp catch.
Running the default 100 here too, and no, never swept it—mostly because the eval never flagged retrieval as the bottleneck, which after reading this probably just means I wasn't measuring recall at fetch depth separately. The bge-reranker token limit thing actually explains a weird ranking drop I saw months back that I never traced to chunk size. Testing 50 vs 150 this week instead of guessing.
That's exactly the trap, the eval can't flag a bottleneck it never measures, so recall at fetch depth stays invisible until you break it out. the bge token limit biting on long chunks is a common one, chunk size and reranker window have to be tuned together. curious what 50 vs 150 does for you, report back.