SABER-Math: An Automated Reranking Benchmark for Mathematical Information Retrieval
SABER-Math
EMNLP 2026  ·  Information Retrieval and Text Mining Track

SABER-Math

An Automated Reranking Benchmark for Mathematical Information Retrieval

Nikolay Georgiev1,* Maria Drencheva1,* Kseniia Ibragimova3,† Ivo Petrov1 Dimitar I. Dimitrov1 Martin Vechev1,2
* Equal contribution. † Work done during an internship at INSAIT.
arXiv 2606.29894 Code on GitHub Hugging Face
≈300K
Source Problem-Solution Pairs
1000
Reranking Instances
150
Candidates per Query
34
Models Evaluated

As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing systems that retrieve useful documents. We address this gap with SABER-Math, the first fully automated reranking benchmark for mathematical IR, built without expert annotation. Starting from 283K high-school-level problems with solutions, it builds challenging tasks in three steps: (i) LLMs extract concise solution summaries and topics for each problem; (ii) candidates are discovered per query using ontology-based topic similarity and lexical overlap between solution summaries; and (iii) a Swiss-style LLM preference tournament assigns every candidate a graded relevance rating. Over these judged candidate pools, we evaluate retrievers across all major paradigms, from lexical and math-specific systems to modern embedding, late-interaction, and reranking models. Reasoning embedders lead, followed by general-purpose models, both far outperforming classical and math-specific baselines. Yet, all still struggle in symbol-heavy domains such as Algebra and Calculus. Importantly, we show that generic IR benchmarks, such as MTEB, track mathematical performance broadly but are less discriminative among the strongest models, highlighting the need for domain-specific evaluation.

Methodology

Overview of SABER-Math construction: sourcing a mathematical corpus, computing topic and solution-summary relevance signals, selecting 1000 queries with 150 candidates each, and ranking candidates with a Swiss tournament.

Figure 1. Query documents are shown in blue. Saturated purple denotes topic relevance, saturated teal denotes solution-summary relevance, and muted colors denote less relevant documents.

The pipeline at a glance Scroll to walk the pipeline
The pipeline at a glance
Four automated stages take 283 000 high-school-level problem–solution pairs to 1,000 reranking tasks with graded relevance labels, with no expert annotation anywhere in the loop. Keep scrolling to walk through each stage.
1
Large Scale Mathematical Corpus
We use 283 000 high-school-level problem–solution pairs from AoPS forums, national and international competitions, and the olympiad split of NuminaMath-1.5.
2
Relevance Signals
Two complementary relevance signals capture different kinds of mathematical relatedness: the topics a problem draws on, and the technique its solution uses. Both are extracted automatically, and each is computed over the whole corpus.
2A
Topic-based similarity
Every problem is annotated with topics from a MathWorld-derived ontology of 17.1K nodes. Two problems are compared by optimally matching their topic sets and averaging Lin's information-content similarity, so a shared rare, specific ancestor counts for more than a shared broad one.
2B
Solution-summary similarity
An LLM writes a short summary of each solution's core idea, and then the core idea relevance is measured using the Jaccard overlap between the summaries' word sets. The metric stays lexical on purpose, so the benchmark is not biased toward the embedding models it evaluates.
3
Build the candidate pools
We select 1000 queries spanning Algebra, Geometry, Number Theory, Combinatorics, and Calculus, each paired with 150 relevant candidates split evenly across topic-only, summary-only, and both-relevant.
4
Judge with a Swiss tournament
Final relevance rankings in our reranking tasks are computed using a pairwise LLM-as-a-judge based on GPT-OSS-120B, run inside a 20-round Swiss-style tournament. Its judgments are aggregated with a Bradley–Terry model into continuous relevance ratings rescaled to [0, 5].

Results

Setting
{{ settingNote }}
{{ countLabel }}
Model
{{ row.name }} {{ row.name }} {{ row.ref }} {{ cell.v }} {{ cell.v }}
Bold marks the best score in each column across the models currently shown.

Takeaways

General IR benchmarks do not predict Math IR

They separate old retrievers from modern ones, but say little about the mathematical ability of the recent models a practitioner would actually choose between. Picking a retriever for mathematics therefore needs domain-specific evaluation.

Strong backbones drive Math IR performance

Non-reasoning retrievers built on Qwen3 beat the best reasoning retrievers built on Qwen2.5, so the backbone matters more than the math-specific data used for retrieval training. Advances in general LLMs are thus very important for retrieval on complex tasks like mathematics.

Reasoning IR methods achieve the best results

On recent backbones, reasoning training gives a further gain. Reason-Embed-8B and Diver-Retriever-4B outperform the best general-purpose system, Octen-Embedding-8B, by large margins.

Symbol-heavy domains remain hard

Geometry and Combinatorics tend to be easier for the strongest systems, while Algebra, Number Theory, and Calculus still lag behind.

Cite

@inproceedings{georgiev2026sabermath,
  title     = {{SABER}-Math: An Automated Reranking Benchmark for
               Mathematical Information Retrieval},
  author    = {Georgiev, Nikolay and Drencheva, Maria and
               Ibragimova, Kseniia and Petrov, Ivo and
               Dimitrov, Dimitar I. and Vechev, Martin},
  booktitle = {The 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026},
  url       = {https://openreview.net/forum?id=Tb2EKLtAS0}
}