SABER-Math: An Automated Reranking Benchmark for Mathematical Information Retrieval
SABER-Math
EMNLP 2026  ·  Information Retrieval and Text Mining Track

SABER-Math

An Automated Reranking Benchmark for Mathematical Information Retrieval

Nikolay Georgiev1,* Maria Drencheva1,* Kseniia Ibragimova† Ivo Petrov1 Dimitar I. Dimitrov1 Martin Vechev1,2
* Equal contribution. † Work done during an internship at INSAIT.
arXiv 2606.29894 Code on GitHub Hugging Face
≈300K
Source Problem-Solution Pairs
1000
Reranking Instances
150
Candidates per Query
49
Models Evaluated

As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing systems that retrieve useful documents. We address this gap with SABER-Math, the first fully automated reranking benchmark for mathematical IR, built without expert annotation. Starting from 283K high-school-level problems with solutions, it builds challenging tasks in three steps: (i) LLMs extract concise solution summaries and topics for each problem; (ii) candidates are discovered per query using ontology-based topic similarity and lexical overlap between solution summaries; and (iii) a Swiss-style LLM preference tournament assigns every candidate a graded relevance rating. Over these judged candidate pools, we evaluate retrievers across all major paradigms, from lexical and math-specific systems to modern embedding, late-interaction, and reranking models. Reasoning embedders lead, followed by general-purpose models, both far outperforming classical and math-specific baselines. Yet, all still struggle in symbol-heavy domains such as Algebra and Calculus. Importantly, we show that generic IR benchmarks, such as MTEB, track mathematical performance broadly but are less discriminative among the strongest models, highlighting the need for domain-specific evaluation.

Methodology

Overview of SABER-Math construction: sourcing a mathematical corpus, computing topic and solution-summary relevance signals, selecting 1000 queries with 150 candidates each, and ranking candidates with a Swiss tournament.

Figure 1. Query documents are shown in blue. Saturated purple denotes topic relevance, saturated teal denotes solution-summary relevance, and muted colors denote less relevant documents.

The pipeline at a glance
Scroll to walk the pipeline ↑ Abstract Results ↓
The pipeline at a glance
Four automated stages take 283 000 high-school-level problem–solution pairs to 1,000 reranking tasks with graded relevance labels, with no expert annotation anywhere in the loop. Keep scrolling to walk through each stage.
1
Large Scale Mathematical Corpus
We use 283 000 high-school-level problem–solution pairs from AoPS forums, national and international competitions, and the olympiad split of NuminaMath-1.5.
2
Relevance Signals
Two complementary relevance signals capture different kinds of mathematical relatedness: the topics a problem draws on, and the technique its solution uses. Both are extracted automatically, and each is computed over the whole corpus.
2A
Topic-based similarity
Every problem is annotated with topics from a MathWorld-derived ontology of 17.1K nodes. Two problems are compared by optimally matching their topic sets and averaging Lin's information-content similarity, so a shared rare, specific ancestor counts for more than a shared broad one.
2B
Solution-summary similarity
An LLM writes a short summary of each solution's core idea, and then the core idea relevance is measured using the Jaccard overlap between the summaries' word sets. The metric stays lexical on purpose, so the benchmark is not biased toward the embedding models it evaluates.
3
Build the candidate pools
We select 1000 queries spanning Algebra, Geometry, Number Theory, Combinatorics, and Calculus, each paired with 150 relevant candidates split evenly across topic-only, summary-only, and both-relevant.
4
Judge with a Swiss tournament
Final relevance rankings in our reranking tasks are computed using a pairwise LLM-as-a-judge based on GPT-OSS-120B, run inside a 20-round Swiss-style tournament. Its judgments are aggregated with a Bradley–Terry model into continuous relevance ratings rescaled to [0, 5].

Results

{{ settingDesc }}

Model
+ {{ part.label }} {{ part.label }} {{ cell.v }} {{ cell.v }}
Bold marks the best value in each column across the models currently shown.

Accuracy against latency

Frontier Frontier Other
{{ t.label }}
{{ t.label }}
{{ tipName }}
Score
Time
{{ tipScore }}
{{ tipLat }}
{{ l.name }}
{{ l.name }}
{{ l.name }}
{{ l.name }}
Score
Median latency per reranking task

Figure 2. Retrieval quality in the {{ settingPhrase }} setting against median latency per reranking task, served with vLLM at batch size B = 16, on a logarithmic time axis. One quality–latency frontier is traced at a time: red for single-pass retrievers (embedding, lexical and API models), black for rerankers and late-interaction models — use the toggle above to switch. A model is on its frontier when no model of the same kind is both faster and better. Hover a point for its name and values.

Score over time

Frontier Frontier Other
{{ t.label }}
{{ t.label }}
pre-2023
{{ l.name }}
{{ l.name }}
{{ l.name }}
{{ l.name }}
{{ tip2Name }}
Score
Released
{{ tip2Score }}
{{ tip2Date }}
Score
Release date

Figure 3. Retrieval quality in the {{ settingPhrase }} setting against public release date. Models released before 2023 share the shaded band on the left. One state-of-the-art trace is shown at a time: red for single-pass retrievers (embedding, lexical and API models), black for rerankers and late-interaction models — use the toggle above to switch. A model is marked when it was the best of its kind available at its release.

Takeaways

Reasoning IR methods achieve the best results

Training on reasoning data is a strong factor for higher performance. ReasonEmbed-Qwen3-8B, RaDeR-7B, and Diver-4B outperform Octen-8B, the strongest general-purpose bi-encoder we evaluate, while being no larger than it.

Strong LLM backbones matter too

The Qwen and Llama versions of ReasonEmbed share a training recipe and a parameter count, yet differ by 0.046 on the strength of their backbone alone. The Llama-based ReasonIR-8B and the ModernBERT-based Reason-ModernColBERT both trail Qwen3-Embedding-4B despite being trained with reasoning data.

IR systems struggle more on symbolic tasks

For the strongest systems, Geometry and Combinatorics tend to be easier, while symbol-heavy domains like Algebra, Number Theory, and Calculus are more challenging. ReasonReranker-Qwen3-32B-Rewrite reaches 0.749 on Geometry and 0.783 on Combinatorics, but only 0.710 on Algebra.

Generic retrieval benchmarks cannot solely predict performance on mathematical IR

Correlations between SABER-Math and MTEB multilingual IR are 0.77 (Pearson) and 0.82 (Spearman), but drop to 0.62 and 0.73 for models released from 2025 onwards. Harrier-27B ranks 1st on MTEB while trailing the Octen, Gemini, and Qwen models here.

Cite

@inproceedings{georgiev2026sabermath,
  title     = {{SABER}-Math: An Automated Reranking Benchmark for
               Mathematical Information Retrieval},
  author    = {Georgiev, Nikolay and Drencheva, Maria and
               Ibragimova, Kseniia and Petrov, Ivo and
               Dimitrov, Dimitar I. and Vechev, Martin},
  booktitle = {The 2026 Conference on Empirical Methods in
               Natural Language Processing},
  year      = {2026},
  url       = {https://openreview.net/forum?id=Tb2EKLtAS0}
}