fixergroup/polish-reranker-base-mse-quantized-onnx

fixergroup/polish-reranker-base-mse-quantized-onnx

This model is an optimized ONNX version of sdadas/polish-reranker-base-mse, designed for efficient CPU inference.
It maintains full compatibility with the original model and can be used directly with the CrossEncoder API from sentence-transformers.
Thanks to quantization and ONNX Runtime, it delivers nearly identical output quality while running up to 2× faster on CPU.

---

Model Overview

This is a Polish text reranking model trained using the mean squared error (MSE) distillation approach on a large dataset of 1.4 million queries and 10 million documents.

Training Data Sources

1. Polish MS MARCO training split (≈800k queries)
2. ELI5 dataset translated into Polish (≈500k queries)
3. Polish medical Q&A collection (≈100k queries)

The teacher model used was unicamp-dl/mt5-13b-mmarco-100k, a large multilingual reranker based on the MT5-XXL architecture.
The student model was sdadas/polish-roberta-base-v2.

In the MSE method, the student learns to directly reproduce the scores predicted by the teacher.

---

Requirements

sentence-transformers>=2.2.0
optimum[onnxruntime]>=1.10.0
onnxruntime>=1.15.0
transformers>=4.20.0
torch>=1.9.0

---

Usage (Sentence-Transformers)

You can use this ONNX model directly with the CrossEncoder class from sentence-transformers:

from sentence_transformers import CrossEncoder
import torch.nn

query = "Jak dożyć 100 lat?"
answers = [
    "Trzeba zdrowo się odżywiać i uprawiać sport.",
    "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]

model = CrossEncoder(
    "fixergroup/polish-reranker-base-mse-quantized-onnx",
    default_activation_function=torch.nn.Identity(),
    max_length=512,
    device="cpu",
    trust_remote_code=True, #<---IMPORTANT
)
pairs = [[query, answer] for answer in answers]
results = model.predict(pairs)
print(results.tolist())

This version runs entirely on CPU and provides approximately 2× faster inference than the original PyTorch model.

---

Usage (Hugging Face Transformers + ONNX Runtime)

You can also use the model with ONNX Runtime directly:

from transformers import AutoTokenizer
from onnxruntime import InferenceSession
import numpy as np

model_name = "fixergroup/polish-reranker-base-mse-quantized-onnx"
tokenizer = AutoTokenizer.from_pretrained(model_name)
session = InferenceSession(f"{model_name}/model.onnx")

query = "Jak dożyć 100 lat?"
answers = [
    "Trzeba zdrowo się odżywiać i uprawiać sport.",
    "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]

texts = [f"{query}</s></s>{answer}" for answer in answers]
tokens = tokenizer(texts, padding="longest", truncation=True, max_length=512, return_tensors="np")

inputs = {k: v for k, v in tokens.items()}
outputs = session.run(None, inputs)
results = np.squeeze(outputs[0])
print(results.tolist())

---

Evaluation Results

The base model sdadas/polish-reranker-base-mse achieves an NDCG@10 score of 57.50 in the Rerankers category of the Polish Information Retrieval Benchmark (PIRB).
The ONNX version preserves equivalent reranking quality while offering improved inference performance.

---

Citation

If you use this model in your research or application, please cite:

@article{dadas2024assessing,
  title={Assessing generalization capability of text ranking models in Polish}, 
  author={Sławomir Dadas and Małgorzata Grębowiec},
  year={2024},
  eprint={2402.14318},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

---

Notes

• This ONNX model was quantized and optimized by FIXER Group.
• Compatible with sentence-transformers CrossEncoder API.
• Ideal for CPU-based reranking in production environments.