What looks like a trivial formatting choice can actually alter research conclusions, so mind the gap!
Big thanks to my co-authors @minhducbui.bsky.social & Katharina von der Wense!
📄 Read the full paper here: arxiv.org/abs/2509.15020
Surprisingly, this small detail:
✅ Shifts model accuracy by up to 11%
✅ Changes which model tops the leaderboard – raising serious concerns about comparability of LLM leaderboards in prior work
✅ Affects calibration (reliability of confidence estimates)
In our #EMNLP2025 paper we study how the space before the answer letter (e.g., "A" vs. "␣A") is tokenized.
Practice is currently split: no community-wide standard exists, and even popular evaluation frameworks differ.
🧐 Evaluating your LLM with multiple-choice question answering?
🧵 A tiny space in the prompt can make accuracy jump by 11% – and even reshuffle model rankings.
#EMNLP2025 #NLP #AI #LLM #Evaluation
