Real user queries often look different from the clean, concise ones in academic benchmarks - ambiguity, full of typos, and much less readable.
We show that even strong RAG systems quickly break under these conditions.
Awesome project led by
@neelbhandari.bsky.social and @tianyucao.bsky.social!!
These days RAG systems have gotten popular for boosting LLMsโbut they're brittle๐. Minor shifts in phrasing (โ๏ธ style, politeness, typos) can wreck the pipeline. Even advanced components donโt fix the issue.
Check out this extensive eval by @neelbhandari.bsky.social and @tianyucao.bsky.social!
11/ This paper has been an incredible effort across institutions @ltiatcmu.bsky.social @uwcse.bsky.social. Huge thanks to my co-first author @tianyucao.bsky.social and co-authors @akhilayerukola.bsky.social @akariasai.bsky.social @maartensap.bsky.social โจ๐
10/ ๐ Paper: "Out of Style: RAGโs Fragility to Linguistic Variation": arxiv.org/abs/2504.08231
๐ฌ Code: github.com/Springcty/RA...
Read our paper for more details on impact of scaling retrieved documents, specific effects of each linguistic variation on RAG pipelines and much more!
9/ ๐จ Takeaway
RAG systems suffer major performance drops from simple linguistic variations.
Advanced techniques offer temporary relief, but real robustness demands fundamental changes - more resilient components and fewer cascading error in order to serve all users effectively.
8/๐ ๏ธ Adding advanced techniques to vanilla RAG improve robustness... sometimes๐ซ
โ
Reranking improves performance on linguistic rewrites, but gaps in performance with original queries remain.
โ ๏ธ HyDE helps rewritten queries but hurts original queries-creating a false sense of robustness
7/๐คWell, maybe scaling generation model size helps?
Scaling up LLM size helps narrow the performance gap between original and rewritten queries. However, this is not consistent across variations. Larger models occasionally worsen the impact, particularly with RTT variations.
6/โ๏ธ RAG is more fragile than LLM-only setups
RAGโs retrieval-generation pipeline amplifies linguistic errors, leading to greater performance drops. On PopQA, RAG degrades by 23% vs. just 11% for the LLM-only setup.
โ ๏ธThe main culprit? Retrieval emerges as the weakest link
5/๐งฉ Generation Fragility
Linguistic variations lead to generation accuracy drops-Exact Match score down by up to ~41%, Answer Match score by up to ~17%.
Structural changes from RTT are particularly damaging, significantly reducing response accuracy.
4/๐Retrieval Robustness
Retrieval recall plummets up to 40.41% due to linguistic variations, especially when exposed to informal queries. Grammatical errors like RTT and typos notably degrade performance, highlighting retrieversโ sensitivity to a number of linguistic variations
3/ We evaluate across an extensive experimental setup:โฃ
๐งฒ 2 Retrievers (Contriever, ModernBERT)โฃ
๐ค 9 open LLMs (3Bโ72B)โฃ
๐ 4 QA datasets (MS MARCO, PopQA, Natural Questions, EntityQuestions)โฃ
๐ Over 50K+ linguistically varied queries per dataset
2/๐ We evaluated RAG robustness against four common linguistic variations:
โ๏ธ Lower formality
๐ Lower readability
๐ Increased politeness
๐ค Grammatical errors (from typos & from round-trip translations (RTT))
1/๐จ ๐ก๐ฒ๐ ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ ๐ฎ๐น๐ฒ๐ฟ๐ ๐จ
RAG systems excel on academic benchmarks - but are they robust to variations in linguistic style?
We find RAG systems are brittle. Small shifts in phrasing trigger cascading errors, driven by the complexity of the RAG pipeline ๐งต
it's so important to make time for yourself, rest, treat yourself gently and with kind words. you, not me, I have to run myself ragged until I collapse in a pile of exhausted self-hatred but you should definitely self care.
Not at all. I just hope the wake up call happens by the end of the month, with the help of a stern winter wind.
