Thanks to my collaborators Sophia Hager, Adi Asija, Nick Andrews, and @danielkhashabi.bsky.social at @jhuclsp.bsky.social !
Arxiv: arxiv.org/abs/2508.11027
Code: github.com/JHU-CLSP/hell-or-high-water
(Data coming soon!)
More tools = worse at handling tool failures
When tool schemas are provided in-context, we find that performance gaps between adversarial and non-adversarial settings increases with the number of schemas.
LLM agents do not handle tool failures well
With RAG on tool schemas, we observe a substantial performance gap between adversarial and non-adversarial settings.
Tools break in the real world all the time, but not much attention has been given to how well LLMs deal with tool failures.
We introduce HOHW, a tool-use benchmark where problems remain solvable even when tools break adversarially.
