My advisor @sarath-chandar.bsky.social is hiring four postdocs! If you're looking to work in academia, his lab at @mila-quebec.bsky.social offers an amazing research environment with tons of opportunities and resources. Highly recommend checking it out!
The figures below illustrate the number of diary entries recalled by Pythia models compared to the target. Color represents model size. Our findings indicate that as model size increases, the ability to accurately recall the correct number of entries emerges.
6/6
If the models consistently recall the correct number of entries—neither omitting nor hallucinating additional ones—it shows that they know how many entries each individual has authored out of all the information they have memorized.
5/6
To assess whether LLMs possess this capability, we design an experiment where fictional individuals each author a random number of diary entries. We then fine-tune the LLMs to memorize these entries and evaluate their ability to recall all entries written by a specific individual.
4/6
This capability is crucial, as many complex problems demand a comprehensive understanding of all relevant facts. Without an awareness of their own knowledge scope, LLMs may struggle to provide well-rounded, reliable insights.
3/6
LLMs absorb vast amounts of information during training, but do they know how much they know about a given topic? For instance, do they know how many news articles they have memorized about a specific world event?
2/6
Do large language models know how much they know? Our EMNLP paper is the first to show that LLMs do seem to have an understanding of how much they know about certain topics. Details in the thread ⬇️ 1/6 #LLM
arxiv.org/abs/2502.19573
