[2207.14251] Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions

Abstract:Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. But what exactly in the training data causes a model to make a certain prediction? We seek to answer this question by providing a language for describing how training data influences predictions, through a causal framework. Importantly, our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone. Addressing the problem of extracting factual knowledge from pretrained language models (PLMs), we focus on simple data statistics such as co-occurrence counts and show that these statistics do influence the predictions of PLMs, suggesting that such models rely on shallow heuristics. Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.

Measuring Causal Effects of Data Statistics on Language Model’s ‘Factual’ Predictions Yanai Elazar1,2 Nora Kassner3 Shauli Ravfogel1,2 Amir Feder4 Abhilasha Ravichander5 Marius Mosbach6 Yonatan Belinkov4 Hinrich Schütze3 Yoav Goldberg1,2 1…

related reading

• Large Language Model: world models or surface statistics?thegradient.pub
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
• [2602.05910] Chunky Post-Training: Data Driven Failures of Generalizationarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Faithful, Interpretable Model Explanations via Causal Abstraction | SAIL Blogai.stanford.edu
• 2308.03958arxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• [2602.06337] Can Post-Training Transform LLMs into Causal Reasoners?arxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• 2202.05262arxiv.org
• [2303.17548] Whose Opinions Do Language Models Reflect?arxiv.org