flâneur

Note on Selection Bias in Observational Estimates of Algorithmic Progress

arxiv.org · 2,986 words · saved by 1 readers

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. Ho et al. [2024] is a very interesting paper that attempts to estimate the degree of algorithmic progress from language models. They collect observational data on language models’ loss and compute over time, and argue that as time has passed, language models’ algorithmic efficiency has been rising. That is, the loss achieved for fixed compute has been dropping over time. In this note, I want to raise one potential methodological problem with the estimation strategy. Intuitively, if part of algorithmic quality is latent (not observed in Ho et al. [2024]’s data), and compute choices are endogenous to algorithmic quality, then the estimation strategy in Ho et al. [2024] will not recover unbiased estimates of the true degree of algorithmic progress because o

Note on Selection Bias in Observational Estimates of Algorithmic Progress Parker Whitfill (August 2025) 1 Introduction Ho et al. [ 2024 ] is a very interesting paper that attempts to estimate the degree of algorithmic progress from language models. They collect observational data on language models’ loss and compute over time, and argue that as time has passed, language models’ algorithmic efficiency has been rising. That is, the loss achieved for fixed compute has been dropping over time. In this note, I want to raise one potential methodological problem with the estimation strategy. Intuitiv

related reading