flâneur

Is the Reversal Curse a generalization problem? – Non_Interactive – Software & ML

nonint.com · 1,429 words · saved by 1 readers

In my last post, I made a claim that the recently discovered reversal curse is not something that worries me. In fact, when I originally learned of it, I can’t say I was very surprised. In this post, I wanted to dig into that a little bit more. My hypothesis is that the reversal curse is a attribute of knowledge look-up, not a problem with the ability of LLMs to perform reasoning. Let me first describe how I think knowledge look-up in neural networks currently works. At a high level, autoregressive neural networks map inputs into high-dimensional vectors (we call them “latents”). At the terminus of the network, we map those latents into sets of probabilities which can be used to make a series of predictions through the process of sampling. The true power of AR models is in the context, which is the sequence of latents which came before the current prediction. Modern AR models learn to use the context to aid their predictions using a mechanism called self attention. Empirically, we find

In my last post, I made a claim that the recently discovered reversal curse is not something that worries me. In fact, when I originally learned of it, I can’t say I was very surprised. In this post, I wanted to dig into that a little bit more. My hypothesis is that the reversal curse is a attribute of knowledge look-up, not a problem with the ability of LLMs to perform reasoning. Lookup in NNs Let me first describe how I think knowledge look-up in neural networks currently works. At a high level, autoregressive neural networks map inputs into high-dimensional vectors (we call them &#822

related reading