Dual representations
In this post, I'll introduce a "new" idea for representing states in reinforcement learning (RL). The one-line summary is: we represent a state as the set of "distances" from every other state. I'll first explain the motivation behind this idea, and explain why it is useful. When studying mathematics and machine learning, one may notice that there is one intriguing theme that recurrently comes up in many different contexts. It has something to do with a "relative" way of looking at things. Here are some examples: The recurring theme in all these examples is that the relation to all other objects often contains enough information to identify the object itself. Can we apply this idea to reinforcement learning? More importantly, do we want to? The answer is yes and yes. Let's see how we can apply this idea to RL. The key question here is: how should we define the "relation" between two states in the context of RL? I'd argue that there's only one natural choice: the temporal distance. For