It’s a hot week for looped Transformers. For example, @Princeton and the University of Pennsylvania proposed an elegant architecture.

Their new Recurrent Looped Transformer (RLT) lets a Transformer carry forward a running state, updating it after each token.

Why is it interesting?
→ A longer sequence creates a longer chain of computation, and RLT doesn't need more layers per token for this.

▪️ RLT's workflow looks like this:

Its distinctive feature is how it separates reading the input from updating that state.

The job is divided into 2 parts:

1. A causal encoder builds input memory
It processes known prompt tokens in parallel and creates cached keys and values shared across decoder layers.

2. A recurrent decoder updates the running state.
It combines the current encoder output with the previous decoder output, attends to recent activations and shared encoder memory, then passes the updated state forward.

On sequences longer than those used in training, RLT reached:

• 100% on parity at 256 bits, after training on at most 40.
• 97% on swap tracking at eight times the training length vs. under 1% for stan
• Up to 93% on modular arithmetic, vs. 33%.

And one interesting detail: different tasks preferred different layer splits. How much computation goes into reading vs. updating may matter as much as the total depth.