#transformers

1 concepts1 episodes3 insights

After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World LabsLatent Space

Future Model Architectures Beyond Transformers

Transformers treat inputs as sets of tokens, which works well for language but is sub-optimal for spatial data that lives in 3-D. The discussion highlights the need for new primitives that map better to distributed hardware and for architectures that can capture physical laws implicitly.

#distributed-compute2 #model-architectures2

“Transformers are set models of token sets, not sequences, limiting spatial reasoning”

Fei-Fei Li

3 insights · 6 quotes