A laboratory seminar will be held on Thursday, May 15th, at 6:00 PM.
Speaker: Artem Alexandrov
Title: Word Synchronization and Metastability of Transformers
Abstract: In 2023, papers arXiv:2305.05465 and arXiv:2312.10794 were published, explaining the self-attention mechanism for the transformer architecture in large language models using the "tokens=particles" correspondence. Using this correspondence, the authors rigorously proved that overfitting is inevitable in a model with a large number of layers. However, this overfitting is not observed empirically. This can be explained by the emergence of metastability in a dynamical system. Subsequent papers (arXiv:2410.06833, arXiv:2411.04551, arXiv:2504.14697) were devoted to this explanation. This talk will review these studies. I'll discuss the correspondence between a transformer and a multiparticle dynamical system, then explain how overfitting and metastability arise. Finally, I'll share my perspective on the emergence of metastability in this system, motivated by graphones.