Transformers, parallelism, and the role of depth

Transformer models have served as the backbone of modern deep learning and large language models for several years, yet their mathematical foundations are sorely underdeveloped. I will talk about recent efforts to put transformers on firmer footing by studying them as a computational model for solving algorithmic problems. These efforts establish new connections to parallel computation and shed light on the role of depth in these models. This talk is based on joint works with Clayton Sanford (Google Research) and Matus Telgarsky (New York University).

Bio:

Daniel Hsu is an associate professor in the Department of Computer Science and a member of the Data Science Institute, both at Columbia University. He works on algorithmic statistics and machine learning, with the goals of designing efficient algorithms for learning and data analysis, and understanding the limits of efficient computation for these tasks. Daniel completed his Ph.D. at UC San Diego and his B.S. at UC Berkeley. He was a postdoc at the Departments of Statistics at Rutgers University and the University of Pennsylvania and also at Microsoft Research New England. He was selected by IEEE Intelligent Systems as one of “AI’s 10 to Watch” in 2015 and received a Sloan Research Fellowship in 2016.