Processing Language, Images and Other Data Modalities

Speaker: Andrew Stuart

Time: 11am – 12pm, Mar. 31, 2026

Location: Center for Health Science Building 73-105

Title: Processing Language, Images and Other Data Modalities

Abstract: A fundamental problem in artificial intelligence is how to simultaneously deploy data from different sources — such as audio, images, text, and video — collectively known as multimodal data. In this talk, I will present a mathematical framework for studying this question, focusing primarily on text and images.

I will begin by describing how large language models (LLMs) operate, addressing the challenging issue of using real-number algorithms to process language. In particular, I will explain next-token prediction — the core of current LLM methodology. I will then focus on the canonical problem of measuring alignment between image and text data (contrastive learning). Finally, I will describe how images can be generated from text prompts (conditional generative modeling).

From a mathematical perspective, a unifying theme underlying this work is the minimization of divergences defined on spaces of probability measures. A second key mathematical idea is the attention mechanism — a form of nonlinear correlation between vector-valued sequences. I aim to explain these concepts — and their relevance to modern machine learning algorithms — in an accessible fashion, suitable for broad  audience from the mathematical and computational sciences.

Bio: Andrew Stuart obtained his undergraduate degree in Mathematics, from Bristol University in 1983, his PhD from the Oxford University Computing Laboratory in 1987 and was then a postdoc at MIT in the period 1987–1989. Before joining Caltech he held permanent positions at Bath University (1989–1992), Stanford University (1992–1999) and Warwick University (1999–2016). His research interests focus on computational applied mathematics; recent interests focus on challenges presented in this age of information, such as the integration of data with mathematical models and the mathematics of machine learning.

He was elected an inaugural SIAM Fellow in 2009.  He delivered invited lectures at the International Congress of Industrial and Applied Mathematics (ICIAM) in 2007 and 2023, at the European Congress of Mathematicians (ECM) in 2012 and at the International Congress of Mathematicians (ICM) in 2014. He was elected a Fellow of The Royal Society in 2020 and selected as a Department of Defense Vannevar Bush Faculty Fellow in 2022.

See this link for more details of the seminar: https://sites.google.com/g.ucla.edu/data-theory-seminar