The Inaugural Workshop on
Frontiers in Statistical Machine Learning (FSML)

Talk Abstract: Weak-to-strong (W2S) generalization is a type of finetuning (FT) where a strong (large) student model is trained on pseudo-labels generated by a weak teacher. Surprisingly, W2S FT often outperforms the weak teacher. We seek to understand this phenomenon through the observation that FT often occurs in intrinsically low-dimensional spaces. Leveraging the low intrinsic dimensionality of FT, we analyze W2S in the ridgeless regression setting from a variance reduction perspective. For a strong student - weak teacher pair with sufficiently expressive low-dimensional feature subspaces Vs, Vw, we provide an exact characterization of the variance that dominates the generalization error of W2S. This unveils a virtue of discrepancy between the strong and weak models in W2S: the variance of the weak teacher is inherited by the strong student in Vs ∩ Vw, while reduced by a factor of dim(Vs)/N in the subspace of discrepancy Vw \ Vs with N pseudo-labels for W2S. Further, our analysis casts
light on the sample complexities and the scaling of performance gap recovery in W2S. The analysis is supported with experiments on synthetic regression and real vision and NLP tasks.

Talk Abstract: This talk introduces recent theoretical advancements on the in-context learning (ICL) capability of sequence models, focusing on the intricate interplay of data characteristics, architectural design, and the implicit algorithms models learn. We discuss how diverse architectural designs—ranging from linear attention to state-space models to gating mechanisms—implicitly emulate optimization algorithms that operate on the context and draw connections to variations of gradient descent and expectation maximization. We elucidate the critical influence of data characteristics, such as distributional alignment, task correlation, and the presence of unlabeled examples, on ICL performance, quantifying their benefits and revealing the mechanisms through which models leverage such information. Furthermore, we will explore the optimization landscapes governing ICL, establishing conditions for unique global minima and highlighting the architectural features (e.g., depth and dynamic gating) that enable sophisticated algorithmic emulation. As a central message, we advocate that the power of architectural primitives can be gauged from their capability to handle in-context regression tasks with varying sophistication.

​

Talk Abstract: In this talk, I will share my perspective on a fundamental challenge facing those of us working at the intersection of statistics and machine learning: how should we engage with large language models (LLMs)? Statistics has a long-standing history of developing principled approaches for reasoning about data in response to emerging technologies, and LLMs represent the most recent example that renders text processing nearly as tractable as numerical computation. However, LLMs constitute a significant departure from previous examples due to their black-box nature, which renders their internal mechanisms largely incomprehensible. In light of this, I will argue that the statistics community should prioritize developing methodologies for LLMs that circumvent their internal workings---such as attention mechanisms---until new mathematical theories become available to (effectively) analyze these complex systems. I will argue that, at least for now, any research that attempts to develop statistical frameworks relying on the attention mechanism or other engineering complexities of Transformers is unlikely to have lasting value. Instead, the focus should be to leverage the probabilistic, autoregressive nature of next-token prediction in LLMs and to emphasize actionable methods that capture quantitative rather than qualitative aspects of observable behaviors. For illustration, I present examples of watermarking, alignment, and evaluation for LLMs. Collectively, these findings demonstrate how statistical reasoning can yield useful insights for improving LLM development, even while the mechanisms of network architecture remain elusive. This talk is based on arXiv:2505.19145, 2506.22343, 2411.13868, 2506.12350, 2505.20627, 2503.10990, and 2506.02058.

Talk Abstract: Learning dynamics, which describes how the learning of specific training examples influences the model’s predictions on other examples, give us a powerful tool for understanding the behaviour of deep learning systems. This talk will cover how we can use a local understanding of training steps, building on empirical neural tangent kernels, to better understand phenomena that happen across the course of training a deep network. Applications include better understandings of knowledge distillation, fine-tuning, and particularly preference tuning for LLM post-training.