paper:doi-10-48550-arxiv-2402-00795LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law
Original abstract (expand)
We study LLMs’ ability to extrapolate the behavior of various dynamical systems, including stochastic, chaotic, continuous, and discrete systems, whose evolution is governed by principles of physical interest. Our results show that LLaMA-2, a language model trained on text, achieves accurate predictions of dynamical system time series without fine-tuning or prompt engineering. Moreover, the accuracy of the learned physical rules increases with the length of the input context window, revealing an in-context version of a neural scaling law. Along the way, we present a flexible and efficient algorithm for extracting probability density functions of multi-digit numbers directly from LLMs.
Similar preprints — Semantic Scholar
Cited by (1)
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
In-context learning and activation steering are the same operation on a latent-concept belief: context adds log-likelihood evidence while a steering vector adds a constant shift to the log prior odds,