Lost in Retraining

Since conversational generative AI chatbots became available a few years ago, the internet has rapidly become populated with AI-generated texts. Many worry that, as these artificial texts are reused as training data for AI chatbots, they could impoverish our language and amplify existing biases.

Over the past few years, this question has become a major research topic, with hundreds of articles published so far. “There have been many studies lately, with different results. Most of them show that the diversity of the data obtained with models retrained on data generated by the same model decreases until it collapses. It can get to creating what they call “garbage,” texts that are just a random collection of words,” explains Matteo Marsili, a Senior Research Scientist in ICTP’s Quantitative Life Sciences section. Marsili recently collaborated with former ICTP Diploma student Fariba Jangjoo from the Kavli Institute for Systems Neuroscience in Norway and from King’s College London, United Kingdom, to understand the problem from a deeper and more general perspective. Their paper was published in Physical Review Letters [1].

While most research groups have so far applied an empirical approach, where they retrain AI models iteratively with texts generated by the model itself, Marsili and his collaborators applied a theoretical approach, focusing on a specific class of models. The group found that the negative effects of retraining, such as the disappearance of certain features and the amplification of some others, can be prevented by choosing a suitable inference technique or by including new data points that were not generated by the same model.

Marsili started working on this idea several years ago and when the question became topical in AI, he had already developed a framework to tackle the problem at a fundamental level. “We studied a specific class of probability distribution models, called exponential families,” Marsili explains, adding, “What makes these models, also called models of maximum entropy, interesting is that they are completely determined by a limited set of parameters.” He and his collaborators used equations to describe how these parameters evolve as they are repeatedly recalculated using data generated by the model itself. Differently from empirical approaches, their theoretical description also provides a controllable framework whose predictions can be tested.

The researchers found that what happens when the parameters are inferred by using input data generated by the model itself depends strongly on the method that is applied. They considered three widely used inference techniques. “In the maximum likelihood method, the parameters of the model are calculated by maximising the probability of the data at hand,” Marsili explains, adding, “We show that in this case models always collapse to singular distributions where some parameters diverge, which dramatically reduces the variability of data and thus amplifies biases.”

Other choices for the inference method lead to more stable results. “Model collapse can be prevented if statistical inference is based on maximum a posteriori method or by introducing regularisations, which are used in many realistic cases,” Marsili explains. The group also considered the role played in the retraining process by data not generated by the model itself and found that a minimum fraction of external data is enough to ensure sufficient variability in the output. “We show that a single new data point is enough to prevent model collapse, even when the widely used maximum likelihood method is employed,” he adds.

These results shed light on the current debate on the effects of retraining in large neural network models. Although Marsili and his collaborators focused on a very specific set of models, different from those that apply in realistic neural networks, their analysis can provide a starting point for addressing issues such as implicit biases arising in artificial neural networks. Marsili also explains that their method could help us describe many complex phenomena where learning takes place without external data. “This is the case, for example, for word-of-mouth communication or cultural transmission across generations. In prenatal development, the brain develops to a large extent in the absence of external stimuli, driven by its own spontaneous activity,” he says, adding, “Although our study focusses on a simple and stylised process, it offers a platform for studying these phenomena.”

[1] Jangjoo, F., Di Sarra, G., Marsili, M., & Roudi, Y. (2026). Lost in retraining: Closed-loop learning and model collapse in exponential families. Physical Review Letters, 136(19), 197301. https://doi.org/10.1103/PhysRevLett.136.197301