The debate between memorization and generalization in the LLM (extensive language models) is not new. But it is rarely addressed with the almost surgical precision with which a group of target researchers, Google Deepmind, Nvidia and the University of Cornell have done so. How much can a Llm of your training data remember? When do you stop memorizing concrete facts and start learning patterns? These questions, apparently philosophical, are actually quantifiable. And that is just what has been achieved in this new study: measure, exactly, how many bits a model can store.
The border between the memorized and the learned in the LLM is not only diffuse: is invisible to the naked eye. Often, a correct answer does not necessarily mean that the model has understood the concept, but could have stored word by word. The work of these researchers seeks precisely to draw a line between both phenomena. To do this, they resorted to a meticulous strategy: train hundreds of language models from scratch, using both synthetic and real datasets, carefully deduced to avoid involuntary repetitions. The sizes of the models ranged between 500,000 and 1.5 billion parameters, with architectures similar to those of GPT.
The most striking innovation of the study is a metric called HK, based on the complexity of Kolmogorov, which It allows estimating how much original and specific information has been really stored by the model. And with this tool, the team obtained a key fact: a MMORIZA, on average, between 3.5 and 3.6 bits for each of its parameters. That is the maximum capacity before the “sature” model its space and begins to generalize, abandoning literal repetition to embrace broader statistical patterns.
This transition is not simply theoretical: it manifests itself in the form of a double fall in the validation error, the well -known phenomenon of the Double Descent, which marks the moment when the model begins to behave more generalized. Upon reaching the saturation threshold, the memorized data stop providing additional value and the model reorganizes its internal “memory” to optimize learning.
Another relevant observation is that the type of precision with which the model is trained – such as BFloat16 compared to FP32 – has a minimum effect on this memorization capacity. Instead, the researchers found that The most prone examples to be memorized were those with rare tokenssyntactically rare sequences or phrases in minority languages. That is, the model tends to store the unusual, which moves away from the dominant pattern.
This type of memorization not only has technical implications. It also raises issues about privacy, audit and traceability. If a model memorizes rare examples that contain sensitive information, it could reproduce them without their designers knowing it. Understanding this quantitative limit becomes, then, a critical tool to evaluate the real behavior of the LLM.
The study does not intend to reduce the size of the models or directly improve their computational efficiency. Your contribution is on another plane: better understand how and how much can “remember” a model. A contribution that also feels the basis for future research on how to control, limit or even audit that memorization process.
Perhaps the most valuable of this work is that, by measuring what seemed immersible, manages to return some transparency to a often treated land like a black box. At a time when LLMs are already part of the infrastructure of our digital life, knowing how much they remember is not an academic curiosity, but an urgent need. Because only understanding what is within a model, we can trust what comes out of it.
