The Transformer architecture revolutionized sequence modeling with the introduction of attention, a mechanism by which models look back on previous inputs and prioritize relevant input data. However, computational costs increase significantly with sequence length, limiting the ability of Transformer-based models to scale to very long contexts, such as those required for complete document understanding or genomic analysis.
The research community has considered various approaches for solutions, including efficient linear recurrent neural networks (RNNs) and state-space models (SSMs) like Mamba-2. These models provide fast, linear scaling by compressing the context to a fixed size. However, this fixed-size compression cannot adequately capture the rich information in very long sequences.
In two new papers, titans and MilasWe present an architecture and theoretical blueprint that combines the speed of RNNs with the accuracy of transformers. Titans is a specific architecture (tool) and MIRAS is a theoretical framework (blueprint) for generalizing these approaches. Together, these advance the concept of test-time memory. This is the ability of AI models to maintain long-term memory by incorporating more powerful “surprise” metrics (i.e., unexpected information) without dedicated offline retraining during model execution.
As Titans demonstrated, the MIRAS framework provides a meaningful transition to real-time adaptation. Rather than compressing information into a static state, this architecture actively learns and updates its parameters as data streams are input. This important mechanism allows the model to instantly incorporate new specific details into its core knowledge.
