Popular transformer-based AI models such as GPT, Gemini, Claude, and Llama have become the standard for natural language processing thanks to their powerful classification and language generation capabilities. Such models rely heavily on multihead attention mechanisms and feedforward networks, which consist of a large number of parameters that consume large amounts of computational resources. These requirements result in these models requiring notoriously high energy consumption and long training times.
But what if there was a way to reduce the resources needed to run a useful model? Researchers could experiment with entirely new architectural ideas, rather than relying solely on incremental improvements to transformers.
Enter CoFrGeNets. This stands for Continued Fraction Generative Networks. These are a new class of model architectures designed by a group within IBM Research. CoFrGeNet replaces traditional transformer-based model components with structures derived from continued fractions, a mathematical representation that can compactly represent complex functions. Relying on continued fractions allows models to operate more expressively with fewer parameters because multiple components can be combined to increase the complexity of the representation without proportionally increasing the size. The research team outlines the evidence for CoFrGeNets in a new paper and presented their results this week at the International Conference on Machine Learning (ICML) in Seoul, South Korea.
According to Amit Dhurandhar, principal investigator at IBM Research, there are two main motivations behind this study. First, the field of LLM architecture design currently relies on a relatively small number of key ideas, particularly transformer models. By introducing alternatives, Dhurandhar and his fellow researchers hope to expand the range of tools available to build more efficient, compact, and scalable models without sacrificing performance. Second, the team wanted to systematically explore the limits and capabilities of this class of functions.
CoFrGeNet builds on the team’s previous work applying continued fractions to classification tasks. The study showed advantages over traditional architectures in a much more restricted environment, Dhurandar said. Our current project extends these ideas into generative models, making significant conceptual and engineering progress.
The core of CoFrGeNet is the use of continued fractions to represent functions. Continued fractions represent values as a series of nested divisions, allowing complex relationships to be captured in a compact way. When adapted to neural networks, these structures form a computational “ladder”, where each layer depends on the inverse of the previous layer.
Ensembles of these ladders can approximate highly complex functions while using fewer parameters than traditional neural networks. This can provide a powerful alternative to dense layers such as feedforward networks. In effect, the input is passed to each “step” of the ladder, where it is multiplied by the model parameters and added to the inverse of the previous step. In this way, the causal chain of token production is preserved, and each subsequent step on the ladder is defined by what happened before it.
Dhurandhar and his colleagues sought to determine whether these ladder-based architectures could directly replace the two core parts of a transformer block by:
Replacement of note: It performs interactions between tokens while preserving the causal structure required for autoregressive language production.
Replacement of feedforward mechanism: It replaces standard feedforward networks and allows mixing of features with fewer parameters.
The development process began with small-scale experiments using a lightweight open-source toolkit called nanoGPT to test the feasibility of different architectures using these two approaches before scaling up. “If it doesn’t work there, it definitely won’t work bigger,” Durandar said. “If things do Just because it works there doesn’t mean it will work elsewhere. But it was a good starting point. ”
To test how the CoFrGeNet components stack up, the team replaced the multihead attention and feedforward network with its own ladder and compared it to strong baselines such as GPT2-xl (1.5B) and Llama-3.2B. The setup was an extensive pre-trained model on large datasets including OpenWebText, GneissWeb, and a mixed Docling data mix. Each combination of continued fraction ladder components was evaluated according to perplexity tests on multiple datasets, in addition to downstream benchmarks such as GLUE (for classification).
For such a different architecture, the results were impressive. Across downstream classification tasks, the CoFrGeNet variants (specifically one called CoFrGeNet-F) often matched or slightly outperformed GPT2-xl, despite using far fewer parameters, often hundreds of millions fewer.
Perplexity scores, which quantify how accurately a model can predict a sample, tell a similar story. The CoFrGeNet model consistently outperforms GPT2-xl and often outperforms efficient attention baselines such as Synthesizer-D and Sparse Attendance. The CoFrGeNet model also demonstrated increased efficiency in the form of reduced training time and faster inference.
Even in large-scale Llama-3.2B experiments, the CoFrGeNet model remained small and fast to train while remaining competitive on tasks such as open-domain Q&A and inference. Taken together, these preliminary findings demonstrate that CoFrGeNet does indeed work and provides comparable or better performance while being leaner and more efficient.
Optimizing new methods
The big challenge with continued fractions is that they are computationally expensive and can be unstable on modern hardware, Durandal said. For example, if the denominator contains very small numbers, the value can be effectively infinite. Such numbers can be dangerous for today’s GPUs.
“The first thing we did to reduce division was to adopt a different formulation of representing continued fractions using what are called continuous fractions,” Durandal said. Continuous is another way to represent a continued fraction in terms of the ratio of two polynomials. Regardless of the depth of your ladder, you only need to make one split. This particular element allows CoFrGeNet to be efficiently trained and inferred on modern digital hardware.
To improve training stability, the team also introduced an “incremental training” technique. This technique introduces model layers gradually during training, rather than optimizing them all at once.
One of the decisive advantages of CoFrGeNet is its flexibility. This approach is primarily “plug and play,” meaning it can be integrated into existing model pipelines with minimal changes. Developers can selectively replace the attention layer, feedforward network, or both, and can mix new and legacy components.
“This approach is very complementary to what we already have,” said study co-author Vijil Chentamarakshan, a senior technical staff member at IBM Research. “If there are new developments in other aspects of the LLM, we can use them alongside our model; it is not contradictory in any way.”
Compared to existing efficiency techniques such as pruning and sparsity techniques that reduce parameters within an unchanged architecture, CoFrGeNet offers a novel solution, Durandhar said. This introduces a fundamentally different way of representing functions and opens new directions for research while remaining compatible with other optimization strategies.
what’s next
In terms of practical impact, this result suggests that it is likely to replace feedforward network components, which account for the majority of parameters in many models. Early collaboration with teams working on the IBM Granite family of models suggests that CoFrGeNet has the potential to reduce parameter counts and improve performance at similar scales. The approach from preliminary testing appears to be even more effective in the Mixed Experts (MoE) architecture that powers most frontier models. This is because each expert typically covers only a subset of tokens, making them more likely to be compressed.
Looking to the future, the team is considering additional hardware acceleration strategies. In an ongoing partnership with researchers at Rensselaer Polytechnic Institute (RPI), the team is currently experimenting with performing division operations on non-GPU hardware, such as analog processors and field-programmable gate array (FPGA) boards. These have the advantage of faster division performance than GPUs can achieve, and they take over the multiplication operations that GPUs are good at.
“This significantly speeds up the work and also results in significant power savings,” says Dhurandhar.
Overall, CoFrGeNet has the potential to be both a practical tool for improving model efficiency and a conceptual shift that expands the way we think about neural architectures.
