In recent years, the recipe for building smarter artificial intelligence has been simple: make it bigger. Add more layers, feed more data, and watch the intelligence emerge.
This brute-force strategy led to significant advances in technology, but it also created major problems. AI data centers are hogging power and engineers are hitting the dreaded “memory wall.” In other words, simply making the model larger will make it unstable.
Meet DeepSeek, the Chinese startup that shocked the AI world last year with cutting-edge models trained with surprisingly few resources. And it's about to disrupt the industry again.
In a new paper published this week, researchers uncover a mechanism called manifold-constrained hyperconnection (mHC). The name is a mouthful, but the concept is elegant. This solves a thorny problem plaguing attempts to make AI models broader and more powerful: the tendency for signals in neural networks to explode or disappear.
Build smarter
The backbone of modern AI is called Residual Connection. This design, introduced several years ago by researchers at Microsoft, allows information to jump through layers, creating a direct path that keeps the signal straight as it passes through a deep neural network. This approach led to AIs like ChatGPT and Gemini. Think of it like installing an elevator in a skyscraper and skipping intermediate floors if they aren't needed.
Now imagine a deep neural network as a high-stakes “telephone” game played across hundreds of layers. As data passes through each layer and is processed, the original signal is often weakened or distorted. As the network grows too large, the risk of losing signal increases. The remaining connections act like express lanes that allow the original signal to hop across layers and recombine into the flow later.
However, there is a limit to the amount that one elevator can handle.
Recently, researchers have started experimenting with hyperconnections (HC). Instead of a single elevator, imagine a complex web of wide shafts where passengers (data) can move between different floors and shafts simultaneously. Sounds great on paper, but in reality it can cause buildings to collapse. In a chaotic web of hyperconnections, data signals are amplified out of control, crashing the AI training process.
DeepSeek's solution is a mathematical constraint called mHC. This forces these wild connections to work and allows the model to remain stable without losing the benefits of the broader network.
When the team tried to train a 27 billion parameter model using standard hyperconnections, the loss (error rate) skyrocketed and the training failed. However, using the mHC approach, we successfully trained a large-scale model. Stability improved significantly, but training time increased by only 6.7%.
classic deep seek
Song Linqi, an associate professor at City University of Hong Kong, told SMCP that this is a typical “DeepSeek-style” move, which takes established technology and improves on it to foster innovation. In fact, a key component of this technique is the Sinhorn-Knopp algorithm, a classic mathematical concept from the 1960s.
Professor Guo Song from the Hong Kong University of Science and Technology pointed out that this could be a turning point in LLM research. We may be moving from incremental “microdesign” tweaks to broader “macrodesign” architectural changes.
As DeepSeek points out, this is not just a stability issue. It's about “Restoring ID Mapping Properties”. In human terms, this means that the AI can ultimately maintain its train of thought, no matter how complex the journey becomes. Get more accurate responses and work with complex queries with more precision.
However, big questions remain. Can it be expanded?
A 27 billion parameter model is impressive, but not huge by modern standards. For comparison, GPT-4 is estimated to have over 1 trillion parameters, and even DeepSeek's own top model has over 600 billion.
“Although mHC has successfully validated up to 27 billion parameters, the effectiveness of mHC for frontier-scale models, i.e., hundreds of billions of parameters, remains an open question,” Guo said.
Ideas are cheap in AI. Running these efficiently on silicon at scale is the difficult part.
this can be very important
We are currently in an arms race to build “smarter” AI, and the conventional wisdom is often that “just make it bigger.” This puts data centers on the path to consuming nearly 5% of U.S. electricity and spending on AI reaching $1.5 trillion last year alone.
DeepSeek's research suggests that how you connect neurons is just as important as the number of neurons, and that there are more efficient ways to do it. If there's a better way to make wheels, we should stop focusing on making them heavier.
This research arXiv.
