Researchers use equilibrium propagation to overcome the disappearance gradient of deep recurrent neural networks

Balanced propagation (EP) offers a promising alternative to traditional backpropagation for training neural networks. Jiaqi Lin, Malyaban Bal and Abhronil Sengupta of Pennsylvania State introduce new EP frameworks that overcome these limitations by incorporating intermediate error signals, improving information flow and enabling successful training of significantly deeper architectures. This study represents an important advance in EP scaling as the team demonstrates cutting-edge performance on challenging image recognition tasks using benchmark datasets such as the CIFAR-10 and CIFAR-100. By integrating knowledge distillation with local error signals, this work unlocks the possibility that EPs will become a viable solution for constrained machine learning applications, paving the way for a more biologically plausible learning system.

Extended equilibrium propagation for stable learning

Researchers have introduced an extended equilibrium propagation (EP) framework to address the instability that is often encountered when training deep neural networks. EP aims to provide a more stable and efficient learning process by seeking stable state within network dynamics that differ from traditional methods that rely on gradient descent. The team strengthened the standard EP framework by adding intermediate learning signals that guided the learning process, improved stability and allowed for deeper architecture training. This method utilizes a mathematical function that represents the overall goal of the network that the EP is trying to minimize.

The loss function measures the difference between the output and desired outcome of the network, both at the output layer and these middle layers, providing a sophisticated learning signal. During training, network weights are adjusted based on these signals, focusing on local connectivity and interactions to improve learning. A technique called Linear mapping coordinates activation of the guide “teacher” network with activation of the learning “student” network, further stabilizing the process. This study contributes to the potential for more stable and efficient training of new training methods, a theoretical understanding of their convergence, and the possibility of more stable and efficient training of deep neural networks, paving the way for application in more complex systems.

Training deep networks with equilibrium propagation

Researchers have designed a new approach to training deep neural networks using equilibrium propagation (EP), a biologically inspired learning rule. This approach overcomes the limitations encountered in deeper architectures and addresses the problem of annihilation gradients by enhancing information flow and neuronal dynamics. The system mimics biological processes that utilize local error signals and knowledge distillation to improve gradient estimation and network stability. The core of this technique involves training a convergent recurrent neural network (CRNN) that propagates backwards through the connection, affecting the previous layer, and adjusting the synaptic weights.

The experiment employs convolution and pooling operations during forward pass, transposed convolution and organisation during reverse pass, effectively reflecting the flow of information within the network. To tackle the problem of vanishing gradients, the team has incorporated intermediate error signals to allow for more robust gradient calculations and significantly trained CRNNs. Furthermore, scientists have used knowledge distillation. There, a smaller “student” network learned from a larger, pre-trained “teacher” network, further stabilizing the training and improving performance. This system provides significant improvements in training performance for the CIFAR-10 and CIFAR-100 datasets, showcasing the scalability of the EP, paving the way for applications in real-world systems.

Deeper recurrence networks trained in equilibrium propagation

Researchers have developed a new framework for equilibrium propagation (EP), a biologically inspired learning rule, dramatically improving scalability for deep neural networks. Previous work with the EP has eliminated gradients in a deeper architecture, impeding convergence and performance. This new approach directly addresses this limitation and allows for training convergent recurring neural networks (CRNNs) that are substantially deeper than before. The team's breakthrough integrates both knowledge distillation and local error signals into the EP process, effectively increasing the flow of information and improving the convergence of neuronal dynamics.

Experiments show that this extended EP framework achieves cutting-edge performance with challenging CIFAR-10 and CIFAR-100 datasets and demonstrates the ability to scale effectively with deep VGG architectures. Specifically, the researchers observed a significant improvement in training loss in the previously lost network, VGG-11 CRNN, when trained with the enhanced EP method. The results clearly show that the integration of local error signals and knowledge distillation could overcome the limitations of previous EP implementations, provide a biologically plausible alternative to traditional backpropagation algorithms, and unleash new avenues of neural computing and artificial intelligence.

Deeper networks and reduced computational costs

Researchers have presented a new framework for equilibrium propagation (EP), a biologically inspired learning rule for neural networks. They addressed the challenge of annihilation of deep EP networks by incorporating intermediate error signals that improve information flow and allow EPs to effectively train deeper architectures. Experiments on CIFAR-10 and CIFAR-100 datasets using the VGG architecture show that this enhanced EP framework achieves cutting-edge performance and demonstrates scalability with no performance degradation as network depth increases. Compared to more traditional backpropagation (BPTT), this EP framework significantly reduces the demand for both GPU memory and computational resources.

This method employs local updates and is particularly suitable for on-chip learning due to its single architecture and weight update mechanism inspired by synaptic plasticity. Although the current research focuses on convolutional networks, the authors acknowledge that future research can explore the generalization of this framework into more complex architectures, such as large-scale vision models and trans-based language models. The authors focus on moderate increases in memory consumption, as they include additional weight matrices used to distill knowledge.