The challenge of efficiently evolving artificial neural networks has long been a major hurdle in machine learning research. Davide Farinati, Frederico JJB Santos, Leonardo Vanneschi, and Mauro Castelli from the University of Nova de Lisboa and the University of Trieste presented a new approach, called NEVO-GSPT, designed to overcome these computational limitations. Their study introduces a new neuroevolutionary algorithm that leverages geometric semantic operators adapted from genetic programming, alongside a new method for controlled network reduction. This innovation allows for more focused and cost-effective exploration of potential network architectures, evaluating only the impact of new components rather than the entire structure. Standard regression benchmark results show that NEVO-GSPT consistently produces compact networks with performance comparable to or better than established techniques such as TensorNEAT and SLIM-GSGP.
Slim-GSGP fights neural network bloat
This research paper investigates the evolution of artificial neural networks (ANNs) using a neuroevolutionary technique called geometric semantic genetic programming (GSGP). This research focuses on addressing common challenges in genetic programming. bloatingEvolved networks grow unnecessarily large and complex without corresponding performance improvements. To overcome this, the authors propose a non-hypertrophic variant called SLIM-GSGP. It incorporates specific constraints and mechanisms within genetic operators to control the growth of networks during evolution.
SLIM-GSGP is evaluated based on several benchmark regression datasets, including human oral bioavailability prediction, concrete compressive strength, and building energy efficiency. The results show that SLIM-GSGP achieves competitive and even superior prediction performance compared to standard GSGP and NEAT while maintaining a significantly smaller and more compact network architecture. Important innovations are deflate mutation is a geometric semantic mutation operator that removes unnecessary nodes and connections, effectively limiting bloat without compromising accuracy.
This study also investigates the hybridization of GSGP and gradient-based optimization techniques and shows that network performance can be further improved by combining evolutionary search and gradient descent. To facilitate adoption and experimentation, the authors released a Python library implementing SLIM-GSGP, providing a practical tool for researchers interested in neuroevolution and compact neural network design.
Overall, this work presents a scalable and efficient approach to evolving neural networks, producing models that are not only accurate, but also more compact and potentially more interpretable. SLIM-GSGP provides a valuable contribution to the neuroevolution and machine learning research community by addressing the problem of bloat and providing an accessible software library.
Neural evolution through geometric semantic perturbations and training
The research team developed NeuroEVOlution through Geometric Semantic perturbation and Population based Training (NEVO-GSPT), a novel neuroevolutionary algorithm designed to overcome the computational limitations inherent in evolving neural network architectures. Traditional methods such as grid search and random search often require a thorough evaluation of the architectural space and lack a clear relationship between structural changes and the resulting network behavior. This study addresses these issues by adapting the principles of geometric semantic genetic programming (GSGP) and the semantic learning algorithm SLIM-GSGP to the evolution of artificial neural networks. Scientists designed NEVO-GSPT to enhance both computational efficiency and semantic recognition over the course of evolution.
This work pioneered the use of geometric semantic operators (GSOs), originally derived from genetic programming, to ensure predictable effects on network semantics and maintain a unimodal error surface. Importantly, the team introduced the Deflate Geometric Semantic Mutation (DGSM) operator. This is a new technique that complements the existing Inflate Geometric Semantic Mutation (IGSM), which enables a controlled reduction in network size and extends the network. This alternating inflationary and deflationary strategy prevents uncontrolled growth and promotes the evolution of compact and interpretable networks. In our experiments, we used a linked list representation of the network as the perturbation component, allowing stepwise fitness evaluation.
This innovative approach means that only newly added or removed components require evaluation, reusing previous evaluations and significantly reducing computational costs compared to methods that require complete retraining. The experimental setup included four regression benchmarks designed to explore the effects of initial population pre-training and the benefits of the NEVO-GSPT approach. Results show that NEVO-GSPT consistently evolves compact networks and achieves performance comparable to or better than established techniques such as standard neural networks, SLIM-GSGP, TensorNEAT, and SLM. This technique represents a significant advance in the efficient exploration of architectural search spaces and provides a computationally viable alternative to resource-intensive techniques such as neural architectural search, which often requires thousands of GPU hours.
Geometric operators accelerate the evolution of neural networks
Scientists have achieved a breakthrough in neuroevolution with the development of NEVO-GSPT, a new algorithm for evolving artificial neural networks. This study addresses the computational demands of traditional network evolution methods by introducing geometric semantic operators (GSOs) adapted from genetic programming. These GSOs ensure that structural changes to the network produce predictable effects, operate within a unimodal error surface, and facilitate a more efficient search process. Because the algorithm only needs to evaluate the semantics of newly added components during population-based training, the team measured a significant reduction in computational cost.
Experiments reveal that NEVO-GSPT consistently evolves compact neural networks across four regression benchmarks. The study details a methodology that incorporates both “inflate” and “deflate” operators. The new “deflate geometric semantic mutation” (DGSM) controls the reduction of network size while preserving the semantic properties established by “inflate geometric semantic mutation” (IGSM). Tests have shown that this alternating expansion and contraction prevents uncontrolled model growth and results in interpretable networks. In this study, we investigated the influence of factors such as initial collective training, post-evolutionary fine-tuning, inflation/deflation probability, and complexity of additional components, and provided a comprehensive analysis of the algorithm’s performance.
Our data show that NEVO-GSPT’s efficient evaluation mechanism significantly reduces the fitness calculation cost by reusing previous evaluations, which is an important advance compared to methods that require complete network retraining. The researchers conducted experiments designed to evaluate four key factors, demonstrating the adaptability and robustness of the algorithm. The results demonstrate comparable or superior performance to established techniques such as standard neural networks, SLIM-GSGP, TensorNEAT, and SLM. This breakthrough provides a computationally efficient and semantically aware approach to neural network design, opening the door to broader accessibility and applications in machine learning tasks.
This effort is built on the Semantic Learning Machine (SLM) framework and extends it with DGSM operators to achieve controlled network complexity. The scientists noted that maintaining the network as a linked list of perturbed components further increases efficiency, since only new or deleted components need to be evaluated during mutation. This step-by-step mechanism shows significant improvements in computational speed and resource utilization and may enable the evolution of more complex and effective neural network architectures.
Geometric semantics drives efficient neural architecture search
In this study, we introduced NEVO-GSPT, a novel neuroevolutionary algorithm designed to address computational demands and improve understanding of structure-behavior relationships in neural architecture search. This work focuses on two key innovations: the adaptation of geometric semantic operators and the development of deflationary geometric semantic modification operators that enable controlled network reduction without compromising semantic properties. These advances enable NEVO-GSPT to achieve efficient population-based training by evaluating only newly modified components, significantly reducing computational costs. Experimental results across four regression benchmarks show that NEVO-GSPT consistently achieves performance comparable to or better than established techniques such as standard networks, SLM, TensorNEAT, and SLIM-GSGP.
Importantly, this performance is achieved with a significantly more compact network than those produced by some comparison algorithms. Although the authors acknowledge that direct runtime comparisons with GPU-accelerated implementations are limited by the CPU-based setup, they highlight the algorithm’s ability to quickly explore a large number of architectural solutions on standard hardware. Future research may focus on investigating the performance of the algorithm with GPU acceleration and its application to more complex problem areas.
