Architectural optimisation in deep neural networks. Tests of a theoretically inspired method

Form factors and temperatures

A deep network (Fig. 1) is built with a set of N neurons that are arranged in an arbitrary amount of K > 2 layers. We refer with L^(p) at the p-th hidden layer within the set of all the ones composing the network p ∈ {1, 2, ⋯ , K} and the number of neurons inside the layer L^(p) is indicated as N^(p). Each of the i-th neurons of a layer L^(p) are connected to each of the j-th neurons of the successive layer L^(p+1) through a set of values called weights\({W}_{ij}^{(p)}\). A way to represent the neuron quantity in each layer is through the vector, α = (α⁽¹⁾, ⋯ , α^(K)) whose entries α^(p) are called form factors (see Fig. 1). Each form factor is defined by the relative amount of neurons with respect to the total number in the network: α^(p) = N^(p)/N. Moreover, another useful vector describing the connections between two successive layers β = (β⁽¹⁾, ⋯ , β^(K)) is introduced. The entries of this second vector are usually referred as inverse temperatures, within a statistical mechanics perspective, and represents the scale of the fluctuations of the coupling strength of each \({W}_{ij}^{(p)}\) between neuron i and j, within layer L^(p) and L^(p+1), as shown in Theory Methods section. Their computation is done through the empirical fluctuation of the weights \({W}_{ij}^{(p)}\) within the layers L^(p) and L^(p+1), as shown in Section “Temperature estimation”

Following the theoretical indications in ref. ²⁴, as explained in Methods, the neurons in the hidden layers should be relocated from the higher-temperature region to the low-temperature one, with respect to the components of β. For example, if the layers \({L}^{(\hat{p})}\) and \({L}^{(\hat{p}+1)}\) register the highest inverse temperature \({\beta }^{(\hat{p})}\), the remaining neurons have to be moved towards this couple. The movement is constrained by maintaining the same form factors between the selected couple i.e., \({\alpha }^{(\hat{p})}={\alpha }^{(\hat{p}+1)}\). Our objective was to implement the theoretical guidance in the most minimal and interpretable manner. This approach ensures structural balance in the network and prevents abrupt architectural changes that might destabilise training. We want to test the theoretical suggestion on a neural network with finite numbers of neurons using a classification task experiment on the MNIST dataset. We aim at improving the performance metrics of accuracy and robustness described by Eqs. (1) and (2). The network used in this experiment is introduced in Fig. 2a, it is composed by one fully-connected input layer of 784 neurons, three hidden layers of 64 neurons each, and an output layer of 10 neurons. Once the coldest layer of couplings has been identified, we cut, at random, two neurons from the remaining layer and we add them each to the boundary layers of such couplings (see Fig. 2b). This operation of trimming and grafting is the simplest possible, given the available information about the temperatures. The limitation of the resources provided by the embedded systems induces the usage of low-dimension neural networks²⁸. We emphasise that the number of neurons is used as a proxy for both the energy consumption and the spatial footprint of the neural network. This choice is motivated by two fundamental constraints on network complexity in nano- and microsatellite applications: first, the severely limited power budget imposed by the satellite’s size; and second, the physical limitations of semiconductor technology, such as minimum gate size²⁹. It is possible to design the hardware utilising the firmware FINN³⁰, a framework provided by AMD³¹, and mixed with the library Brevitas³² and the framework PYNQ³³, such models can be implemented into electronic systems.

Accuracy and robustness

For the evaluation of the performance, we use two metrics: the Accuracy (Acc) that measures how accurate are the network predictions, and their sturdiness with the Robustness (Rob). Their definitions depend on the classification behaviour of the network that is explained in the following. Each of the ten classes representing the MNIST digits, have a label that we express with the index i ∈ {0, 1, . . . , 9}. We call M_i the total amount of images in the testing set for an i-digit. For each provided j-image in M_i, to be tested, an output layer vector V_ij is provided. We can define the set of its ten components as

only the highest \({V}_{ij}^{\ell }\) is associated with the prediction ℓ. The definition of the metrics are therefore:

Since the number of images of each label in the testing set M_i is not constant, Acc weights the total amount of correct prediction with M_i, and a uniform average on all the classes is performed. Rob provides the difference between the prediction associated to the correct label \({V}_{ij}^{i}\) and the highest score among the remaining ones \(ma{x}_{\ell \ne i}({{\mathbb{V}}}_{ij})\). In particular, the higher is Rob, the higher is the distance between the correct prediction and the first possible misleading digit. For this reason, we consider Rob as a measure of sturdiness of the answer given by the network. The rationale for selecting these metrics is thoroughly detailed in the Metrics motivations Methods section. Both the metrics have been developed for having a better understanding of the behaviour of the networks when noise is induced by the field radiation^34,35,36.

Temperature estimation

In order to study the consequences of the theoretical indications over the choice of the architecture and its performance, an estimation of the temperature must be added. It turns out that the empirical fluctuations of the weight matrix elements, connecting each couple of layers, is linked to the inverse temperature^26,27. We therefore evaluate it as:

where 〈W^(p)〉 is the mean value of the weights \({W}_{ij}^{(p)}\) of a particular couple of nearest neighbours layers. Assigned the inverse temperature β^(p) and the form factors α^(p), we introduce following the work in ref. ²⁴ the matrix

The spectral radius ρ(β, α) carries an important thermodynamic meaning. In order for the network to be able to have a properly classifying structure, one must have ρ(β, α) > 1, as illustrated in the Theory Methods section.

As we are going to show in the following, besides the strategy of reaching thermal equilibrium β⁽¹⁾ ~ β⁽²⁾ we discovered that the topologies where ρ(β, α) is higher are the architectures that maximise the Robustness. This is a remarkable fact since the Spectral Radius is a pure thermodynamic property of an infinite dimensional network, while the Robustness is an empirical quantity measured in a real finite dimensional system.

Architectural optimisation 1. The MNIST case study

In this section, we refer to a network configuration with the vector α_k, containing the form factors of each hidden layer of our test network. The k-index stands for the k-th step of moving neurons, starting from the initial network configuration α₀. The initial condition is the equidistributed neurons network \({{\boldsymbol{\alpha }}}_{{\bf{0}}}=({\alpha }_{0}^{(1)},{\alpha }_{0}^{(2)},{\alpha }_{0}^{(3)})\) where \({\alpha }_{0}^{(1)}={\alpha }_{0}^{(2)}={\alpha }_{0}^{(3)}=64/N\) and N = 192. For a network configuration, we can also associate \({{\boldsymbol{\beta }}}_{{\boldsymbol{k}}}=({\beta }_{k}^{(1)},{\beta }_{k}^{(2)})\). Once the network is trained, we compute the associated inverse temperature values, using (4) and we obtain \({{\boldsymbol{\beta }}}_{{\bf{0}}}=({\beta }_{0}^{(1)}=7.42,{\beta }_{0}^{(2)}=7.35)\). We notice that \({\beta }_{0}^{(p)} > 1\) implies ρ(β₀, α₀) > 1 and therefore the exit from the paramagnetic phase. The accuracy of the model that we obtain is Acc = 85.815, therefore, confirming the theoretical modelisation is catching the structure of the database. Since \({\beta }_{0}^{(1)} > {\beta }_{0}^{(2)}\), the elementary redistribution is shifting two neurons from the last hidden layer to the first two layers, ensuring that \({\alpha }_{1}^{(1)}={\alpha }_{1}^{(2)}=65/192\). We continue this movement for every k, as observed in Fig. 2b. The results of this movement are shown in Fig. 3a and b. In Fig. 3a are represented the Accuracy and \(({\beta }_{k}^{(1)},{\beta }_{k}^{(2)})\), Fig. 3b illustrates the Robustness and ρ(β, α). After three neuronal modifications, we observe that β⁽¹⁾ and β⁽²⁾ reach a region where they overlap within their fluctuations, and we continue it until reaching \({{\boldsymbol{\alpha }}}_{30}=({\alpha }_{30}^{(1)}={\alpha }_{30}^{(2)}=94/N,{\alpha }_{30}^{(3)}=4/192)\). We observe that β⁽¹⁾ and β⁽²⁾ diverge from each other, till reaching the bottleneck effect at α₂₈ where the Accuracy drops drastically. Within the architectural changes, the Accuracy is a slow increasing function, with its maximum reached at α₁₈, where it gains the 2.17% with respect to α₀. The total variation of this metric between α₈ (inside the overlapping region) and α₁₈, turns out to be 0.87%.

The maximum value reached for the Robustness at α₈ comes with a 13.5% improvement with respect to α₀. With these two metrics, we can compare better each architectural configuration and find the optimal tradeoff between the two. The configuration obtained with α₈ maximises the Robustness and makes it the most suitable configuration for resisting the radiation field (see Metrics motivations, “Methods” section). From Fig. 3b it turns out that the maximum value of ρ is reached in proximity of configuration displaying the highest Robustness. This is a hint to the fact that not only the trimming and grafting is valuable, but it suggests a deeper link between its ideal thermodynamic properties with the practical behaviour of the neural network.

To validate such statements, we produce a series of trained networks, each time with the neuronal exchange introduced in Fig. 2b, and averaging the results. In this way we can obtain an average behaviour of our metrics and identify the networks belonging to the overlapping region that we name thermal equilibrium. Its quantitative evaluation is explained fully in the Thermal Equilibrium, Methods section. Denoting as \({\mathcal{O}}\) the generic observable representing one of the metrics or the inverse temperatures, we define its average and fluctuation over repetitive trainings as:

where \({\mathcal{T}}\) is the number of trainings, \({{\mathcal{O}}}_{t}\) is the observable measured at the end of the t-th training. The plots illustrating all the observables, together with their fluctuations, are displayed in Fig. 4. In these tests, we observe the maximum value of the Robustness for α₈, which proves to be the most stable value under repetitive trainings. This architecture is achieved within the thermal equilibrium region, suggesting that the equilibration of temperatures across synaptic connections serves as a meaningful criterion for identifying an adequate architectural configuration. A further indication of optimality emerges from the spectral radius, whose functional behaviour during the trimming and grafting process closely correlates with that of Robustness. Notably, both metrics reach their peak at the same configuration, α₈. As one can see from Fig. 4, α₈ represents the correct tradeoff between performance and the network’s Robustness, reaching an enhancement of 4.8% with respect to α₀.

To fully investigate the effect of the topological changes induced by the trimming and grafting, we also tested alternative movement schemes. Fig. 5 shows two cases, the former moving neurons from the middle layer towards the outer layer, as shown in Fig. 5a, the latter in the opposite direction of the temperature criterion, as shown in Fig. 5b. From the computation of the metrics, we observe that these architectural choices yield lower performance results, as shown in Fig. 5c, d. Moving the neurons towards the coldest couple of layers appears to be the best choice among the tested ones.

Architectural optimisation 2. The other case studies

The previous work has been performed utilising the same dataset, number of classes and training procedure. Therefore, we aim to visualise how the network would respond to perturbations of these attributes.

The first modification we implement is reducing the number of classes of the MNIST dataset while maintaining the same code implementation. Instead of assigning each digit to its own class, we sort them into two classes: odd and even numbers. The output layer of the network is then changed from 10 to 2 neurons. We perform the same test as before by transferring neurons from the hottest area towards the coldest one. The results are visualised in Fig. 6a, which displays the mean over different trainings for all architectures. Compared to the results obtained with the MNIST dataset in Section “Architectural optimisation 1. The MNIST case study”, both the Accuracy and the Robustness show enhanced results. This is due to the fact that the task is simplified by performing binary classification. Here, the SpectralRadius has a plateau in the thermal equilibrium region, where the maximum value of the Accuracy, α₁₆, is registered. At this architecture \(\overline{Acc}\) increases by 0.7% and \(\overline{Rob}\) increases by 1.7% with respect to α₀ performance. We also observe that \(\overline{\rho }({\boldsymbol{\alpha }},{\boldsymbol{\beta }})\) tends to have a similar functional form of the Robustness as we continue the architectural changes.

In particular, they peak at the very last architecture model, which no longer suffers from the bottleneck effect and \(\overline{Rob}\) reaches 6% improvement compared to α₀. However, a slight decline in accuracy is observed compared to previous configurations. This can be attributed to the role of the spectral radius ρ as an order parameter signalling phase transitions within the network (see Methods). As previously discussed, when ρ > 1, the network transitions from a stable energy landscape to a more complex one. Yet, further increasingρ does not necessarily translate into improved performance. Excessively large spectral radius values may give rise to highly irregular energy landscapes, marked by sharp and narrow minima that are difficult to access during training—ultimately leading to degraded performance.

In the second test, we change the dataset from MNIST to fashion MNIST²⁵. It consists of ten classes of clothes items, with the same input dimensions as the standard MNIST images, as described in Methods. The results of the study using this dataset are displayed in Fig. 6b, showing the mean results of five different trainings. From the scale of the two metrics, both Robustness and Accuracy are significantly smaller compared to the standard MNIST results. However, we once again observe the same behaviour described in Section “Architectural optimisation 1. The MNIST case study”. At the boundary of the thermal equilibrium region we detect α₅ as the most robust architecture. At this configuration, the Robustness peaks and the three metrics exhibit the smallest uncertainties. Here, \(\overline{Acc}\) and \(\overline{Rob}\) respectively reach <1% and 2.8% with respect to the initial condition α₀. In Fig. 4, the higher scores of \(\overline{Rob}\) and \(\overline{\rho }\) correspond to a narrow region of architectures, whereas, in this second test, the higher values of \(\overline{\rho }\) alone correspond to a broader region forming a plateau in Fig. 6b.