ReactorNet based on machine learning framework to identify control rod position for real time monitoring in PWRs

In order to increase performance and speed up training convergence, our proposed architecture uses pre-trained weights and is based on EfficientNetB0, as illustrated in Fig. 8. This network is designed for multi-task learning, integrating classification and regression tasks inside a singular, efficient architecture. The process starts with an input layer that receives RGB pictures measuring 224 $\times$ 224 pixels. The backbone employs EfficientNetB0, a lightweight convolutional neural network tuned for performance, which extracts high-level features and produces a 7 $\times$ 7 $\times$ 1280 feature map. A Global Average Pooling 2D layer then compresses these characteristics into a one-dimensional vector of length 1280, reducing spatial dimensions while maintaining channel information. The model then processes this vector via a Dense layer with 256 units with ReLU activation, therefore creating non-linearity and projecting characteristics into a lower-dimensional space. Batch Normalization stabilizes training by normalizing activations, followed by a Dropout layer (rate = 0.5) to reduce overfitting by randomly deactivating fifty percent of the units throughout training. The architecture diverges into many task-specific branches. First, Classification heads forecast categorical results for four separate tasks (6-class, 5-class, 5-class, and 5-class outputs, designated B, T, D, and R). Second, Regression heads forecast continuous numerical values (e.g., bounding box coordinates or scores), with a minimum of one explicit regression output (K).

The ideal starting point for feature extraction in this complicated multiclass prediction issue is EfficientNetB0, which is renowned for its balance of computing efficiency and accuracy. EfficientNet employs a compound scaling algorithm to consistently scale the depth, breadth, and resolution dimensions; it is a CNN design. First, Neural Architecture Search (NAS) has been used to build a baseline network. NAS is a tool for automating neural network construction to optimize accuracy and efficiency, quantified by floating-point operations per second. Then, the architecture utilizes mobile inverted bottleneck convolutions (MBConv), and the EfficientNets models are created by scaling up the baseline network, as shown in Fig. 9.

The EfficientNet family of convolutional neural network models uses compound scaling to evenly scale depth, breadth, and resolution⁴⁴. The baseline model in this family, EfficientNetB0, optimizes accuracy and computing efficiency in FLOPS. EfficientNetB0 has a significantly bigger architecture than MnasNet, mainly because to a higher FLOPS objective⁵⁴. EfficientNet models improve performance using MBConv and squeeze-and-excitation layers. The EfficientNetB0 design is illustrated in Table 2. EfficientNetB0 is scaled up via compound scaling to create the remainder of the EfficientNet family. Suitable scaling coefficients have been found based on grid searching as $\alpha =1.2, \beta = 1.1, \gamma = 1.15, \phi = 1$. These fixed coefficients were used to generate EfficientNetB1−EfficientNetB7 models with increasing capacity. Optimal for mobile and embedded systems, EfficientNet models decrease computing costs, power consumption, and training and inference times for deep learning applications on edge devices.

Table 2 Architecture of EfficientNetB0⁴⁴.

EfficientNetB0 customization

EfficientNetB0 employs a compound scaling approach, which balances network depth, width, and resolution. This scaling is defined as $\text {depth} = \alpha ^d$, $\quad \text {width} = \beta ^d$, and $\quad \text {resolution} = \gamma ^d$ where $\alpha$, $\beta$, and $\gamma$ are constants that control the scaling, and $d$ is a factor that determines the scaling for each dimension, ensuring an efficient balance across network parameters. The architecture processes an input image $X$ through convolutional, Squeeze-and-Excitation (SE) blocks, and pooling layers, generating feature maps $F(X)$ that capture relevant spatial and hierarchical information⁴⁴:

$$\begin{aligned} F(X) = \text {EfficientNetB0}(X) \end{aligned}$$

(9)

These feature maps serve as a high−dimensional representation, which is fed into customized layers for classification and regression tasks. The EfficientNetB0 Layers and Parameters are summarized as: Input Layer, which accepts images of size $224 \times 224 \times 3$, preparing the model for image data. Rescaling and Normalization, which Standardizes input data, and ensuring consistent values across the network. Convolutional Layers are Incorporated batch normalization and activation layers, and learning spatial hierarchies progressively. SE Blocks are used to enhance the model’s ability to capture essential features by adjusting the importance of different channels. Global average pooling is defined as condensing spatial information after convolutional layers and preparing for dense layers. Finally, custom dense layers are added to meet the specific output requirements.

For the classification heads, each output (B, T, D and R) has a dense layer with a softmax activation function for probability distributions⁵⁰. For each classification output $\hat{y}_i$, the softmax activation is defined as $\hat{y}_i = \frac{e^{z_i}}{\sum _{j=1}^{C} e^{z_j}}$ where $z_i$ is the logit for class $i$, and $C$ is the total number of classes for each output. While the K output (Regression Head) has a dense layer with a linear activation function to predict continuous values. The regression error is minimized using MAE, calculated as⁵⁵:

$$\begin{aligned} \text {MAE} = \frac{1}{N} \sum _{i=1}^{N} \left| \hat{y}_i – y_i \right| \end{aligned}$$

(10)

where $N$ is the number of samples, $\hat{y}_i$ is the predicted value, and $y_i$ is the true value. Also, the pre-trained weights on the ImageNet dataset provide a strong baseline of visual representations, allowing the model to leverage learned features effectively. Initially freezing the base layers prevents overfitting to domain-specific noise and speeds up convergence. Fine-tuning these layers later adapts the model to our specific data, enhancing its ability to recognize complex patterns unique to reactor monitoring.

To examine the Layer-by-Layer Breakdown and Advanced Techniques, we focus on Depth wise Convolution and SE Blocks. Depth wise convolutions diminish parameter quantities while preserving spatial feature extraction efficacy. While SE blocks readjust feature maps, augmenting channel significance for enhanced classification and regression efficacy. Additionally, there exist Custom Dense Layers: Each dense layer incorporates batch normalization and dropout layers, with dropout serving as a regularization technique to mitigate overfitting by randomly excluding units during training. By employing EfficientNetB0, our model attains significant efficiency, retaining a compact size (about 4.38 million parameters) while providing robust performance in classification and regression tests. This equilibrium renders it optimal for real-time and resource-constrained applications, such as reactor parameter monitoring.

Training strategy

Our training strategy was designed to leverage transfer learning, optimize performance for both classification and regression tasks, and ensure model robustness through cross-validation and rigorous hyperparameter tuning.

Transfer learning and efficientNetB0 utilization

We utilized EfficientNetB0 with pre-trained ImageNet weights as the feature extractor. Initially, the base layers were frozen to prevent overfitting and allow the model to utilize general image features from the pre-trained weights. Later, these layers were unfrozen and fine-tuned, allowing the model to adapt specifically to our dataset. This transfer learning approach provided two primary benefits. First, Faster Convergence; EfficientNetB0’s pre-trained weights enabled the model to start from a knowledgeable baseline, reducing training time and improving convergence speed. Second, Improved Accuracy; By leveraging ImageNet-trained features, the model could adapt high-level visual features relevant to our task, achieving a high level of accuracy even in complex, multi-output prediction scenarios.

Cross-validation

To ensure our model’s robustness and generalizability, we applied 5-fold cross-validation. Each fold used 80% of the data for training and 20% for validation, providing a comprehensive evaluation across various data subsets and reducing the risk of overfitting. The performance was averaged over all folds, yielding a reliable estimate. Mathematically, the average loss, $\mathscr {L}_{\text {cv}}$, across folds can be expressed as:

$$\begin{aligned} \mathscr {L}_{\text {cv}} = \frac{1}{K} \sum _{k=1}^{K} \mathscr {L}_k \end{aligned}$$

(11)

where $K = 5$ is the number of folds and $\mathscr {L}_k$ is the loss for each fold $k$.

Data augmentation

To enhance the model’s generalization capabilities, we applied data augmentation techniques such as random rotations, zooms, and horizontal flips. These transformations simulated variations in the input data, making the model more robust to unseen instances and helping reduce overfitting. The augmentation transformations $T_a$ applied to each sample $X$ created a diverse training set $T_a(X)$ where:

$$\begin{aligned} T_a(X) = \text {Augment}(X; \text {rotation}, \text {zoom}, \text {flip}) \end{aligned}$$

(12)

Loss functions

Since our model handles both classification and regression tasks, we designed a multi-loss function to balance these outputs. The total loss $\mathscr {L}$ for each sample combines classification losses for each output class $(B, T, D, R)$ using categorical cross-entropy, and the regression loss for $K$ using Mean Absolute Error (MAE). This is expressed as:

$$\begin{aligned} \mathscr {L} =\sum _{i \in \{\text {B, T, D, R}\}}\alpha _i \cdot \text {CrossEntropy}(y_i, \hat{y}_i)+\beta \cdot \text {MAE}(y_{\text {K}}, \hat{y}_{\text {K}}) \end{aligned}$$

(13)

where $\alpha _i$ and $\beta$ are weighting factors for each classification and regression task, determined by their relative importance. The categorical cross-entropy for the ith class is given by:

$$\begin{aligned} \text {CrossEntropy}(y, \hat{y}) = – \sum _{c=1}^{C} y_c \log (\hat{y}_c) \end{aligned}$$

(14)

with $y$ as the true one-hot encoded label, $\hat{y}$ as the predicted probability vector, and $C$ is the total number of classes for each output. For regression, the MAE loss for $K$ is:

$$\begin{aligned} \text {MAE} = \frac{1}{N} \sum _{j=1}^{N} \left| y_{\text {K}, j} – \hat{y}_{\text {K}, j} \right| \end{aligned}$$

(15)

where $N$ is the batch size, and $y_{\text {K}, j}$ and $\hat{y}_{\text {K}, j}$ are the true and predicted values for the $K$ output.

Custom heads for classification and regression

Each output (B, T, D, R) is assigned a softmax layer for classification, while the regression output $K$ is configured with a linear layer. These custom heads enable specialized processing for each target, further improving model performance. The softmax classification prediction $\hat{y}_i$ for a class $i$ is defined as:

$$\begin{aligned} \hat{y}_i = \frac{e^{z_i}}{\sum _{j=1}^{C} e^{z_j}} \end{aligned}$$

(16)

where $z_i$ represents the logit for class $i$. For the regression output $K$, a linear activation function allows for continuous value prediction, essential for real-valued targets.

Hyperparameter tuning

After initial experiments, we fine-tuned key hyperparameters, such as learning rate and batch size. The final learning rate was set to $0.0001$ with a reduction schedule, allowing for gradual convergence as training progressed. A decay schedule for the learning rate $\eta$ at epoch $e$ is given by:

$$\begin{aligned} \eta _e = \eta _0 \cdot \text {decay}\_\text {rate}^{\frac{e}{\text {decay}\_\text {steps}}} \end{aligned}$$

(17)

where $\eta _0$ is the initial learning rate, ensuring stable convergence towards an optimal solution.

Evaluation metrics

Classification outputs (B, T, D, R) were evaluated using accuracy, while the regression output $K$ was evaluated using MAE. This combination of metrics enabled us to comprehensively assess both discrete and continuous predictions, aligning evaluation closely with the multi-output structure.

Test-time augmentation (TTA)

TTA is a powerful technique applied during inference to improve the robustness and generalization of model predictions by introducing multiple, slightly modified versions of each test sample. By making predictions on these augmented versions and aggregating the results, TTA reduces model sensitivity to minor input variations, which is particularly useful in tasks requiring stable predictions, such as reactor behaviour analysis.

During TTA, we generate $n$ augmented versions $x_i′$ of each test sample $x$, where $i = 1, \ldots , n$. Each augmented sample $x_i′$ produces a prediction $\hat{y}_i$. The final prediction $\hat{y}$ is obtained by averaging the individual predictions for regression tasks or taking the mode for classification tasks:

$$\begin{aligned} \hat{y}= {\left\{ \begin{array}{ll} \frac{1}{n} \sum _{i=1}^{n} \hat{y}_i & \textrm{regression}\\ \text {mode}(\{\hat{y}_i\}_{i=1}^{n}) & \textrm{classification} \end{array}\right. } \end{aligned}$$

(18)

This approach helps ensure consistent model performance across varied input conditions, enhancing generalization by effectively creating an ensemble of predictions based on different transformations of the same data. Thus, we choose augmentations to imitate real-world input fluctuations to improve model generalization on varied scenarios, where minor rotations ($\pm 10^\circ$) imitate perspective shifts, enhancing orientation variation robustness. In addition, Flipping has been used during data augmentation to improve the model’s robustness, ensuring it is invariant to changes caused by horizontal or vertical reflections. Finally, minor scaling and cropping tweaks increased model scale robustness.

These augmentations are consistent with transformations applied during training, ensuring the model encountered familiar data variations at test time. By focusing on aggregation of predictions, for classification outputs (B, T, D, and R), we average the predictions from each augmented version to obtain the final class prediction. For the regression output, K, the MAE was averaged across augmented predictions, yielding a smoother and more reliable output:

$$\begin{aligned} \hat{y}_K = \frac{1}{n} \sum _{i=1}^{n} \hat{y}_{K_i} \end{aligned}$$

(19)

The advantages of TTA include enhanced model robustness to input variations. Real-world data frequently contains little discrepancies that the model may not have encountered during training. TTA addresses this by presenting the model with a range of enhanced inputs, resulting in more dependable predictions across different contexts. Additionally, aggregating predictions from various enhanced inputs diminishes the model’s dependence on particular details within an individual input. This method diminishes the probability of overconfident forecasts, successfully mitigating bias and variation. Furthermore, utilizing several augmentations, TTA operates akin to an ensemble, resulting in more reliable and stable predictions. The diversity in input circumstances enables the model to discern intricate patterns. Additionally, TTA enhances the model’s capacity to generalize to unfamiliar data by evaluating it against minor modifications of each input sample. This method is especially advantageous for edge circumstances that may be inadequately represented in the training data. Furthermore, testing data frequently contains noise or defects that may influence predictions. Through the average of predictions over augmentations, TTA mitigates the effects of noise, resulting in more dependable outputs.

Nonetheless, TTA possesses several drawbacks. As TTA produces several predictions for each input, the inference duration escalates in direct correlation to the number of augmentations. The selection of augmentation kinds and the quantity of augmented samples necessitate meticulous calibration. Inadequately selected augmentations may result in worse outcomes. TTA is especially advantageous for intricate models, like CNNs, but may not provide substantial advantages for simpler models.

In summary, TTA improves model resilience and generalization by implementing various augmentations during inference, hence increasing the reliability of predictions, particularly in critical applications such as reactor behaviour analysis. Although TTA incurs greater inference time, its enhancements to prediction accuracy and stability render it essential for workloads demanding high dependability and consistency.

Source link