Comparison of deep learning models for predictive maintenance in industrial manufacturing systems using sensor data

System framework

In this paper, we propose a comprehensive framework for comparing deep learning models in predictive maintenance (PdM) applications for industrial manufacturing systems using sensor data. The proposed framework consists of several key components, including data acquisition and preprocessing, feature extraction and selection, model training and evaluation, and model deployment and integration.

Figure 1 presents the proposed system framework for deep learning-based predictive maintenance using sensor data. The framework consists of data acquisition, preprocessing, feature extraction, model construction, and evaluation components, which work together to enable accurate and reliable prediction of equipment failures and optimization of maintenance strategies.

The first component of the framework is data acquisition and preprocessing. Sensor data is collected from various equipment and processes in the industrial manufacturing system, including condition monitoring data and process data. The raw sensor data may contain noise, outliers, and missing values, which need to be addressed through preprocessing techniques such as data cleaning, filtering, and normalization. The preprocessed data is then split into training, validation, and testing sets for model development and evaluation.

The second component of the framework focuses on feature extraction and selection. Deep learning models have the ability to automatically learn hierarchical features from raw sensor data. However, in some cases, it may be beneficial to extract domain-specific features or select relevant features to improve model performance and interpretability. Techniques such as time-frequency analysis, wavelet transforms, or statistical feature extraction can be applied to the preprocessed sensor data to obtain informative features. Feature selection methods, such as correlation analysis or wrapper-based approaches, can be used to identify the most relevant features for PdM tasks.

The third component of the framework involves model training and evaluation. Various deep learning architectures, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variants, are employed to learn patterns and representations from the sensor data. The choice of architecture depends on the specific characteristics of the data and the PdM task, such as fault detection, diagnosis, or prognostics. The models are trained using the preprocessed sensor data and the extracted features, optimizing their parameters to minimize a predefined loss function. The performance of the trained models is evaluated using appropriate metrics, such as accuracy, precision, recall, or mean squared error, on the validation and testing sets. Model selection and hyperparameter tuning are performed to identify the best-performing models for each PdM task.

The fourth component of the framework focuses on model deployment and integration. The trained deep learning models are deployed in the industrial manufacturing system for real-time PdM applications. The models continuously process incoming sensor data, providing predictions or alerts regarding the health and remaining useful life (RUL) of equipment. The predictions are integrated with existing maintenance systems and decision support tools to optimize maintenance scheduling, resource allocation, and spare parts inventory management. The deployed models are monitored and periodically retrained or updated to adapt to changes in the manufacturing environment or equipment behavior.

The proposed framework also incorporates techniques to address the challenges associated with deep learning-based PdM, such as data quality, model interpretability, and scalability. Data quality assessment and improvement methods, such as outlier detection and data augmentation, are applied to ensure the reliability and representativeness of the sensor data. Model interpretability techniques, such as attention mechanisms or layer-wise relevance propagation, are employed to provide insights into the decision-making process of the deep learning models. Scalable architectures and distributed computing techniques are utilized to handle large volumes of sensor data and enable efficient model training and deployment.

By integrating these components into a cohesive framework, we aim to provide a comprehensive approach for comparing and evaluating deep learning models in PdM applications using sensor data. The framework enables the systematic development, validation, and deployment of deep learning-based PdM solutions, facilitating the adoption of proactive maintenance strategies in industrial manufacturing systems.

Data acquisition and preprocessing

Data acquisition and preprocessing are crucial steps in the development of deep learning-based predictive maintenance (PdM) models for industrial manufacturing systems. The quality and relevance of the sensor data directly impact the performance and reliability of the PdM models. In this section, we discuss the methods for collecting operational data from manufacturing equipment using various sensors and the preprocessing steps required to prepare the data for deep learning models.

Data acquisition

In industrial manufacturing systems, a wide range of sensors are employed to monitor the condition and performance of equipment. These sensors capture different types of data, such as vibration, temperature, pressure, acoustic emissions, and electrical signals, which can provide valuable insights into the health and degradation patterns of the equipment.

Vibration sensors, such as accelerometers and proximity probes, are commonly used to monitor the mechanical condition of rotating machinery, such as motors, pumps, and bearings. These sensors measure the acceleration or displacement of the equipment, capturing information about vibration amplitude, frequency, and phase. Vibration data can reveal abnormalities in the equipment’s operation, such as imbalance, misalignment, or bearing defects.

Temperature sensors, including thermocouples and resistance temperature detectors (RTDs), are used to monitor the thermal condition of equipment. Abnormal temperature readings can indicate issues such as overheating, friction, or insufficient lubrication. Temperature data can be collected from various locations on the equipment, such as bearings, gearboxes, or electrical components.

Pressure sensors, such as strain gauges and piezoelectric transducers, are employed to monitor the pressure levels in hydraulic and pneumatic systems. Abnormal pressure readings can indicate leaks, blockages, or equipment wear. Pressure data can be collected from different points in the system, such as pumps, valves, or cylinders.

Acoustic emission sensors detect high-frequency elastic waves generated by the rapid release of energy from sources within the equipment, such as crack propagation or material deformation. Acoustic emission data can provide early indications of structural degradation or fatigue in equipment components.

Electrical sensors, such as current transformers and voltage transducers, monitor the electrical characteristics of equipment, including motors, generators, and power electronics. Abnormalities in electrical data, such as current imbalances or voltage distortions, can indicate faults in the electrical components or control systems.

In addition to these condition monitoring sensors, process sensors are used to collect data related to the manufacturing process itself. These sensors measure variables such as production rate, quality metrics, or energy consumption. Process data can provide context for interpreting the condition monitoring data and help identify the impact of process variations on equipment health.

The data acquisition process involves the installation of sensors on the critical components or subsystems of the manufacturing equipment. The sensors are connected to data acquisition systems, such as programmable logic controllers (PLCs) or industrial gateways, which sample and digitize the sensor signals. The sampled data is then transmitted to a central database or cloud platform for storage and further processing.

Data preprocessing

Raw sensor data collected from manufacturing equipment often contains noise, outliers, and missing values, which can adversely affect the performance of deep learning models. Therefore, data preprocessing is essential to clean, transform, and prepare the data for model training and evaluation.

The first step in data preprocessing is data cleaning, which involves handling missing values and outliers. Missing values can occur due to sensor malfunctions, communication issues, or data transmission errors. These missing values can be imputed using techniques such as mean imputation, median imputation, or k-nearest neighbor imputation, depending on the characteristics of the data and the missing value patterns. Outliers, which are data points that significantly deviate from the normal behavior, can be detected using statistical methods, such as Z-score or interquartile range (IQR), and either removed or replaced with interpolated values.

The next step is data normalization or scaling, which brings the sensor data from different sources and ranges to a common scale. Normalization helps improve the convergence and stability of deep learning models during training. Common normalization techniques include min-max scaling, which scales the data to a fixed range (e.g., [0, 1]), and standardization, which transforms the data to have zero mean and unit variance.

Feature extraction is another important preprocessing step that aims to extract informative and discriminative features from the raw sensor data. Deep learning models have the ability to automatically learn hierarchical features from data, but in some cases, extracting domain-specific features can improve model performance and interpretability. Time-domain features, such as statistical moments (mean, standard deviation, skewness, kurtosis), root mean square (RMS), and peak-to-peak value, can be extracted from the sensor data to capture the overall characteristics of the signals. Frequency-domain features, obtained through techniques like Fourier transform or wavelet transform, can reveal the spectral content and dynamics of the sensor data. Time-frequency domain features, such as short-time Fourier transform (STFT) or wavelet packet transform (WPT), can capture the time-varying spectral characteristics of the signals.

After feature extraction, feature selection techniques can be applied to identify the most relevant and informative features for PdM tasks. Feature selection helps reduce the dimensionality of the data, mitigate overfitting, and improve model interpretability. Techniques such as correlation analysis, mutual information, or wrapper-based methods can be used to rank and select the most discriminative features.

Finally, the preprocessed and transformed sensor data is split into training, validation, and testing sets for model development and evaluation. The training set is used to train the deep learning models, the validation set is used to tune the hyperparameters and assess the model’s performance during training, and the testing set is used to evaluate the final model’s performance on unseen data.

By following these data acquisition and preprocessing steps, we ensure that the sensor data is of high quality, informative, and suitable for training deep learning models for PdM applications. The preprocessed data is then fed into the deep learning models, which learn patterns and representations from the data to detect, diagnose, and predict equipment failures.

Deep learning model construction

In this section, we focus on the construction and training of deep learning models for predictive maintenance (PdM) tasks using sensor data from industrial manufacturing systems. We explore several mainstream deep learning architectures, including Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks, and discuss their network design and training methodologies.

Convolutional neural networks (CNNs)

CNNs have shown remarkable success in various domains, particularly in image and signal processing tasks. In the context of PdM, CNNs can be employed to learn hierarchical features from raw sensor data, capturing local dependencies and patterns that are indicative of equipment health and degradation.

The typical CNN architecture consists of convolutional layers, pooling layers, and fully connected layers. The convolutional layers apply a set of learnable filters to the input data, extracting local features at different spatial scales. The pooling layers downsample the feature maps, reducing the spatial dimensions and providing translation invariance. The fully connected layers then aggregate the learned features and perform the final prediction or classification task.

For PdM tasks using sensor data, the CNN architecture can be designed to handle time-series data by treating the sensor signals as 1D or 2D images. In the case of 1D CNNs, the convolutional filters move along the time axis, capturing temporal dependencies in the sensor data. For 2D CNNs, the sensor data can be transformed into a 2D representation, such as a spectrogram or a wavelet scalogram, allowing the network to learn both temporal and frequency-domain features.

The hyperparameters of the CNN, such as the number of convolutional layers, filter sizes, and number of filters, can be tuned to optimize the network’s performance for specific PdM tasks. Table 1 presents an example of the network architecture parameters for a CNN model.

Long short-term memory (LSTM) networks

LSTM networks are a type of recurrent neural network (RNN) that are well-suited for modeling sequential data with long-term dependencies. In the context of PdM, LSTM networks can capture the temporal evolution of sensor data, learning patterns and relationships that span multiple time steps.

The core component of an LSTM network is the LSTM cell, which consists of an input gate, a forget gate, an output gate, and a memory cell. The input gate controls the flow of new information into the cell, the forget gate determines what information to discard from the cell state, and the output gate controls the output of the cell. The memory cell maintains a state vector that can store long-term information, allowing the network to capture long-range dependencies in the sensor data.

The forward propagation equations for an LSTM cell are as follows:

where \(\:{i}_{t}\), \(\:{f}_{t}\), and \(\:{o}_{t}\) are the input, forget, and output gates, respectively; \(\:{\stackrel{\sim}{C}}_{t}\) is the candidate cell state; \(\:{C}_{t}\) is the cell state; \(\:{h}_{t}\) is the hidden state; \(\:{W}_{i}\), \(\:{W}_{f}\), \(\:{W}_{o}\), and \(\:{W}_{C}\) are the weight matrices; \(\:{b}_{i}\), \(\:{b}_{f}\), \(\:{b}_{o}\), and \(\:{b}_{C}\) are the bias vectors; \(\:\sigma\:\) is the sigmoid activation function; and \(\:\odot\:\) denotes element-wise multiplication.

For PdM tasks, the LSTM network can be designed to process the sensor data sequentially, with each time step corresponding to a sample or a window of samples. The hidden states of the LSTM cells capture the temporal dependencies and patterns in the sensor data, allowing the network to make predictions based on the historical context.

Model training and optimization

The training of deep learning models for PdM tasks involves optimizing the network parameters to minimize a predefined loss function. The choice of loss function depends on the specific task, such as binary cross-entropy for fault detection or mean squared error for remaining useful life (RUL) estimation.

The binary cross-entropy loss function is commonly used for binary classification tasks, such as fault detection, and is defined as:

where \(\:N\) is the number of samples, \(\:{y}_{i}\) is the true label (0 or 1), and \(\:{\widehat{y}}_{i}\) is the predicted probability.

The training process typically involves the following steps:

1. Initialize the network parameters randomly or using pre-trained weights.

2. Feed the preprocessed sensor data to the network in mini-batches.

3. Compute the forward pass through the network to obtain the predictions.

4. Calculate the loss function based on the predicted and true labels.

5. Perform backpropagation to compute the gradients of the loss with respect to the network parameters.

6. Update the network parameters using an optimization algorithm, such as stochastic gradient descent (SGD) or Adam.

7. Repeat steps 2–6 for a specified number of epochs or until a convergence criterion is met.

During training, techniques such as learning rate scheduling, regularization (e.g., L1/L2 regularization, dropout), and early stopping can be employed to improve model generalization and prevent overfitting. The trained models are then evaluated on a separate test set to assess their performance on unseen data.

Model evaluation and selection

The performance of the trained deep learning models is evaluated using appropriate metrics based on the PdM task. For fault detection, metrics such as accuracy, precision, recall, and F1-score can be used to assess the model’s ability to correctly identify faulty conditions. For RUL estimation, metrics such as mean absolute error (MAE), mean squared error (MSE), or root mean squared error (RMSE) can be used to measure the model’s prediction accuracy.

Model selection involves comparing the performance of different deep learning architectures and hyperparameter configurations to identify the best-performing model for the specific PdM task. This can be done using techniques such as cross-validation, where the data is split into multiple subsets, and the models are trained and evaluated on different combinations of these subsets. The model with the best average performance across the cross-validation folds is selected as the final model.

In summary, the construction and training of deep learning models for PdM tasks using sensor data involve designing appropriate network architectures, such as CNNs and LSTM networks, and optimizing their parameters using suitable loss functions and training methodologies. The trained models are evaluated using relevant performance metrics, and the best-performing model is selected for deployment in the industrial manufacturing system.

Model evaluation metrics

Evaluating the performance of deep learning models for predictive maintenance (PdM) tasks is crucial to assess their effectiveness and reliability in real-world industrial manufacturing systems. In this section, we define several evaluation metrics commonly used to measure the predictive performance of deep learning models, including accuracy, precision, recall, and F1-score.

Accuracy

Accuracy is a widely used metric that measures the overall correctness of the model’s predictions. It is defined as the ratio of the number of correct predictions to the total number of predictions. In the context of PdM, accuracy can be calculated as:

where \(\:TP\) (true positives) is the number of correctly predicted faulty samples, \(\:TN\) (true negatives) is the number of correctly predicted healthy samples, \(\:FP\) (false positives) is the number of healthy samples incorrectly predicted as faulty, and \(\:FN\) (false negatives) is the number of faulty samples incorrectly predicted as healthy.

Accuracy provides an overall measure of the model’s performance but may not be sufficient when the data is imbalanced, i.e., when the number of samples in different classes is significantly different.

Precision

Precision, also known as positive predictive value, measures the proportion of correctly predicted faulty samples among all samples predicted as faulty. It is calculated as:

Precision focuses on the model’s ability to minimize false positives, which is important in PdM tasks to avoid unnecessary maintenance actions and associated costs.

Recall

Recall, also known as sensitivity or true positive rate, measures the proportion of correctly predicted faulty samples among all actual faulty samples. It is calculated as:

Recall focuses on the model’s ability to minimize false negatives, which is critical in PdM tasks to prevent unexpected equipment failures and downtime.

F1-score

The F1-score is the harmonic mean of precision and recall, providing a balanced measure of the model’s performance. It is calculated as:

The F1-score is particularly useful when both precision and recall are important, as it penalizes extreme values and favors a balance between the two metrics.

In addition to these metrics, other evaluation measures can be used depending on the specific requirements of the PdM task. For example, in the case of remaining useful life (RUL) estimation, metrics such as mean absolute error (MAE), mean squared error (MSE), or root mean squared error (RMSE) can be employed to assess the model’s prediction accuracy.

It is important to consider multiple evaluation metrics when assessing the performance of deep learning models for PdM tasks, as each metric provides a different perspective on the model’s predictive capabilities. By combining these metrics, a comprehensive understanding of the model’s strengths and limitations can be obtained, enabling informed decision-making and model selection.

Furthermore, the evaluation metrics should be calculated using a separate test set that the model has not seen during training. This ensures an unbiased assessment of the model’s generalization ability and performance on unseen data, which is crucial for the reliable deployment of PdM models in real-world industrial manufacturing systems.

In summary, the evaluation of deep learning models for PdM tasks using sensor data involves the use of various performance metrics, including accuracy, precision, recall, and F1-score. These metrics provide a quantitative assessment of the model’s predictive capabilities and help in selecting the most suitable model for deployment in industrial manufacturing systems.