An intelligent object detection and classification framework for assisting visually challenged persons using deep learning and improved crow search optimization

This study introduced an HDLMODC-ICSA method. The HDLMODC-ICSA method primarily focuses on an accurate and real-time object recognition method to aid visually challenged persons. To accomplish that, the proposed HDLMODC-ICSA method involves various methods such as image pre-processing, OD, feature extraction, classification, and hyperparameter tuning. Figure 1 depicts the overall workflow of the HDLMODC-ICSA approach.

Image Pre-processing using MF

In the initial stage, the image pre-processing stage applies MF to remove noise or distortions to make the image more transparent²⁹. This model is chosen due to its robust capability to remove impulse (salt-and-pepper) noise while preserving crucial edge details. Unlike mean or Gaussian filters that blurrs image features, MF maintains structural integrity, which is significant for accurate OD and feature extraction. Its non-linear nature allows it for handling high-frequency noise efficiently without altering the underlying image content. MF is also computationally effectual and easy to implement, making it appropriate for real-time or large-scale applications. Compared to more intrinsic denoising techniques, MF presents a balanced trade-off between performance and simplicity. This makes it an ideal choice for improved image quality in the proposed pipeline. Figure 2 specifies the structure of the MF model.

MF is a non-linear image processing mode employed to decrease noise, particularly salt-and-pepper noise while maintaining edges in an image. In object recognition and classification tasks, it aids in improving image quality by eliminating unwanted noise, which can inhibit precise object localization and feature extraction. MF smoothing an image creates more effective edge recognition and segmentation models. It is mainly beneficial in lower-quality or noisy images, enhancing the performance of conventional and DL-based methods. However, extreme filtering can blur significant features, so it must be used sensibly. Overall, MF is a beneficial pre-processing stage for enhancing classification and detection accuracy.

OD using faster R-CNN method

Next, the OD process is performed by the Faster R-CNN model to generate precise region proposals and detect objects within images efficiently³⁰. This model is chosen due to its high accuracy and efficiency in detecting objects with varying scales and complex backgrounds. Unlike conventional methods or earlier R-CNN variants, it combines region proposal and classification into a single, end-to-end trainable network, significantly mitigating processing time. Its region proposal network (RPN) allows for fast and precise localization of objects, which is crucial for downstream tasks like feature extraction. Compared to models such as YOLO or SSD, Faster R-CNN gives superior detection performance, particularly in conditions needing high localization precision. Its robustness across diverse datasets makes it a reliable choice for real-world applications. This justifies its role as the OD backbone in the proposed model. Figure 3 demonstrates the structure of the Faster R-CNN method.

Faster R-CNN is the renowned OD model for its higher precision. It contains dual modules: a detection system and RPN. The key basis of Faster R‐CNN includes the succeeding stages:

1. 1)

   base network: utilize a pre-trained CNN to remove features from the input image

2. 2)

   RPN: It makes candidate targeted areas by sliding windows over the feature mapping, considering them as anchor boxes, and forecasting whether the anchor boxes comprise objects and how to fine-tune the limits of the anchor boxes.

3. 3)

   Region of interest pooling (RoI Pooling): Split all candidates’ areas into predetermined size subfields and map them on the static size feature maps.

4. 4)

   Object classification networks: Utilize fully connected (FC) systems for classifying all candidate areas, seizing the RoI Pooling output as input, and outputting the possibilities of all candidate areas characterized by various targeted classes.

5. 5)

   Bounding box regression: Implement this regression to fine-tune the coordinates of the bounding box targets in all candidate areas.

To carry out OD in Faster R-CNN, they utilize the RPN to make candidate areas. The RPN phase includes dual primary equations, one to compute the coordinate of anchor boxes and the other to calculate the loss amongst ground truth bounding boxes and the anchor boxes. At last, they utilize the succeeding equations for computing the anchor boxes coordinates and establish their locations within the images:

\(\:{x}_{anchor}\) and \(\:{y}_{anchor}\): The upper left coordinate of the anchor boxes represents their location within the images. \(\:{x}_{center}\) and \(\:{y}_{center}\): The coordinates of the central point of the anchor or target boxes are applied to define the location of the anchor boxes. \(\:{w}_{anchor}\) and \(\:{h}_{anchor}\): The width and height of the anchor boxes utilized to establish their dimensions. \(\:widt{h}_{anchor}\) and \(\:heigh{t}_{anchor}\): The pre-defined width and height of the anchor boxes, normally fixed as static values in training. Continuing these, they present the Smooth Ll Loss as the loss function amongst ground truth bounding and anchor boxes. The calculation equation for these loss functions is demonstrated:

Whereas \(\:{t}_{i}\) characterizes the forecast bounding boxes offset, \(\:{t}_{i}^{{\prime\:}}\) characterizes the consistent ground truth bounding boxes offset, and \(\:{1}_{i}\) means positive is the indicator function, which captures a value of 1 after the anchor boxes \(\:i\) represent positive instances. During Faster \(\:R-CNN\), there is additionally an essential phase of computing the output of the \(\:RoI\) pooling layer. The pooling layer of \(\:RoI\) is applied for mapping RoIs of dissimilar dimensions on a predetermined size feature mapping to maintain the RoI feature’s spatial alignment. The succeeding equation is utilized to calculate the \(\:RoI\) pooling layer output:

\(\:{F}_{conv}\) characterizes the convolution feature mapping gained from the feature extraction system, and \(\:p\) symbolizes the input parameters for the pooling layer of\(\:\:RoI\), using the size information and coordinates of the RoI. The pooling layer of \(\:RoI\) maps RoI’s of various sizes on predetermined size feature maps to maintain the RoI feature’s spatial alignment.

The principle of this stage is to carry out pooling processes on the feature mapping areas comparable to \(\:RoIs\) of dissimilar dimensions, leading to predetermined size \(\:RoI\) features. This permits mapping \(\:RoIs\) of dissimilar dimensions on the similar-sized feature mapping, enabling succeeding bounding box regression and object classification. The detection system captures the candidate boxes from the RPN as input and implements bounding box regression and object classification. They utilize the following formulations to calculate the detection system outputs. The equation for object classification is as shown:

The equation for bounding box regression is as shown:

Whereas \(\:{W}_{cls},{b}_{cls},{W}_{reg}\), and \(\:{b}_{reg}\) are learned parameters. The softmax function is applied to transform the output of object classification into the likelihood distribution above class labels. Finally, they utilize the bounding box regression and object classification outcomes to filter out the end recognition outcomes. By establishing a threshold, they choose the targeted boxes with higher confidence as the last recognition outcomes and enhance their bounding box locations utilizing the bounding box regression outcomes for more precise localization. By incorporating the detection network and RPN, Faster R-CNN attains an accurate OD level and has proven essential performance developments on numerous benchmark datasets.

Feature extraction using improved LeNet-5 method

Furthermore, the proposed HDLMODC-ICSA method employs the Improved LeNet-5 model to extract meaningful and discriminative features from the identified regions³¹. This model is chosen due to its simplicity, efficiency, and adaptability to modifications that improve performance. By extending the original architecture with deeper layers, advanced activation functions, or batch normalization, the model can capture more complex and discriminative features. Compared to heavier networks namely VGG or ResNet, the improved LeNet-5 presents a lightweight alternative with lesser computational cost while maintaining robust feature representation. It is specifically efficient for mid-level feature extraction from localized regions, as given by Faster R-CNN. Its fast training and reduced parameter count make it ideal for integration in hybrid pipelines. These merits make it a balanced and effectual choice for the proposed framework. Figure 4 demonstrates the Improved LeNet-5 approach.

LeNet-5 method is presented as a CNN base method. The approach contains a 7‐layer structure and can execute pooling, convolution, and incorporating information functions. LeNet‐5 model architecture in which \(\:C\) represents the convolutional layer, \(\:S\) signifies the pooling layer, and \(\:{F}^{1}\) indicates an FC layer. All neurons on layer Cl are linked to an input neighbourhood. Layer S2 is a pooling layer gained by sampling from layer Cl. All neurons in layer F6 are connected to each of the neurons in the preceding layer. Convolutional processes improve the novel signal features and decrease noise.

Whereas \(\:l\) characterizes the layer, \(\:W\) signifies the convolutional kernel weighting parameter, \(\:{M}_{j}\) symbolizes a selection of input features, and \(\:b\) denotes the bias.

The combination of feature categories is carried out throughout the down-sampling process to decrease the spatial sizes and, therefore, mitigate the occurrence of overfitting. When the input feature count is \(\:n\), the feature counts after the down-sampling layer might be lower than or equivalent to \(\:n\), and the output feature dimensions should be smaller.

During this FC process, a single neuron in the FC layer is linked to each of the neurons in the previous layer. The FC layer incorporates the class-distinctive local information from the pooling and convolutional layers and transforms the feature information into a 1D vector. At last, the output of the FC layer is given to the output layer. To model the reduction of calculation and training time, LeNet‐5 CNN is enhanced in this study.

Initially, a small serial convolutional kernel was applied to replace the convolutional kernel at layer C3 within the LeNet-5 model. Dual 3 × 3 size convolutional kernels substitute the 5 × 5 size convolutional kernel. It is discovered that the parameter counts for convolutional cost utilizing dual 3 × 3 convolutional kernels are lower than one 5 × 5 convolutional kernel. Simultaneously, the network layer counts improve after a small convolution kernel is applied rather than the unique convolution kernel. The rise in the network layer counts assists in enhancing the precision of the model classification. Adding an activation function after the added convolutional layers enhances the method’s non-linear ability and enables the fitting of more composite functions. The probability of a convolutional layer regarding a single hidden neuron is:

On the other hand, \(\:E(v,\:h)\) characterizes the energy function under the Bernoulli distribution.

The probability-based solution reflects only 1 neuron; formerly, the probability of \(\:n\) hidden neurons presenting on the convolutional layer is stated as:

The probability of \(\:m\) neurons in the convolutional layer over the hidden layer (HL) is stated as shown:

The probability that the \(\:ith\) hidden neuron is activated for \(\:m\) visible neurons is demonstrated as:

Whereas \(\:w\) denotes the weight between the dual layers, \(\:c\) and \(\:b\) signify the respective offsets. For \(\:n\) hidden neurons, the probability that the \(\:jth\) visual unit is activated is indicated as shown:

Here, \(\:\sigma\:\left(\bullet\:\right)\) signifies the function of machine probability.

Formerly, the activation function of \(\:ReLU\) is applied to replace the activation function of Sigmoid. During this similar training atmosphere, the activation function of \(\:ReLU\) needs less training than the Sigmoid function. For \(\:N\) input samples \(\:v=\left\{{v}_{0},\:{v}_{1},\:\dots\:,\:{v}_{N}\right\}\), and \(\:{v}_{0},\) \(\:{v}_{1},\dots\:,{v}_{N}\) follow independent distributions.

The probability estimation for the sample set \(\:v\) is said as follows:

Here \(\:\theta\:=\{w,\:c,\:b\}\) signifies the energy parameter.

Solving for the maximal value of \(\:L\left(\theta\:\right)\) converts into finding a solution for the maximum value of \(\:L\left(\theta\:\right)\).

Now, \(\:\eta\:\) refers to the learning rate, and \(\:\eta\:>0.\).

The sample is logarithmically resolved for a single sample \(\:{v}_{0}=\left\{{v}_{01},\:{v}_{02},\:\dots\:,\:{v}_{0m}\right\}\).

Employ the partial derivative to \(\:\theta\:=\{w,\:c,\:b\}.\)

.

The probability distribution is exposed as shown:

The partial derivatives of \(\:\theta\:=\{w,\:c,\:b\}\) are gained for all three parameters \(\:w,c,b.\)

.

Then, the S4 pooling layer of the LeNet-5 CNN is adjusted with the spatial pyramid pooling (SPP) model to reduce the influence of the pooling process on the feature values. The cubes characterize the output of the feature maps from the convolutional layers. These feature maps are connected to the three pooling layers to obtain size 4 × 4, 2 × 2, and 1 × 1 outputs. The outputs of the three pooling layers are connected to get a 21‐dimensional vector. The results of performance are compared with the method with and without SPP. After assuming the SPP model, the error rate of the single-size and multiple-size trained network methods was reduced by 0.62% and 1.12%, respectively. Lastly, an FC layer is applied to replace the C5 layer of the LeNet‐5CNN. After the development, three convolutional layers are earlier than the C5 network layer. With no reduction in the convolutional layers, substituting the C5 layer with an FC layer additionally increases the stability and classification precision of the method.

Classification using ABS-BiLSTM model

For OD and classification, the hybrid of the ABS-BiLSTM technique is employed³². This model is chosen due to its capability to capture both past and future context in sequential data, which is significant for comprehending intrinsic patterns. The bidirectional structure improves the capability of the method to learn temporal dependencies, while the attention mechanism concentrates on the most relevant features, enhancing interpretability and accuracy. Compared to standard LSTM or unidirectional models, this technique exhibits superior performance in tasks needing deep contextual understanding. It is particularly effectual in scenarios with variable-length input and subtle feature variations. Its incorporation confirms robust classification outcomes in the proposed system. These advantages make it an ideal choice for high-level decision-making tasks. Figure 5 represents the structure of the ABS-BiLSTM model.

A stacked bi-directional LSTM neural network (NN) is the attention-based method fundamental to the architecture. After transforming the text into a fixed-length vector contribution, the structure is initially used to clean the data of the text, remove specific capitalization and characters, and keep only the particulars, which are retrieved as semantics. To discover the syntax of the sentence here, the features of the sequence of the text are initially examined by utilizing stacked Bi‐LSTM. Then, the self‐attention (SA) layer powerfully weighs characteristics, emphasizing contextual significance and tackling ambiguous words. Finally, a multi-layer view is applied to make the classification results. Embedding layer: By embedding usage, words are currently identified by computers and transformed into vectors. The NN language method allows NN and probability-based language processing for text methods by defining probability-based language method features through NNs. After a sentence is lengthy, \(\:h\{{C}_{1},\:{C}_{2},\:\dots\:{C}_{i},\:\dots\:{C}_{h}\}\), then utilize the learned word vector modelling to perform word vector mapping, which makes a word vector matrix from the sequence of words \(\:\{{D}_{1},\:{D}_{2},\:\dots\:{D}_{j},\:\dots\:{D}_{t}\}\), while the size of the matrix is \(\:h\text{*}d\), in such cases, \(\:d\) denotes the size of the word vectors.

Stacked Bi-LSTM-NN layer: An individual NN could efficiently remove complex data features. Deeper frameworks, namely the attention-based stacked Bi‐LSTM method, utilize the initial Bi‐LSTM layer output as input for the next. These models more efficiently solve complex things, thus improving the feature representation abilities of the model. Primarily the layer of Bi‐LSTM \(\:\{{D}_{1},\:{D}_{2},\:\dots\:{D}_{j},\:\dots\:{D}_{t}\}\), HL is typically transferred by the initial layer of Bi‐LSTM, while \(\:i\) represents an \(\:{i}^{th}\) word, the parameters \(\:{C}_{1},\) \(\:{z}_{1}\) be spread to the members of the layers. The connection of the forward and the reversed LSTM’s HL gives the information for the following layer of Bi‐LSTM, helping as the initial Bi‐LSTM layer output. The features vector representation is made through the next layer of Bi‐LSTM \(\:{t}_{i}\) of text by uniting HLs, which are forward and reverse. \(\:{t}_{i}\) refers to a graph that contains implied deeper‐level relations in a text that are important to increase the classification precision. The succeeding Eqs characterize the HL ti1. (25)\(\:-\)(27):

The following equations define the stacked Bi-LSTM’s final output within the next layer Eqs. (28)\(\:-\)(30):

Attention layer: Afterward, the stacked layer of Bi-LSTM, an SA mechanism, is used for weighting all contextual representations of word vectors, making the attention layer imitate every word’s importance to sentence semantics. Lastly, the proportionate summary is applied to make the global semantical representation of the sentence \(\:S\).

Stage1: Calculate the weighted score \(\:{e}_{i}\) that computes how much \(\:ith\) word gives to the understanding of uncertain terms. Equation (31)’ \(\:s\) output \(\:{h}_{j}\) through the FC technique having an activation objective of \(\:tanh\) acts as a computing process.

The computation of the activation function of \(\:tanh\:\)is Eq. (32):

Whereas \(\:{h}_{i}\) denotes a layer that particularly hides the adequate for the \(\:{C}_{c}\) training parameter and the \(\:{z}_{c}\) terms for bias denotes a state, \(\:ith\) expression is applied to define a hyperbolic at a tangent function, which captures some actual integer as input and outputs a value between \(\:(0\), 1). It is applied to process information and outputs; the nearer the output is to 1.0.

Stage 2: Define the attention weight of the word \(\:{\alpha\:}_{i}\). The function of Softmax is applied next standardization \(\:{e}_{i}^{T}\) and \(\:{u}_{c}\) dot-creation exposed in Eq. (33):

Here, \(\:{e}_{c}\) denotes the learnable context vector by an arbitrary beginning.

Stage 3: Eq. (34) displays how the computed weights for attention are used to stack Bi-LSTM’s HL outputs or clause vector \(\:S\).

The representation vector might be calculated using the model mentioned above for phrases with an SA mechanism.

The layer of output: The focal layer outputs a higher-dimensional depiction, \(\:S\), which is used as the feature vector. This feature vector, therefore, connects to the FC-HL, whereas the activation function of softmax, as defined in Eq. (35), transforms it into the \(\:N-dimensional\) vectors.

For training the projecting method, calculate the loss function of cross-entropy (\(\:P\)) between the ground‐truth label \(\:{a}_{j}\) and an anticipated label \(\:\widehat{a}\) utilizing Eq. (36) as shown:

\(\:\theta\:\) signifies the ability of the training parameter vector. Adjusting a parameter, which reflects the varying slope of the loss process, \(\:\theta\:.{\nabla\:}_{\theta\:}P\left(\theta\:\right)\) reduces the cross-entropy loss function utilizing a gradient descent optimizer method \(\:\theta\:\). Equation (37) denotes the value of the gradient descent model updated equation.

\(\:\gamma\:\) displays the learning rate.

Hyperparameter tuning using ICSA model

Eventually, the ICSA design will implement the hyperparameter selection of the ABS-BiLSTM model. This technique is chosen due to its effectual global search capability, inspired by the natural behavior of crows. ICSA outperforms in exploring the solution space with lesser iterations and averting local optima, which is significant for finding the optimal combination of hyperparameters. Unlike gradient-based methods, ICSA does not depend on differentiability and can efficiently tune non-differentiable or complex objective functions. Its capability to balance exploration and exploitation confirms robust optimization without excessive computational cost. Furthermore, the flexibility of the ICSA model makes it appropriate for fine-tuning the hyperparameters of DL methods, giving significant enhancements in performance and generalization. This makes ICSA an ideal choice for fine-tuning the proposed model. Figure 6 indicates the flowchart of the ICSA model.

The CSA is the population-based intelligent optimizer model stimulated by the crow’s foraging behaviour³³. By mimicking crows’ defence and memory tactics, the CSA seeks out optimum solutions inside the solution area. It is considered because of its extensive applicability and simplicity, having established efficient outcomes through different optimizer difficulties. Nevertheless, the CSA tends to get stuck in local bests in the searching procedure, and its balance between exploitation and exploration is narrow, resulting in slow convergence speeds and inadequate accuracy in multimodal, complex, higher‐dimension issues. To deal with these restrictions, this work presents an ICSA that incorporates reinforcement learning (RL) strategies and adaptive neighbourhood search. Initially, adaptive neighbourhood search powerfully fine-tunes the crow’s search radius, guaranteeing exploration flexibility throughout dissimilar searching stages; this permits wide-ranging searching in the initial phases while slowly concentrating on adjustment in the end phases. Additionally, the combination of RL permits the crows to dynamically choose optimum searching behaviours depending on the response from the atmosphere, thus improving the model’s decision-making abilities. This development supports the CSA’s local searching precision and global exploration abilities and considerably improves convergence speed and solution accuracy, improving stability and robustness in challenging composite optimizer issues. The complete stages and main expressions for the ICSA are as demonstrated:

(1) Population Initialization: Initializing the location \(\:{x}_{i}\) and memory location \(\:{m}_{i}\) of all crows within the population, in addition to their respective fitness values. Set the adjustable neighbourhood radius \(\:{\delta\:}_{j}\) and rate of learning \(\:{\alpha\:}_{i}\), along with the \(\:Q\)-table for RL, which should be applied to learning the optimum behaviours for various conditions.

(2) Adaptive Neighborhood Search: Establishing the neighbourhood radius \(\:{\delta\:}_{i}\) and dynamically fine-tuning the searching neighbourhoods. Describe the primary radius. \(\:{\delta\:}_{init}\) and scaling features \(\:\gamma\:\) (whereas \(\:\gamma\:\in\:\left(\text{0,1}\right))\) Fine-tune the neighbourhood radius according to the present population fitness and density, permitting the searching variety to be dynamically modified in the optimizer procedure (38):

Whereas \(\:t\) denotes the present iteration, and \(\:\gamma\:\) controls the rate of contraction.

(3) RL Approach: Applying \(\:Q\)-learning in RL to determine a behaviour choice approach. Describe the state \(\:S\) (with information such as present fitness and location) and the actions \(\:A\) (selecting extended or neighbourhood searches). Updated all crow’s \(\:Q\)‐value through the rate of learning \(\:\eta\:\) and factor of discount \(\:\lambda\:\). The \(\:Q\)‐value updated equation is presented in Eq. (39):

Here \(\:r\) denotes the reward gained from the present action, and \(\:{S}^{{\prime\:}}\) denotes a novel state deriving from action \(\:A.\).

(4) Updated Position: According to neighbourhood search and RL decisions, select whether to use the memory location (for local search) or implement global exploration in Eq. (40):

Now, \(\:{r}_{i}\) denotes a randomly generated number between \(\:0\:and\:1\).

(5) Updates Memory Location: When the novel location \(\:{X}_{i}^{t+1}\) is superior to the memory location \(\:{m}_{i}\), updated \(\:{m}_{i}\) through \(\:{X}_{i}^{t+1}.\).

(6) Adaptive Modification of Learning Rate and Neighborhood Radius: According to the crow’s population diversity and fitness, adaptively fine-tune the learning rate \(\:{\alpha\:}_{j}\) and neighbourhood radius \(\:{\delta\:}_{i}\) to guarantee a more excellent exploration range in the initial phases and more concentrated local searches later in Eq. (41):

Here, \(\:\gamma\:\) and \(\:\beta\:\) control the decay rates of the learning rate and radius, correspondingly, and \(\:T\) denotes maximal iteration counts.

(7) Stopping Conditions: The iteration procedure stops when a pre-defined maximal iteration count is reached or when convergence conditions are encountered (for example, the fitness value displays no vital development), and the optimum solution is produced. Algorithm 1 represents the ICSA model.

Table 2 Specifies the hyperparameter settings for the ICSA model. Each parameter is defined along with its symbol, functional role within the optimization process, and general value ranges based on standard experimental setups. These parameters are significant for balancing exploration and exploitation, controlling adaptive search behavior, and ensuring robust convergence in complex optimization environments.

The ICSA model presents FF to achieve enhanced classification outcomes. It expresses a positive number to signify the improved performance of the candidate solution. The reduction of the classification rate of error is reflected as FF in this study, as expressed in Eq. (42).