Hybrid pre trained model based feature extraction for enhanced indoor scene classification in federated learning environments

Indoor image classification is becoming increasingly important in applications like smart cities, smart healthcare, robotics, and security systems. Several research works have been done to propose and implement robust computational approaches and facilitate indoor scene classifications in various research domains¹⁸.

Traditional indoor image classification methods were performed using feature extraction methods like Scale-Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG)¹⁹. Such techniques implemented manually designed algorithms for detecting textures and edges in an image. Though effective, these methods seemed ineffective in complex indoor environments like variable lighting or occlusions and had several limitations such as reliance and expertise on the domain.

The advent of ML algorithms like SVMs and Random Forest (RF) significantly enhanced the learning patterns from the features extracted using traditional methods²⁰. Though being comparatively more effective, the quality of these models was constrained which was eventually removed by making a transition to DL models²¹. CNNs improved the performance of tasks involving diverse and complex indoor scenes by ensuring automatic feature learning from the raw images. Such evolution allows better generalization across various application domains and datasets by minimizing the dependence on manually extracting features.

1. a.

   Applications of ML in indoor image classification.

Various researchers have understood the importance of classifying indoor images in diverse application areas. Several ML methods have efficiently been used to classify indoor scenes into distinct classes. The lightweight architecture of the GenericConv model is designed and proposed in²² to classify images using max-pooling, dense layers, integration of convolutional layers, dropout, and few-shot learning which helps in the prevention of overfitting and efficiently extracts features. MiniSun, MiniPlaces, and MIT-Indoor 67 datasets were used to test the model results, where superior model performances were highlighted. In²³, the authors used S3DIS dataset to classify 3D indoor point clouds using XGBoost, Multi-Layer Perceptron (MLP), Random Forest (RF), and TabNet al.gorithms, with RF achieving 86% accuracy. A novel ML model named RepConv is introduced in²⁴ to classify scenes on the Intel scenes dataset which was recategorized for binary and multi-class classification. The authors demonstrated a comparative performance of RepConv on ResNet 50 and SE ResNext 101 using fewer parameters and epochs. For multi-task classification, 93.55% and 75.54% accuracies are achieved on training and validation data, respectively. Whereas, for binary classification, 98.08% and 92.70% are achieved on training and validation data, respectively.

A comprehensive review of ML is provided in various image processing techniques, highlighting the challenges of foggy images and enhancing, segmenting, and denoising images using ML. The advantages of transfer learning for small datasets are emphasized and the need for better datasets and lightweight architectures is also emphasized for real-time defogging applications. Scene-Aware Label Graph Learning (SALGL) framework to classify multi-label images is introduced in²⁵ which captures label co-occurrence dynamically by associating them with scene categories. A semantic attention module is incorporated for aligning visual label features, a graph propagation mechanism is used to refine label predictions, and a scene-aware co-occurrence module is implemented to model label interactions. Robust label handling dependencies are implemented in diverse datasets, which outperforms the existing methods significantly. In²⁶, the authors present a semantic classification of LiDAR-based sensory data of indoor robot navigation. A cost and memory-efficient framework is introduced to distinguish between doorways, rooms, halls, and corridors and missing or infinite LiDAR values are handled using the preprocessing techniques. SVM achieved the best testing accuracy of 97.21%. Table 1 summarizes the various ML approaches used for classifying indoor images.

ML uses algorithms and statistical methods that can be trained using data patterns without explicit programming. ML models extract features to classify images into identified classes. Some of the most widely used ML methods for image classification are:

• SVMs classify images by segregating features into specific classes using hyperplanes. They can classify indoor images from outdoor images using edges, colors, or textures²⁷.

• RF and DT classify images by learning decision rules based on their extracted features. They generally detect indoor objects based on their shapes²⁸.

• KNN is capable of identifying the image class based on the nearest neighbor of the image. They can classify images based on their visual similarity²⁹.

Though ML algorithms are efficient in classification methods, they are encountered with several challenges. Some of the challenges encountered by ML algorithms are that they require manual feature extraction thereby making them more time-consuming. They also exhibit limited performance of large-scale, complex datasets³⁰.

1. b.

   Applications of DL in indoor image classification.

Various DL approaches and frameworks are also used in indoor scene classification. The extensive literature review demonstrates diverse applications of DL in this domain. A novel DL framework for multi-label image classification is presented in³¹, where the label-specific pooling (LSP) is introduced. MS-COCO and PASCAL VOC datasets were used to evaluate the methods and achieve better classification results, mean average precision (mAP), recall, and precision compared to the existing models. A multi-scale CNN and LSTM-based scene classification method is proposed and optimized by a Whale Optimization Algorithm (WOA). 98.91% and 94.35% classification accuracies were obtained on lab and FR079 public datasets. On optimizing the learning rate and regularization, the WOA elevated the accuracy to 99.76³².

The authors proposed a fusion of hand-crafted methods for indoor scene classification, with DL features from the EfficientNet-B7 model. An enhanced accuracy of 93.87% is obtained using the fusion model which outperformed the state-of-the-art methods³³. The authors present a lightweight DL model for big data applications in indoor scene classification. IRDA-YOLOv3, an enhanced lightweight architecture is proposed which improves the YOLOv3-Tiny model for scene classification and object detection. IRDA-YOLOv3 demonstrated improved efficiency and forward computation time by 0.2 ms, reduced parameters by 56.2%, and computation by 46.3%³⁴. Multimodal DL techniques for indoor and outdoor scene classification and recognition are reviewed, demonstrating advancements in CNN and computer vision. The integration of transfer learning and CNNs is highlighted to enhance parameter optimization and feature extraction. Significantly enhanced accuracy was attained on Kaggle’s Indoor Scene dataset³⁵. The various DL models used for indoor scene classification are summarized in Table 2.

These methods are capable of automating feature extraction and achieving higher accuracy while classifying images. Some of the commonly used DL methods are enlisted as follows:

• CNNs can classify large datasets with thousands of categories. They use convolutional layers to learn hierarchical features like shapes, edges, or textures for classification. The most popular CNN models are AlexNet, VGG, ResNet, and DenseNet³⁶.

• RNNs with CNNs: In sequence-based or video classifications like indoor surveillance videos, RNNs capture temporal relationships while CNNs perform feature extraction³⁷.

• Generative Adversarial Networks (GANs): These networks augment training datasets on synthetic data to improve classification accuracy. It is generally used in training models with limited indoor images by generating realistic samples³⁸.

• Transfer Learning: Transfer learning leverages pre-trained models such as ResNet or VGG. These models are fine-tuned for indoor image classification tasks with limited datasets. This approach also enhances accuracy by reducing training time and requiring fewer computational resources³⁹.

Though ML methods like SVM, KNN, etc., have laid the basis for image classification, DL methods like CNNs have improved their scalability and accuracy. These aspects help in diverse applications across various fields like healthcare, robotics, security and surveillance, drones, autonomous vehicles, and real-time indoor and outdoor classifications.

1. c.

   Applications of FL in indoor image classification.

FL can be applied to classify indoor scenes to address challenges in distributed data across multiple devices. It creates and empowers robust and privacy-preserving models, especially in distributed environments⁴⁰. The indoor images are collected from different devices and classified locally without transferring the data to a centralized server. Some of the major applications of FL in indoor scene classification are in smart homes, indoor robotics, healthcare, retail, and smart public places where there are multifold benefits like preservation of data privacy, scalability, and learning from diverse heterogeneous data⁴¹. Another crucial aspect that affects the model performance and training process in FL is Independent and Identically Distributed (IID) and Not Independent and/or Not Identically Distributed (Non-IID) data⁴².

IID refers to the identically distributed independent data across various devices. It has the additional advantages of faster convergence during training and lower model bias risk. Whereas, non-IID refers to the non-uniform data distributed across various clients. Unlike IID, the non-IID exhibits challenges like comparatively slower convergence, biases in the model, and challenges in aggregation.

In⁴³, the authors have designed a blockchain-based FL scheme, namely FedBG, to address privacy concerns and insufficient medical image classification data challenges. It incorporates EC-GAN, an enhanced classification-GAN to improve image diversity along with optimizing classification accuracy. Results after experimentation demonstrate that FedBG exhibits a 27–38% reduced training time and 0.9-2% better accuracy along with data privacy preservation while high-quality medical images are generated. An FL-based indoor image recognition positioning system is proposed in⁴⁴ to address privacy concerns in image-based indoor positioning methods. The system exhibits client privacy and is 94% accurate by implementing the MobileNet model with FedAvg and FedOpt algorithms. Whereas, in⁴⁵, four FL algorithms, namely FedBABU, FedAvg, FedProto, and APPLE in non-IID data in the Fashion MNIST dataset. 76.04% and 74.62% accuracies were obtained from FedAvg and FedBABU algorithms respectively which were surpassed by APPLE and FedProto algorithms attaining 99.21% and 99.40% accuracies. Personalized FL (P-FL) and generalized FL (G-FL) were bridged for image classification and FED-ROD framework was proposed which outperformed the other methods with non-IID data in diverse datasets with superior scalability and balanced accuracy⁴⁶. A FL-based MRI brain tumor classification was proposed leveraging the FedAvg algorithm and the EfficientNet-B0 model to minimize privacy breach issue and enhance accuracy in diagnosis. ResNet-50 was outperformed by EfficientNet-50, with a maximum of 80.17% test accuracy and minimum of 0.612 loss as compared to ResNet-50 which attained 65.32% accuracy and 1.017 loss. The FL model was resilient to heterogeneous data and achieved 99% accuracy⁴⁶. In the research done⁴⁷, the transformative impact of FL on medical imaging is highlighted while addressing data privacy concerns. The results demonstrate that FL showed comparable results as centralized models in diagnostic conditions, disease classification, etc. Along with active ongoing optimization improvements, there are several prevalent issues like communication bottlenecks, heterogeneity, and computational costs.

The effectiveness of FL in analyzing histopathological images of breast cancer in a secure manner using BreakHis dataset is done in⁴⁸. The research compares federated, independent, and centralized training models where FL achieves results comparable to centralized learning with negligible differences in F1 scores, accuracy, and precision. DenseNet-201 and ResNet-152 achieved high diagnostic odds ratios and Kappa values and outperformed other models in reliability and consistency, thus validating FL as a solution to address privacy concerns in the collaborative analysis of medical images. In another research work⁴⁹, the authors introduce the FLBIC-CUAV framework as a combination of blockchain and FL for efficient and secure classification of UAV network images in Industrial IoT (IIoT) environments. The proposed framework leverages FL with ResNet for classification, beetle swarm optimization (BSO) for UAV clustering, and blockchain for secure data transmission. The results showed that the proposed framework produced superior outcomes as compared to existing methods with up to 99.15% accuracy, reduced delay, improved throughput, and lower energy consumption. A concise representation of various FL approaches used to classify indoor images is represented in Table 3.

Problem description

The extensive literature survey found that the existing research does not address several aspects in various domains. ML and DL algorithms play a pivotal role in classifying indoor images providing reliable and robust techniques for essential decision-making based on pattern recognition and feature extraction. The roles of ML and DL algorithms in indoor image classification include better feature extraction, enhanced classification accuracy, and integration with multimodal data. Apart from these, DL proves to be a dominant paradigm in this domain, providing further advantages over ML in terms of scalability and generalization, where DL models can scale up to larger datasets and provide better generalization as compared to ML models, which have limited capacity to learn from large datasets⁵⁰. ML also lags in using contextual information in classification, which is overcome by using DL models, along with handling occlusions, cluttering, and lighting variations⁵¹. The pre-trained DL models like ResNet, MobileNet, and VGG can help in reducing the indoor classification time and improving performance.

Various smart domains like smart homes, indoor robotic navigation, and augmented reality (AR) require privacy preservation as the foremost context of scene classification. Multiple devices are used in the data collection process, where either the data distribution may be IID or Non-IID⁵². FL models are effective in these cases where privacy preservation and data integrity are the key concerns. Unlike the centralized data, in FL models, the data is trained in their respective devices and the gradients are shared with the central server⁵³.

The key contributions of this work are:

• To analyze machine learning and DL algorithms for indoor scene classification.

• To design and develop hybrid pre-trained model-based feature extraction for better classification.

• To implement the proposed hybrid pre-trained model-based feature extraction technique in an FL environment with identical and non-identical data distribution.