Hybrid LLM and machine learning framework for early detection of fire in subway tunnels

Sensitivity analysis of operating parameters

Table 8 shows the sensitivity analysis of the LLM augmented classifier across different decision thresholds. \((\Tau)\) and a temporary persistent window (k) total dataset. This analysis evaluates how the F1 score, detection delay, and pre-alarm rate (PAR) change as the alarm formation rules become more permissive or more conservative. The default behavior settings used in the main experiments are: \(\tau =0.50\) and \(k=1.0\text {s}\).

Table 8 Sensitivity analysis of LLM enhanced classifier on total dataset across decision thresholds (\(\Tau\)) and temporary persistent windows. Results are presented as mean (± standard deviation) across five seeds. Bold value Indicates the default behavior settings (\(\tau =0.50\) and \(k=10\) frames (equivalent to 1.0 s)) were used in the main experiment. These results are consistent with the main table.

Overall, the results show that there is a clear trade-off between detection sensitivity and early alarm suppression. In general, lower thresholds maintain higher F1 scores, but significantly increase PAR, indicating more aggressive alarm behavior. for example, \(\tau =0.25\) and \(k=1.0\text {s}\)SVM + LLM and GBM + LLM achieve high F1 scores of 87.45% and 90.85%, respectively, while the PAR values increase to 99.25% and 51.32%. In contrast, a higher threshold suppresses premature alarms more strongly, but often at the expense of lowering F1. For example, if you increase the threshold as follows: \(\tau =0.75\) in \(k=1.0\text {s}\) reduces the PAR to 8.68% for SVM + LLM, 6.04% for RF + LLM, and 12.08% for GBM + LLM, but reduces the F1 score to 64.78%, 69.21%, and 69.66%, respectively.

Temporal persistence windows further adjust this trade-off. Increasing the persistence window from 1.0 seconds to 2.0 or 3.0 seconds generally reduces PAR, but excessive smoothing can significantly reduce F1, especially for SVM + LLM and RF + LLM. GBM + LLM is relatively robust under moderate smoothing. \(\tau =0.50\) and \(k=2.0\text {s}\)while maintaining an F1 score of 88.33% while reducing the PAR from 47.55% in the default setting to 28.68%. However, if I enlarge the window further, \(k=3.0\text {s}\) reduced F1 to 83.46%, indicating that overly conservative alarm persistence can also suppress effective fire detection.

Comparative analysis with direct LLM prompt baseline

To justify the architectural design of using LLM as a semantic feature extractor rather than as an end-to-end decision maker, Table 9 evaluates direct LLM classification based on zero-shot, one-shot, and few-shot chain of thought (CoT) prompting strategies.^52–54. The results show that direct prompting is highly sensitive to both the prompting settings and the LLM backbone when applied to raw multivariate numerical sensor snapshots.

These results indicate that the final alarm behavior is highly sensitive to the chosen threshold and duration window. hold \(\tau =0.50\) and \(k=1.0\text {s}\) This is used as the default setting for the main experiments as it provides a consistent operating point to compare classifiers. Meanwhile, sensitivity analysis reveals how alternative alarm rules change the balance between F1, delay, and pre-ignition false alarms.

Table 9 Comparison of the performance of direct LLM classification using pure prompting strategies (zero-shot, one-shot, and few-shot CoT) on the entire dataset. Results are presented as mean % (± standard) across five seeds. Bold value Shows the best performance within each prompt setting.

We evaluated several backbones, including Llama-3-8B, under a zero-shot CoT configuration.⁴¹Kwen-2.5-3B⁵⁸Qwen-2.5-7B, and Ministral-8B⁵⁹– Produces near-zero recall and F1. This indicates that it is difficult to directly map raw numerical sensor values to stable fire/no-fire decisions. Llama-3.2-3B is a notable exception for zero-shot prompts (F1 = 78.47%), but its performance drops sharply for one-shot settings (F1 = 3.90%), demonstrating the sensitivity of direct prompts to demonstration context.

The few-shot CoT improved the strongest result of prompt-only, reaching Llama-3-8B’s accuracy of 65.67% and F1 score of 70.32%. However, this is still below the supervised hybrid approach. When LLM is used as a semantic feature generator rather than as a standalone classifier, GBM + Llama-3-8B achieves an F1 score of 90.77%. These comparisons suggest that in this setting, LLM is more reliable when used to generate intermediate semantic representations for supervised downstream classifiers than when used directly as a numerical classifier.

Additional implementation details

This subsection provides supplementary implementation details of the proposed framework. The prompt design used for contextual interpretation and ML-based classification is shown in Figure 4 in the Appendix. The hyperparameter settings for the ML classifier, LLM semantic extractor, TF-IDF vectorizer, and deep learning baseline are summarized in Table 10 in the Appendix. The scenario-specific thresholds used for the enhanced rules + ML ablation baseline are reported in Table 11 in the Appendix.

Table 10 Hyperparameter settings for ML classifier, LLM, TF-IDF vectorizer, and deep learning baseline.

Table 11 Scenario-specific multilevel thresholds used for enhanced rules + ML ablation baseline. Threshold generates handcrafted semantic tags from temperature, smoke, and O.\(\vphantom{0}_2\)Colorado, Colorado\(\vphantom{0}_2\)and inter-sensor differential features provide a stronger symbolic feature baseline than the original two threshold rule settings.

Source link