Pith. sign in

REVIEW 3 major objections 4 minor 23 references

Learning to Calibrate for Reliable Visual Fire Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A differentiable ECE loss, trained with a curriculum schedule, cuts calibration error on two fire-detection benchmarks by up to half while keeping accuracy loss under 0.7 percent.

desk verdict The central equation (6) drops the label and is not a differentiable accuracy surrogate, so the claimed ECE-loss mechanism fails; the paper is a flawed but instructive application idea. read the letter →

arxiv 2502.09872 v1 pith:PI57BEID submitted 2025-02-14 cs.CV cs.LG

Ziqi Zhang , Xiuzhuang Zhou , Xiangyang Gong This is my paper · ORCID
classification cs.CVcs.LG
keywords firedetectionuncertaintycalibrationexpectederrordifferentiablelosscurriculumlearningconfidencemulti-classclassificationdeep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Fire-detection models tend to be overconfident: they report high confidence for predictions that are wrong. This paper claims that a training loss built directly from the Expected Calibration Error (ECE) can fix that, making calibration part of the optimization instead of a post-training patch. The method smooths the non-differentiable ECE metric with a sigmoid function so gradients can flow, and it ramps up the calibration loss's weight over training following curriculum-learning ideas. On the DFAN and EdgeFireSmoke benchmarks, the reported test ECE drops from 0.05436 to 0.04013 and from 0.01208 to 0.00596, while classification accuracy falls by less than 0.7 percent. If the result holds, reliable visual fire detection can be trained directly, without extra validation data or post-hoc recalibration.

What carries the argument

The load-bearing object is the smoothed bin accuracy in Eq. (6), $\text{acc}(B_m) = \sum_{i \in B_m} S(\tan(\pi \hat{p}_i - \pi/2)) / |B_m|$, where $S(x)=1/(1+e^{-x})$. This replaces the non-differentiable indicator $\mathbf{1}(\hat{y}_i = y_i)$ used to compute bin accuracy in the ECE metric, turning the metric into a differentiable loss. The paper then combines this ECE loss with NLL loss using a curriculum schedule (Eq. 8), in which the weight of the ECE loss grows from the epoch it is introduced ($s_e$) to a preset value $\gamma_E$ over $N$ epochs. The machinery is doing the work of making the training objective itself penalize miscalibration, rather than correcting probabilities after training.

What would settle it

Compute the standard indicator-based ECE (Eq. 5) on held-out test data across several bin counts, say M=5, 10, 15, and 20, while also recording the differentiable surrogate's value at those bins; if the reported improvement exists only at M=15, or if the surrogate drops while the indicator-based ECE rises, the claim that the loss improves calibration is not supported.

Watch

Extended reading notes

Core claim

The central claim is that online calibration for multi-class fire detection can be achieved by replacing the hard 0-1 indicator inside the ECE calculation with a smooth sigmoid function, namely $\text{acc}(B_m)=\sum_{i\in B_m} S(\tan(\pi \hat{p}_i - \pi/2))/|B_m|$, and using that differentiable quantity as a loss. The paper states this conversion is made "without altering the underlying calculation logic" (Section III-C). Combined with cross-entropy loss, and with the ECE loss weight increased gradually during training, this yields calibrated confidence for multi-class fire detection models: test ECE falls from 0.05436 to 0.04013 on DFAN and from 0.01208 to 0.00596 on EdgeFireSmoke, with accuracy loss kept under 0.7 percent. The authors also report that the curriculum-style weight schedule outperforms a constant ECE weight, with constant-weight training producing ECE 0.11289 on EdgeFireSmoke versus 0.00596 for the curriculum-calibrated model, and 0.05254 versus 0.04013 on DFAN.

Load-bearing premise

The method depends on treating a smooth curve of the model's confidence as a stand-in for whether the prediction was correct; that curve contains no information about the true label, so if the stand-in stops tracking real correctness, minimizing the loss can shrink the calibration score without making predictions more trustworthy.

Editorial extensions

If this is right

  • Fire-detection models can be trained to output calibrated confidence directly, without a post-hoc recalibration step.
  • The calibration benefit appears even when the baseline already has low ECE: EdgeFireSmoke drops from 0.01208 to 0.00596 with accuracy loss under 0.5 percent.
  • Applying the ECE loss at constant weight from the start hurts both accuracy and calibration; the curriculum ramp is necessary to balance the two.
  • The method is reported to work across two different architectures and dataset scales, DFAN and EdgeFireSmoke.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to compare the label-free surrogate in Eq. (6) with a soft-label variant that uses ground-truth correctness, isolating how much of the reported ECE gain comes from matching true accuracy rather than from reshaping confidence values.
  • The paper's evaluation is on in-distribution test splits; an open question is whether the calibration gain survives out-of-distribution conditions such as novel fire-like distractors, where overconfidence is most dangerous.
  • Because the loss only modifies the training objective, it should attach to any classification backbone; the same curriculum schedule could be tested on other high-stakes vision tasks such as medical image screening.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a differentiable Expected Calibration Error (ECE) loss for multi-class visual fire detection. The idea is to replace the non-differentiable 0-1 accuracy indicator in the ECE computation with a sigmoid of a transformed confidence value, combine this 'ECE Loss' with cross-entropy, and anneal its weight during training following a curriculum-learning schedule. Experiments are reported on two datasets, DFAN and EdgeFireSmoke, with the claim that the calibrated models achieve lower test ECE (e.g., DFAN from 0.05436 to 0.04013; EdgeFireSmoke from 0.01208 to 0.00596) while losing less than 0.7% accuracy. The central mechanism is unsound because the proposed differentiable surrogate in Eq. (6) does not depend on the true label and therefore does not approximate the correctness indicator in Eq. (2).

Significance. If the proposed method worked as stated, it would provide a practical way to train fire-detection models with calibrated confidence directly, which is valuable for safety-critical applications. The paper also addresses an important gap in the fire-detection literature, where overconfidence is common, and it includes experiments on two public datasets with reliability diagrams. However, the core contribution rests on a mathematically invalid surrogate for classification accuracy. The reported ECE reductions may reflect confidence regularization or optimization artifacts rather than genuine calibration improvement, so the paper does not currently support its central claim. I do not see a circularity problem with the evaluation metric, which is computed on held-out test data with true labels; the problem is a soundness failure in the training objective.

major comments (3)
  1. [Section III-C, Eq. (6)] Equation (6) defines the differentiable accuracy surrogate as S(tan(pi * p_hat_i - pi/2)), which is a function of the model's confidence p_hat_i alone; the true label y_i does not appear. Equation (2) defines accuracy as an average of indicators 1(y_hat_i = y_i), so Eq. (6) cannot approximate it. A model that outputs confidence 0.5 for every sample would give acc(B_m) = 0.5 in every occupied bin, and therefore zero ECE Loss, even when all predictions are wrong. Consequently, the statement in Section III-C that the conversion is made 'without altering the underlying calculation logic' is false, and the proposed training objective does not implement online calibration. The ECE reductions in Tables 3 and 4 may reflect reshaping of the confidence distribution rather than an improved alignment between confidence and correctness.
  2. [Section IV-D, Tables 3-5] Even if Eq. (6) were a valid surrogate, the empirical evidence is insufficient to establish the calibration claim. The tables report single runs with no error bars or significance tests; the hyperparameters gamma_E and s_e are selected using 'observations from the experimental process' (Section III-D), with no explicit statement of whether this selection is made on a validation set or on test data; and no standard calibration baselines such as temperature scaling, vector scaling, or MC dropout are compared. Given the small ECE differences (for example, DFAN from 0.05436 to 0.04013 in Table 3), the absence of variance information and baseline comparisons leaves regularization or optimization artifacts as plausible alternative explanations.
  3. [Section III-D, Eq. (8)] Equation (8), as printed as L = L_n + ce - se/(N - se) * gamma_E * L_e, is ambiguous and does not, by standard operator precedence, realize the described curriculum schedule, which requires a coefficient of (ce - se)/(N - se). The paper should provide a correctly parenthesized formula and state explicitly how ce, se, and gamma_E are chosen, including whether the reported best value se = 0 is selected using a held-out validation set or the test set.
minor comments (4)
  1. [Section III-B, Eq. (4)] Equation (4) is not fully specified: the summation over p needs an explicit probability measure or density, and its relationship to the ECE formula in Eq. (5) should be stated more precisely.
  2. [Section III-C] The paper should clarify the behavior of the proposed surrogate at the extremes p_hat = 0 and p_hat = 1, since tan(pi * p_hat - pi/2) diverges there; although the sigmoid maps the limits to 0 and 1, the derivative behavior is not the same as that of a smooth indicator, and this should be discussed.
  3. [Section IV-B] The ECE loss is computed with M = 10 bins in Section III-C, while the evaluation ECE is computed with M = 15 bins in Section IV-B; the discrepancy is acceptable but should be explicitly justified, since bin count affects the numerical value of ECE.
  4. [References] Reference [12] contains a typo: 'Transactions of Beijing Institute of Techonology' should be 'Technology'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: evaluation ECE uses held-out true labels, and the proposed loss is a training surrogate, not a refitting of the metric.

full rationale

The paper's central claim is that training with the differentiable surrogate in Eq. (6), combined with NLL loss and a curriculum-like weight ramp, improves calibration. The reported ECE values in Tables 3 and 4 are computed on held-out test splits using the standard ECE definition in Eq. (5), with acc(B_m) from Eq. (2) based on true labels y_i. The training loss is therefore not fitted to the evaluation target; the test ECE is an independent measurement. I checked the only candidate self-citation: Ref. [4] shares co-author X. Gong, but it is background on smoke-detection features and is not load-bearing for the calibration claim. There is no uniqueness theorem, no parameter fitted to test ECE and renamed a prediction, and no definitional equivalence between the training objective and the reported metric. The main weakness is Eq. (6): the smooth surrogate S(tan(pi p_i - pi/2)) contains no true label, so it is not an approximation of the indicator 1(yhat_i = y_i) and may not track correctness. That is a soundness/validity concern about whether the loss truly calibrates, not a circularity, and under the review rules it should not raise the circularity score. The paper also acknowledges limited dataset availability, but that limitation is unrelated to circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The core of the method rests on a single unstated and false premise: that the sigmoid-of-tan transform of the model's confidence is a smooth stand-in for correctness. The remaining assumptions are standard binning approximations and reliance on prior model descriptions. No new physical entities are introduced.

free parameters (3)
  • gamma_E (ECE loss weight) = 0.05 for DFAN, 5 for EdgeFireSmoke
    Chosen from observed NLL-to-ECE loss magnitude ratio; a hand-set hyperparameter that controls the accuracy-calibration tradeoff.
  • se (start epoch for ECE loss) = 0 for DFAN and Edgecali1, 10 for Edgecali2
    Selected by comparing test calibration performance; the paper notes the best effect at se=0, indicating post-hoc selection.
  • M (number of bins) = 10 for ECE loss, 15 for evaluation
    Chosen by convention; affects the granularity of the calibration measure but is not fitted.
assumptions (3)
  • ad hoc to paper S(tan(pi * p_hat - pi/2)) is a valid smooth approximation of the correctness indicator 1(y_hat = y).
    Equation (6) replaces the indicator in equation (2) with this confidence-only expression, but the approximation omits the label and therefore does not track correctness. The whole ECE loss rests on this premise.
  • domain assumption The DFAN and EdgeFireSmoke model architectures and training settings from the original papers are sufficient to reproduce the baselines.
    Section IV-C says training 'largely adheres to the parameter settings outlined in the original papers' and does not restate architecture details, relying on Refs. [22] and [23].
  • standard math Binned estimates of accuracy and confidence with M=15 are unbiased approximations of the ideal calibration condition in equation (1).
    Section III-B and IV-B use standard binning; this is conventional but not formally justified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Calibrate for Reliable Visual Fire Detection." pith.science (2026). https://pith.science/paper/PI57BEID

@misc{pith2026250209872,
  author       = {Pith},
  title        = {Pith review of: Learning to Calibrate for Reliable Visual Fire Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PI57BEID}},
  note         = {Machine review of arXiv:2502.09872}
}
read the original abstract

Fire is characterized by its sudden onset and destructive power, making early fire detection crucial for ensuring human safety and protecting property. With the advancement of deep learning, the application of computer vision in fire detection has significantly improved. However, deep learning models often exhibit a tendency toward overconfidence, and most existing works focus primarily on enhancing classification performance, with limited attention given to uncertainty modeling. To address this issue, we propose transforming the Expected Calibration Error (ECE), a metric for measuring uncertainty, into a differentiable ECE loss function. This loss is then combined with the cross-entropy loss to guide the training process of multi-class fire detection models. Additionally, to achieve a good balance between classification accuracy and reliable decision, we introduce a curriculum learning-based approach that dynamically adjusts the weight of the ECE loss during training. Extensive experiments are conducted on two widely used multi-class fire detection datasets, DFAN and EdgeFireSmoke, validating the effectiveness of our uncertainty modeling method.

Figures

Figures reproduced from arXiv: 2502.09872 by the authors.

Figure 1
Figure 1. Non-fire images with interfering objects. II. RELATED WORK Fire detection is designed to identify and confirm the occurrence of a fire in a monitored scene. Currently, fire detection relies mainly on computer vision technology, with works focusing on areas such as distinguishing interfering objects, recognizing complex fire images, and enhancing the efficiency of detection systems. Fire detection environments are di… view at source ↗
Figure 2
Figure 2. Reliability diagram under perfect calibration. classification accuracy of samples in the corresponding sub￾interval. Ideally, the two areas coincide completely. Reliability diagrams provide an intuitive means of reflecting a model’s uncertainty. However, when the differences between two reliability diagrams are subtle, as illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Two indistinguishable reliability diagrams. In such cases, representing the model’s uncertainty using a scalar value proves to be more practical. A widely adopted method for quantification is illustrated as X pˆ |P(Yˆ = Y |Pˆ = p) − p| (4) This method of uncertainty quantification is derived from equation 1, which assesses the predictive uncertainty by evalu￾ating the gap between the model’s confidence in its predic… view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Curve of accuracy with predicted probability changing from zero to one. D. Online Calibration In this study, we combine ECE Loss with cross-entropy loss (NLL Loss) to jointly supervise the training of a multi-class [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Example images of the DFAN dataset. The EdgeFireSmoke dataset [23] consists of wildfire images captured by drones and is organized into four categories: Burned-area images, which typically feature blackened ground or withered tree trunks; Fire-smoke images, where the s…
Figure 7
Figure 7. Figure 7: Example images of EdgeFireSmoke dataset. while the ECE metric is intended to measure the decisions’ reliability. In the multi-classification task of this paper, precision, recall, F1 score, and accuracy are first calculated for each individual class, and then the avera…
Figure 8
Figure 8. Figure 8: Reliability diagram of DFAN model before and after calibration. The experimental comparison between the vanilla and cali￾brated EdgeFireSmoke models is shown in [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: In the interval [0.6, 1.0], the predicted accuracy and [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Reliability diagram of EdgeFireSmoke model before and after calibration. commonly used datasets. The development of more classifica￾tion datasets will further promote the adoption of uncertainty modeling techniques in visual fire detection. Future work will focus on i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

  1. [1]

    Research on fire recognition method of inspection robot based on video image,

    J. Liang, “Research on fire recognition method of inspection robot based on video image,” Ph.D. dissertation, Hebei University of Technology, 2023

    work page 2023

  2. [2]

    Video fire detection methods based on deep learning: Datasets, meth- ods, and future directions,

    C. Jin, T. Wang, N. Alhusaini, S. Zhao, H. Liu, K. Xu, and J. Zhang, “Video fire detection methods based on deep learning: Datasets, meth- ods, and future directions,” Fire, vol. 6, no. 8, p. 315, 2023

    work page 2023

  3. [3]

    A triple interference removal network based on temporal and spatial attention interaction for forest smoke recognition in videos,

    H. Tao, “A triple interference removal network based on temporal and spatial attention interaction for forest smoke recognition in videos,” Computers and Electronics in Agriculture , vol. 218, p. 108756, 2024

    work page 2024

  4. [4]

    Efficient attention based deep fusion cnn for smoke detection in fog environment,

    L. He, X. Gong, S. Zhang, L. Wang, and F. Li, “Efficient attention based deep fusion cnn for smoke detection in fog environment,” Neurocom- puting, vol. 434, pp. 224–238, 2021

    work page 2021

  5. [5]

    Fire and smoke detection algorithm based on improved yolov8,

    D. Li, J. Zhou, and Q. Liu, “Fire and smoke detection algorithm based on improved yolov8,” pp. 1–9, 2024

    work page 2024

  6. [6]

    F2m: Ensemble-based uncertainty estimation model for fire detection in indoor environments,

    M. Arlovi ´c, M. Patel, J. Balen, and F. Hr ˇzi´c, “F2m: Ensemble-based uncertainty estimation model for fire detection in indoor environments,” Engineering applications of artificial intelligence , vol. 133, p. 108428, 2024

    work page 2024

  7. [7]

    Monte carlo dropblock for modeling uncertainty in object detection,

    S. H. Yelleni, D. Kumari, P. Srijith et al. , “Monte carlo dropblock for modeling uncertainty in object detection,” Pattern Recognition, vol. 146, p. 110003, 2024

    work page 2024

  8. [8]

    Learning discriminative feature representation with pixel-level supervision for forest smoke recognition,

    H. Tao, Q. Duan, M. Lu, and Z. Hu, “Learning discriminative feature representation with pixel-level supervision for forest smoke recognition,” Pattern Recognition, vol. 143, p. 109761, 2023

    work page 2023

Show all 23 references
  1. [9]

    Advanced wildfire detection using generative adversarial network-based augmented datasets and weakly supervised object localization,

    M. Park, J. Bak, S. Park et al. , “Advanced wildfire detection using generative adversarial network-based augmented datasets and weakly supervised object localization,” International Journal of Applied Earth Observation and Geoinformation , vol. 114, p. 103052, 2022

    2022

  2. [10]

    Anchor-free smoke and flame recognition algorithm with multi-loss,

    G. Li, P. Chen, C. Xu, C. Sun, and Y . Ma, “Anchor-free smoke and flame recognition algorithm with multi-loss,” Fire, vol. 6, no. 6, p. 225, 2023

    2023

  3. [11]

    Cubic-cross convo- lutional attention and count prior embedding for smoke segmentation,

    F. Yuan, Z. Dong, L. Zhang, X. Xia, and J. Shi, “Cubic-cross convo- lutional attention and count prior embedding for smoke segmentation,” Pattern Recognition, vol. 131, p. 108902, 2022

    2022

  4. [12]

    Multi-scale fire detection algorithm with adaptive attention,

    Y . Liang, T. Chen, and W. Zhang, “Multi-scale fire detection algorithm with adaptive attention,” Transactions of Beijing Institute of Techonol- ogy, vol. 44, no. 01, pp. 91–101, 2024

    2024

  5. [13]

    Sustainable collaboration: Federated learning for environ- mentally conscious forest fire classification in green internet of things (iot),

    A. A. Siddique, N. Alasbali, M. Driss, W. Boulila, M. S. Alshehri, and J. Ahmad, “Sustainable collaboration: Federated learning for environ- mentally conscious forest fire classification in green internet of things (iot),” Internet of Things , vol. 25, p. 101013, 2024

    2024

  6. [14]

    Fast forest fire detection and segmentation application for uav-assisted mobile edge computing system,

    C. Li, G. Li, Y . Song, Q. He, Z. Tian, H. Xu, and X. Liu, “Fast forest fire detection and segmentation application for uav-assisted mobile edge computing system,” IEEE Internet of Things Journal , 2023

    2023

  7. [15]

    Fire-and-smoke detection based on convolutional attention and feature fusion,

    J. Tian, G. Qin, and W. Zhang, “Fire-and-smoke detection based on convolutional attention and feature fusion,” pp. 1–12, 2024

    2024

  8. [16]

    Smoke and flame detection method with yolov5-resnet cascade network,

    Q. Zhang, W. Zhang, and X. Yang, “Smoke and flame detection method with yolov5-resnet cascade network,” vol. 23, no. 02, pp. 397–405, 2023

    2023

  9. [17]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International conference on machine learning. PMLR, 2017, pp. 1321–1330

    2017

  10. [18]

    Uncertainty-guided learning for improving image manipulation detection,

    K. Ji, F. Chen, X. Guo, Y . Xu, J. Wang, and J. Chen, “Uncertainty-guided learning for improving image manipulation detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 22 456–22 465

    2023

  11. [19]

    Modeling the distributional uncertainty for salient object detection models,

    X. Tian, J. Zhang, M. Xiang, and Y . Dai, “Modeling the distributional uncertainty for salient object detection models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 660–19 670

    2023

  12. [20]

    Obtaining well calibrated probabilities using bayesian binning,

    M. P. Naeini, G. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in Proceedings of the AAAI conference on artificial intelligence , vol. 29, no. 1, 2015

    2015

  13. [21]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th annual international conference on machine learning , 2009, pp. 41–48

    2009

  14. [22]

    Optimized dual fire attention network and medium-scale fire classification benchmark,

    H. Yar, T. Hussain, M. Agarwal, Z. A. Khan, S. K. Gupta, and S. W. Baik, “Optimized dual fire attention network and medium-scale fire classification benchmark,” IEEE Transactions on Image Processing , vol. 31, pp. 6331–6343, 2022

    2022

  15. [23]

    Edgefiresmoke: A novel lightweight cnn model for real-time video fire–smoke detection,

    J. S. Almeida, C. Huang, F. G. Nogueira, S. Bhatia, and V . H. C. de Albuquerque, “Edgefiresmoke: A novel lightweight cnn model for real-time video fire–smoke detection,” IEEE Transactions on Industrial Informatics, vol. 18, no. 11, pp. 7889–7898, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.