{"id":"ba3b0bb1-d663-436e-b753-5603fe29aa52","arxiv_id":"2502.09872","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A differentiable approximation of Expected Calibration Error is added to the training loss of fire detection models, with a curriculum schedule, and is reported to reduce calibration error on DFAN and EdgeFireSmoke, though the approximation omits labels.","lead":"This paper adds a penalty to the training of fire detection models that is meant to lower the gap between a model's confidence and its accuracy. The authors report better calibration on two fire image datasets, but the penalty as written ignores the true labels, so it may not be doing what it claims.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (6) contains no label and is not a differentiable accuracy surrogate, so the claimed mechanism for reducing ECE is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Eq. (6) omits the label, so the 'differentiable ECE loss' does not approximate accuracy and cannot be justified as an online calibration objective. This is not a stylistic or incremental issue; it invalidates the paper's central claimed mechanism. The paper's own notation in Section III-C and Eq. (3) defines p_i as confidence (the maximum predicted probability), and the proposed accuracy surrogate depends only on that confidence. A concrete manifestation is the constant-confidence model for which the loss is zero despite terrible calibration. The empirical results are also underpowered: single runs, no error bars, and se selected post hoc. However, the load-bearing flaw is the label-free Eq. (6). Because the reader already reached REJECT for this reason, no verdict adjustment is needed. The underlying idea of training with a differentiable calibration penalty remains worth pursuing, but with a surrogate that actually uses labels, such as a soft top-1 accuracy term or a binning-free calibration loss, and with proper experimental protocols.","tokens_in":9617,"tokens_out":4065,"duration_ms":45200,"concrete_test":"Take the trained DFANcali model's test logits and compute Eq. (6) twice: once with the true labels and once with randomly permuted labels. Because Eq. (6) contains no y_i, the two values and their gradients will be identical. If confirmed, Eq. (6) is label-free and cannot be an accuracy surrogate. Additionally, on a synthetic constant-confidence model (p = 0.5 for all samples), show that the proposed loss is zero while true ECE is large; this settles whether the loss measures calibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (6) in Section III-C replaces the indicator 1(yhat_i = y_i) from Eq. (2) with S(tan(pi p_i - pi/2)), where p_i is the model's confidence (the largest softmax probability). The true label y_i does not appear. This term is therefore a smooth function of confidence alone, not an approximation of accuracy. For example, a model that always outputs p = 0.5 gives S(tan(0)) = 0.5, so acc(Bm) = conf(Bm) = 0.5 in the occupied bin and the proposed loss is zero even if true accuracy is near chance. Optimizing this objective can lower the reported ECE by reshaping confidence values without making confidence track correctness. Thus the statement in Section III-C that the conversion is made 'without altering the underlying calculation logic' is false, and the central mechanism of online calibration is unsupported. The empirical ECE reductions in Tables 3 and 4 may reflect regularization or confidence smoothing rather than calibration; the comparison also lacks repeated runs, error bars, and matched calibration baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a differentiable Expected Calibration Error (ECE) loss for multi-class visual fire detection. The idea is to replace the non-differentiable 0-1 accuracy indicator in the ECE computation with a sigmoid of a transformed confidence value, combine this 'ECE Loss' with cross-entropy, and anneal its weight during training following a curriculum-learning schedule. Experiments are reported on two datasets, DFAN and EdgeFireSmoke, with the claim that the calibrated models achieve lower test ECE (e.g., DFAN from 0.05436 to 0.04013; EdgeFireSmoke from 0.01208 to 0.00596) while losing less than 0.7% accuracy. The central mechanism is unsound because the proposed differentiable surrogate in Eq. (6) does not depend on the true label and therefore does not approximate the correctness indicator in Eq. (2).","tokens_in":9867,"tokens_out":3625,"duration_ms":37834,"significance":"If the proposed method worked as stated, it would provide a practical way to train fire-detection models with calibrated confidence directly, which is valuable for safety-critical applications. The paper also addresses an important gap in the fire-detection literature, where overconfidence is common, and it includes experiments on two public datasets with reliability diagrams. However, the core contribution rests on a mathematically invalid surrogate for classification accuracy. The reported ECE reductions may reflect confidence regularization or optimization artifacts rather than genuine calibration improvement, so the paper does not currently support its central claim. I do not see a circularity problem with the evaluation metric, which is computed on held-out test data with true labels; the problem is a soundness failure in the training objective.","major_comments":[{"comment":"Equation (6) defines the differentiable accuracy surrogate as S(tan(pi * p_hat_i - pi/2)), which is a function of the model's confidence p_hat_i alone; the true label y_i does not appear. Equation (2) defines accuracy as an average of indicators 1(y_hat_i = y_i), so Eq. (6) cannot approximate it. A model that outputs confidence 0.5 for every sample would give acc(B_m) = 0.5 in every occupied bin, and therefore zero ECE Loss, even when all predictions are wrong. Consequently, the statement in Section III-C that the conversion is made 'without altering the underlying calculation logic' is false, and the proposed training objective does not implement online calibration. The ECE reductions in Tables 3 and 4 may reflect reshaping of the confidence distribution rather than an improved alignment between confidence and correctness.","section":"Section III-C, Eq. (6)"},{"comment":"Even if Eq. (6) were a valid surrogate, the empirical evidence is insufficient to establish the calibration claim. The tables report single runs with no error bars or significance tests; the hyperparameters gamma_E and s_e are selected using 'observations from the experimental process' (Section III-D), with no explicit statement of whether this selection is made on a validation set or on test data; and no standard calibration baselines such as temperature scaling, vector scaling, or MC dropout are compared. Given the small ECE differences (for example, DFAN from 0.05436 to 0.04013 in Table 3), the absence of variance information and baseline comparisons leaves regularization or optimization artifacts as plausible alternative explanations.","section":"Section IV-D, Tables 3-5"},{"comment":"Equation (8), as printed as L = L_n + ce - se/(N - se) * gamma_E * L_e, is ambiguous and does not, by standard operator precedence, realize the described curriculum schedule, which requires a coefficient of (ce - se)/(N - se). The paper should provide a correctly parenthesized formula and state explicitly how ce, se, and gamma_E are chosen, including whether the reported best value se = 0 is selected using a held-out validation set or the test set.","section":"Section III-D, Eq. (8)"}],"minor_comments":[{"comment":"Equation (4) is not fully specified: the summation over p needs an explicit probability measure or density, and its relationship to the ECE formula in Eq. (5) should be stated more precisely.","section":"Section III-B, Eq. (4)"},{"comment":"The paper should clarify the behavior of the proposed surrogate at the extremes p_hat = 0 and p_hat = 1, since tan(pi * p_hat - pi/2) diverges there; although the sigmoid maps the limits to 0 and 1, the derivative behavior is not the same as that of a smooth indicator, and this should be discussed.","section":"Section III-C"},{"comment":"The ECE loss is computed with M = 10 bins in Section III-C, while the evaluation ECE is computed with M = 15 bins in Section IV-B; the discrepancy is acceptable but should be explicitly justified, since bin count affects the numerical value of ECE.","section":"Section IV-B"},{"comment":"Reference [12] contains a typo: 'Transactions of Beijing Institute of Techonology' should be 'Technology'.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central contribution of the manuscript is not defensible: Eq. (6), the core of the proposed differentiable ECE loss, is not a surrogate for accuracy because it omits the true label. This is a load-bearing error that cannot be repaired without changing the method's essential mechanism. The empirical comparisons also lack repeated runs, error bars, and standard calibration baselines, so even a weaker claim that the proposed loss acts as a useful regularizer for ECE would need additional evidence. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core mechanism is broken. Equation (6) replaces the indicator in the ECE accuracy term with a smooth function of confidence alone; the true label never appears. That is not an approximation of accuracy, and the claim that this 'does not alter the underlying calculation logic' is simply false. Optimizing this loss can reshape confidence values and lower measured ECE without making confidence track correctness. The stress-test note is right on target.\n\nWhat the paper does well: it identifies a real practical problem — fire detection models are overconfident and existing work mostly ignores calibration — and it is, as far as this literature goes, a new application of the online-calibration idea to multi-class fire detection. The curriculum-learning weighting scheme is sensible: start with CE, then ramp up the calibration penalty. The empirical results on DFAN and EdgeFireSmoke are consistent with the claim of lower ECE, but they are not confirming evidence for the proposed mechanism.\n\nThe soft spots go beyond the main equation. The paper never cites the existing differentiable calibration loss literature (e.g., trainable calibration measures, focal calibration loss), so the novelty is overstated. The experiments report single runs with no error bars, the start epoch se is selected post hoc, and there are no matched baselines like temperature scaling or other regularization methods. These would matter less if the math were correct, but they compound the problem.\n\nMy bottom line: this paper should not be published as is. The claimed mechanism is unsupported and the central equation is a mathematical error, not a debatable modeling choice. The underlying idea — train fire detectors online to be calibrated — is worth pursuing with a correct surrogate, and this paper could serve as a cautionary example of how not to build one. For a serious editor, I would desk reject, but I would not ignore the topic.","headline":"The central equation (6) drops the label and is not a differentiable accuracy surrogate, so the claimed ECE-loss mechanism fails; the paper is a flawed but instructive application idea.","tokens_in":10337,"tokens_out":3512,"would_cite":false,"duration_ms":37778,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A differentiable ECE loss, trained with a curriculum schedule, cuts calibration error on two fire-detection benchmarks by up to half while keeping accuracy loss under 0.7 percent.","keywords":["fire detection","uncertainty calibration","expected calibration error","differentiable loss","curriculum learning","confidence calibration","multi-class classification","deep learning"],"falsifier":"Compute the standard indicator-based ECE (Eq. 5) on held-out test data across several bin counts, say M=5, 10, 15, and 20, while also recording the differentiable surrogate's value at those bins; if the reported improvement exists only at M=15, or if the surrogate drops while the indicator-based ECE rises, the claim that the loss improves calibration is not supported.","tokens_in":9421,"feed_emoji":"🔥","tokens_out":7803,"duration_ms":67845,"temperature":0.7,"pith_summary":"Fire-detection models tend to be overconfident: they report high confidence for predictions that are wrong. This paper claims that a training loss built directly from the Expected Calibration Error (ECE) can fix that, making calibration part of the optimization instead of a post-training patch. The method smooths the non-differentiable ECE metric with a sigmoid function so gradients can flow, and it ramps up the calibration loss's weight over training following curriculum-learning ideas. On the DFAN and EdgeFireSmoke benchmarks, the reported test ECE drops from 0.05436 to 0.04013 and from 0.01208 to 0.00596, while classification accuracy falls by less than 0.7 percent. If the result holds, reliable visual fire detection can be trained directly, without extra validation data or post-hoc recalibration.","feed_headline":"Fire-detection AI: calibration loss halves overconfidence error","feed_subtitle":"A differentiable calibration term cuts expected calibration error by up to 51 percent, with accuracy falling less than 0.7 percent.","key_machinery":"The load-bearing object is the smoothed bin accuracy in Eq. (6), $\\text{acc}(B_m) = \\sum_{i \\in B_m} S(\\tan(\\pi \\hat{p}_i - \\pi/2)) / |B_m|$, where $S(x)=1/(1+e^{-x})$. This replaces the non-differentiable indicator $\\mathbf{1}(\\hat{y}_i = y_i)$ used to compute bin accuracy in the ECE metric, turning the metric into a differentiable loss. The paper then combines this ECE loss with NLL loss using a curriculum schedule (Eq. 8), in which the weight of the ECE loss grows from the epoch it is introduced ($s_e$) to a preset value $\\gamma_E$ over $N$ epochs. The machinery is doing the work of making the training objective itself penalize miscalibration, rather than correcting probabilities after training.","core_discovery":"The central claim is that online calibration for multi-class fire detection can be achieved by replacing the hard 0-1 indicator inside the ECE calculation with a smooth sigmoid function, namely $\\text{acc}(B_m)=\\sum_{i\\in B_m} S(\\tan(\\pi \\hat{p}_i - \\pi/2))/|B_m|$, and using that differentiable quantity as a loss. The paper states this conversion is made \"without altering the underlying calculation logic\" (Section III-C). Combined with cross-entropy loss, and with the ECE loss weight increased gradually during training, this yields calibrated confidence for multi-class fire detection models: test ECE falls from 0.05436 to 0.04013 on DFAN and from 0.01208 to 0.00596 on EdgeFireSmoke, with accuracy loss kept under 0.7 percent. The authors also report that the curriculum-style weight schedule outperforms a constant ECE weight, with constant-weight training producing ECE 0.11289 on EdgeFireSmoke versus 0.00596 for the curriculum-calibrated model, and 0.05254 versus 0.04013 on DFAN.","pith_inferences":["A testable extension is to compare the label-free surrogate in Eq. (6) with a soft-label variant that uses ground-truth correctness, isolating how much of the reported ECE gain comes from matching true accuracy rather than from reshaping confidence values.","The paper's evaluation is on in-distribution test splits; an open question is whether the calibration gain survives out-of-distribution conditions such as novel fire-like distractors, where overconfidence is most dangerous.","Because the loss only modifies the training objective, it should attach to any classification backbone; the same curriculum schedule could be tested on other high-stakes vision tasks such as medical image screening."],"forward_implications":["Fire-detection models can be trained to output calibrated confidence directly, without a post-hoc recalibration step.","The calibration benefit appears even when the baseline already has low ECE: EdgeFireSmoke drops from 0.01208 to 0.00596 with accuracy loss under 0.5 percent.","Applying the ECE loss at constant weight from the start hurts both accuracy and calibration; the curriculum ramp is necessary to balance the two.","The method is reported to work across two different architectures and dataset scales, DFAN and EdgeFireSmoke."],"supporting_citations":[{"why":"Defines ECE and the binning procedure that Eq. (5) and the differentiable loss are built from.","marker":"[20]"},{"why":"Documents neural-network overconfidence and post-hoc calibration methods that motivate online calibration.","marker":"[17]"},{"why":"Supplies the curriculum-learning idea that motivates the dynamic ECE-loss weight schedule.","marker":"[21]"},{"why":"Provides the DFAN dataset and baseline model used for the first experimental validation.","marker":"[22]"},{"why":"Provides the EdgeFireSmoke dataset and baseline model used for the second experimental validation.","marker":"[23]"}],"fun_headline_variants":["Calibrated fire detection: differentiable ECE loss cuts error 51%","Curriculum calibration tames fire-detection AI overconfidence","Fire AI: smooth ECE loss improves calibration, keeps accuracy","Learn to calibrate: fire detection uncertainty halved","Reliable fire detection via differentiable calibration loss"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on treating a smooth curve of the model's confidence as a stand-in for whether the prediction was correct; that curve contains no information about the true label, so if the stand-in stops tracking real correctness, minimizing the loss can shrink the calibration score without making predictions more trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Calibrated fire detection: differentiable ECE loss cuts error 51%","Curriculum calibration tames fire-detection AI overconfidence","Fire AI: smooth ECE loss improves calibration, keeps accuracy","Learn to calibrate: fire detection uncertainty halved","Reliable fire detection via differentiable calibration loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2612,"prompt_tokens":951,"completion_tokens":1661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1580}},"tokens_in":567,"tokens_out":1661,"duration_ms":12707,"temperature":1.0,"reasoning_tokens":1580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:12:07.598599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the standard indicator-based ECE (Eq. 5) on held-out test data across several bin counts, say M=5, 10, 15, and 20, while also recording the differentiable surrogate's value at those bins; if the reported improvement exists only at M=15, or if the surrogate drops while the indicator-based ECE rises, the claim that the loss improves calibration is not supported.","supporting_citations":[{"cited_title":"Optimized dual fire attention network and medium-scale fire classification benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides the DFAN dataset and baseline model used for the first experimental validation."},{"cited_title":"Edgefiresmoke: A novel lightweight cnn model for real-time video fire–smoke detection,","cited_arxiv_id":null,"evidence_quote":"Provides the EdgeFireSmoke dataset and baseline model used for the second experimental validation."}],"review_version":1}