{"id":"6d6eb589-1d8b-4c43-9d94-57fae8110bfd","arxiv_id":"2412.19018","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FuRud selects per-class triangular fuzzy membership functions via simulated annealing to transform ICL probabilities, improving accuracy and class balance.","lead":"This paper introduces FuRud, a post-hoc method that corrects large language model in-context learning class probabilities to balance class accuracies without retraining. Across seven benchmarks it reports a 21% relative accuracy gain and a 56% relative reduction in accuracy bias.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 21% relative accuracy gain is dominated by the DDI outlier; without it, FuRud no longer outperforms DNIP, so the cross-benchmark claim is not robust.","rationale":"The reader's weakest assumption concerned the stability of the probability-magnitude-to-accuracy relationship. That is a valid concern, but the more immediately load-bearing issue is the fragility of the headline numbers themselves. The reported 21% relative accuracy improvement and the claim of outperforming state-of-the-art methods are both driven overwhelmingly by the DDI dataset, which has an implausibly low ICL baseline (7.2% for a 5-class task) and a dramatic FuRud improvement (69.3%). Leave-one-out arithmetic shows that without DDI, FuRud's average accuracy is 72.45% versus DNIP's 74.00%, so the method does not actually lead on accuracy across the other six benchmarks. This does not require any assumption about data distributions; it is directly visible in the published table. The paper should report a leave-one-out robustness analysis or provide a thorough explanation of the DDI result before the cross-benchmark claim is taken at face value. Since the reader already flagged DDI as suspicious and assigned a CONDITIONAL verdict, this concern reinforces but does not change that verdict.","tokens_in":13400,"tokens_out":9241,"duration_ms":90532,"concrete_test":"Recompute the Table 1 summary statistics for all four methods excluding DDI: average accuracy, average COBias, and per-dataset win counts. If FuRud loses its average-accuracy advantage over DNIP and BC, and the relative accuracy gain over ICL falls below 10%, then the 'across seven benchmarks' claim is not supported. As a secondary check, inspect DDI's original ICL probability distributions to confirm they are not degenerate and reproduce the 69.3% accuracy with held-out demonstrations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim depends heavily on a single dataset. In Table 1, DDI's ICL accuracy is 7.2% (far below random for 5 classes), and FuRud raises it to 69.3%, a +62.1 point jump. Recomputing the seven-dataset averages without DDI changes the picture: ICL average accuracy becomes (59.4×7−7.2)/6 = 68.1% and FuRud's becomes (72.0×7−69.3)/6 = 72.45%, a relative improvement of only 6.4% instead of the claimed 21%. More importantly, DNIP's average accuracy excluding DDI is (69.2×7−40.4)/6 = 74.0%, so FuRud no longer outperforms DNIP on average accuracy. Per-dataset, DNIP beats FuRud on AGNews (87.9 vs 85.7), DBpedia (93.4 vs 92.2), and PubMedQA (63.1 vs 55.9), with near ties on TREC and RTE; FuRud's only clear win is DDI. Thus the statement that FuRud 'outperforms state-of-the-art debiasing methods across seven benchmarks' rests on a single unexplained outlier. The paper provides no analysis of DDI's unusually low ICL baseline or why the correction succeeds there, so the robustness of the central claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FuRud, a post-hoc debiasing method for in-context learning (ICL) text classification. For each class, FuRud selects one of 19 triangular fuzzy membership functions by solving a weighted multi-objective integer program via simulated annealing on a labeled optimization set, then applies the selected functions to test-time class probabilities to reduce class-accuracy bias (COBias) and improve overall accuracy. Experiments on seven benchmarks with Llama-2-13B report average accuracy rising from 59.4% to 72.0% and COBias falling from 40.5% to 17.8%, with additional results on two more models, a few-shot optimization setting, and letter-based prompts.","tokens_in":13719,"tokens_out":7072,"duration_ms":53714,"significance":"The idea of using fuzzy membership functions to provide interpretable, range-specific probability corrections is novel and practically attractive because it requires no LLM fine-tuning and operates purely at inference time. The paper follows reasonable experimental hygiene: separate optimization/development/test splits, three runs per dataset, and supplementary analyses on additional models and optimization sizes. If the reported improvements were robust, FuRud would be a useful addition to the ICL debiasing toolbox. However, the headline claim of outperforming state-of-the-art methods is not robust to the removal of the DDI dataset, and the COBias comparison to DNIP is not favorable. The central empirical claim therefore needs re-examination and reframing.","major_comments":[{"comment":"The headline claim that FuRud 'outperforms state-of-the-art debiasing methods across seven benchmarks' is not robust to the DDI outlier. In Table 1, DDI has an ICL accuracy of 7.2% (far below random for a 5-class task) and FuRud raises it to 69.3%, a 62.1-point jump. Excluding DDI from the seven-dataset averages, ICL accuracy becomes 68.1% and FuRud becomes 72.4%, a relative improvement of only 6.4% rather than the claimed 21%; DNIP's average without DDI is 74.0%, so FuRud no longer outperforms DNIP. Per-dataset, DNIP is higher on AGNews, DBpedia, and PubMedQA. The paper gives no analysis of why DDI's ICL baseline is so low or why FuRud succeeds there, so the cross-benchmark claim is not established.","section":"Table 1, Abstract"},{"comment":"The COBias comparison to DNIP is unfavorable and should be reported honestly. FuRud's average COBias is 17.8% versus DNIP's 14.3%, and DNIP achieves lower COBias on six of the seven datasets (AGNews, DBpedia, SST-5, TREC, RTE, DDI). The abstract's 'outperforming state-of-the-art' is therefore only defensible on accuracy, and even that claim depends on the DDI outlier discussed above.","section":"Table 1, Section 4.2"},{"comment":"The paper's motivating assumption that, within a ground-truth class, instances with lower ICL probability have lower accuracy is asserted in Section 1 as a 'general observation' but supported only by a single example in Figure 4 (class Business on AGNews). Since the interpretability and range-specific correction claims rest on this relationship, the authors should provide a systematic validation across all classes and datasets, for example by reporting the correlation between probability bins and per-bin accuracy before correction.","section":"Section 1, Figure 4"},{"comment":"The hyperparameters of the optimization are not reported. The paper says α and β are 'tuned on the development set' and λ is set to 0.5, but the actual values of α and β, the simulated annealing schedule, the number of iterations, and the number of restarts are missing. Because the method is an integer-programming heuristic with a search space of 19^N, these details are needed to assess stability and reproducibility.","section":"Section 4.1"}],"minor_comments":[{"comment":"The table header contains the typo 'PubMedaQA'; it should be 'PubMedQA'.","section":"Table 1"},{"comment":"The acronym is written inconsistently as 'FuRuD' (Section 4.2) and 'FuRud' elsewhere; please unify.","section":"Throughout"},{"comment":"Equation (3) is self-referential: p'_mi is defined using the sum of p'_mi on the right-hand side. Please clarify the intended computation order, e.g., first compute each f_k(p_mi) for the selected κ_i, then reset to the original probability if all are zero.","section":"Equation (3)"},{"comment":"There is a typo in the sentence 'although he DC partition alone can obtain 15% higher accuracy'; 'he' should be 'the'.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The DDI result is so extreme that the authors should be asked to verify the DDI experiments and to provide per-dataset significance tests. If the DDI result is genuine, the paper should still be accepted only after reframing the central claim to focus on interpretability and per-dataset improvements rather than aggregate outperformance. The fit with the journal is otherwise fine."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nFuRud is a useful extension of DNIP but the main claim is shakier than the abstract suggests. The idea is simple: after ICL, transform each class's predicted probabilities with one of 19 triangular membership functions, chosen per class by simulated annealing on a labeled optimization set. That is new relative to DNIP's class-level weights, and the paper does some things right: a clean optimization/development/test split, three runs per dataset, ablations on membership granularity, and extra experiments on other models and prompt styles. The interpretability analysis, e.g., showing which ranges of a class's probabilities get amplified or suppressed, is a genuine attempt to make the corrections explainable.\n\nThe soft spot is the DDI outlier. DDI's ICL accuracy is 7.2%, far below random; FuRud raises it to 69.3%, a 62-point swing. Remove that one dataset and the averaged results change a lot: FuRud's relative accuracy gain over ICL drops from 21% to about 6%, and DNIP's average accuracy (74.0%) actually beats FuRud (72.45%). Per-dataset, DNIP wins on AGNews, DBpedia, and PubMedQA. So the claim that FuRud 'outperforms state-of-the-art debiasing methods across seven benchmarks' is not robust. The paper acknowledges DDI's importance in one ablation but never addresses why DDI behaves so differently or how much the headline numbers depend on it. That needs to be fixed.\n\nOther issues are smaller. The code isn't released, and the values of α and β, which control the COBias/accuracy tradeoff, are not disclosed. The interpretability claim is illustrated but not evaluated in any quantitative way. The underlying assumption that probability range tracks class accuracy is plausible and the optimization/test split means the gains aren't circular, but the paper doesn't test whether that relationship is stable across datasets or demonstrations.\n\nThis is a solid incremental paper, not a breakthrough. It deserves a serious referee, but with major revision: the authors should report results with DDI excluded, analyze why DDI is such an outlier, release code, and disclose the tuning parameters. I wouldn't cite it in its current form, but it's worth a reading group discussion.\n\nBest,\n[Your name]","headline":"FuRud is a plausible incremental extension of DNIP, but the headline 21% gain is driven almost entirely by one outlier dataset (DDI); without it, FuRud no longer beats DNIP on average accuracy.","tokens_in":14217,"tokens_out":2982,"would_cite":false,"duration_ms":25598,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fuzzy rules cut LLM class-accuracy bias by 56%","keywords":["fuzzy rule","in-context learning","debiasing","class imbalance","interpretability","probability correction","triangular membership functions","COBias"],"falsifier":"If, on a held-out split of a benchmark, the per-class accuracy within probability ranges is inverted or flat compared with the optimization set—for instance, low-probability instances turn out to be more accurate than high-probability ones—then the learned membership functions would amplify the wrong ranges and FuRud's reported gains should disappear. A direct check is to recompute FuRud's corrections on a dataset where the optimization set and test set come from different distributions or time periods and see whether accuracy and COBias revert toward or below the un-corrected ICL baseline.","tokens_in":13214,"feed_emoji":"🎯","tokens_out":5486,"duration_ms":47894,"temperature":0.7,"pith_summary":"In-context learning (ICL) lets large language models classify text by prompting, but per-class accuracy is often lopsided: some classes are over-predicted while others are under-predicted. The paper claims this imbalance can be fixed at inference time, without retraining or updating the model, by transforming each class's output probability through a range-specific correction curve. The curves are triangular membership functions from fuzzy logic, and each class picks one of 19 candidates by solving a small combinatorial optimization on labeled examples. On seven classification benchmarks, the method, called FuRud, cuts the average pairwise class-accuracy gap (COBias) by 56% and raises overall accuracy from 59.4% to 72.0%. If right, this gives an interpretable, cheap way to debias ICL outputs.","feed_headline":"Fuzzy rules cut LLM class-accuracy bias by 56%","feed_subtitle":"Post-hoc triangular corrections raise average ICL classification accuracy to 72% without retraining the model.","key_machinery":"The central machinery is a family of 19 triangular membership functions, each a piecewise-linear map f(p) defined by three parameters (a, b, c), together with a per-class integer selection variable κ_i that picks one map per output class. Each map transforms a class probability according to which probability range it falls into, allowing asymmetric amplification and reduction. A multi-objective integer program combines three objectives—minimize COBias, maximize overall accuracy, and penalize classes below an accuracy threshold—into a single energy function that is minimized by simulated annealing on a labeled optimization set. At inference, the learned maps are applied to each test instance's class probabilities and the final class is the argmax of the corrected probabilities.","core_discovery":"The paper's central discovery is that the accuracy imbalance in ICL classifiers is range-dependent: within a ground-truth class, instances whose model-assigned probability falls in a low range tend to be misclassified, while instances with higher probability in that class are usually correct. Consequently, a single class-level weight—as used by prior debiasing—misses this structure, and the right correction is a piecewise-linear map that amplifies the low range, sometimes leaves middle ranges alone, and can shrink high ranges of competing classes. FuRud learns one such map per class from a labeled optimization set, choosing among 19 triangular membership functions by minimizing a weighted sum of class-accuracy bias (COBias), overall error, and an extreme-class penalty. The selected maps then transform test-time probabilities, and a 'Don't Change' identity function is selected when a class is already accurate. The paper reports average test accuracy rising from 59.4% to 72.0% and average COBias falling from 40.5% to 17.8% across AGNews, DBpedia, SST-5, TREC, RTE, DDI, and PubMedQA.","pith_inferences":["Because the method operates purely on output probabilities, the same machinery could be applied to any probabilistic classifier—vision models, rerankers, or regression-to-class reductions—whenever the range-accuracy monotonicity holds, not just to decoder-only LLMs.","The interpretability output could be used as a diagnostic for prompt design: if a class frequently selects an amplification function, the prompt's demonstrations or verbalizer are likely biasing that class down, guiding targeted prompt edits.","A natural stress test would be to learn membership functions on one domain and apply them to another, to see whether the selected fuzzy rules are generalizable calibration priors or overfit to the optimization set.","The 'Don't Change' selection rate could itself serve as a metric of how imbalanced a task is, giving a probe into model bias before debiasing."],"forward_implications":["FuRud gives a post-hoc, training-free debiasing route: no LLM parameter updates, only a few minutes of CPU search, so it applies to models whose weights are frozen or inaccessible.","The selected membership functions double as an explanation: a class assigned 'Don't Change' is already accurate, while an amplified low-range or shrunk high-range points to where the model's probability scale is miscalibrated.","The method transfers to harder prompting setups: under letter-answer prompts it raises average accuracy by 44% relative and cuts COBias by 54% relative, and under k-shot demonstrations it raises average accuracy to 73.5%.","FuRud works with as few as 10 labeled optimization examples, suggesting it can be deployed quickly on new tasks without collecting large validation sets.","The accuracy-COBias tradeoff is controllable through the granularity of the fuzzy partitions, with mixed partitions giving the best joint result."],"supporting_citations":[{"why":"Defines COBias, the pairwise class-accuracy imbalance metric, and contributes the DNIP re-weighting baseline that FuRud improves upon.","marker":"[1]"},{"why":"Introduces dummy-prompt output calibration for few-shot learning, one of the calibration baselines FuRud compares against.","marker":"[15]"},{"why":"Provides Batch Calibration, a test-time contextual correction baseline, and a main comparison point.","marker":"[20]"},{"why":"Introduces fuzzy sets, the mathematical foundation for the membership functions used in FuRud.","marker":"[12]"},{"why":"Supplies triangular membership functions and fuzzy rule-based classification methods that motivate the design of the 19 functions.","marker":"[8]"},{"why":"Establishes that integer programming is NP-complete, the justification for using simulated annealing in the search.","marker":"[21]"}],"fun_headline_variants":["FuRud: fuzzy triangular rules cut LLM class bias by 56% without retraining","Interpretable fuzzy maps rebalance LLM classes, cutting bias by 56%","Range-aware fuzzy corrections improve ICL accuracy from 59% to 72%","No retraining: fuzzy membership functions halve class bias in LLM ICL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the relationship between the magnitude of an in-context output probability for a class and the true accuracy of instances in that probability range is stable enough to transfer from the labeled optimization set to the unlabeled test distribution.","fun_headline_variants_meta":{"raw":{"variants":["FuRud: fuzzy triangular rules cut LLM class bias by 56% without retraining","Interpretable fuzzy maps rebalance LLM classes, cutting bias by 56%","Range-aware fuzzy corrections improve ICL accuracy from 59% to 72%","No retraining: fuzzy membership functions halve class bias in LLM ICL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001795,"raw_usage":{"total_tokens":7122,"prompt_tokens":1045,"completion_tokens":6077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":5987}},"tokens_in":661,"tokens_out":6077,"duration_ms":39976,"temperature":1.0,"reasoning_tokens":5987,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:57:25.162327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If, on a held-out split of a benchmark, the per-class accuracy within probability ranges is inverted or flat compared with the optimization set—for instance, low-probability instances turn out to be more accurate than high-probability ones—then the learned membership functions would amplify the wrong ranges and FuRud's reported gains should disappear. A direct check is to recompute FuRud's corrections on a dataset where the optimization set and test set come from different distributions or time periods and see whether accuracy and COBias revert toward or below the un-corrected ICL baseline.","supporting_citations":[{"cited_title":"Calibrate Before Use: Improving Few-shot Performance of Language Models","cited_arxiv_id":null,"evidence_quote":"Introduces dummy-prompt output calibration for few-shot learning, one of the calibration baselines FuRud compares against."},{"cited_title":"Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering","cited_arxiv_id":null,"evidence_quote":"Provides Batch Calibration, a test-time contextual correction baseline, and a main comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces fuzzy sets, the mathematical foundation for the membership functions used in FuRud."},{"cited_title":"Hybridization of Fuzzy GBML Approaches for Pattern Classification Problems","cited_arxiv_id":null,"evidence_quote":"Supplies triangular membership functions and fuzzy rule-based classification methods that motivate the design of the 19 functions."},{"cited_title":"Garey and D.S","cited_arxiv_id":null,"evidence_quote":"Establishes that integer programming is NP-complete, the justification for using simulated annealing in the search."}],"review_version":1}