{"id":"129ee17a-abc2-4d92-b4ad-73862908ae37","arxiv_id":"2501.11852","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A cross-entropy optimization attack (CEA) with sememe- and MLM-based candidate words improves black-box adversarial attacks on classifiers and machine translation models.","lead":"This paper proposes CEA, an attack method that uses cross-entropy optimization to find word substitutions that fool text classifiers, machine translation models, and small language models. It reports better attack success, fewer edits, and better sentence quality than several baselines across multiple datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard-label CEA cannot learn from its samples as written: with rho=0.5 and a 0/1 objective, Eq. 11 selects all samples unless more than half of a batch already flips, so the hard-label results do not demonstrate CE optimization.","rationale":"The reader identified candidate-set emptiness as the weakest assumption; that is a relevant missing diagnostic, but the more decisive issue is algorithmic. For hard-label attacks, f is binary after multiplying by m in Eq. 6: non-flips have f = 0 and flips have f = Sem(x|x). With rho = 0.5, the threshold gamma_t is the median, which remains 0 unless more than half of the 100 sampled candidates flip. Therefore Eq. 11's elite set is either all samples (no learning) or only appears when success is already common. The final argmax over an essentially uniform distribution would also produce a near-random full perturbation, which is inconsistent with the reported low modification rates unless candidate sets cover very few words. This internal inconsistency is more load-bearing than the candidate-set coverage question because it means the hard-label experiments, including the LLM hard-label results in Table 8, do not test the proposed CE mechanism at all. I still keep the reader's conditional verdict rather than moving to reject: the soft-label and NMT branches use continuous objectives and may be salvageable with a corrected rho or elite-selection strategy, and the paper provides code and broad experiments. But the hard-label claim as written is not supported and requires a major revision, not just additional statistics.","tokens_in":19761,"tokens_out":18247,"duration_ms":202299,"concrete_test":"Instrument Algorithm 1 for a hard-label attack (e.g., SST2-BERT) by logging gamma_t, the number of samples with f >= gamma_t, and the distance between p_t and the uniform distribution at each iteration. If gamma_t = 0 and all N samples are selected for the first 49 iterations, then run a single-pass uniform-random substituter with the same candidate sets and modification budget and compare SAR, Mod, and SS. If the random substituter matches CEA, the reported hard-label performance is not evidence for CE optimization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The hard-label branch of CEA cannot perform the claimed CE optimization. With m in Eq. 6 equal to 1(F(x) != y), every non-flipping candidate has f = 0 in Eq. 4 because the Sem factor is multiplied by zero; only flipping candidates have f = Sem(x|x). The paper fixes rho = 0.5 (Sec. 4.1), so Eq. 10 sets gamma_t to the median of f over N = 100 samples. If fewer than half of the sampled candidates flip, the median is 0, and Eq. 11 selects all N samples because every f >= 0. The updated distribution is then just the unconditioned empirical frequency of substitutions, with no information about which substitutions caused flips. The distribution becomes informative only when more than half of a batch already flips, which is the regime where the attack has essentially succeeded. Consequently, under the reported hyperparameters the hard-label CEA is a uniform random substituter over the candidate sets, and the hard-label results in Tables 3 and 8 cannot be attributed to CE or rare-event optimization. This directly undercuts the claim that CEA is a single effective black-box attack across soft-label, hard-label, and NMT settings.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes CEA, a black-box textual adversarial attack based on the cross-entropy (CE) optimization method for rare-event simulation. The attack defines a multiplicative objective f(x|x)=m(F(x))·Sem(x|x) with model-specific performance measures for soft-label classifiers (Eq. 5), hard-label classifiers (Eq. 6), and NMTs (Eq. 7), constructs substitution candidates as the intersection of MLM suggestions and HowNet sememe-based synonyms (Sec. 3.2), and iteratively samples N candidates, updates a threshold gamma by the (1-rho)-quantile, and updates categorical distributions over the elite samples (Sec. 3.3). The empirical section compares CEA with PSO, TextFooler, CLARE, RJA, HLBB, LimeAttack, Seq2Sick, HAA, and others on document classification and English-Chinese translation, and additionally reports results on small LLMs, defenses, transferability, targeted attacks, adversarial retraining, and efficiency. The central claim is that CEA consistently outperforms existing methods in attack success, imperceptibility, and sentence quality across soft-label, hard-label, and NMT settings.","tokens_in":20123,"tokens_out":13071,"duration_ms":130817,"significance":"If the empirical claims held, CEA would be a valuable single black-box attack framework spanning classifiers and sequence-to-sequence models, with the practical strengths of a unified objective, a public code repository, and a broad experimental coverage including LLMs, defenses, transferability, and ablation studies. The paper is less convincing as a demonstration of cross-entropy optimization: the hard-label branch as written degenerates under the reported hyperparameters, the reported tables are single runs without variance estimates, and the closest CE-based textual attack baselines are cited but never compared. These issues directly affect the paper's advertised contribution, so the current evidence does not support the strong 'consistently excels' conclusion.","major_comments":[{"comment":"The hard-label branch of CEA cannot perform the claimed CE optimization under the reported hyperparameters. With the hard-label objective m(F(x))=1(F(x)!=y) in Eq. 6, every non-flipping candidate has f=0 (its Sem factor is multiplied by zero). With rho=0.5 (Sec. 4.1), Eq. 10 sets gamma_t to the median of the N=100 objective values. Whenever fewer than half of the sampled candidates flip the victim, the median is 0, and Eq. 11 selects all N candidates because every f>=0. The updated distribution is then just the empirical frequency of unconditional substitutions, with no information about which substitutions caused flips; CE learning begins only in the regime where more than half of a batch already flips, i.e., after the attack has essentially succeeded. The hard-label results in Tables 3, 8, and 10 therefore cannot be attributed to cross-entropy or rare-event optimization. Please re-run the hard-label experiments with an elite-selection rule that can select a proper subset under sparse rewards (for example, a small fixed rho or a positive threshold when any flip occurs), and report whether the final x' constructed by Algorithm 1 lines 13-16 actually flips the victim.","section":"Sec. 3.3 (Eqs. 6, 10, 11) and Sec. 4.1"},{"comment":"The objective and the soft-label threshold are internally inconsistent as printed. Eq. 4 imposes Sem(x|x) <= epsilon, but f=m·Sem is increasing in Sem, so the constraint as written caps the quantity the attacker is trying to maximize; if the intended constraint is semantic preservation, it should read Sem(x|x) >= epsilon, and the values of epsilon and eta must be specified. In Eq. 5, the text states that 'a consistent m(F(x)) <= 1/K value will guarantee a misclassified example', but with m=1-F_c this condition reads 1-F_c <= 1/K, i.e., F_c >= 1-1/K, which is a high-confidence correct prediction and does not imply misclassification. The intended piecewise definition of Eq. 5 and the pigeonhole argument need to be restated carefully.","section":"Sec. 3 (Eqs. 4 and 5)"},{"comment":"The hyperparameters N=100, rho=0.5, and T=50 were selected in the ablation study on IMDB and WMT T1 (Sec. 4.8.2) and then used to produce the headline results on those same datasets (Tables 2-4), which is a selection-on-test procedure; no held-out tuning split is described. In addition, all reported metrics are point estimates from a single run with no error bars, confidence intervals, or significance tests. The 'consistently achieves' comparison to baselines in Sec. 4.6 is therefore not statistically supported. Please report results over multiple random seeds and a separate validation split for any hyperparameter choice.","section":"Secs. 4.1, 4.8.2 and Tables 2-4"},{"comment":"The related-work section cites [32] and [33] as existing applications of cross-entropy optimization to adversarial text generation, yet neither method is included in any experiment. Because the paper's central claim is that CEA is a new and superior CE-based attack, omitting the closest CE baselines makes the specific contribution over prior CE work impossible to assess. Please add these baselines to the comparison or justify their exclusion with concrete reasons.","section":"Secs. 2.3 and 4.4"},{"comment":"The candidate substitution set si = sM_i ∩ ssyn_i underlies every experiment, but the paper reports no statistics on the size of si, the fraction of positions for which the intersection is empty, or the value of the MLM top-K parameter. If the intersection is frequently empty, many words cannot be modified, and if it omits the words that actually flip the victim, the CE sampler has no path to success. Reporting the average candidate-set size and the empty-intersection rate per dataset would directly test this load-bearing assumption.","section":"Sec. 3.2 and Tables 2-4"},{"comment":"In Table 4, CEA's BLEU Drop on WMT T2 with T5 is 16, which is lower than HAA's 17, yet Sec. 4.6.1 states that CE 'surpasses static baselines in attack performance' and the abstract claims consistent superiority. This row contradicts the 'consistently' claim and should be acknowledged or explained in the text.","section":"Table 4, WMT T2/T5 row"}],"minor_comments":[{"comment":"There are many typographical and wording errors, including 'the an customised' (Sec. 3), 't‘ e performance' (Sec. 3.3), 'smenatics' (Sec. 4.6.1), 'Rostbustness' (Fig. 6), 'TexCNN' (Fig. 4), 'deprave' (Sec. 3.1.2), 'substations' (Sec. 3.3), and '4Seq2sick' (Table 7). A careful proofread is needed.","section":"Throughout"},{"comment":"The MLM top-K value is never reported; please specify K and whether it varies across datasets or positions.","section":"Sec. 3.2"},{"comment":"Constructing the final example x' by taking the per-position argmax of the learned marginals can yield a combination that was never sampled as a whole; please describe how ties are broken and whether the final constructed example is verified to satisfy the attack objective.","section":"Algorithm 1, lines 13-16"},{"comment":"The initial-probability ablation is not well-defined because p(i,j)=0.1-0.5 cannot be varied independently for all positions while maintaining sum_j p(i,j)=1 for arbitrary candidate-set sizes ni; please describe exactly how this ablation was performed.","section":"Sec. 4.8.2"},{"comment":"The Emotion dataset is described in Sec. 4.2.1 and used in Sec. 4.11 but is missing from the dataset statistics in Table 1.","section":"Table 1"},{"comment":"The code-availability statements list inconsistent repository URLs (github.com/MingzeLucasNi/RCEAgit and github.com/MingzeLucasNi/CEA.git); please unify them.","section":"Declarations"}],"recommendation":"major_revision","confidential_remarks":"The paper's relationship to the CE-based attacks it cites in [32] and [33] needs to be clarified and empirically quantified; the current text cites them as related work but never compares against them. I would also ask the authors to verify the reference entries for [32] and [33], which have unusually formatted volume/page information. The hard-label issue in Eq. 10-11 is the main technical concern, but it is fixable within the scope of the manuscript by changing the elite-selection rule and re-running the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper has a real bug in its hard-label branch. With rho=0.5 and a 0/1 objective, Eq. 11 selects every sample unless more than half of a batch already flips. So hard-label CEA is just random substitution over the candidate sets in the typical case, and Tables 3 and 8 don't demonstrate cross-entropy optimization. That's load-bearing because the paper markets CEA as a single effective attack across soft-label, hard-label, and NMT.\n\nWhat's actually new: the unified objective for soft-label, hard-label, and NMT is a legitimate packaging, and the sememe-plus-MLM intersection for candidate generation is a reasonable heuristic. Experiments are wide-ranging: classifiers, NMTs, small LLMs, defenses, transferability, targeted attacks, and adversarial retraining. If it worked, it would be a useful tool, though not paradigm-shifting, since the paper's own related work cites prior CE text attacks in [32] and [33].\n\nSoft spots, in proportion:\n1. The hard-label threshold issue is the big one. With N=100 and rho=0.5, gamma is the median of f. If fewer than half the candidates flip, gamma is 0 and all candidates pass, so the probability update reduces to the marginal substitution frequency, not conditioned on success. The distribution becomes informative only after a majority already flips, which is too late.\n2. No error bars anywhere; single runs for a stochastic attack.\n3. Hyperparameters N, rho, T were tuned on IMDB and WMT T1 and then evaluated on the same datasets. That's test-set leakage.\n4. The two closest CE baselines are cited but not compared, which weakens the superiority claim.\n5. The paper doesn't report candidate set sizes or how often the intersection is empty. If it's often empty, the attack has no path.\n\nThe Eq. 5 justification is sloppy but not central. The soft-label branch looks fine. The stress-test concern holds up on my reading; the authors can fix it by using an elite threshold or a smaller rho for hard-label and re-running.\n\nBottom line: This deserves a serious referee, not a desk reject. The idea is plausible, the experiments are broad, and the flaws are identifiable and fixable. It needs major revision: rerun hard-label with a proper elite threshold, add variance, include the CE baselines, and redo the hyperparameter tuning honestly. I wouldn't cite it in current form, but I'd bring it to a reading group as a cautionary example of a thresholding bug.\n\nRecommendation: send to peer review with the expectation of heavy revision.","headline":"Hard-label branch of CEA degenerates to random substitution under the reported hyperparameters, undercutting the central unified-attack claim, but the broad experiments and identifiable, fixable flaws make it worth a serious referee.","tokens_in":20567,"tokens_out":3444,"would_cite":false,"duration_ms":31456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single black-box attack built on cross-entropy optimization, CEA, treats adversarial examples as rare events and outperforms existing text attacks on classifiers, machine translation models, and small LLMs while…","keywords":["adversarial attacks","cross-entropy optimization","rare event simulation","black-box attacks","hard-label attacks","machine translation attacks","text classification attacks"],"falsifier":"Count, on the reported datasets, the fraction of word positions whose intersection set $s_i$ is empty or contains no substitution that changes the target model's output, and compare CEA against uniform random sampling from the same sets; a high empty fraction or similar success under random sampling would show the cross-entropy refinement is not the active ingredient.","tokens_in":19583,"feed_emoji":"🎯","tokens_out":9099,"duration_ms":86284,"temperature":0.7,"pith_summary":"This paper tries to establish that textual adversarial attacks can be driven by one optimization method across settings that are usually treated separately: soft-label classifiers, hard-label classifiers, and neural machine translation. The method, Cross-Entropy Attack (CEA), treats a successful adversarial example as a rare event in a well-trained model and uses cross-entropy optimization to sample and refine word substitutions until the model's behavior changes. The paper reports that CEA achieves higher attack success rates, lower modification rates, higher semantic similarity, and better fluency than existing attacks on document classification, translation, and small LLM classification tasks. The practical point is that a unified black-box attack lowers the cost of robustness evaluation and forces defenses to handle distribution-refining sampling attacks, not just saliency-ranked word swaps.","feed_headline":"Rare-event sampling yields one universal black-box text attack","feed_subtitle":"One optimizer produces adversarial examples for classifiers, translators, and small LLMs while editing fewer words.","key_machinery":"The load-bearing mechanism is a sequence of rare-event simulations over a multivariate Categorical distribution. Each word position $i$ is assigned a substitution set $s_i$, formed as the intersection of masked-language-model top candidates and synonyms from a sememe-based thesaurus, and a candidate rewrite is drawn position-by-position from $\\text{Cat}(\\theta_i)$. At each iteration the threshold $\\hat{\\gamma}$ is set to the $(1-\\rho)$-quantile of sampled objective values, and each substitution probability $\\hat{p}_{(i,j)}$ is updated as the frequency of that substitution among the samples above threshold. This distribution-refinement loop converts discrete replacement search into tractable sampling, and the same loop is reused for soft-label, hard-label, and NMT objectives; only the performance measure $m(F(\\cdot))$ changes.","core_discovery":"The central claim is that the hard combinatorial problem of finding a minimal, meaning-preserving word substitution can be reformulated as maximizing a single scalar objective $f(\\tilde{x}|x)=m(F(\\tilde{x}))\\cdot \\text{Sem}(\\tilde{x}|x)$ under constraints on modification rate and semantic similarity, and that cross-entropy optimization solves that objective without gradients or saliency rankings. For classifiers, the paper sets $m(\\cdot)$ to a truncated confidence term that gives no extra reward once the model's confidence on the true class falls below the uniform threshold $1/K$ (soft-label), or to the indicator of a label flip (hard-label). For NMT, it sets $m(\\cdot)=1-\\text{BLEU}(F(\\tilde{x})|y)\\cdot \\text{Sem}(F(\\tilde{x})|y)$, so successful attacks lower BLEU and semantic fidelity together. The cross-entropy loop samples $N$ candidate texts from a categorical distribution over per-word substitution sets, raises the acceptance threshold to the $(1-\\rho)$-quantile of sampled scores, and re-estimates substitution probabilities from the samples above threshold; after $T$ iterations it outputs the highest-probability substitution at each position. The paper claims this one loop attains the best attacking performance, imperceptibility, and sentence quality in its experiments.","pith_inferences":["Editorial inference: If the gain comes from concentrating probability on rare effective substitutions, the same loop should transfer to other discrete search problems with a cheap scalar score, such as prompt-level attacks on instruction-following language models.","Editorial inference: A controlled comparison that replaces the CE update with uniform random sampling from the same candidate sets would isolate how much of the reported success comes from the optimizer rather than from the MLM-plus-thesaurus candidate construction.","Editorial inference: The paper's threshold and candidate-count ablations suggest a query-budget trade-off; a word-level adaptive threshold or early-stopping rule could reduce attack cost further on easy examples."],"forward_implications":["A single optimizer, CEA, covers soft-label classifiers, hard-label classifiers, and NMTs, so an adversary does not need to pick a method by model type.","Because CEA needs only predicted labels in the hard-label setting, it works in realistic black-box scenarios where confidence scores are hidden.","The reported low modification rates and high semantic similarity imply adversarial texts that are harder for a casual reader to notice.","The NMT objective, which combines BLEU drop with semantic drop, generates translations that are both lower quality and less faithful, giving robustness evaluators a stronger stress test.","Adversarial retraining with CEA examples improves downstream robustness up to a point, beyond which clean accuracy declines, so the attack is useful for generating training data as well as for evaluation."],"supporting_citations":[{"why":"Supplies the cross-entropy method for combinatorial and continuous optimization that CEA adapts to text attack.","marker":"[25]"},{"why":"Establishes the rare-event simulation perspective for adversarial example generation that motivates the attack.","marker":"[32]"},{"why":"Applies cross-entropy optimization to adversarial text, the direct predecessor CEA builds on.","marker":"[33]"},{"why":"Supplies the masked-language-model contextual substitution construction used to build candidate sets.","marker":"[6]"},{"why":"Supplies the sememe-based lexical resource used to filter candidates for semantic consistency.","marker":"[35]"},{"why":"Supplies the BLEU score used in the NMT objective and in evaluation.","marker":"[19]"},{"why":"Formulates word-level adversarial attack as combinatorial optimization and serves as a soft-label baseline.","marker":"[4]"},{"why":"Provides a budgeted hard-label attack baseline that CEA is compared against.","marker":"[8]"},{"why":"Provides a sampling-based attack baseline (reversible jump) that CEA is compared against.","marker":"[24]"},{"why":"Provides a sequence-to-sequence attack baseline (Seq2Sick) for the NMT comparison.","marker":"[2]"}],"fun_headline_variants":["One cross-entropy loop attacks classifiers, translators, and LLMs","Cross-entropy attack edits fewer words while keeping meaning intact","Rare-event simulation powers a single attack for many text models","No gradients needed: cross-entropy finds text adversarial examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack succeeds only when the substitution sets built in Section 3.2 contain enough replacement words that actually change the target model's behavior; the paper does not report how often those sets are empty or miss the decisive word.","fun_headline_variants_meta":{"raw":{"variants":["One cross-entropy loop attacks classifiers, translators, and LLMs","Cross-entropy attack edits fewer words while keeping meaning intact","Rare-event simulation powers a single attack for many text models","No gradients needed: cross-entropy finds text adversarial examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2656,"prompt_tokens":941,"completion_tokens":1715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1645}},"tokens_in":557,"tokens_out":1715,"duration_ms":12764,"temperature":1.0,"reasoning_tokens":1645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:47:40.783315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count, on the reported datasets, the fraction of word positions whose intersection set $s_i$ is empty or contains no substitution that changes the target model's output, and compare CEA against uniform random sampling from the same sets; a high empty fraction or similar success under random sampling would show the cross-entropy refinement is not the active ingredient.","supporting_citations":[{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence, pp","cited_arxiv_id":null,"evidence_quote":"Provides a budgeted hard-label attack baseline that CEA is compared against."},{"cited_title":"Methodology and Computing in Applied Probability 1(2), 127–190 (1999)","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-entropy method for combinatorial and continuous optimization that CEA adapts to text attack."},{"cited_title":"Advances in Neural Information Processing Systems 33, 2345–2356 (2020)","cited_arxiv_id":null,"evidence_quote":"Establishes the rare-event simulation perspective for adversarial example generation that motivates the attack."},{"cited_title":"Proceedings of the Association for Computational Linguistics (ACL) 58, 2141–2152 (2020)","cited_arxiv_id":null,"evidence_quote":"Applies cross-entropy optimization to adversarial text, the direct predecessor CEA builds on."},{"cited_title":"In: Proceedings of 27 the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the masked-language-model contextual substitution construction used to build candidate sets."},{"cited_title":"In: Coling 2010: Demonstrations, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the sememe-based lexical resource used to filter candidates for semantic consistency."},{"cited_title":"In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp","cited_arxiv_id":null,"evidence_quote":"Formulates word-level adversarial attack as combinatorial optimization and serves as a soft-label baseline."},{"cited_title":"Machine Learning, 1–31 (2024)","cited_arxiv_id":null,"evidence_quote":"Provides a sampling-based attack baseline (reversible jump) that CEA is compared against."},{"cited_title":"In: Pro- ceedings of the AAAI Conference on Artificial Intelligence, vol","cited_arxiv_id":null,"evidence_quote":"Provides a sequence-to-sequence attack baseline (Seq2Sick) for the NMT comparison."}],"review_version":1}