{"id":"eddb44b6-6fe6-4784-b35f-f83d60689e2d","arxiv_id":"2412.10535","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper claims model-specific correlations between adversarial and OOD robustness in LLMs, but these are based on a tiny number of strategies and are not statistically reliable.","lead":"This paper tests whether methods that improve one type of LLM robustness (adversarial or out-of-distribution) also improve the other, using three models and four benchmark datasets. It reports model-specific correlations between these robustness types, but the correlations are computed from only three or four strategies per benchmark and lack statistical validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed neutral/negative/positive correlations rest on regression slopes fitted to at most a handful of min-max-normalized points, with no stability check; a single benchmark can flip the sign, as the paper itself concedes for LLaMA2-13b.","rationale":"The reader's weakest assumption is precisely that a linear regression slope computed from min-max-normalized accuracies across three or four strategies is a meaningful and stable measure of the intrinsic correlation. My analysis confirms this is the load-bearing concern: the correlation findings are the paper's central contribution, and they rest entirely on the sign of slopes fit to a very small number of non-independent points. The paper's own Section 8 acknowledges that the LLaMA2-13b negative correlation is driven by one benchmark, which is direct evidence that the slope is not robust. I also considered whether the paper has independent support: it reports raw accuracy and F1 tables for all benchmark-strategy combinations, which is useful, but the correlation analysis step itself lacks any uncertainty quantification or sensitivity analysis. I therefore see no reason to change the reader's REJECT verdict, though the underlying data tables could support a more careful follow-up study.","tokens_in":12914,"tokens_out":2766,"duration_ms":23914,"concrete_test":"Recompute each model's slope with leave-one-out: drop each strategy-benchmark point (or each strategy) and refit the regression; for LLaMA2-13b, if excluding the AHP-on-PromptRobust point changes the slope from negative to non-negative, the claimed negative correlation is not robust. Also compute a 95% confidence interval via bootstrap over the 2×2 benchmark pairings and report whether the interval excludes zero.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—neutral, negative, and positive correlations—depends entirely on the sign of the slope from a linear regression over min-max-normalized accuracies, as described in Section 8. Each model contributes only 3–4 strategy values per benchmark, and the points are non-independent because each adversarial benchmark is paired with each OOD benchmark. With so few points and no confidence interval, bootstrap, or significance test, the slope is not a stable estimator of the claimed correlation. The paper's own Section 8 concedes that the negative LLaMA2-13b result is strongly influenced by AHP on PromptRobust; the AHP accuracy drops to .583 from .774 on that benchmark while AHP improves OOD accuracy on Flipkart, so removing that single strategy-benchmark point or changing the normalization would likely change the sign. The same fragility applies to the positive Mixtral slope and the neutral LLaMA2-7b slope. Moreover, the points are not independent samples from a population; they are outcomes of a small set of interventions, so the slope conflates method effects with any intrinsic robustness relationship. Without reporting the exact number of points, the pairing procedure, or any uncertainty, the abstract's 'neutral/negative/positive' summary overstates the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether adversarial robustness and out-of-distribution (OOD) robustness are correlated in large language models. The authors evaluate Llama2-7b, Llama2-13b, and Mixtral-8x7b on two adversarial benchmarks (PromptRobust, AdvGLUE++) and two OOD benchmarks (Flipkart, DDXPlus), applying three robustness-enhancement strategies (AHP, AHP2, ICR). For each model they compute a linear regression slope over min-max normalized accuracy values of the strategies on adversarial-OOD benchmark pairings, interpreting the sign of the slope as the correlation between the two robustness types. They report a neutral correlation for Llama2-7b, a negative correlation for Llama2-13b, and a positive correlation for Mixtral-8x7b, and discuss implications for model architecture and size. The paper includes per-attack and per-task tables in the appendix and describes the prompts used for each strategy.","tokens_in":13177,"tokens_out":4286,"duration_ms":38461,"significance":"If the correlation findings were reliable, they would inform whether robustness interventions transfer across adversarial and distributional shift threat models, a question of practical importance. The paper also provides a transparent account of its experimental limitations and makes its prompt templates available in the appendix, which aids reproducibility. However, the central claims rest on a fragile statistical analysis: the correlation coefficients are slopes from regressions over only three to four strategies per benchmark, with no error bars, confidence intervals, or significance tests, and the paper itself concedes that a single benchmark-strategy point drives the negative correlation for Llama2-13b. As such, the neutral/negative/positive classification is not supported by the evidence, and the study is better characterized as a preliminary pilot than a definitive finding.","major_comments":[{"comment":"The central claim of the paper—neutral, negative, and positive correlations—depends entirely on regression slopes computed from 3-4 min-max normalized accuracy values per benchmark, as described in Section 8. The paper provides no confidence intervals, bootstrap, or any significance test for these slopes, and the data points are not independent because each adversarial benchmark is paired with each OOD benchmark. The instability of the result is acknowledged in Section 8: the negative correlation for LLaMA2-13b is 'strongly influenced by AHP on PromptRobust', and removing that single strategy-benchmark point would likely change the sign. This fragility means the abstract's model-specific correlation claims overstate what the data actually support.","section":"Section 8"},{"comment":"There is a direct inconsistency in the definition of the regression variables. Equation (3) states that Y is the dependent variable (Adversarial Metric) and X is the independent variable (OOD Metric), but Section 8 says 'X values are adversarial benchmark metrics and Y values are OOD benchmark metrics.' This ambiguity makes the reported slope sign impossible to interpret. Moreover, a slope from a linear regression on min-max normalized variables is not a correlation coefficient; a correlation requires standardized variables and an accompanying measure of uncertainty. As reported, the slopes do not provide a valid measure of correlation between adversarial and OOD robustness.","section":"Section 4.1.3 / Section 8"},{"comment":"The OOD benchmarks are small and possibly not OOD for all models. Flipkart is subsampled to only the first 300 reviews with character lengths between 150 and 160, and DDXPlus contains just 100 samples. The paper asserts that these datasets are out-of-distribution without checking against the models' training corpora; indeed, Section 7.2 suggests that Flipkart 'may not be as out-of-distribution for Mixtral as they are for LLaMA models.' Because the OOD axis is one of the two variables in the correlation analysis, this undermines the validity of the correlation findings.","section":"Section 3.2 / Tables 3-4"},{"comment":"The paper's own 'Higher Level Observations and Shortcomings' paragraph concedes that the negative correlation for LLaMA2-13b is driven by a single benchmark (PromptRobust AHP) and that 'further experimentation is needed.' Despite this explicit admission of instability, the abstract and conclusion present the neutral/negative/positive correlations as established results. Given the load-bearing nature of these correlations, the conclusions should be substantially weakened or the analysis extended with additional benchmarks and proper statistical inference before such claims can be made.","section":"Section 7.3 / Section 8"}],"minor_comments":[{"comment":"The prompt table reference is left as 'Table?? in the appendix', which should be resolved to the actual table number.","section":"Section 6.1"},{"comment":"There is a typo: 'LLaMa2:13b outperformed it's smaller counterpart' should use 'its' instead of 'it's'.","section":"Section 7.1"},{"comment":"The distinction between AHP and AHP2 is only introduced in the caption of Table 3; it should be clearly defined in the main text before the results are presented.","section":"Section 7.2"},{"comment":"The notation for the error term ε is defined but never used later, which is harmless but could be removed for clarity.","section":"Section 4.1.3"},{"comment":"The term 'neutral correlation' is never defined. A threshold or a hypothesis test should specify what slope magnitude counts as neutral versus positive or negative, especially since no significance intervals are provided.","section":"Section 8"},{"comment":"The exact number of points used in each regression is not reported. The reader cannot infer whether the regressions are based on 8, 12, or 16 points, which is essential for assessing the reliability of the slope estimates.","section":"Section 8"}],"recommendation":"reject","confidential_remarks":"The paper is transparent about its limitations and provides helpful experimental details, which is commendable. However, the central statistical analysis is not sufficiently rigorous for a journal publication: the correlation coefficients are based on too few non-independent points and lack any uncertainty quantification. The authors explicitly note that a single benchmark flips the sign for one model, so the conclusions are not robust. The study may be more suitable for a workshop or as a pilot report, but it does not meet the bar for a serious journal at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the question it asks is legitimate: do robustness interventions that help against adversarial inputs also help out-of-distribution inputs, and does that relationship vary by model? That is a real gap, and the paper deserves credit for trying to look at it across model families and scales. Second, the central claim — neutral correlation for LLaMA2-7b, negative for 13b, positive for Mixtral — is not backed by the evidence in the paper. The correlation is a regression slope over min-max normalized accuracy scores from three or four strategies per benchmark. There are no error bars, no bootstrap, no significance test, and the points are not independent observations of a population; they are outcomes of specific interventions. The authors even admit that the negative LLaMA2-13b result depends heavily on one strategy’s poor performance on one benchmark. That is not a stable estimate.\n\nWhat the paper does well is mostly in the details. The result tables are transparent about per-attack and per-task performance, which is more than many papers do. The discussion section is appropriately hedged, acknowledging the limited number of benchmarks and the influence of single points. The authors also flag their own sampling shortcuts, like taking the first 300 Flipkart reviews and the DDXPlus size. That honesty is worth acknowledging.\n\nThe soft spots are structural, not cosmetic. The correlation analysis is the core of the paper, and it cannot support the abstract’s plain statements of neutral, negative, and positive correlations. With only two adversarial and two OOD benchmarks, and only baseline plus two or three strategies, the regression is fitting a line through a handful of points that are themselves products of the same methods. Changing the normalization or removing one strategy flips the sign. There are also smaller issues: the appendix prompt table is referenced as Table??, the naming is inconsistent (LLaMA vs Llama, AdvGlue vs AdvGLUE++), and there is no code or data to reproduce the numbers.\n\nThis reads like a solid course project rather than a research paper. It identifies a real problem and runs some experiments, but it does not yet deliver a result that would survive scrutiny. The right call is to not send this to peer review in its current form. If the authors extend the study with more benchmarks, more independent strategies, proper uncertainty quantification, and full release of prompts and code, it could become a worthwhile contribution. For now, the abstract overstates what the evidence supports.","headline":"Asks a good question about whether adversarial and OOD robustness transfer across LLMs, but the headline correlations are built on a handful of normalized points and fall apart under the paper's own admission.","tokens_in":13703,"tokens_out":1628,"would_cite":false,"duration_ms":17516,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims the correlation between adversarial and out-of-distribution robustness in large language models is model-specific: neutral for LLaMA2-7b, negative for LLaMA2-13b, and positive for Mixtral-8x7b.","keywords":["adversarial robustness","out-of-distribution robustness","large language models","correlation analysis","linear regression","in-context rewriting","AHP","natural language inference"],"falsifier":"Recompute the regression for LLaMA2-13b without the single strategy-benchmark pair that drives the negative slope, AHP on PromptRobust, where accuracy drops from .774 to .583 while OOD accuracies rise, and see whether the correlation stays negative. More generally, add one more robustness strategy or one more benchmark per type to each model's point set: if any model's slope changes sign, the claimed neutral/negative/positive trichotomy is an artifact of the small point set rather than a property of the model.","tokens_in":12671,"feed_emoji":"🛡️","tokens_out":8078,"duration_ms":60742,"temperature":0.7,"pith_summary":"This paper asks whether a large language model that resists adversarial attacks, inputs deliberately crafted to fool it, is also robust to out-of-distribution text, inputs unlike what it was trained on. The authors take two methods, each designed for one of these robustness types, apply both to both kinds of benchmarks across three LLMs, and measure performance with accuracy, precision, recall, and F1 scores. Their central claim is that the correlation between the two robustness types is not universal but model-specific: neutral for LLaMA2-7b, negative for LLaMA2-13b, and positive for Mixtral-8x7b. If this is right, robustness improvements do not transfer between the two settings, and defenses and evaluations must be chosen per model and per domain rather than assumed to generalize.","feed_headline":"LLM robustness link varies by model, not universal","feed_subtitle":"Three models give three answers: neutral, negative, positive. Defenses must be chosen per model.","key_machinery":"The machinery is a per-model regression: for each model, the accuracy of three or four strategies (baseline, AHP, AHP2 where applicable, and ICR) on adversarial benchmarks is paired with the same strategies' accuracy on OOD benchmarks, min-max normalized per benchmark, and a line $Y = \\beta_0 + \\beta_1 X + \\epsilon$ is fitted, with the slope $\\beta_1$ read as the correlation coefficient between adversarial and OOD robustness. The Analytic Hierarchy Process, a prompt-driven framework that decomposes inference into safety-and-validity assessment, input refinement, and verified response synthesis, using one LLM to clean or rewrite inputs before the model answers, supplies the adversarial-oriented strategy; In-Context Rewriting, which prompts a model to rewrite OOD or perturbed inputs to look like in-distribution examples before the downstream task, supplies the OOD-oriented strategy. The regression turns the small set of strategy-benchmark accuracy points into a single signed claim, neutral, negative, or positive, for each model.","core_discovery":"The paper's central discovery is that adversarial robustness and out-of-distribution robustness do not move together in any fixed way across language models. Using the slope of a regression line fitted through per-strategy, min-max-normalized accuracies on adversarial versus OOD benchmarks, the authors find a neutral correlation for LLaMA2-7b, a negative correlation for LLaMA2-13b, and a positive correlation for Mixtral-8x7b, and they attribute the differences to parameter scale and architecture. At the same time, no improvement strategy, whether the adversarial-oriented Analytic Hierarchy Process or the OOD-oriented In-Context Rewriting, improved both robustness types consistently on all benchmarks for any model, and the authors state plainly that the correlation signs are fragile, with a single benchmark's outcome able to flip the slope. The conclusion they defend is that the two robustness types are largely decoupled and interact differently depending on the model, so robustness work must be evaluated and designed at the level of a specific model and domain.","pith_inferences":["With only three or four data points per model, the regression slope measures which strategy happens to win on each benchmark family more than any intrinsic property of the model; adding strategies or benchmarks could plausibly change the sign for any of the three models.","A clean way to separate scale from architecture would be to run the same protocol on LLaMA2-70b and on a dense model of roughly Mixtral's parameter count; the paper stops at Mixtral versus the LLaMA2 pair, which confounds the two factors.","The AHP collapses on LLaMA models may be partly a prompt-formatting artifact, since the paper notes JSON-format overloading and prompt sensitivity, in which case the reported correlations inherit that artifact rather than reflecting a true robustness trade-off.","The finding connects to an open question in computer vision about whether adversarial and OOD robustness are in tension; a multilingual or multimodal extension of this protocol could test whether the model-specific trichotomy survives across modalities."],"forward_implications":["No strategy in the study improves both adversarial and OOD robustness across all models and benchmarks, so hybrid robustness frameworks that combine both kinds of defenses would be needed rather than a single universal method.","Scaling the LLaMA2 family from 7b to 13b flips the correlation from neutral to negative, so parameter count alone does not predict how the two robustness types relate.","Mixtral-8x7b, a sparse mixture-of-experts architecture, shows a positive correlation, indicating that some architectures may balance both robustness types at once.","Robustness conclusions are provisional: the paper shows that the negative correlation for LLaMA2-13b can be driven by one strategy failing on one benchmark, so evaluations should report per-benchmark, per-model results rather than pooled numbers."],"supporting_citations":[{"why":"Supplies the Analytic Hierarchy Process framework whose adversarial performance across models generates the main correlation data points.","marker":"(Liu et al., 2023)"},{"why":"Supplies In-Context Rewriting, the OOD-oriented strategy applied to both benchmark families.","marker":"(O'Brien et al., 2024)"},{"why":"Establishes the zero-shot OOD evaluation setup and the choice of Flipkart and DDXPlus as OOD benchmarks.","marker":"(Wang et al., 2023)"},{"why":"Provides the PromptRobust adversarial benchmark with its attack types.","marker":"(Zhu et al., 2024)"},{"why":"Provides AdvGLUE, the benchmark that AdvGLUE++ expands for adversarial evaluation.","marker":"(Wang et al., 2021)"},{"why":"Supplies the AdvGLUE++ enhanced adversarial benchmark used in the experiments.","marker":"(Wang et al., 2024)"},{"why":"Defines the LLaMA-2 models whose scale comparison drives the neutral-to-negative finding.","marker":"(Touvron et al., 2023)"},{"why":"Defines the Mixtral mixture-of-experts architecture behind the positive-correlation finding.","marker":"(Jiang et al., 2023)"},{"why":"Provides the DDXPlus medical-diagnosis OOD dataset.","marker":"(Tchango et al., 2022)"},{"why":"Provides the Flipkart product-review OOD dataset.","marker":"(Vaghani & Thummar, 2023)"}],"fun_headline_variants":["LLM robustness: model-specific, no universal link","Adversarial vs OOD robustness: it depends on the model","Three LLMs, three robustness correlations: neutral, negative, positive","Robustness transfer in LLMs varies by scale and architecture"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats the slope of a line fitted through min-max-normalized accuracies from only three or four strategies per benchmark as a stable measure of the true correlation between adversarial and OOD robustness, a limitation the authors themselves state in section 8, where they note that a single benchmark's outcome can flip the sign.","fun_headline_variants_meta":{"raw":{"variants":["LLM robustness: model-specific, no universal link","Adversarial vs OOD robustness: it depends on the model","Three LLMs, three robustness correlations: neutral, negative, positive","Robustness transfer in LLMs varies by scale and architecture"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1350,"prompt_tokens":986,"completion_tokens":364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":293}},"tokens_in":602,"tokens_out":364,"duration_ms":3681,"temperature":1.0,"reasoning_tokens":293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:52:02.521922+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the regression for LLaMA2-13b without the single strategy-benchmark pair that drives the negative slope, AHP on PromptRobust, where accuracy drops from .774 to .583 while OOD accuracies rise, and see whether the correlation stays negative. More generally, add one more robustness strategy or one more benchmark per type to each model's point set: if any model's slope changes sign, the claimed neutral/negative/positive trichotomy is an artifact of the small point set rather than a property of the model.","supporting_citations":[{"cited_title":"Translation of Gulmanelli's seminar notes \"On a theory of isotopic spin\" (1954)","cited_arxiv_id":"2311.08261","evidence_quote":"Supplies the Analytic Hierarchy Process framework whose adversarial performance across models generates the main correlation data points."},{"cited_title":"Improving Black-box Robustness with In-Context Rewriting","cited_arxiv_id":"2402.08225","evidence_quote":"Supplies In-Context Rewriting, the OOD-oriented strategy applied to both benchmark families."},{"cited_title":"and Thummar, M","cited_arxiv_id":null,"evidence_quote":"Provides the Flipkart product-review OOD dataset."}],"review_version":1}