{"id":"6c1d003c-a12d-45f5-b79d-a52d2319b16e","arxiv_id":"2412.20595","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLMs show large out-of-domain drops in few-shot genre classification and generated-text detection, and detailed prompt instructions that forbid topical cues narrow the gap by up to 20 points.","lead":"This paper measures how well LLMs classify texts when demonstration examples come from a different topic than the test texts, finding clear accuracy drops. It then shows that prompting the model to focus on style and ignore topic can shrink that gap in genre classification and AI-text detection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7–20 point gains may be inflated by prompt selection: the detailed control prompt was tuned on the same benchmark and no held-out prompt development is reported.","rationale":"The reader's weakest assumption was measurement validity: small test sets, topic-model misassignment, and single-generator synthetic data. These are real reliability concerns, and the reader also noted in the rationale that the detailed prompt was tuned on the same setup. I agree that measurement noise matters, but the most load-bearing concern for the central claim is prompt selection bias. Even with perfect measurement, the reported effect size could be optimistic because the prompt was developed on the benchmark used for evaluation. The paper's own ablation studies and rephrasing checks do not rule this out: ablations only show that parts of the tuned prompt matter, and rephrasing only checks lexical variants of the same tuned prompt. The method's practical promise is that a user can write a style-focused prompt and get domain transfer gains; if the gains require benchmark-specific tuning, that promise is weakened. This does not make the paper's internal observations false, but it makes the headline effect size less trustworthy and should be tested with a proper held-out prompt development protocol. The reader's conditional verdict already accommodates this kind of concern, so I do not recommend changing the verdict.","tokens_in":15490,"tokens_out":6076,"duration_ms":61995,"concrete_test":"Split the 25 topics into 20 development topics and 5 held-out topics (stratified by genre pair). Use only the 20 development topics to select and tune the detailed-control prompt among several candidate phrasings, including the paper's original. Freeze the selected prompt and evaluate on the held-out 5 topics for all six models, computing the off-topic accuracy gain and OOD-gap reduction relative to the basic prompt. Repeat with 5-fold rotation of the held-out topics. If the average held-out gain falls substantially below the 7–20 point range in Table 1, or is not consistently positive across models, the reported improvement is partly an artifact of prompt selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the detailed-control prompt reduces the OOD gap by up to 20 points rests on the assumption that the prompt is not selected on the evaluation data. Section 2.6 states that the prompts 'were developed based on our preliminary experiments,' and the ablation (Table 2) and GPT-4o rephrasing runs are evaluated on the same 25-topic, 10-test-text benchmark used in Table 1. No topics, tasks, or test portions are held out during prompt construction. Because the detailed prompt contains many components (formality, structure, purpose, perspective, voice, citations, plus explicit topic prohibitions), there is ample room to tune wording on the benchmark until the desired gap reduction appears. The rephrasing experiments only vary the surface form of an already-tuned prompt, so they do not address selection bias. If the 7–20 point reductions depend on this tuning, the method may not transfer to a prompt chosen without access to the target test set. This is the most load-bearing threat because the paper's headline is an empirical effect size, and selection on the evaluation set directly biases that effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates out-of-domain (OOD) performance gaps in few-shot in-context learning (ICL) for two non-topical classification tasks: genre classification and generated-text detection. Using six LLMs from the GPT and Claude families, the authors first confirm that accuracy drops when demonstration examples are drawn from a different topic than the test texts. They then propose a 'Detailed Control' prompt that instructs the model to focus on stylistic and structural indicators and to ignore topical content. The main reported result is that this prompt reduces the OOD gap by up to 20 percentage points in the generated-text detection task and by up to 7 points in genre classification, with ablation and paraphrase-robustness checks supporting the contribution.","tokens_in":15705,"tokens_out":3932,"duration_ms":40524,"significance":"If the effect is reliable, the paper offers a practical, fine-tuning-free intervention for improving domain transfer in few-shot ICL for non-topical classification. The study covers a useful range of models and tasks, provides a replication repository, and includes ablation studies and prompt-rephrasing robustness checks, which are strengths. The main finding, however, rests on a single headline cell in Table 1 and on prompt development that is not protected against selection on the evaluation set. The paper would be a valuable empirical contribution after the statistical reliability and prompt-selection concerns are addressed.","major_comments":[{"comment":"The method's headline effect size is vulnerable to prompt-selection bias. The authors state in Section 2.6 that the prompts 'were developed based on our preliminary experiments,' and all evaluations, ablations, and rephrasing checks in Tables 1 and 2 use the same 25-topic, 10-test-text benchmark. Because the Detailed Control prompt contains many free components (formality, structure, purpose, perspective, voice, citations, explicit topic prohibitions), tuning it on this same evaluation set can inflate the reported 7–20 point gap reductions. The rephrasing experiments in Table 2 vary only the surface form of an already-tuned prompt and therefore do not address this selection. Please provide a held-out evaluation (e.g., prompts fixed before any test topics are examined, or a separate development set) or report the prompt development history and show results for a pre-registered prompt.","section":"Section 2.6, Table 1, Table 2"},{"comment":"The statistical evidence for the central claim is underreported. Each cell in Table 1 is based on 10 test texts per topic (250 classifications per condition), but the paper gives only point accuracies, with no confidence intervals, per-topic variance, or significance tests for the detailed-control versus baseline differences. The 'up to 20 percentage points' claim is a single cell (GPT-4o, generated-text detection), and its uncertainty is not quantified. Report bootstrap or per-topic confidence intervals and significance tests for the main comparisons, not only for the OOD gap itself.","section":"Section 2.5, Table 1"},{"comment":"The generated-text detection task has two design features that threaten the validity of the claimed effect. First, the synthetic texts were generated by Claude 3 Sonnet, and Claude 3 Sonnet is also one of the evaluated detectors, so generator–detector overlap may inflate detection accuracy for that model and potentially for models from the same family. Second, three of six genres are excluded because they were 'trivially distinguishable'; this changes the task distribution and limits generalization to the full genre set. Please discuss these threats and, if possible, test whether the gap reduction persists when the generator and detector are from different families and when the excluded genres are included with a more difficult generation protocol.","section":"Section 2.4, Table 1"}],"minor_comments":[{"comment":"The baselines are asymmetric: the genre-classification baseline uses a two-stage Chain-of-Thought prompt, while the generated-text-detection baseline is single-stage without CoT. The authors should justify this asymmetry or use parallel baselines for both tasks.","section":"Section 2.6.1"},{"comment":"The column headers in Table 2 are difficult to parse; the first row of numbers appears to lack clear alignment with the column descriptions. Please reformat the table or add a caption that explains each column (including the three rephrased runs for each prompt type).","section":"Table 2"},{"comment":"The abstract and introduction refer to 'GPT-4', while the experiments use 'GPT-4o'; Section 4.2 mentions 'GPT-4.5'. Please standardize the model naming throughout.","section":"Section 1 and Section 4.2"},{"comment":"The text says the approach 'consistently enhances domain transfer performance,' but in Table 1, Claude 3.5 Haiku's genre classification with simple control (69.6) is below its baseline (70.8), and GPT-3.5's generated-text detection with detailed control (66.0) is essentially unchanged from the baseline (64.8). Please qualify the consistency claim or add significance tests to show these differences are within noise.","section":"Section 3.3"},{"comment":"The y-axis starts at 59.2 rather than 0, which visually exaggerates the difference between baseline and detailed control. Please use a full-scale axis or add error bars.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim is plausible, but the headline 'up to 20 percentage points' is a single point estimate without uncertainty quantification and is vulnerable to prompt-selection bias. The generated-text detection task also has a generator/detector overlap concern. I would like to see the authors provide held-out prompt evaluation and confidence intervals before publication; the current evidence is not strong enough for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The qualitative finding is solid: across GPT-4 and Claude families, few-shot ICL accuracy drops when demonstrations are off-topic, and a detailed prompt that tells the model to rely on style and structure rather than topic narrows the gap in most model-task rows. The headline 'up to 20 points' is not a trustworthy effect size, because the detailed prompt was developed on the same test benchmark used to measure it.\n\nWhat is actually new: prior work had reported OOD gaps qualitatively in ChatGPT and quantitatively in BERT-sized models; this paper quantifies the gap systematically across six API models and two non-topical tasks, and tests an explicit prompt-control method. The ablations are informative: dropping the style-feature list costs most of the gain, and rephrasing the prompt leaves the trend intact. The authors also release their setup, which is reproducible evidence.\n\nThe soft spot is prompt selection. Section 2.6 states the prompts were 'developed based on our preliminary experiments,' and the evaluation uses the same 25-topic, 10-text-per-topic benchmark. No topics or test items are held out during prompt construction. Rephrasing a prompt after it is tuned is not a control for tuning. The 7-20 point gains are point estimates without confidence intervals, and the generated-text detection benchmark uses only three genres (three were excluded as trivially distinguishable) and relies on one synthesizer model, Claude 3 Sonnet. These are disclosed, but they limit how far the numbers generalize.\n\nThis does not sink the paper. The qualitative trend is consistent across models and tasks, the method is cheap and easy to try, and the writing is honest about limitations. Just read the headline as 'up to 20 points in this configuration,' not as a stable estimate.\n\nThe paper is for people working on practical few-shot ICL, domain transfer, and non-topical classification. It won't reshape the theory of OOD robustness, but it is a useful empirical datapoint.\n\nI would send it to peer review. A serious referee should ask for repeated sampling with confidence intervals, a held-out prompt-development protocol, and either an independent test set or full release of the exact test texts.","headline":"Useful, honestly presented demonstration that style-control prompts shrink OOD gaps in few-shot ICL; the headline 20-point gain is probably inflated by prompt tuning on the same test set.","tokens_in":16233,"tokens_out":2903,"would_cite":true,"duration_ms":28636,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that modern large language models still show an out-of-domain accuracy gap in few-shot in-context learning, and that explicitly instructing the model to ignore topic and use stylistic indicators reduces that gap by up to…","keywords":["out-of-domain gap","in-context learning","few-shot learning","genre classification","generated text detection","prompt control","domain transfer","large language models"],"falsifier":"An experiment that would settle it: rerun the 5-shot comparison with the same topics but 50 test texts per topic, several genre pairs per topic, and synthetic texts from several different generators (not only Claude 3 Sonnet); if the detailed-control advantage does not persist, the central claim is an artifact of the original small sample.","tokens_in":15270,"feed_emoji":"🤖","tokens_out":6811,"duration_ms":58244,"temperature":0.7,"pith_summary":"This paper argues that the out-of-domain (OOD) gap in text classification, known from smaller BERT-era models, still affects modern large language models used with in-context learning: when the five demonstration examples come from one topic (say, sport) and the test texts come from another (say, history), accuracy drops. It then shows a fix: give the model a detailed list of style- and structure-based indicators to use, tell it to ignore topical content and text length, and require that its criteria be applicable to any topic. On genre classification, this reduces the OOD gap by up to 7 percentage points; on generated-text detection, by up to 20 percentage points, and in two of four models the gap disappears. The practical stakes are that labeled examples from one domain can be reused in another without fine-tuning, saving annotation effort. The claims are made for two tasks, six model variants, and a 5-shot setting.","feed_headline":"Prompt control cuts LLM out-of-domain gaps by up to 20 points","feed_subtitle":"Guiding models to ignore topic and focus on style shrinks transfer losses in genre and AI-text detection.","key_machinery":"The mechanism that carries the argument is prompt control over which predictive indicators the model may use. The detailed-control prompt lists concrete stylistic and structural features to attend to, such as formality, tone, sentence length, purpose, person perspective, voice, citations, and personal experience, while explicitly prohibiting topical content and text length and insisting that criteria must apply to any topic. The test apparatus is a topic-model split of a large natural-genre corpus into on-topic and off-topic documents, which creates measurable OOD transfer scenarios. Chain-of-Thought prompting, defined here as a two-stage instruction asking the model to articulate criteria before classifying, serves as the baseline that this controlled prompting is compared against.","core_discovery":"The central claim is that modern LLMs reproduce the OOD gap observed in earlier PLMs, and that the gap can be controlled by prompt design rather than by model retraining. The authors test two non-topical tasks—genre classification and generated-text detection—using a corpus with topic labels from a topic model, and compare a basic chain-of-thought prompt with prompts that add simple or detailed control over which features the model may use. The detailed-control prompt, which names stylistic criteria (formality, tone, sentence structure, purpose, perspective, voice, citations, personal experience) and explicitly forbids topic and length, is the best configuration for the more capable models. It reduces the OOD gap by up to 7 percentage points in genre classification and up to 20 percentage points in generated-text detection, and eliminates the detection gap entirely for two of the four models tested. Ablations show the effect depends on the detailed instruction rather than on generic chain-of-thought: removing the style-feature list cuts the gain by more than half, while paraphrasing the prompt leaves the trend stable.","pith_inferences":["The paper's numeric gap estimates rest on only 10 test texts per topic and a single random genre pair per topic, so the 7-20 point figures should be read as approximate; a larger test sample would give a sharper estimate.","Because the AI-generated side of the detection data was written by a single generator (Claude 3 Sonnet), the 20-point gain could partly reflect that generator's stylistic fingerprint; testing other generators would show whether the control prompt transfers.","The paper does not test languages other than English; since genre conventions and topic cues differ across languages, the benefit of topical suppression could be stronger or weaker in translation.","The contrast between simple and detailed control suggests a testable hypothesis: models with weaker instruction-following may need more explicit examples of what 'topic' means before they can ignore it."],"forward_implications":["The method offers a recipe for cross-topic use of few-shot examples: a task-specific list of permitted indicators plus an explicit ban on topical content.","More capable models (GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet) gain more from detailed control than older or smaller models, suggesting prompt control and model scale interact.","The same control approach should transfer to other non-topical classification tasks, such as sentiment, authorship, or gender classification, because the methodology is task-independent.","The OOD gap remains non-zero in most conditions, so prompt control reduces rather than eliminates the cost of domain shift.","Ablated and rephrased prompts confirm that the specificity of the instructions, not the exact wording, is what drives the improvement."],"supporting_citations":[{"why":"Supplies the corpus, topic model, and OOD evaluation methodology that the paper reuses and extends to LLMs.","marker":"Roussinov and Sharoff (2023)"},{"why":"Establishes prior genre classification results with LLMs and the OOD gap, and provides the working definition of genre.","marker":"Kuzman et al. (2023)"},{"why":"Introduces chain-of-thought prompting, the baseline method the paper adapts and compares against.","marker":"Wei et al. (2022)"},{"why":"Introduced in-context learning and few-shot prompting, the setting in which the paper measures OOD transfer.","marker":"Brown et al. (2020)"},{"why":"Latent Dirichlet allocation is the topic model used to define on-topic versus off-topic domains.","marker":"Blei et al. (2003)"},{"why":"Provides the BERT model fine-tuned as a small-model baseline for the same 5-shot setup.","marker":"Devlin et al. (2018)"},{"why":"Documents OOD robustness problems in LLMs and motivates the need for domain-transfer remedies.","marker":"Yuan et al. (2024)"},{"why":"Supplies the McNemar test used to establish statistical significance of the OOD differences.","marker":"Dror et al. (2018)"}],"fun_headline_variants":["Style-focused prompts shrink LLM domain gaps by 20 points","Prompt tweak reduces LLM out-of-domain drop by up to 20 pts","Teaching LLMs to ignore topic cuts transfer loss 20 points","Genre and AI-text detection: prompt control beats chain-of-thought"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on a small benchmark: 10 test texts per topic, one randomly chosen genre pair per topic, and topic-model scores that decide which texts count as on-topic or off-topic; if those scores are noisy or the 10-text samples are not representative, the measured OOD gaps and the 7-20 point gains could be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Style-focused prompts shrink LLM domain gaps by 20 points","Prompt tweak reduces LLM out-of-domain drop by up to 20 pts","Teaching LLMs to ignore topic cuts transfer loss 20 points","Genre and AI-text detection: prompt control beats chain-of-thought"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000848,"raw_usage":{"total_tokens":3698,"prompt_tokens":962,"completion_tokens":2736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2660}},"tokens_in":578,"tokens_out":2736,"duration_ms":20053,"temperature":1.0,"reasoning_tokens":2660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:16:19.139523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that would settle it: rerun the 5-shot comparison with the same topics but 50 test texts per topic, several genre pairs per topic, and synthetic texts from several different generators (not only Claude 3 Sonnet); if the detailed-control advantage does not persist, the central claim is an artifact of the original small sample.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the corpus, topic model, and OOD evaluation methodology that the paper reuses and extends to LLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents OOD robustness problems in LLMs and motivates the need for domain-transfer remedies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the McNemar test used to establish statistical significance of the OOD differences."}],"review_version":1}