{"id":"d16c6a00-3b6c-4e22-93b7-b7e73f7fa4e5","arxiv_id":"2411.08979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CoCoP, which formats text classification as code completion, improves LLM accuracy over few-shot prompting and lets small code models approach large general models.","lead":"A new prompting method, CoCoP, turns text classification into a code completion task by formatting examples as variable assignments and function calls, asking the LLM to fill in the missing label argument. On SST2, CoLA, MRPC, and SNLI, it often beats standard few-shot prompting, and a 7B code model can rival a 70B general model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CoCoP advantage is measured against an unspecified and possibly malformed few-shot baseline; the code-format attribution and the 'small model beats 70B' claim depend on that comparison.","rationale":"The reader's weakest assumption is that the few-shot baseline is unspecified and may be weak; my review converges on the same point and adds evidence that the baseline may be actively malformed for code models. This concern is load-bearing because the strongest claim is comparative: CoCoP is said to improve over few-shot prompting, and CodeLLaMA-7B with CoCoP is said to match LLaMA2-70B with few-shot prompting. Without the baseline template and with no released code, the comparison cannot be independently reproduced, and the non-monotonic few-shot scores for CodeLLaMA make the baseline quality doubtful. I do not see an internal inconsistency in the CoCoP construction itself, and the method is described precisely enough to be tested, so the appropriate outcome remains CONDITIONAL rather than ACCEPT or REJECT. The concrete test above would settle whether the concern lands; until then, the conditional verdict is the fair one.","tokens_in":10728,"tokens_out":5007,"duration_ms":50515,"concrete_test":"Run a matched-pair experiment with the same demonstrations, label vocabulary, and example draws under three conditions: (a) the CoCoP template, (b) a plain natural-language template such as 'Sentence: ... Sentiment: ...', and (c) the authors' exact few-shot template once it is released, using CodeLLaMA-7B/13B/34B on SST2 and SNLI at temperature 0. If the plain template raises CodeLLaMA-34B few-shot well above 23.0 or lifts CodeLLaMA-7B few-shot toward CoCoP levels, the format-driven improvement and the small-model-vs-70B comparison are not established; if the plain baseline remains far below CoCoP, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 describes the CoCoP prompt but never specifies the 'traditional few-shot learning' prompt template used for the baseline in Table 1. The paper's causal claim is that the code-completion format itself improves classification, yet the comparison does not control for prompt quality, response format, or label-extraction procedure. This is not a purely hypothetical concern: the few-shot baseline for CodeLLaMA is non-monotonic in model size (SST2: 79.7, 49.7, 23.0 for 7B, 13B, 34B; SNLI: 58.6, 50.6, 60.9), which is implausible for a well-formed prompt and suggests the baseline template or extraction is mismatched to code-specialized models. If CodeLLaMA few-shot is a weak baseline, the headline result that CodeLLaMA-7B with CoCoP matches LLaMA2-70B few-shot becomes a comparison against an unfairly weak comparator rather than evidence for CoCoP. Additionally, Table 1 shows CoCoP is not uniformly better within LLaMA2-7B (SST2 58.3 vs 67.7; CoLA 72.7 vs 75.0), so the broad claim of improvement requires qualification or a matched baseline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoCoP (Code Completion Prompt), a prompting method that recasts text classification as a code-completion task. Given a handful of labeled examples, an Incomplete-Code Generator renders each example as a variable assignment plus a function call (e.g., apply_sentence_sentiment(sentence, LABEL)); the query appears as an incomplete function call whose label argument is missing. A frozen LLM completes the call, and a Label Extractor reads off the label. The authors evaluate CoCoP against a \"traditional few-shot learning\" baseline on SST2, CoLA, MRPC, and SNLI, using LLaMA2-chat (7B, 13B, 70B) and CodeLLaMA-Instruct (7B, 13B, 34B), with ten runs and reported standard deviations. They report that CoCoP often improves accuracy over few-shot prompting, that CodeLLaMA models benefit more than LLaMA2, and that small code models with CoCoP can match or exceed LLaMA2-70B few-shot accuracy. The paper also includes ablations on the order and type of demonstrations, plus the effect of adding task names or label lists to the prompt.","tokens_in":10954,"tokens_out":4417,"duration_ms":44484,"significance":"If the central comparison is reliable, the result is practically significant: it suggests that a 7B code-specialized model can outperform a 70B general model on text classification through prompt design alone, which would be useful for cost-constrained deployment. The paper's strengths include repeated runs with standard deviations, four datasets covering binary and multi-class settings, and a simple, clearly described method that does not require fine-tuning. There is no fitted-parameter circularity: the paper reports accuracy on held-out test sets, and the only manually chosen parameter is the function name. However, the significance is currently conditional on the unreported and apparently fragile few-shot baseline, and on the availability of implementation details that are not yet provided. The central empirical claim is plausible and worth verifying, but the manuscript in its present form does not fully support it.","major_comments":[{"comment":"The baseline \"traditional few-shot learning\" prompt is never specified. Section 3.1 states only that few-shot examples are drawn from training data and that two examples per class are used, but it does not give the prompt template, the label verbalization, the response format, or the label-extraction procedure. This matters because the paper's causal claim is that the code format itself drives the improvement. The baseline numbers in Table 1 are also non-monotonic in model size for CodeLLaMA: SST2 accuracy is 79.7, 49.7, and 23.0 for the 7B, 13B, and 34B models, and SNLI is 58.6, 50.6, and 60.9. Large degradation of this kind is implausible for a well-formed natural-language prompt and suggests that the baseline template or extraction is mismatched to CodeLLaMA. Please provide the exact baseline prompt template and extraction procedure, and add a control condition in which the same demonstrations, same label words, and same extraction are used in a natural-language format, so that the code format is the only varying factor.","section":"Section 3.1, Table 1"},{"comment":"The broad claim that \"CoCoP can improve performance in classification tasks\" is not supported uniformly by the data. For LLaMA2-7B-chat, CoCoP is worse than few-shot on SST2 (58.3 vs 67.7) and on CoLA (72.7 vs 75.0). The paper acknowledges this reversal in passing but still summarizes the result as a general improvement and states that the impact grows with model size. Please report the reversal explicitly, discuss its cause, and qualify the conclusion. In addition, no statistical significance tests are reported; with ten runs, several differences that are discussed in the text fall within one standard deviation of each other, so claims of \"better,\" \"comparable,\" and \"robust\" need paired significance tests or confidence intervals on the key comparisons (e.g., CoCoP vs few-shot on the same model, and CodeLLaMA-7B with CoCoP vs LLaMA2-70B few-shot).","section":"Section 3.2.1, Table 1"},{"comment":"Reproducibility is incomplete. The abstract promises that source code will be made available, but no code, hyperparameters, inference settings (e.g., maximum new tokens, decoding details beyond temperature 0), or label-extraction failure rates are included. The exact few-shot baseline template is absent, as noted above, and no seeds are reported for the example sampling. These omissions prevent a reviewer from reproducing the central comparison. Please release the code and a complete configuration file, and document the baseline prompt and parsing procedure so that the experiments can be rerun exactly.","section":"Section 3, Abstract"}],"minor_comments":[{"comment":"There is a typo: \"SLI\" should be \"SNLI\" in the sentence beginning \"LLaMA2 with few-shot learning showed slightly better performance...\".","section":"Section 3.2.2"},{"comment":"Figures 3 and 4 do not show error bars or confidence intervals, even though Table 1 reports standard deviations from ten runs. Adding variance information would make the ablation conclusions, especially about the F1-score of the negative class, more convincing.","section":"Figures 3 and 4"},{"comment":"The citation for GPT-2 is incorrect: Brown et al. (2020) is the GPT-3 paper, not GPT-2. Please cite the appropriate GPT-2 reference.","section":"Section 4.1"},{"comment":"Raffel et al. 2020a and 2020b appear to be the same T5 paper, cited twice with different entries. Please consolidate them.","section":"References"},{"comment":"The model-size comparison is described inconsistently: the abstract says \"one-tenth of the model size,\" while Section 3.2.4 says \"twice-as-small\" for CodeLLaMA-34B versus LLaMA2-70B, and CodeLLaMA-13B is roughly one-fifth of 70B. Please state the size ratios for each comparison explicitly.","section":"Section 3.2.4 and Abstract"},{"comment":"The function name is described as a user-provided hyperparameter, but the paper never reports how the function name was chosen or whether performance is sensitive to it. Please state the chosen function names for all datasets and, ideally, include a small sensitivity analysis.","section":"Section 2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical study whose central comparison depends on an unreported baseline prompt. The non-monotonic few-shot numbers for CodeLLaMA strengthen my concern that the baseline is not a fair comparator. I would like to see the baseline template, the code, and significance tests before this can be accepted. The paper fits the scope of the journal if the empirical claims are verified, but the current submission does not yet permit verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoCoP isn't a conceptual breakthrough—casting a task as code completion has been around since CaP and TidyBot—but applying it to text classification is new, and the empirical pattern is real: across SST2, CoLA, MRPC, and SNLI, the code-format prompt beats a standard few-shot prompt in most cells, especially with CodeLLaMA models. The authors ran ten seeds, report standard deviations, and include ablations on example order and example type. The finding that code-specialized models benefit more is the most useful part of the paper; it suggests a cheap way to squeeze classification accuracy out of small models.\n\nThe soft spot is the baseline. Section 3.1 describes the CoCoP prompt but never specifies the 'traditional few-shot learning' prompt used for comparison. Without that template, you can't attribute the gain to the code format rather than to prompt quality. The non-monotonic few-shot results for CodeLLaMA (SST2: 79.7 → 49.7 → 23.0 for 7B/13B/34B) reinforce the worry that the baseline is malformed for code models, which would inflate CoCoP's apparent advantage. Also, CoCoP isn't uniformly better: LLaMA2-7B few-shot beats CoCoP on SST2 (67.7 vs 58.3) and CoLA (75.0 vs 72.7), so the broad 'outperforms few-shot' claim needs qualification. There are no significance tests, and the code is promised only 'upon acceptance,' which doesn't help verification now.\n\nThese are fixable issues. The paper's central claim is plausible and the experiments are above average for a prompt-engineering paper. The authors just need to disclose the baseline prompt, ideally with a matched control (same label words, same extraction), and temper the claims where the data don't support them.\n\nThis paper is for people working on prompting tricks or trying to get more from small deployed models. It deserves a serious referee, but the review should demand the baseline specification and, if possible, a code release.","headline":"Solid prompt-engineering result with a fixable flaw: the few-shot baseline is unspecified, so the size of the CoCoP advantage is uncertain.","tokens_in":11466,"tokens_out":2625,"would_cite":false,"duration_ms":23999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recasting text classification as a code completion task—examples as variable assignments, the query as an unfinished function call—improves accuracy over standard few-shot prompting and lets small code models rival much larger general ones.","keywords":["text classification","code completion prompt","CoCoP","few-shot learning","large language models","in-context learning","CodeLLaMA","prompt engineering"],"falsifier":"Take the same few-shot examples and label wording and run the classification twice—once wrapped in CoCoP's incomplete-code structure and once in a plain natural-language prompt with explicit instructions of matched length and clarity—using the same models, seeds, and datasets; if accuracy is no higher in the code condition, the paper's causal claim that the code-completion format drives the improvement is refuted.","tokens_in":10541,"feed_emoji":"💻","tokens_out":7050,"duration_ms":57648,"temperature":0.7,"pith_summary":"CoCoP turns text classification into a code completion exercise: each training example becomes a variable assignment plus a function call carrying its label, and the query becomes the same function call with the label missing. The frozen LLM finishes the call, and a label extractor reads the completed argument. The paper reports that this format beats a traditional few-shot prompt on SST2, CoLA, MRPC, and SNLI, with especially large gains for code-specialized models such as CodeLLaMA. As a result, a 7B or 13B code model with CoCoP can match or exceed the few-shot accuracy of a 70B general model on several of these benchmarks, at about one-tenth the size.","feed_headline":"Code-form prompts let a 7B model beat a 70B LLM on sentiment","feed_subtitle":"Recasting classification as incomplete code lifts accuracy on four benchmarks and shrinks the model needed.","key_machinery":"The central object is the Incomplete-Code Generator, which converts demonstrations and the query into a snippet of code: examples become string-variable assignments followed by calls to a user-chosen function such as apply_sentence_sentiment(sentence, LABEL), and the query becomes a call with the label argument omitted. The LLM completes the call, and the Label Extractor pulls the label from that position. The function name is a user-chosen hyperparameter meant to signal the task, and optional additions such as a list of all possible labels can be embedded in the code.","core_discovery":"The central claim is that the format of the prompt, not just the model's general language ability, is the lever: recasting a classification problem as incomplete code activates an LLM's code-completion competence. In CoCoP, each demonstration is converted into a string-variable assignment followed by a call to a user-named function such as apply_sentence_sentiment(sentence, LABEL), and the query becomes the same call with the label argument omitted. The model completes the call, and the Label Extractor pulls the label from that position. On SST2, CoCoP raises CodeLLaMA-7B-Instruct accuracy to 92.2, compared with 67.7 for LLaMA2-7B-chat under few-shot prompting and 82.9 for LLaMA2-70B-chat under few-shot prompting. Across the four datasets, CoCoP with CodeLLaMA-34B-Instruct outperforms LLaMA2-70B-chat few-shot on SST2, CoLA, and MRPC and is within about one point on SNLI.","pith_inferences":["A direct test of the mechanism would hold examples and label wording fixed and vary only the code wrapping, isolating the format effect the paper attributes to code completion.","The same incomplete-code trick could be tried on other structured outputs drawn from a fixed set, such as named entity spans, relation triples, or structured medical codes.","Because the function name is a hyperparameter, systematically searching function names may yield further gains or reveal which task verbs best activate code-completion behavior.","If the mechanism is code-completion competence, models pretrained on more code should gain more from CoCoP; this prediction is testable across other code-model families."],"forward_implications":["Changing the prompt format alone, with no fine-tuning and no extra training data, yields large accuracy gains on SST2, CoLA, MRPC, and SNLI.","Code-specialized models benefit more from CoCoP than general models, and smaller code models can match or exceed much larger general models under few-shot prompting.","The method extends beyond binary sentiment to multi-class tasks such as SNLI.","CoCoP shows lower standard deviations across random example selections than few-shot prompting, suggesting greater stability to example variation.","Adding a list of possible labels to the code prompt slightly helps most datasets, while adding the task name as a comment hurts performance, so prompt content beyond format still matters."],"supporting_citations":[{"why":"Defines few-shot in-context learning, the baseline approach CoCoP is compared against.","marker":"Brown et al., 2020"},{"why":"Supplies the LLaMA2-chat models used as general-purpose baselines.","marker":"Touvron et al., 2023b"},{"why":"Supplies CodeLLaMA-Instruct, the code models whose completion ability CoCoP exploits.","marker":"Roziere et al., 2023"},{"why":"Provides SST2, CoLA, and MRPC, three of the four evaluation datasets.","marker":"Wang et al., 2018"},{"why":"Provides SNLI, the multi-class evaluation dataset.","marker":"Bowman et al., 2015"},{"why":"Motivates the ablation on example type, showing the role demonstrations play in in-context learning.","marker":"Min et al., 2022"}],"fun_headline_variants":["Turn classification into code, and 7B beats 70B","Code-completion prompts boost accuracy by 20%+","Recast text tasks as code to slash model size needed","Small code models outdo giant LLMs via code prompts","Coding prompts unlock small LLMs for text classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CoCoP advantage is measured against a traditional few-shot baseline whose prompt template is not specified; if that baseline is weaker than a well-designed natural-language prompt, the gap attributed to the code format could shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Turn classification into code, and 7B beats 70B","Code-completion prompts boost accuracy by 20%+","Recast text tasks as code to slash model size needed","Small code models outdo giant LLMs via code prompts","Coding prompts unlock small LLMs for text classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3114,"prompt_tokens":955,"completion_tokens":2159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2077}},"tokens_in":571,"tokens_out":2159,"duration_ms":15694,"temperature":1.0,"reasoning_tokens":2077,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:11:07.614756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same few-shot examples and label wording and run the classification twice—once wrapped in CoCoP's incomplete-code structure and once in a plain natural-language prompt with explicit instructions of matched length and clarity—using the same models, seeds, and datasets; if accuracy is no higher in the code condition, the paper's causal claim that the code-completion format drives the improvement is refuted.","supporting_citations":[],"review_version":1}