Pith. sign in

REVIEW 3 major objections 6 minor 12 references

CoCoP: Enhancing Text Classification with LLM through Code Completion Prompt

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Recasting text classification as a code completion task—examples as variable assignments, the query as an unfinished function call—improves accuracy over standard few-shot prompting and lets small code models rival much larger general ones.

desk verdict Solid prompt-engineering result with a fixable flaw: the few-shot baseline is unspecified, so the size of the CoCoP advantage is uncertain. read the letter →

arxiv 2411.08979 v1 pith:F3N67CJA submitted 2024-11-13 cs.CL cs.AI

Mohammad Mahdi Mohajeri , Mohammad Javad Dousti , Majid Nili Ahmadabadi This is my paper · ORCID
classification cs.CLcs.AI
keywords textclassificationcodecompletionpromptCoPfew-shotlearninglargelanguagemodelsin-contextLLaMAengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoCoP turns text classification into a code completion exercise: each training example becomes a variable assignment plus a function call carrying its label, and the query becomes the same function call with the label missing. The frozen LLM finishes the call, and a label extractor reads the completed argument. The paper reports that this format beats a traditional few-shot prompt on SST2, CoLA, MRPC, and SNLI, with especially large gains for code-specialized models such as CodeLLaMA. As a result, a 7B or 13B code model with CoCoP can match or exceed the few-shot accuracy of a 70B general model on several of these benchmarks, at about one-tenth the size.

What carries the argument

The central object is the Incomplete-Code Generator, which converts demonstrations and the query into a snippet of code: examples become string-variable assignments followed by calls to a user-chosen function such as apply_sentence_sentiment(sentence, LABEL), and the query becomes a call with the label argument omitted. The LLM completes the call, and the Label Extractor pulls the label from that position. The function name is a user-chosen hyperparameter meant to signal the task, and optional additions such as a list of all possible labels can be embedded in the code.

What would settle it

Take the same few-shot examples and label wording and run the classification twice—once wrapped in CoCoP's incomplete-code structure and once in a plain natural-language prompt with explicit instructions of matched length and clarity—using the same models, seeds, and datasets; if accuracy is no higher in the code condition, the paper's causal claim that the code-completion format drives the improvement is refuted.

Watch

Extended reading notes

Core claim

The central claim is that the format of the prompt, not just the model's general language ability, is the lever: recasting a classification problem as incomplete code activates an LLM's code-completion competence. In CoCoP, each demonstration is converted into a string-variable assignment followed by a call to a user-named function such as apply_sentence_sentiment(sentence, LABEL), and the query becomes the same call with the label argument omitted. The model completes the call, and the Label Extractor pulls the label from that position. On SST2, CoCoP raises CodeLLaMA-7B-Instruct accuracy to 92.2, compared with 67.7 for LLaMA2-7B-chat under few-shot prompting and 82.9 for LLaMA2-70B-chat under few-shot prompting. Across the four datasets, CoCoP with CodeLLaMA-34B-Instruct outperforms LLaMA2-70B-chat few-shot on SST2, CoLA, and MRPC and is within about one point on SNLI.

Load-bearing premise

The CoCoP advantage is measured against a traditional few-shot baseline whose prompt template is not specified; if that baseline is weaker than a well-designed natural-language prompt, the gap attributed to the code format could shrink or disappear.

Editorial extensions

If this is right

  • Changing the prompt format alone, with no fine-tuning and no extra training data, yields large accuracy gains on SST2, CoLA, MRPC, and SNLI.
  • Code-specialized models benefit more from CoCoP than general models, and smaller code models can match or exceed much larger general models under few-shot prompting.
  • The method extends beyond binary sentiment to multi-class tasks such as SNLI.
  • CoCoP shows lower standard deviations across random example selections than few-shot prompting, suggesting greater stability to example variation.
  • Adding a list of possible labels to the code prompt slightly helps most datasets, while adding the task name as a comment hurts performance, so prompt content beyond format still matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the mechanism would hold examples and label wording fixed and vary only the code wrapping, isolating the format effect the paper attributes to code completion.
  • The same incomplete-code trick could be tried on other structured outputs drawn from a fixed set, such as named entity spans, relation triples, or structured medical codes.
  • Because the function name is a hyperparameter, systematically searching function names may yield further gains or reveal which task verbs best activate code-completion behavior.
  • If the mechanism is code-completion competence, models pretrained on more code should gain more from CoCoP; this prediction is testable across other code-model families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CoCoP (Code Completion Prompt), a prompting method that recasts text classification as a code-completion task. Given a handful of labeled examples, an Incomplete-Code Generator renders each example as a variable assignment plus a function call (e.g., apply_sentence_sentiment(sentence, LABEL)); the query appears as an incomplete function call whose label argument is missing. A frozen LLM completes the call, and a Label Extractor reads off the label. The authors evaluate CoCoP against a "traditional few-shot learning" baseline on SST2, CoLA, MRPC, and SNLI, using LLaMA2-chat (7B, 13B, 70B) and CodeLLaMA-Instruct (7B, 13B, 34B), with ten runs and reported standard deviations. They report that CoCoP often improves accuracy over few-shot prompting, that CodeLLaMA models benefit more than LLaMA2, and that small code models with CoCoP can match or exceed LLaMA2-70B few-shot accuracy. The paper also includes ablations on the order and type of demonstrations, plus the effect of adding task names or label lists to the prompt.

Significance. If the central comparison is reliable, the result is practically significant: it suggests that a 7B code-specialized model can outperform a 70B general model on text classification through prompt design alone, which would be useful for cost-constrained deployment. The paper's strengths include repeated runs with standard deviations, four datasets covering binary and multi-class settings, and a simple, clearly described method that does not require fine-tuning. There is no fitted-parameter circularity: the paper reports accuracy on held-out test sets, and the only manually chosen parameter is the function name. However, the significance is currently conditional on the unreported and apparently fragile few-shot baseline, and on the availability of implementation details that are not yet provided. The central empirical claim is plausible and worth verifying, but the manuscript in its present form does not fully support it.

major comments (3)
  1. [Section 3.1, Table 1] The baseline "traditional few-shot learning" prompt is never specified. Section 3.1 states only that few-shot examples are drawn from training data and that two examples per class are used, but it does not give the prompt template, the label verbalization, the response format, or the label-extraction procedure. This matters because the paper's causal claim is that the code format itself drives the improvement. The baseline numbers in Table 1 are also non-monotonic in model size for CodeLLaMA: SST2 accuracy is 79.7, 49.7, and 23.0 for the 7B, 13B, and 34B models, and SNLI is 58.6, 50.6, and 60.9. Large degradation of this kind is implausible for a well-formed natural-language prompt and suggests that the baseline template or extraction is mismatched to CodeLLaMA. Please provide the exact baseline prompt template and extraction procedure, and add a control condition in which the same demonstrations, same label words, and same extraction are used in a natural-language format, so that the code format is the only varying factor.
  2. [Section 3.2.1, Table 1] The broad claim that "CoCoP can improve performance in classification tasks" is not supported uniformly by the data. For LLaMA2-7B-chat, CoCoP is worse than few-shot on SST2 (58.3 vs 67.7) and on CoLA (72.7 vs 75.0). The paper acknowledges this reversal in passing but still summarizes the result as a general improvement and states that the impact grows with model size. Please report the reversal explicitly, discuss its cause, and qualify the conclusion. In addition, no statistical significance tests are reported; with ten runs, several differences that are discussed in the text fall within one standard deviation of each other, so claims of "better," "comparable," and "robust" need paired significance tests or confidence intervals on the key comparisons (e.g., CoCoP vs few-shot on the same model, and CodeLLaMA-7B with CoCoP vs LLaMA2-70B few-shot).
  3. [Section 3, Abstract] Reproducibility is incomplete. The abstract promises that source code will be made available, but no code, hyperparameters, inference settings (e.g., maximum new tokens, decoding details beyond temperature 0), or label-extraction failure rates are included. The exact few-shot baseline template is absent, as noted above, and no seeds are reported for the example sampling. These omissions prevent a reviewer from reproducing the central comparison. Please release the code and a complete configuration file, and document the baseline prompt and parsing procedure so that the experiments can be rerun exactly.
minor comments (6)
  1. [Section 3.2.2] There is a typo: "SLI" should be "SNLI" in the sentence beginning "LLaMA2 with few-shot learning showed slightly better performance...".
  2. [Figures 3 and 4] Figures 3 and 4 do not show error bars or confidence intervals, even though Table 1 reports standard deviations from ten runs. Adding variance information would make the ablation conclusions, especially about the F1-score of the negative class, more convincing.
  3. [Section 4.1] The citation for GPT-2 is incorrect: Brown et al. (2020) is the GPT-3 paper, not GPT-2. Please cite the appropriate GPT-2 reference.
  4. [References] Raffel et al. 2020a and 2020b appear to be the same T5 paper, cited twice with different entries. Please consolidate them.
  5. [Section 3.2.4 and Abstract] The model-size comparison is described inconsistently: the abstract says "one-tenth of the model size," while Section 3.2.4 says "twice-as-small" for CodeLLaMA-34B versus LLaMA2-70B, and CodeLLaMA-13B is roughly one-fifth of 70B. Please state the size ratios for each comparison explicitly.
  6. [Section 2, Figure 1] The function name is described as a user-provided hyperparameter, but the paper never reports how the function name was chosen or whether performance is sensitive to it. Please state the chosen function names for all datasets and, ideally, include a small sensitivity analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: CoCoP is evaluated empirically on held-out test sets, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper derives nothing from its target result: CoCoP is a fixed prompt construction, and its accuracy is measured on held-out test sets for SST2, CoLA, MRPC, and SNLI across several frozen LLMs. The label returned by the method is the model's code-completion output, not a value fitted to the test data. No parameter is estimated from the target labels and then reported as a prediction, so the fitted-input-called-prediction pattern does not apply. The method is introduced by the authors rather than imported through a self-citation: the cited works (CodeLLaMA, LLaMA2, GLUE, SNLI, and prior code-format prompting papers) are external and do not themselves establish the CoCoP result. The comparison baseline 'traditional few-shot learning' is not specified in detail, and the few-shot results for CodeLLaMA are non-monotonic across model sizes, but that is a methodological/experimental-control concern about the strength of the comparator, not circularity: the paper's central claim is an empirical accuracy comparison, and the claimed improvement is not guaranteed by the definition of CoCoP or by any equation that ties the output to the input. The ablation studies in Section 3.3 vary prompt structure and example types to probe mechanism, which further indicates the claimed effect is treated as an empirical finding rather than a definitional consequence. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or model entities are introduced. The method relies on the transferability of code-completion skill and a parseable output assumption, and uses one hand-chosen hyperparameter.

free parameters (1)
  • function name
    The function name (e.g., apply_sentence_sentiment) is a user-provided hyperparameter that guides the LLM; the paper does not test sensitivity to this choice.
assumptions (3)
  • domain assumption Code-specialized LLMs develop code completion abilities that transfer to text classification when the task is formatted as code.
    Core premise of CoCoP; Section 2 assumes the LLM will complete the function call with a label from the examples.
  • domain assumption LLaMA2-chat and CodeLLaMA-Instruct differ only in code training, so performance differences can be attributed to code specialization.
    Comparison in Section 3.2.2 relies on this.
  • domain assumption Frozen LLM output can be reliably parsed by the Label Extractor with no accuracy loss.
    Section 2 Label Extractor assumes a label appears at a particular position; unparseable outputs are not reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CoCoP: Enhancing Text Classification with LLM through Code Completion Prompt." pith.science (2026). https://pith.science/paper/F3N67CJA

@misc{pith2026241108979,
  author       = {Pith},
  title        = {Pith review of: CoCoP: Enhancing Text Classification with LLM through Code Completion Prompt},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3N67CJA}},
  note         = {Machine review of arXiv:2411.08979}
}
read the original abstract

Text classification is a fundamental task in natural language processing (NLP), and large language models (LLMs) have demonstrated their capability to perform this task across various domains. However, the performance of LLMs heavily depends on the quality of their input prompts. Recent studies have also shown that LLMs exhibit remarkable results in code-related tasks. To leverage the capabilities of LLMs in text classification, we propose the Code Completion Prompt (CoCoP) method, which transforms the text classification problem into a code completion task. CoCoP significantly improves text classification performance across diverse datasets by utilizing LLMs' code-completion capability. For instance, CoCoP enhances the accuracy of the SST2 dataset by more than 20%. Moreover, when CoCoP integrated with LLMs specifically designed for code-related tasks (code models), such as CodeLLaMA, this method demonstrates better or comparable performance to few-shot learning techniques while using only one-tenth of the model size. The source code of our proposed method will be available to the public upon the acceptance of the paper.

Figures

Figures reproduced from arXiv: 2411.08979 by the authors.

Figure 1
Figure 1. The figure illustrates the CoCoP method used for a sentiment analysis task. The process begins by [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The accuracy metric evaluated the comparison [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The impact of examples’ order in the CoCoP method on the MRPC and SST2 datasets. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Effect of example type variation in CoCoP models on the MRPC and SST2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [4]

    arXiv preprint arXiv:2307.10169

    Challenges and applications of large language models. arXiv preprint arXiv:2307.10169. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova

    arXiv

  2. [5]

    In NeurIPS 2023 F oundation Models for Decision Making Workshop

    Chain of Code: Reasoning with a language model-augmented code emulator. In NeurIPS 2023 F oundation Models for Decision Making Workshop. Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng

    work page 2023

  3. [6]

    In 2023 IEEE In- ternational Conference on Robotics and Automation (ICRA)

    Code as policies: Language model programs for embodied control. In 2023 IEEE In- ternational Conference on Robotics and Automation (ICRA). IEEE. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qing- wei Lin, and Daxin Jiang

    work page 2023

  4. [8]

    arXiv preprint arXiv:2308.12950

    Code Llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. CodeGemma Team

    arXiv

  5. [10]

    arXiv preprint arXiv:2312.11805

    Gemini: a family of 9 highly capable multimodal models. arXiv preprint arXiv:2312.11805. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. LLaMA: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971. Hugo ...

    arXiv

  6. [12]

    arXiv preprint arXiv:2104.14690

    Entailment as few-shot learner. arXiv preprint arXiv:2104.14690. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al

    arXiv

  7. [2015]

    In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda ...

    work page 2015

  8. [2018]

    In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

    GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP . Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma

    work page 2018

Show all 12 references
  1. [2021]

    arXiv preprint arXiv:2107.03374

    Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen

    arXiv

  2. [2022]

    Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu

    Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li...

    2022

  3. [2023]

    arXiv preprint arXiv:2303.08774

    GPT-4 technical re- port. arXiv preprint arXiv:2303.08774. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning

    arXiv

  4. [2024]

    arXiv preprint arXiv:2406.11409

    Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.