REVIEW 5 major objections 6 minor 16 references
Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification
T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Task-level principles distilled from a handful of examples by multiple LLM agents, consolidated into one rule set, beat zero-shot, CoT, and stepback prompting on text classification and match few-shot prompting at lower inference cost.
desk verdict A useful, clearly-written prompting recipe with consistent gains, but the headline numbers rest on a single unexamined demonstration draw and no variance reporting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the consolidated task-level principle set: a short natural-language 'standard operating procedure' that lists the cues, tones, and distinguishing features of each class. It is produced by a three-stage pipeline: multiple LLM agents independently generate candidate principles from n=4, 8, or 16 demonstrations (with or without labels); a finalizer agent consolidates all candidates into one clear, non-redundant set; and a classifier agent receives that set as the only task-specific context for every test instance. Because the principle set is built once per task, its generation cost is negligible compared with classifying a whole dataset, and inference on each example remains a single short prompt.
What would settle it
Run the full pipeline several times on the same dataset, each time drawing a fresh random sample of n demonstrations for principle generation, and compare the macro-F1 of the resulting principle sets. If the gains over vanilla prompting disappear or vary widely across draws, the reported improvement depends on favorable demonstration selection rather than on principle distillation itself.
Extended reading notes
Core claim
The central claim is that task-level principles generated once per task can serve as the sole task-specific context for LLM text classification, replacing per-instance demonstrations. With the consolidation-based multi-agent framework, macro-F1 gains over vanilla zero-shot prompting average 6.92% on FLAN-T5-XXL and 10.69% on FLAN-UL2 across five datasets, and the approach outperforms CoT and step-back prompting in those averages. On one private product-classification dataset, the automatically generated principle set beats the human-crafted guidelines for both models (16.21% vs. 14.89% on FLAN-T5-XXL; 19.37% vs. 13.26% on FLAN-UL2). The paper also shows that principle-based prompting is competitive with few-shot ICL up to eight demonstrations while keeping input token lengths at or below roughly the two-shot or four-shot level.
Load-bearing premise
The approach assumes that a few randomly sampled demonstrations are representative enough that the single principle set distilled from them improves classification across the entire test distribution; the paper does not test how the final principles change with different random draws of demonstrations.
Editorial extensions
If this is right
- A single reusable principle set can replace per-instance demonstrations, reducing input token length and inference cost while matching or beating few-shot ICL on most tested datasets.
- Gains are larger for the stronger classifier (FLAN-UL2) than for FLAN-T5-XXL, indicating that the approach pays off most when the classifier can actually exploit abstract instructions.
- Automatically generated principles can match or outperform human-crafted annotation guidelines on private product-classification tasks, suggesting a route to automated SOP generation.
- Under low-resource settings, principle-based prompting outperforms fine-tuned RoBERTa-large, narrowing the gap to supervised classification when labels are scarce.
- Because principle-based prompts are shorter than few-shot prompts, the method sidesteps the input-length limits that block few-shot ICL on long-document tasks.
Reading between the lines
- Because the paper does not vary the demonstration draws, testing stability across random samples would show whether the principle set is robust or whether the reported gains partly reflect favorable sampling; a bootstrap-style stability check would settle this.
- The consolidation step uses one specific commercial finalizer; whether a weaker finalizer preserves the gains is unknown, and swapping the finalizer while keeping the generators fixed would isolate its contribution.
- If principles are stable across demonstration draws, the resulting rule sets could serve as auditable artifacts of what the model learned, allowing humans to inspect or edit task instructions before deployment.
- The same generation-consolidation pipeline could be repurposed to produce retrieval documents for RAG; the paper names this as future work, so it remains an untested extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes PRINCIPLE-BASED PROMPTING, a multi-agent LLM strategy for zero-shot text classification. Candidate principles are generated by six LLM agents from randomly sampled demonstrations with or without labels (n=4, 8, 16), then consolidated or ranked by a finalizer into a single task-level principle set, which is appended to prompts for FLAN-T5-XXL and FLAN-UL2 classifiers. Experiments on Irony2018, Emotion20, Financial Phrasebank, and two private product-classification datasets report macro-F1 improvements over vanilla zero-shot prompting (1.55%–19.37%) and favorable comparisons against CoT, stepback, few-shot ICL, and human-crafted principles. Ablations examine the effects of label information, number of demonstrations, and single-agent versus multi-agent generation.
Significance. If the empirical claims hold, the paper offers a useful and cost-effective alternative to few-shot ICL for text classification: the amortized cost of principle generation is small, the input token length is shorter than most few-shot settings, and the method can match or exceed human-crafted principles on two private datasets. The learning protocol is legitimate, since principles are generated from training demonstrations and evaluated on held-out test data. However, the current evidence lacks statistical grounding and robustness checks, so the practical significance depends on whether the gains persist across random demonstration draws and finalizer choices.
major comments (5)
- [Methods, Principle Generation] The central claim depends on a single, unreported random draw of demonstrations for each dataset (n in {4,8,16}, labeled/unlabeled), and the paper never tests how robust the finalized principle set is to that draw. The Discussion itself acknowledges that "sub-optimal sampling of demonstrations might provide a biased perspective," but no experiment varies the draw or reports which demonstrations were used. Because the five-seed classification runs reuse one fixed principle set, the headline gains (1.55%–19.37%) could be specific to one fortunate sample. Please report repeated principle-generation runs over independent draws with standard deviations or a sensitivity analysis.
- [Results, Tables 1 and 2] Tables 1 and 2 report only macro-F1 deltas averaged over five inference seeds, with no standard deviations, confidence intervals, or significance tests. Several decisive comparisons are small (e.g., FLAN-T5-XXL on Irony2018: 0.45 consolidation versus 0.68 for n=8 few-shot in Table 2; PC1: 1.43 versus 1.49), so the claims that the method "outperforms" baselines or is "on par" with few-shot ICL are not statistically supported. Report per-cell variance, bootstrap confidence intervals, or pairwise significance tests across the five seeds.
- [Ablation Studies, Figure 4] The single-agent versus multi-agent ablation is confounded by generator model strength. The single-agent baseline uses the classifier models FLAN-T5-XXL and FLAN-UL2 to generate principles, whereas the multi-agent framework includes Claude 3.5 Sonnet and Llama-3-70B-Instruct as generator agents. The improvement attributed to the multi-agent framework in Figure 4 could therefore reflect the use of stronger principle generators rather than collaboration. Please hold the generator model fixed across single- and multi-agent conditions (e.g., a single Claude agent versus a multi-agent set that includes Claude) or otherwise separate model capability from the number of agents.
- [Methods, Principle Consolidation] The finalizer for the consolidation variant is selected post hoc: the paper states that Claude 3.5 Sonnet was chosen "based on the overall high quality of principles generated in the previous step." Since the same test sets were used both to make this design choice and to report final numbers, the comparison between consolidation and ranking/random may be optimistically biased. Please justify the finalizer choice on a validation split, or report results across several candidate finalizers to show robustness to this design choice.
- [Results, Table 1] The broad claim that principle-based prompting outperforms CoT and stepback prompting is not uniformly supported by Table 1. For example, on FLAN-T5-XXL the stepback baseline achieves 17.56 on PC2 while the best multi-agent principle variant achieves 16.21, and CoT achieves 17.25 there; on FLAN-UL2, stepback is 13.99 on PC2 versus 19.37 for consolidation, but many per-dataset differences are within a few points. Please qualify the headline comparison and provide significance tests per dataset/model rather than relying only on averages.
minor comments (6)
- [Methods] The text refers to "FLAN-T5-XX" when it should say "FLAN-T5-XXL" (typo).
- [Table 1] The column header "A VG" should be "AVG".
- [Appendix, Table 4] The consolidation prompt for Emotion20 asks the model to create a consolidated set for "identifying irony in statements," which appears to be a copy-paste error from the Irony2018 prompt.
- [Related Work / Discussion] CARP is mentioned as a strong baseline in the Discussion, but no CARP results are reported anywhere in the experiments; please either add the comparison or remove the mention.
- [Tables 1 and 2] The "principle + human" rows do not specify how human-crafted principles are incorporated into the prompt; please state whether they are appended in exactly the same way as the generated principles.
- [Figures 3 and 4] Figures 3 and 4 are referenced as ablation results in the main text but appear only in the appendix; please indicate this placement explicitly or move them into the main text with full captions.
Circularity Check
No significant circularity: the principle-generation pipeline is a legitimate train/eval protocol; the only self-citation is background and not load-bearing.
full rationale
The paper's derivation chain is: sample demonstrations from the training set, ask multiple LLM agents to produce candidate principles, consolidate them into a task-level principle, then append that principle to the prompt and classify held-out test instances. This is a standard learning/evaluation separation. The principles are not constructed from test labels, and the macro-F1 gains over vanilla prompting, CoT, stepback, few-shot ICL, and RoBERTa fine-tuning are measured on held-out test sets of public benchmarks and private datasets. There is no equation-level reduction in which an output variable is defined as its own input, and no fitted parameter is renamed as a prediction. The only self-citation is De Silva et al. (2023), which is invoked in the introduction as background motivation ('Previous studies show that adding complex class descriptions as additional inputs to a pre-trained transformer backbone via cross-encoder architecture can significantly boost classification performance') and is not used to justify the central claim, to select the method, or to forbid alternatives; thus it is not load-bearing. The skeptic's concern about sensitivity to the single random demonstration draw is a robustness/generalizability issue, not circularity: even if the reported gains were draw-specific, the method's derivation would not reduce to its inputs by construction. The paper itself acknowledges that 'sub-optimal sampling of demonstrations might provide a biased perspective' but does not quantify the effect; that omission affects the strength of the empirical claim, not the circularity of the derivation. No self-referential or imported uniqueness theorem is used to force the chosen approach, and no known result is merely renamed. Overall, the central claim has independent content and the paper is self-contained against external benchmarks, so the circularity score is low.
Assumptions & free parameters
free parameters (4)
- number of demonstrations for principle generation =
n in {4, 8, 16}
- principle generator agent set =
FLAN-T5-XXL, FLAN-UL2, Llama-3-70B-Instruct, Mistral 7B, Mixtral 8x7B, Claude 3.5 Sonnet
- finalizer agent =
Claude 3.5 Sonnet
- sampling hyperparameters =
temperature=0.2, top_p=0.9
assumptions (4)
- domain assumption Randomly sampled demonstrations are representative of the task distribution for principle generation.
- domain assumption The finalizer agent produces a consolidated principle set that is accurate, non-redundant, and comprehensive enough to guide classification.
- domain assumption Macro-F1 averaged over five seeds is a sufficient evaluation protocol for comparing prompting strategies.
- domain assumption The prompt templates in the appendix elicit the intended behavior from all models consistently.
Cite this review
Pith. "Pith review of Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification." pith.science (2026). https://pith.science/paper/WUBQTOVW
@misc{pith2026250207165,
author = {Pith},
title = {Pith review of: Don't Just Demo, Teach Me the Principles: A Principle-Based Multi-Agent Prompting Strategy for Text Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/WUBQTOVW}},
note = {Machine review of arXiv:2502.07165}
}
read the original abstract
We present PRINCIPLE-BASED PROMPTING, a simple but effective multi-agent prompting strategy for text classification. It first asks multiple LLM agents to independently generate candidate principles based on analysis of demonstration samples with or without labels, consolidates them into final principles via a finalizer agent, and then sends them to a classifier agent to perform downstream classification tasks. Extensive experiments on binary and multi-class classification datasets with different sizes of LLMs show that our approach not only achieves substantial performance gains (1.55% - 19.37%) over zero-shot prompting on macro-F1 score but also outperforms other strong baselines (CoT and stepback prompting). Principles generated by our approach help LLMs perform better on classification tasks than human crafted principles on two private datasets. Our multi-agent PRINCIPLE-BASED PROMPTING approach also shows on-par or better performance compared to demonstration-based few-shot prompting approaches, yet with substantially lower inference costs. Ablation studies show that label information and the multi-agent cooperative LLM framework play an important role in generating high-quality principles to facilitate downstream classification tasks.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
- [3]
-
[4]
Semantic matching for text classi- fication with complex class descriptions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 7654–7680. Du, Y .; Li, S.; Torralba, A.; Tenenbaum, J. B.; and Mor- datch, I
work page 2023
-
[5]
arXiv preprint arXiv:2305.14325
Improving factuality and reasoning in lan- guage models through multiagent debate. arXiv preprint arXiv:2305.14325. Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al
-
[6]
arXiv preprint arXiv:2310.06825
Mistral 7B. arXiv preprint arXiv:2310.06825. Kossen, J.; Rainforth, T.; and Gal, Y
-
[7]
Levy, I.; Bogin, B.; and Berant, J
In-context learn- ing in large language models learns label relationships but is not conventional learning.arXiv preprint arXiv:2307.12375. Levy, I.; Bogin, B.; and Berant, J
-
[9]
arXiv preprint arXiv:2305.02156
Zero-shot listwise document reranking with a large language model. arXiv preprint arXiv:2305.02156. Malo, P.; Sinha, A.; Korhonen, P.; Wallenius, J.; and Takala, P
-
[10]
Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837. Sailunaz, K.; and Alhajj, R
-
[11]
arXiv preprint arXiv:2212.00193
Distilling reasoning capabilities into smaller language models. arXiv preprint arXiv:2212.00193. Sun, W.; Yan, L.; Ma, X.; Ren, P.; Yin, D.; and Ren, Z. 2023a. Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542. Sun, X.; Li, X.; Li, J.; Wu, F.; Guo, S.; Zhang, T.; and Wang, G. 2023b. Text cl...
Show all 16 references
-
[12]
arXiv preprint arXiv:2205.05131
Ul2: Unifying language learning paradigms. arXiv preprint arXiv:2205.05131. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q.; Chi, E.; Narang, S.; Chowdhery, A.; and Zhou, D
-
[13]
arXiv preprint arXiv:2203.11171
Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V .; Zhou, D.; et al
-
[14]
arXiv preprint arXiv:2212.10375
Self-adaptive in-context learning: An information compression perspec- tive for in-context example selection and ordering. arXiv preprint arXiv:2212.10375. Xiong, K.; Ding, X.; Cao, Y .; Liu, T.; and Qin, B
-
[15]
arXiv preprint arXiv:2305.11595
Examining inter-consistency of large language models col- laboration: An in-depth analysis via debate. arXiv preprint arXiv:2305.11595. Zheng, H. S.; Mishra, S.; Chen, X.; Cheng, H.-T.; Chi, E. H.; Le, Q. V .; and Zhou, D
-
[16]
arXiv preprint arXiv:2310.06117
Take a step back: Evoking reasoning via abstraction in large language models. arXiv preprint arXiv:2310.06117. Appendix Figure 3: Effects of label information in sampled demonstrations on generating high-quality principles for downstream classi- fication Figure 4: Effects of s...
-
[2020]
arXiv preprint arXiv:2010.12421
TweetEval: Unified benchmark and com- parative evaluation for tweet classification. arXiv preprint arXiv:2010.12421. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al
2010 arXiv
-
[2022]
arXiv preprint arXiv:2212.06800
Diverse demon- strations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800. Ma, X.; Zhang, X.; Pradeep, R.; and Lin, J
-
[2023]
arXiv preprint arXiv:2308.07201
Chateval: Towards better llm- based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201. Chen, J.; Chen, L.; Zhu, C.; and Zhou, T
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.