{"id":"7a01eed4-1eb6-4cb4-b768-a360d3fc39c5","arxiv_id":"2412.06432","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On a public dataset of 769 report passages, automatic instruction refinement by GPT-4o mini outperformed dynamic few-shot example selection for emission goal detection, but adding few-shot examples to optimized prompts provided limited extra benefit.","lead":"Companies publish sustainability reports, and analysts must find passages where the company commits to cutting carbon emissions. The authors used GPT-4o mini to classify each passage as containing such a goal or not. They compared two ways to give the model expert knowledge. The first selects a few labeled example passages that are similar to the new passage and adds them to the prompt.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single optimization trajectory per condition leaves the headline ranking inside noise; no prompt-seed variance is reported.","rationale":"I agree with the reader's weakest assumption and would make it even more specific: the missing unit of replication is the optimization trajectory, not the final-prompt evaluation. The strongest illustrative result (simple prompt 68.6 to 88.2) is probably stable, but the abstract makes a general claim and the table contains smaller, order-sensitive gaps, especially in the expert-start and example-during-tuning rows. A second, related caveat is the label-budget asymmetry: automatic prompt design consumes all 562 labeled training passages for error diagnosis and candidate selection, whereas example selection uses 5 demonstrations. The paper should state this asymmetry explicitly; a subsample variant of my proposed test would show whether the advantage is really about prompt design or simply about using more labels. The Limitations section is candid about single-model and non-objective-data issues, and I would credit the paper for reporting the held-out report-level split and public dataset. The o4-mini vs gpt-4o-mini inconsistency in the Limitations section should be corrected for reproducibility. These concerns do not invalidate the experiments but do mean the headline should stay conditional until prompt-level replication and confidence intervals are supplied.","tokens_in":9303,"tokens_out":8859,"duration_ms":99086,"concrete_test":"Rerun the full Section 3.2 greedy optimization independently at least 10 times per condition, changing the random permutation of the 562 training passages and the API seed, for the four cells that decide the headline: simple/expert start x no-examples/static-examples during tuning. Evaluate each resulting prompt 7 times on the test set and report per-run test F1, mean, and 95% CI (or bootstrap over the 207 test passages). If the auto-prompt CI overlaps the corresponding example-selection CI, the superiority claim must be qualified; if the expert-start and example-addition rankings flip across seeds, the 'limited benefit' conclusion is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing problem is that each cell in Table 2 is produced by one greedy prompt-tuning run, while the reported '7 repetitions' only re-evaluate the final prompt. Section 3.2's optimization is stochastic in two ways that the test-time averaging does not capture: the training passages are visited in random order, and the LLM's instruction-modification step is non-deterministic (the paper itself notes OpenAI outputs are non-deterministic). The resulting accepted prompt is therefore a single draw from a distribution, and no seed-level variance is reported. The headline comparison includes cells where the decisive gap is small: expert-start zero-shot auto-prompt (F1 86.9) vs expert-start similar-example selection (F1 86.3), and simple-start static-tuning/static-testing (89.6) vs zero-shot/zero-shot (88.2). On a 207-passage test set, a handful of flips moves F1 by more than a point, and epsilon=0.01 is asserted to be about 2x the observed evaluation standard deviation without any trajectory-level error bar. With only point estimates and no significance tests, the abstract's unqualified 'automatic prompt optimization is superior' is not yet established; several finer ordering claims could reverse under replication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two ways of integrating expert-labeled passages into LLM prompts for emission goal detection in corporate reports: (1) dynamic few-shot example selection based on embedding similarity, and (2) automatic prompt optimization via a greedy ProTeGi-style refinement loop. Experiments on the public NetZeroFact-BIG dataset (769 passages, GPT-4o mini) report F1 scores under simple and expert starting instructions, with and without few-shot examples during tuning and testing. The headline findings are that automatic prompt optimization outperforms example selection and that adding few-shot examples to automatically optimized prompts gives limited or negative benefit. The appendix documents the starting prompts, static examples, the modification prompt, and qualitative instruction evolution.","tokens_in":9453,"tokens_out":5769,"duration_ms":57878,"significance":"The task is practically relevant for sustainability monitoring and carbon accounting, and the experimental design has clear strengths: a public dataset, report-level train/test splitting to prevent leakage, held-out test evaluation, averaging over seven model calls for the final prompts, and transparent appendices containing all prompts and examples. If the ranking of strategies is stable, the paper provides useful practitioner guidance for integrating expert labels into LLM pipelines. However, the headline claims are currently supported only by point estimates from single optimization trajectories, with no confidence intervals or significance tests, so the strength of the conclusions exceeds what the reported evidence establishes.","major_comments":[{"comment":"The central comparison in Table 2 is based on a single greedy prompt-tuning trajectory per condition. Section 3.2 describes a stochastic process: training passages are visited in random order and the LLM's instruction-modification step is non-deterministic, as Section 4 itself notes. The reported '7 repetitions' average only re-evaluations of the final accepted prompt; they do not measure the variability of the accepted prompt itself. No confidence intervals or significance tests are reported. This matters because the decisive gaps are small: for the expert start, zero-shot auto-prompting gives F1 86.9 versus Similar example selection at 86.3, and for the simple start, Static/Static (89.6) versus zero-shot/zero-shot (88.2) is a 1.4-point gap on a 207-passage test set, where a handful of label flips changes F1 by more than a point. The abstract's claim that 'automatic prompt optimization is the superior approach' is therefore not yet established at the reported precision. Please report distributions over multiple independent tuning runs and/or significance tests, and make the number of runs explicit.","section":"Section 4, Table 2; Section 3.2"},{"comment":"The acceptance margin epsilon=0.01 is justified as 'approximately two times the standard deviation observed in many of our experiments,' but no supporting data are shown, and the standard deviation of repeated evaluations of a fixed final prompt is not the same as the variability of the optimization trajectory. Since epsilon controls how many instruction modifications are accepted, the ranking in Table 2 may depend on this free parameter. Please provide a sensitivity analysis over epsilon or trajectory-level standard deviations based on repeated optimization runs.","section":"Section 3.2; Section 4, Setup and Technical Details"},{"comment":"The abstract and conclusion state that automatic prompt optimization is 'the superior approach' without qualification, but the experiments use a single model (GPT-4o mini), a single dataset, and a single prompt-tuning run per condition. The Limitations section appropriately acknowledges model-specificity and mentions initial LLaMA-3 evaluations, but no numbers or details for those evaluations are reported. Please qualify the claims to the evaluated model and dataset, or add multi-model evidence, so that the general wording matches the evidence.","section":"Abstract; Section 1; Limitations"}],"minor_comments":[{"comment":"The Limitations section says 'we only focus on OpenAI's o4-mini model,' but the experiments use GPT-4o mini (gpt-4o-mini-2024-07-18). This model name should be corrected for consistency.","section":"Limitations"},{"comment":"The selection rule is unclear: 'we select up to three examples per class ... to form five few-shot demonstrations.' With two classes this could yield up to six examples; please clarify how the total is constrained to five (e.g., class-balancing rule or maximum total).","section":"Section 3.1"},{"comment":"There are typos in the ProTeGi description: 'LLM-generated rasonings' should be 'LLM-generated reasonings,' and 'stears a beam search' should be 'steers a beam search.'","section":"Section 2.2"},{"comment":"Given the paper's own point about non-deterministic outputs, it would be helpful to report the standard deviations across the seven evaluation repetitions next to the F1 values, not only mention them in the text as justification for epsilon.","section":"Section 4, Table 2"},{"comment":"The dataset is referred to both as 'NetZero-Facts' and 'NetZeroFact-BIG'; please use one consistent name.","section":"Abstract; Section 1; Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical contribution for an applied NLP venue, and I do not see a circularity problem: the optimized prompt is selected using training-set F1 and evaluated on a held-out test set, and the self-citation to NetZeroFacts is appropriate given the dataset. The main gate is statistical presentation: the single-trajectory point estimates and absent variance measures make the headline ranking potentially fragile. This is fixable within the manuscript's scope by adding repeated optimization runs and significance tests or confidence intervals."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honestly-written empirical comparison of two ways to use expert labels without fine-tuning — semantic few-shot example selection vs. LLM self-refinement of the prompt — on emission goal detection in corporate reports. The task is real, the dataset is public, the split is at the report level, and the appendix shows the actual prompt evolution, which is nice. The big headline difference — simple zero-shot prompt at 68.6% F1 vs. auto-tuned at 88.2% — is large and credible. If you work on ESG monitoring or applied ICL, this is worth a look.\n\nWhat's actually new: nobody else has compared these two strategies on this task, and the finding that few-shot examples can hurt after auto-prompting is a useful counterintuitive data point. The authors are upfront about domain-specificity and the non-objective nature of the data.\n\nWhere it gets soft: the stress-test is on target. Each condition is a single greedy prompt-tuning run. The stochasticity of the LLM's instruction-modification step and the random traversal of training passages mean you get one draw from a distribution of optimized prompts. The 7 repetitions only re-evaluate the final prompt; they don't capture trajectory variance. No confidence intervals or significance tests. Some of the decisive gaps are small — e.g., 86.9 vs 86.3 for expert-start zero-shot vs. similar-example selection, and 89.6 vs 88.2 for static vs. zero-shot under simple start. On 207 test passages, a few label flips shift F1 by more than a point. The claim that epsilon=0.01 is \"approximately two times the standard deviation observed in many of our experiments\" conflates test-time evaluation noise with optimization trajectory noise. So the abstract's unqualified \"automatic prompt optimization is superior\" is too strong as stated; the simple-prompt condition supports it, but the finer ordering claims could flip on replication.\n\nThere's also a model-name inconsistency: the methods say gpt-4o-mini-2024-07-18, but the Limitations section says o4-mini. That's the kind of slip that makes a referee want to see the exact configuration and code.\n\nVerdict: this deserves a serious referee, not a desk reject, but the revision needs multiple prompt-tuning seeds, error bars, artifact sharing, and a fix to the model identity. It's a useful practitioner-focused empirical study, not a theoretical breakthrough. I'd bring it to reading group if your group cares about applied LLM workflows; otherwise it's a pass.","headline":"Useful empirical comparison of example selection vs. automatic prompt design for ESG text, but the headline ranking rests on single optimization runs with no error bars — treat as provisional.","tokens_in":10037,"tokens_out":2559,"would_cite":true,"duration_ms":24639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic prompt optimization outperforms few-shot example selection for detecting emission reduction goals in corporate reports.","keywords":["emission goal detection","in-context learning","automatic prompt optimization","few-shot example selection","large language models","climate reporting","NetZeroFact-BIG","sustainability NLP"],"falsifier":"Repeat the same greedy prompt-tuning procedure multiple times with different random orders of the training passages (or different API nondeterminism) and compute F1 distributions for the best prompt-tuning and best example-selection conditions; the central ranking is falsified if the distributions overlap to the point that the ordering can reverse.","tokens_in":9038,"feed_emoji":"🎯","tokens_out":5536,"duration_ms":49928,"temperature":0.7,"pith_summary":"This paper asks how to make the best use of analyst-labeled example passages when a large language model is asked to detect emission reduction goals in corporate reports. It compares two in-context learning strategies: dynamically retrieving a few semantically similar examples to include in the prompt, and letting the LLM iteratively rewrite its own instructions after inspecting its mistakes. On 769 real-world business-report passages (NetZeroFact-BIG), the automatic-prompt approach is the clear winner: a simple starting prompt improves from 68.6% to 88.2% F1, and an already expert-tuned prompt from 81.4% to 86.9%. Adding few-shot examples to an already optimized prompt gives little benefit and, for some selection strategies, hurts. The paper's practical claim is that, at least for this kind of classification task, prompt self-refinement is a stronger way to inject expert knowledge than example selection.","feed_headline":"Automatic prompt tuning beats few-shot examples for climate-goal detection","feed_subtitle":"On 769 corporate passages, a greedy prompt-refinement loop lifts F1 from 68.6% to 88.2%.","key_machinery":"The central mechanism is the automatic prompt-design loop: the LLM is given a training passage, makes a prediction, and when that prediction is wrong the LLM is asked to analyze the root cause of the error and propose a revised instruction that fixes the mistake without reversing correct labels; the revised instruction is accepted only if its F1 on the training set exceeds the current one by at least a margin of 0.01, and the search proceeds greedily without beam search. This is a simplified version of ProTeGi's gradient-descent-plus-beam-search, chosen for computational economy. The competing mechanism is similarity-based example selection, which embeds passages with Sentence-BERT (all-MiniLM-L6-v2) and retrieves up to three examples per class by cosine similarity to form five demonstrations.","core_discovery":"The paper's central finding is that automatic prompt design, implemented as a simplified greedy version of ProTeGi, is a more effective way to integrate expert labels than selecting few-shot examples based on embedding similarity. Starting from an ad-hoc instruction, the greedy self-refinement loop raises test F1 from 68.6% to 88.2% on the NetZeroFact-BIG test set, exceeding the best few-shot result (77.0% with static examples) by a wide margin; starting from a manually expert-tuned instruction, it raises F1 from 81.4% to 86.9%, while similarity-based example selection reaches 86.3%. The authors find that once the prompt has been optimized, attaching few-shot examples usually degrades performance, and that optimizing with static examples and testing zero-shot can reach 89.5% F1. They interpret this as evidence that the optimization process encodes the task's edge-case knowledge into the instruction itself, making explicit demonstrations redundant or even confusing.","pith_inferences":["The results suggest that for classification tasks dominated by edge cases, the bottleneck is instruction specificity rather than exemplar supervision; a separate study could test whether this transfers to other extraction tasks (e.g., reported emissions, ESG disclosures) or other models.","The lack of significance testing and the single tuning run per condition mean the ranking between prompt tuning and example selection may be less stable than the headline numbers suggest; a replication with multiple random orders and models would be the natural next step.","If prompt optimization encodes domain knowledge from the training set's errors, then the method's success is bound to the representativeness of the training set; on a more diverse set of reports or languages, example selection might regain value as an in-context fallback."],"forward_implications":["A practitioner with an ad-hoc prompt can get a large accuracy gain (68.6% to 88.2% F1) simply by letting the model rewrite its own instructions from its mistakes, without collecting more labels.","Few-shot examples are most helpful when the prompt is weak; once the prompt has been automatically optimized, adding examples can reduce accuracy, so the two strategies should not be naively combined.","Optimized prompts are human-interpretable: the instruction evolution shows the model learning task-specific criteria such as requiring a target year and a percentage for relative reductions, which supports manual review and auditing of the classifier.","The greedy one-epoch prompt-tuning procedure is cheap enough to be a practical alternative to maintaining curated example sets."],"supporting_citations":[{"why":"Provides the NetZeroFact-BIG dataset, the public benchmark whose passages and analyst labels define the emission goal detection task and the train/test splits.","marker":"Wrzalik et al., 2024"},{"why":"ProTeGi is the method that the greedy automatic prompt design simplifies; it supplies the error-analysis-and-modify loop core to the approach.","marker":"Pryzant et al., 2023"},{"why":"Shows that semantically similar in-context examples improve GPT-3 performance, motivating the embedding-based 'Similar' example-selection baseline.","marker":"Liu et al., 2022"},{"why":"Introduces Sentence-BERT embeddings, used to compute cosine similarity between passages for example retrieval.","marker":"Reimers and Gurevych, 2019"},{"why":"Chain-of-Thought prompting is the step used to elicit root-cause analyses of prediction errors during prompt optimization.","marker":"Wei et al., 2022"},{"why":"The API and model (gpt-4o-mini) on which all reported experiments were run.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Prompt self-refinement outshines few-shot examples for climate goals","LLM prompt tuning hits 88% F1, beating example selection in climate reports","Automatic prompt design beats expert-labeled examples for emission goals","Greedy prompt loop lifts climate-goal detection to 88.2% F1","Better climate-goal detection: let the LLM refine its own prompt"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions assume the measured F1 gaps are stable across runs, even though each tuning condition was run only once and the reported numbers come from averaging seven evaluations without confidence intervals or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["Prompt self-refinement outshines few-shot examples for climate goals","LLM prompt tuning hits 88% F1, beating example selection in climate reports","Automatic prompt design beats expert-labeled examples for emission goals","Greedy prompt loop lifts climate-goal detection to 88.2% F1","Better climate-goal detection: let the LLM refine its own prompt"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1440,"prompt_tokens":863,"completion_tokens":577,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":479,"tokens_out":577,"duration_ms":5210,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:40:11.673635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the same greedy prompt-tuning procedure multiple times with different random orders of the training passages (or different API nondeterminism) and compute F1 distributions for the best prompt-tuning and best example-selection conditions; the central ranking is falsified if the distributions overlap to the point that the ordering can reverse.","supporting_citations":[],"review_version":1}