Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design

Adrian Ulges; Anne Uersfeld; Florian Faust; Marco Wrzalik; Viola Campos

REVIEW 3 major objections 5 minor 18 references

Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automatic prompt optimization outperforms few-shot example selection for detecting emission reduction goals in corporate reports.

desk verdict Useful empirical comparison of example selection vs. automatic prompt design for ESG text, but the headline ranking rests on single optimization runs with no error bars — treat as provisional. read the letter →

arxiv 2412.06432 v2 pith:3QYFWAUI submitted 2024-12-09 cs.LG cs.CL

Marco Wrzalik , Adrian Ulges , Anne Uersfeld , Florian Faust , Viola Campos This is my paper · ORCID

classification cs.LGcs.CL

keywords emissiongoaldetectionin-contextlearningautomaticpromptoptimizationfew-shotexampleselectionlargelanguagemodelsclimatereportingNetZeroFact-BIGsustainabilityNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to make the best use of analyst-labeled example passages when a large language model is asked to detect emission reduction goals in corporate reports. It compares two in-context learning strategies: dynamically retrieving a few semantically similar examples to include in the prompt, and letting the LLM iteratively rewrite its own instructions after inspecting its mistakes. On 769 real-world business-report passages (NetZeroFact-BIG), the automatic-prompt approach is the clear winner: a simple starting prompt improves from 68.6% to 88.2% F1, and an already expert-tuned prompt from 81.4% to 86.9%. Adding few-shot examples to an already optimized prompt gives little benefit and, for some selection strategies, hurts. The paper's practical claim is that, at least for this kind of classification task, prompt self-refinement is a stronger way to inject expert knowledge than example selection.

What carries the argument

The central mechanism is the automatic prompt-design loop: the LLM is given a training passage, makes a prediction, and when that prediction is wrong the LLM is asked to analyze the root cause of the error and propose a revised instruction that fixes the mistake without reversing correct labels; the revised instruction is accepted only if its F1 on the training set exceeds the current one by at least a margin of 0.01, and the search proceeds greedily without beam search. This is a simplified version of ProTeGi's gradient-descent-plus-beam-search, chosen for computational economy. The competing mechanism is similarity-based example selection, which embeds passages with Sentence-BERT (all-MiniLM-L6-v2) and retrieves up to three examples per class by cosine similarity to form five demonstrations.

What would settle it

Repeat the same greedy prompt-tuning procedure multiple times with different random orders of the training passages (or different API nondeterminism) and compute F1 distributions for the best prompt-tuning and best example-selection conditions; the central ranking is falsified if the distributions overlap to the point that the ordering can reverse.

Watch

Extended reading notes

Core claim

The paper's central finding is that automatic prompt design, implemented as a simplified greedy version of ProTeGi, is a more effective way to integrate expert labels than selecting few-shot examples based on embedding similarity. Starting from an ad-hoc instruction, the greedy self-refinement loop raises test F1 from 68.6% to 88.2% on the NetZeroFact-BIG test set, exceeding the best few-shot result (77.0% with static examples) by a wide margin; starting from a manually expert-tuned instruction, it raises F1 from 81.4% to 86.9%, while similarity-based example selection reaches 86.3%. The authors find that once the prompt has been optimized, attaching few-shot examples usually degrades performance, and that optimizing with static examples and testing zero-shot can reach 89.5% F1. They interpret this as evidence that the optimization process encodes the task's edge-case knowledge into the instruction itself, making explicit demonstrations redundant or even confusing.

Load-bearing premise

The conclusions assume the measured F1 gaps are stable across runs, even though each tuning condition was run only once and the reported numbers come from averaging seven evaluations without confidence intervals or significance tests.

Editorial extensions

If this is right

A practitioner with an ad-hoc prompt can get a large accuracy gain (68.6% to 88.2% F1) simply by letting the model rewrite its own instructions from its mistakes, without collecting more labels.
Few-shot examples are most helpful when the prompt is weak; once the prompt has been automatically optimized, adding examples can reduce accuracy, so the two strategies should not be naively combined.
Optimized prompts are human-interpretable: the instruction evolution shows the model learning task-specific criteria such as requiring a target year and a percentage for relative reductions, which supports manual review and auditing of the classifier.
The greedy one-epoch prompt-tuning procedure is cheap enough to be a practical alternative to maintaining curated example sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The results suggest that for classification tasks dominated by edge cases, the bottleneck is instruction specificity rather than exemplar supervision; a separate study could test whether this transfers to other extraction tasks (e.g., reported emissions, ESG disclosures) or other models.
The lack of significance testing and the single tuning run per condition mean the ranking between prompt tuning and example selection may be less stable than the headline numbers suggest; a replication with multiple random orders and models would be the natural next step.
If prompt optimization encodes domain knowledge from the training set's errors, then the method's success is bound to the representativeness of the training set; on a more diverse set of reports or languages, example selection might regain value as an in-context fallback.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a solid, honestly-written empirical comparison of two ways to use expert labels without fine-tuning — semantic few-shot example selection vs. LLM self-refinement of the prompt — on emission goal detection in corporate reports. The task is real, the dataset is public, the split is at the report level, and the appendix shows the actual prompt evolution, which is nice. The big headline difference — simple zero-shot prompt at 68.6% F1 vs. auto-tuned at 88.2% — is large and credible. If you work on ESG monitoring or applied ICL, this is worth a look.

What's actually new: nobody else has compared these two strategies on this task, and the finding that few-shot examples can hurt after auto-prompting is a useful counterintuitive data point. The authors are upfront about domain-specificity and the non-objective nature of the data.

Where it gets soft: the stress-test is on target. Each condition is a single greedy prompt-tuning run. The stochasticity of the LLM's instruction-modification step and the random traversal of training passages mean you get one draw from a distribution of optimized prompts. The 7 repetitions only re-evaluate the final prompt; they don't capture trajectory variance. No confidence intervals or significance tests. Some of the decisive gaps are small — e.g., 86.9 vs 86.3 for expert-start zero-shot vs. similar-example selection, and 89.6 vs 88.2 for static vs. zero-shot under simple start. On 207 test passages, a few label flips shift F1 by more than a point. The claim that epsilon=0.01 is "approximately two times the standard deviation observed in many of our experiments" conflates test-time evaluation noise with optimization trajectory noise. So the abstract's unqualified "automatic prompt optimization is superior" is too strong as stated; the simple-prompt condition supports it, but the finer ordering claims could flip on replication.

There's also a model-name inconsistency: the methods say gpt-4o-mini-2024-07-18, but the Limitations section says o4-mini. That's the kind of slip that makes a referee want to see the exact configuration and code.

Verdict: this deserves a serious referee, not a desk reject, but the revision needs multiple prompt-tuning seeds, error bars, artifact sharing, and a fix to the model identity. It's a useful practitioner-focused empirical study, not a theoretical breakthrough. I'd bring it to reading group if your group cares about applied LLM workflows; otherwise it's a pass.

Referee Report

3 major / 5 minor

Summary. The paper compares two ways of integrating expert-labeled passages into LLM prompts for emission goal detection in corporate reports: (1) dynamic few-shot example selection based on embedding similarity, and (2) automatic prompt optimization via a greedy ProTeGi-style refinement loop. Experiments on the public NetZeroFact-BIG dataset (769 passages, GPT-4o mini) report F1 scores under simple and expert starting instructions, with and without few-shot examples during tuning and testing. The headline findings are that automatic prompt optimization outperforms example selection and that adding few-shot examples to automatically optimized prompts gives limited or negative benefit. The appendix documents the starting prompts, static examples, the modification prompt, and qualitative instruction evolution.

Significance. The task is practically relevant for sustainability monitoring and carbon accounting, and the experimental design has clear strengths: a public dataset, report-level train/test splitting to prevent leakage, held-out test evaluation, averaging over seven model calls for the final prompts, and transparent appendices containing all prompts and examples. If the ranking of strategies is stable, the paper provides useful practitioner guidance for integrating expert labels into LLM pipelines. However, the headline claims are currently supported only by point estimates from single optimization trajectories, with no confidence intervals or significance tests, so the strength of the conclusions exceeds what the reported evidence establishes.

major comments (3)

[Section 4, Table 2; Section 3.2] The central comparison in Table 2 is based on a single greedy prompt-tuning trajectory per condition. Section 3.2 describes a stochastic process: training passages are visited in random order and the LLM's instruction-modification step is non-deterministic, as Section 4 itself notes. The reported '7 repetitions' average only re-evaluations of the final accepted prompt; they do not measure the variability of the accepted prompt itself. No confidence intervals or significance tests are reported. This matters because the decisive gaps are small: for the expert start, zero-shot auto-prompting gives F1 86.9 versus Similar example selection at 86.3, and for the simple start, Static/Static (89.6) versus zero-shot/zero-shot (88.2) is a 1.4-point gap on a 207-passage test set, where a handful of label flips changes F1 by more than a point. The abstract's claim that 'automatic prompt optimization is the superior approach' is therefore not yet established at the reported precision. Please report distributions over multiple independent tuning runs and/or significance tests, and make the number of runs explicit.
[Section 3.2; Section 4, Setup and Technical Details] The acceptance margin epsilon=0.01 is justified as 'approximately two times the standard deviation observed in many of our experiments,' but no supporting data are shown, and the standard deviation of repeated evaluations of a fixed final prompt is not the same as the variability of the optimization trajectory. Since epsilon controls how many instruction modifications are accepted, the ranking in Table 2 may depend on this free parameter. Please provide a sensitivity analysis over epsilon or trajectory-level standard deviations based on repeated optimization runs.
[Abstract; Section 1; Limitations] The abstract and conclusion state that automatic prompt optimization is 'the superior approach' without qualification, but the experiments use a single model (GPT-4o mini), a single dataset, and a single prompt-tuning run per condition. The Limitations section appropriately acknowledges model-specificity and mentions initial LLaMA-3 evaluations, but no numbers or details for those evaluations are reported. Please qualify the claims to the evaluated model and dataset, or add multi-model evidence, so that the general wording matches the evidence.

minor comments (5)

[Limitations] The Limitations section says 'we only focus on OpenAI's o4-mini model,' but the experiments use GPT-4o mini (gpt-4o-mini-2024-07-18). This model name should be corrected for consistency.
[Section 3.1] The selection rule is unclear: 'we select up to three examples per class ... to form five few-shot demonstrations.' With two classes this could yield up to six examples; please clarify how the total is constrained to five (e.g., class-balancing rule or maximum total).
[Section 2.2] There are typos in the ProTeGi description: 'LLM-generated rasonings' should be 'LLM-generated reasonings,' and 'stears a beam search' should be 'steers a beam search.'
[Section 4, Table 2] Given the paper's own point about non-deterministic outputs, it would be helpful to report the standard deviations across the seven evaluation repetitions next to the F1 values, not only mention them in the text as justification for epsilon.
[Abstract; Section 1; Section 4] The dataset is referred to both as 'NetZero-Facts' and 'NetZeroFact-BIG'; please use one consistent name.

Circularity Check

0 steps flagged · score 1.0 of 10

No circularity in the derivation: prompts are tuned on training F1 and evaluated on a report-disjoint test split; the only self-citation (NetZeroFact-BIG dataset) is not load-bearing.

full rationale

The central comparison is not circular. The paper tunes prompts using training-set F1 ('The accuracy of this new prompt with instructions I′ is assessed by computing its F1 score on the training set') and evaluates on a held-out split constructed along reports ('Four reports have been chosen whose 207 passages form the test set, while 562 passages from the remaining 12 reports form the training set'). Thus the abstract's claim that automatic prompt optimization is superior is not fitted to the test data. The main self-citation is the NetZeroFact-BIG dataset (Wrzalik et al., 2024) by overlapping authors, but the evaluation is performed on an unseen report-level test split, so the citation is not load-bearing in a circular way. The 'expert instruction' was manually tuned before the experiments and serves as a baseline input rather than a predicted output. Robustness concerns about the single greedy optimization trajectory and missing confidence intervals are legitimate statistical-risk issues, not evidence of circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical comparison rather than a derivation, so the ledger records the experimental hyperparameters chosen by hand and the unverified assumptions on which the comparison rests. The main free parameters are the acceptance margin epsilon, the number of few-shot examples, and the number of evaluation repetitions; the main assumptions are that the expert labels are correct, that embedding similarity retrieves useful examples, that training-set F1 is a good prompt-acceptance criterion, that report-level splitting prevents leakage, and that 7-run averaging gives stable point estimates. No invented entities are introduced.

free parameters (3)

prompt acceptance margin epsilon = 0.01
Chosen by hand to exceed two standard deviations observed in evaluations; controls how many instruction modifications are accepted during prompt tuning, directly shaping the final optimized prompt and hence the headline F1 gains.
number of few-shot examples = 5 (up to 3 per class)
Chosen by the authors; the paper does not test other counts, so conclusions about few-shot examples are specific to this setting.
evaluation repetitions = 7
Averaging over 7 outputs is used to mitigate non-determinism, but no confidence intervals are reported and prompt tuning itself is not repeated.

assumptions (5)

domain assumption The expert labels in NetZeroFact-BIG are correct ground truth for emission goal detection.
Used to train and evaluate all prompts; the paper itself notes the lack of external validation and that company reports are non-objective (Limitations).
domain assumption Cosine similarity in the all-MiniLM-L6-v2 embedding space is a valid proxy for task-relevant similarity for few-shot example selection.
Section 3.1; if retrieval surface is not aligned with label-relevant features, example selection results would change.
domain assumption F1 on the training set is a sufficient criterion for accepting prompt modifications.
Section 3.2; the optimized prompt is selected on training F1 with margin epsilon, and this is assumed to generalize to the test split.
domain assumption The report-level split prevents data leakage and yields independent train and test sets.
Section 4; duplicate statements within the same report are assumed to be the only leakage channel, and report-level splitting is assumed sufficient.
domain assumption Averaging seven non-deterministic GPT-4o mini outputs estimates expected performance without bias.
Section 4; no confidence intervals, no repeated tuning runs, and decoding details (temperature) are not reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design." pith.science (2026). https://pith.science/paper/3QYFWAUI

@misc{pith2026241206432,
  author       = {Pith},
  title        = {Pith review of: Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QYFWAUI}},
  note         = {Machine review of arXiv:2412.06432}
}

read the original abstract

We address the detection of emission reduction goals in corporate reports, an important task for monitoring companies' progress in addressing climate change. Specifically, we focus on the issue of integrating expert feedback in the form of labeled example passages into LLM-based pipelines, and compare the two strategies of (1) a dynamic selection of few-shot examples and (2) the automatic optimization of the prompt by the LLM itself. Our findings on a public dataset of 769 climate-related passages from real-world business reports indicate that automatic prompt optimization is the superior approach, while combining both methods provides only limited benefit. Qualitative results indicate that optimized prompts do indeed capture many intricacies of the targeted emission goal extraction task.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages

[1]

Emphasizes specific emission reduction targets

work page
[2]

Focus on explicit statements of intent or measurable targets

work page
[3]

Excludes vague or general strategies

work page
[4]

Adds requirement for numerical values or timelines in commitments

work page
[5]

< target label >

We are also committed to excluding oil exploration from our investments by 2030. AI : False Human : < input passage > AI : < output prediction > A.4 Automatic Prompt Tuning: Prompt for Instruction Modification System : < instruction > Human : < example input > AI : < example output >... Human : < input passage > AI : < output prediction > Human : Your pre...

work page
[6]

Focuses on specific commitments, not just data or ambi- tions

work page
[7]

Clarifies that numerical values alone are insufficient

work page
[8]

Emphasizes intent to achieve targets within a timeframe

work page

Show all 18 references

[9]

Notes need for long-term commitment or endpoint
[10]

Stresses actionable targets with clear timelines
[15]

reduce to zero by 2050

Recognizes specific phrasing like "reduce to zero by 2050." Tuned Instruction: Determine if the text explicitly describes a commitment to reducing carbon emissions, achieving net zero, or setting specific, measurable emission reduction tar- gets with clear numerical values or ...
[16]

Adds emphasis on the need for both a clearly stated percentage and a target year for relative reductions, and a target year for net zero/carbon neutrality
[17]

Includes checks to ensure the presence of a target year for all goals

Stresses the validity of the target only if it explicitly states the target year for both relative reductions and net zero/carbon neutrality. Includes checks to ensure the presence of a target year for all goals
[18]

Reinforced the need for specific target years and percentages for all commit- ments

Refines further by requiring that net zero or carbon neutrality commitments specify a particular year (e.g., 2030), not just a vague timeframe. Reinforced the need for specific target years and percentages for all commit- ments. Tuned Instruction: You are an information extrac...
[2022]

Learning To Retrieve Prompts for In-Context Learning. In Proc. NAACL 2022, July 10-15, 2022, pages 2655–2671. Tobias Schimanski, Andrin Reding, Nico Reding, Julia Bingler, Mathias Kraus, and Markus Leippold. 2024. Bridging the gap in esg measurement: Using nlp to quantify envi...

2022 arXiv
[2023]

True " if it does , otherwise return

OpenReview.net. Appendix A Prompts A.1 Simple Instruction System : Determine if the text describes a commitment to reducing carbon emissions or achieving net zero ; return " True " if it does , otherwise return " False ". Human : < example input > AI : < example output >... Hu...

2019
[2024]

Large Language Models as Optimizers. In Proc. ICLR 2024, May 7-11, 2024. OpenReview.net. Yiming Zhang, Shi Feng, and Chenhao Tan. 2022. Ac- tive Example Selection for In-Context Learning. In Proc. EMNLP 2022, December 7-11, 2022 , pages 9134–9148. Yongchao Zhou, Andrei Ioan Mu...

2024
[2025]

Gradient Descent

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. Dario Garigliotti. 2024. SDG target detection in envi- ronmental reports using retrieval-augmented gener- ation with LLMs. In Proceedings of the 1st Work- shop on Natural Lang...

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Emphasizes specific emission reduction targets

work page

[2] [2]

Focus on explicit statements of intent or measurable targets

work page

[3] [3]

Excludes vague or general strategies

work page

[4] [4]

Adds requirement for numerical values or timelines in commitments

work page

[5] [5]

< target label >

We are also committed to excluding oil exploration from our investments by 2030. AI : False Human : < input passage > AI : < output prediction > A.4 Automatic Prompt Tuning: Prompt for Instruction Modification System : < instruction > Human : < example input > AI : < example output >... Human : < input passage > AI : < output prediction > Human : Your pre...

work page

[6] [6]

Focuses on specific commitments, not just data or ambi- tions

work page

[7] [7]

Clarifies that numerical values alone are insufficient

work page

[8] [8]

Emphasizes intent to achieve targets within a timeframe

work page